Best PC platform for running Esxi/Docker at home?

[NeverDie]

@BearWithBeard said in Best PC platform for running Esxi/Docker at home?:

I wonder, where would you get that Ryzen 7 5800U from? Isn't that a non-socketable BGA package meant for notebooks?

Good question. It may be a gotcha. I found this picture:

Is that some kind of carrier that it's affixed to, or is it all BGA on the other side?

[NeverDie]

@mfalkvidd said in Best PC platform for running Esxi/Docker at home?:

@NeverDie linux runs fine on amd, no hassles for many years. There were some small issues back in the 90’s but it worked well even back then.

What abuot esxi virtualization? If I'm not mistaken, a few years ago it seemed much more likely to support Intel chip virtualization. Not sure if that has changed since then or not though. Is Esxi virtualization on AMD chips now equally well supported? Anyone know?

[NeverDie]

@BearWithBeard said in Best PC platform for running Esxi/Docker at home?:

AMD Athlon 5350 APU

Were you using it for Esxi or Docker?

[mfalkvidd]

@NeverDie you may be right about esxi. https://www.vmware.com/resources/compatibility/search.php only lists a few (old) amd models, so there seems to be very little official support.

https://communities.vmware.com/t5/vSphere-Hypervisor-Discussions/Does-ESXi-work-with-AMD-Ryzen/td-p/1856913 and https://www.servethehome.com/amd-ryzen-working-with-vmware-esxi-6-5/ have some additional info.

But you could use Proxmox instead of esxi. It runs windows and linux well.

[monte]

+1 for proxmox. Didn't have good experience with esxi. Proxmox is opensource, free and debian-based. Using it on HP microserver gen 8.
As for CPU choice I don't see how there would be any issue with Ryzen. All you need from processor is hardware virtualization, and I think all except the lowest level ones have it nowadays.

[BearWithBeard]

@NeverDie said in Best PC platform for running Esxi/Docker at home?:

Were you using it for Esxi or Docker?

I do not use any hardware virtualization / hypervisors. Just a regular Linux distribution as a host running Docker containers and other stuff.

Is that some kind of carrier that it's affixed to, or is it all BGA on the other side?

AFAIK, all Ryzen CPUs ending with an U or H are soldered BGA packages for the mobile OEM market. U indicates low TDP variants for long battery runtime, whereas H or HS stands for high performance setup.

[Ikes 72000]

@monte said in Best PC platform for running Esxi/Docker at home?:

+1 for proxmox. Didn't have good experience with esxi. Proxmox is opensource, free and debian-based. Using it on HP microserver gen 8.
As for CPU choice I don't see how there would be any issue with Ryzen. All you need from processor is hardware virtualization, and I think all except the lowest level ones have it nowadays.

I have two HP gen8 microserver running Esxi 6.7U3, working fine.
I tried Proxmox earlier, and encountered some issues.
Both HP run old cpu (core i3 and xeon e3-1265) and 7 or 8 VM each (OMV, pfsense, Nodered, Mosquitto, HA, Nextdom, seedbox, other).
Cpu are not used much, less than 25%.
So with recent cpu, you shoudn't have any issues.

[monte]

@Ikes-72000 I guess I just love open source software that doesn't rely on some vendor and it's good will to give you access to use it or find solution to your problem.

[NeverDie]

@BearWithBeard said in Best PC platform for running Esxi/Docker at home?:

I wonder, where would you get that Ryzen 7 5800U from? Isn't that a non-socketable BGA package meant for notebooks?

Looks like you're right. The closest thing at the moment would be something like this:

https://www.newegg.com/asrock-4x4-box-4800u/p/N82E16856158069
which is the prior generation chip but which runs at the same average 15w TDP. Presumably there will be something similar for the 5800u sometime after it starts to ship. However, when it's all said and done, not much savings, if any, as compared to simply buying a laptop. At least a laptop, because of its battery, wouldn't need a separate UPS, which would somewhat defeat the point of getting such a small box in the first place!

[BearWithBeard]

I get that efficieny is important for a device that runs 24/7, but looking at the options you have with a 5800U, I think you sacrifice a lot. How do you add hard drives for storage, via USB3? What about RAID? How do you know if the laptop's thermal design is sufficient for 24/7 uptime?

Do not focus too much on the nominal TDP. It is basically just an approximation for the cooling solution required to achieve the rated performance of the CPU. In the specific case of AMD Ryzen CPUs, the rated TDP roughly equals the power draw in P0 state, which is the highest active power state the CPU can be in.

For example, a Ryzen 5 5600X rated at 65W may draw 60 to 75W (once the thermal limit is reached and everything is settled in) under high demanding workloads (prime95 and such). Modern CPUs are smart enough to switch to quite efficient P-states when they have nothing to do, lowering the clock to the sub-GHz range, drastically reducing the core voltage. So if the CPU is idling, it may only draw 10 to 15% of the rated TDP - and a home server will likely idle 95% of the time.

At this point, other components (mainboard chipset, hard drives and other peripherals, power losses of the power supply, ...) might start to contribute more to the total power consumption of the PC than the CPU itself. And I haven't even mentioned that you could underclock (which doesn't void warranty) the CPU the increase the efficiency even further.

[NeverDie]

@BearWithBeard said in Best PC platform for running Esxi/Docker at home?:

Modern CPUs are smart enough to switch to quite efficient P-states when they have nothing to do, lowering the clock to the sub-GHz range, drastically reducing the core voltage. So if the CPU is idling, it may only draw 10 to 15% of the rated TDP - and a home server will likely idle 95% of the time.

I can believe that may be true of some CPU's, but I wonder if that really does hold true for the Ryzen 5 5600X. Instead of a min, max, and "default" TDP, it's specs list just one TDP at 65w:
https://www.amd.com/en/products/cpu/amd-ryzen-5-5600x

In contrast, the ryzen 7 5800u gives a range of 10-25w, with a "default" of 15w:
https://www.amd.com/en/products/apu/amd-ryzen-7-5800u

That said, maybe there's a better place I could look where I could confirm that it draws only 10 to 15% of its nominal power draw while idling?

I do very much like that the Ryzen 5 5600X has a very good single-thread passmark score for its roughly $299 price, which is quite favorable compared to much of its competition. If its power draw really were 10-15% of its nominal 65w about 95% of the time, then I think it would be a great choice. But how to confirm that without actually buying it and testing it? Surely it must be documented somewhere. But where exactly?

[BearWithBeard]

The cTDP is generally only provided for APUs (CPU + GPU on a single chip). It allows OEMs to easily limit the power draw (mainly the all core boost clock) to match the CPU to their thermal design requirements. That's why two equally equipped notebooks can have vastly different performance results. It's a pity that OEMs almost never advertise their cTDP setting.

The desktop CPU market is a little different. Thermal design isn't that critical, CPUs are also sold directly to end users, etc. While it may depend on the mainboard manufacturer to enable the cTDP setting in their UEFI, it is generally possible to adjust it on desktop Ryzens as well. Many do that to find their efficiency sweet spot. I guess nothing could stop you from setting a 5600X's cTDP to 15W. That's what I did with my Athlon as well, albeit only from 25W to 15W. Besides the cTDP, there are other ways to increase efficiency: Lowering the PPT threshold, setting a negative vCore offset, etc. But that whole topic is too complex to explain or discuss here. You may search on PC hardware websites like AnandTech, Tom's Hardware, GamersNexus, Guru3D and such for more details if you're interested.

Here's a 5600X review by Tom's Hardware. They suggest that it draws 13W in idle at stock settings, which are 20% of the nominal 65W TDP. Other reviewers may get different results, due to hundreds of different combinations of configurations, hardware choice and measurement methods. I assume that you can reduce those 13W even further with some adjustments, but don't take my word on that. Unfortunately, I don't own a Ryzen right now to test that myself.

That being said, I don't think you can realistically underclock a desktop CPU to match the power draw of their mobile pendants. So the mobile CPU may always be more economical.

Maybe the embedded lineup of CPUs like the Intel Xeon D or AMD V / Epic Embedded may suit you? They are generally more targeted towards industrial use and have far more connectivity options than those NUC-style mini PCs or notebooks. Commercial NAS vendors like Synology use those. Unfortunately, at first glance, it looks like both Intel and AMD solutions seem rather outdated at the moment.

[monte]

Look at this thread, if you haven't already: https://forums.serverbuilds.net/t/guide-nas-killer-4-0-fast-quiet-power-efficient-and-flexible-starting-at-125/

[NeverDie]

Results are all over the map, but it looks as though idle generally is low across a range of platforms:

These are all mini-pc's, so it's not surprising that the idle power numbers are low.

At any rate, it looks as though the idle wattage for just the Ryzen 5 5600X cpu is around 26 watts:

Not particularly awesome, but it can do a lot more work when needed than the mini-pc's above.
26w is manageable with a quiet fan. Total system draw will, of course, be a higher number.

And here are the loaded numbers for contrast and comparison:
source: https://thinkcomputers.org/amd-ryzen-5-5600x-processor-review/8/

[BearWithBeard]

@NeverDie said in Best PC platform for running Esxi/Docker at home?:

At any rate, it looks as though the idle wattage for just the Ryzen 5 5600X cpu is around 26 watts:

See what I mean with "other reviewers may get different results"? ThinkComputers says 26W idle. Tom's Hardware says 13W. AnandTech says 11W. TPU says 50W (whole system). Who is right? It's unfortunate that there is no standard testing procedure every reviewer adheres to.

Part of the different results may be due to mainboard selection. Some (I don't know how many and if they are rare or the majority) AM4 mainboards seem to be rather power hungry. Especially those with an X570 chipset. Some may add another 10 to 20W to the total power consumption, even if you use the same CPU. See here for example.

26w is manageable with a quiet fan.

Oh yeah, absolutely! Even under load. My 80W Xeon sitting right next to me here is inaudible for most people unless they hold their ear close to the PC case.

Oh, and please note that I'm not trying to sell you the 5600X just because I used it as a comparison a few times. IMHO, all CPUs in that category are grossly overpowered for the average home server unless you know that you will need it sooner or later.

[NeverDie]

It seems that a minimum of 3 disks is required for running ProxMox: one to boot from, one for ISO's, and one for VM's. I'm surprised it's that literal and not able to partition one disk into 3 equivalent disks. It advises not to use a USB flash for the boot disk.

[mfalkvidd]

@NeverDie I use proxmox with 2 drives (in mirror, so proxmox only sees 1 drive).

[NeverDie]

Well, good to know for the future. Right now I'm not finding good beginner instructions for a single drive system, or even a two drive system, so I guess I'll just use a cheap SSD over USB 3.0 as the bootdrive. I'll add a second SSD to my Intel NUC, and in theory that should work well enough in the short-term to evaluate ProxMox.

Presently, running just ProxMox with no VM's and no load: it's drawing 8 watts at idle on on an Intel NUC-6i5SYS (that's a i5-6260u CPU) with 16GB memory and a 1TB Samsung NVME. After adding a SATA SSD and a USB SSD, it should meet the nominal minimum criteria for ProxMox's beginner setup. Not an ideal arrangement, so I'll consider something bigger with more drive bays as a better setup in the future.

[mfalkvidd]

@NeverDie it certainly is possible

[NeverDie]

On second thought, I'll try ProxMox on a system I previously built for Esxi that has 9 drives:

Out with the old hypervisor and in the with the new! This way I'll have plenty of drives to try BTRFS in a Linux VM and whatever else.