Best PC platform for running Esxi/Docker at home?
-
Since prices for both GPU's and Ryzen 9 5950x CPU's have gone crazy, I ended up ordering a Ryzen 5700G, which is 65w nominal. CPU performance is a little less than the 5800x (at 105w), but the 5700G is priced about the same and has integrated graphics. I'd like to use the 5700G as the host for virtual machines, and I'm wondering now whether the integrated graphics gets shared among multiple virtual machines or whether at most one VM can make use of it. Offhand, I'm not sure how I would even test for that, except maybe with some kind of graphics benchmark.
-
I've been looking at building a new server myself.. Currently I have an old hp desktop machine with an I5 / 16Gb of ram, running as server.. Works fine (and it was cheap). But I want to play more with docker / kubernetes, and perhaps do some more machinelearning on zoneminder images, so could use a cpu with a bit more oompf.. Perhaps an nvidia gfx for the ML part..
Only problem is that I'm not ready to spend 1000$ or more on hardware, so I'll probably just wait for the prices to drop again..
(Currently I just run ubuntu server, and then docker on top of that.. No VMs or the like..)
-
I've been looking at building a new server myself.. Currently I have an old hp desktop machine with an I5 / 16Gb of ram, running as server.. Works fine (and it was cheap). But I want to play more with docker / kubernetes, and perhaps do some more machinelearning on zoneminder images, so could use a cpu with a bit more oompf.. Perhaps an nvidia gfx for the ML part..
Only problem is that I'm not ready to spend 1000$ or more on hardware, so I'll probably just wait for the prices to drop again..
(Currently I just run ubuntu server, and then docker on top of that.. No VMs or the like..)
@tbowmo I have Nvidia GT1030 running yolo object detection on camera feeds with 7fps on full model and 20-30fps on yolo-tiny. Considering you don't need to process every frame your cameras produce and 1 or even 0.5 fps per camera is practical enough, you can get this thing going on a budget.
-
I've been looking at building a new server myself.. Currently I have an old hp desktop machine with an I5 / 16Gb of ram, running as server.. Works fine (and it was cheap). But I want to play more with docker / kubernetes, and perhaps do some more machinelearning on zoneminder images, so could use a cpu with a bit more oompf.. Perhaps an nvidia gfx for the ML part..
Only problem is that I'm not ready to spend 1000$ or more on hardware, so I'll probably just wait for the prices to drop again..
(Currently I just run ubuntu server, and then docker on top of that.. No VMs or the like..)
@tbowmo For the machine learning part I bet there are cloud based RTX cards you could use, or perhaps AI cloud servers. Last I checked, which was a while ago, Google was offering tensorflow in the cloud for free. I'm using nvidia Now, which is meant for gaming, but as a data point it offers access to a dedicated RTX3080 on a VM for a mere $10 per month. That's a trivially low price compared to buying physical RTX3080 hardware at current market rates.
-
@tbowmo For the machine learning part I bet there are cloud based RTX cards you could use, or perhaps AI cloud servers. Last I checked, which was a while ago, Google was offering tensorflow in the cloud for free. I'm using nvidia Now, which is meant for gaming, but as a data point it offers access to a dedicated RTX3080 on a VM for a mere $10 per month. That's a trivially low price compared to buying physical RTX3080 hardware at current market rates.
@NeverDie i prefer to keep video footage local. Not saying that I need an rtx3080/3090. They're too pricey. Looking more towards 1050ti or the like. Should be enough, if I'm not going for coral tpu. Waiting for a friend to get hold on some tpus, and give his verdict.
-
@tbowmo I have Nvidia GT1030 running yolo object detection on camera feeds with 7fps on full model and 20-30fps on yolo-tiny. Considering you don't need to process every frame your cameras produce and 1 or even 0.5 fps per camera is practical enough, you can get this thing going on a budget.
-
@monte currently I'm at the order of 0.2fps, would be nice with a bit more throughput, as I'm planning for more cameras along the way. (currently I have 2 cameras running 24/7,but have 2 more that I just need to put up..)
-
@tbowmo I don't know your setup, but optimizing the code that serves frames to the detector can help more then you may think.
@monte said in Best PC platform for running Esxi/Docker at home?:
@tbowmo I don't know your setup, but optimizing the code that serves frames to the detector can help more then you may think.
I'm using zoneminder, and zmeventserver for this.. All running in docker at the moment.. Haven't had the time for digging in to this machine learning, other than scratch the surface, and trying to apply prior work to my own setup..
-
The ryzen 9 5900x seems to be the sweet spot of bang per buck (lots of hefty cores for the price), so I've put an order in for one of those as well.[Edit: I canceled my ryzen 9 5900x order, which seemed perpetually backordered, and decided to buy a ryzen 9 5950x at scalper's prices instead to get it more quickly. Now I understand how the disproportionately high price of the ryzen 9 5950x (as compared to the ryzen 9 5900x) can be justfied: when you take into account the full system cost, rather than just the CPU cost in isolation, the marginal cost per core of the ryzen 9 5950x becomes more or less a wash as compared to the marginal cost per core of the ryzen 9 5900x, if it were underpinned by the same motherboard and memory. If the total sytem cost is even higher (if, say, it's a NAS or running video cards), then the marginal cost per core of the ryzen 9 5950x becomes lower than a 5900x, even considering scalper pricing on the CPU itself.
I'll be putting the Ryzen 9 5950x on an ASUS X570 workstation motherboard that comes with 3 PCIe 4.0 slots, where the board's bios is advertised as being capable to divvy up the various pcie lanes in a somewhat flexible manner. So, that should facilitate assigning different graphics cards to different VM's, or possibly adding 10Gbe ethernet and/or a SATA controller with no interference between devices. It's the only motherboard I found with these programmable bifurcation features that also supports ECC memory, though the Asrock X570D4I-2T may have something similar and also looks like a good board, though possibly a bit cramped.
Something I want to try: using bifurcation to add more than one nvme card in parallel to get higher throughput. From what I've read, it should be possible to get more than 20gbps. Maybe 28gbps read speed if four pcie 4.0 nvme cards are used in a single pcie 4.0 slot using an inexpensive adapter board? And would I notice significantly faster VM load times with that approach, assuming a VM is somehow spread over the four nvme cards using one of the ZFS raid configurations?]
-
I'm fairly happy with ProxMox, but FreeBSD's Bhyve hypervisor can clone (or snapshot) a VM in less than a second--even if running on a 10 year old laptop computer--whereas ProxMox cannot (even if, as I have done, I install ZFS as the file system everywhere used by ProxMox). Is it because the FreeBSD ZFS is either more complete or more tightly integrated? Anyone here done a compare/contrast of Bhyve vs ProxMox? I'm also intrigued because FreeBSD/Bhyve has "jails" for added security of VM's. Also, FreeBSD is a more complete OS, and so I suspect the entire thing is better scrubbed than the way Linux distros are thrown together, with each distro incorporating different packages.
Apple built its current OS on top of FreeBSD rather than Linux, so that's at least a minimal endorsement of FreeBSD, even if it were to turn out that the main reason Apple chose FreeBSD over Linux was due to FreeBSD's more favorable licensing terms.
-
I'm fairly happy with ProxMox, but FreeBSD's Bhyve hypervisor can clone (or snapshot) a VM in less than a second--even if running on a 10 year old laptop computer--whereas ProxMox cannot (even if, as I have done, I install ZFS as the file system everywhere used by ProxMox). Is it because the FreeBSD ZFS is either more complete or more tightly integrated? Anyone here done a compare/contrast of Bhyve vs ProxMox? I'm also intrigued because FreeBSD/Bhyve has "jails" for added security of VM's. Also, FreeBSD is a more complete OS, and so I suspect the entire thing is better scrubbed than the way Linux distros are thrown together, with each distro incorporating different packages.
Apple built its current OS on top of FreeBSD rather than Linux, so that's at least a minimal endorsement of FreeBSD, even if it were to turn out that the main reason Apple chose FreeBSD over Linux was due to FreeBSD's more favorable licensing terms.
@NeverDie said in Best PC platform for running Esxi/Docker at home?:
FreeBSD is a more complete OS, and so I suspect the entire thing is better scrubbed than the way Linux distros are thrown together
Haven't you heard the last controversy about wireguard driver merged into FreeBSD core by Pfsense, which had awful quality and was written by some pretty shady person? And it was only a few day before the new release of FreeBSD, when the author of wireguard wrote a letter about it and stopped the commit from being released with the kernel.
https://arstechnica.com/gadgets/2021/03/in-kernel-wireguard-is-on-its-way-to-freebsd-and-the-pfsense-router/
Can't find the long story I've read about it, but this article can explain the matter good enough.
I would say that broader adoption and even segmentation in some way, help more to make robust opensource OS. -
OK, reporting back: it turned out to be a wild goose chase all because at time index 3:10 this guy:
https://youtu.be/uV61mVYsFM8appeared to demonstrate that he could clone a VM using Bhyve in about 1 second on a 10-year old laptop, whereas proxmox was taking him around 2 minutes to clone a VM. That's the claim that sent me off on this red herring.
Well, after digging into it, I've come to the conclusion that he wasn't truly cloning a VM but rather cloning a containerized VM. That option is also available in proxmox.
Either way, I'm hoping that when my Beast computer (above) is set up then ProxMox will be able to do rapid clone clones of either type, and that's why I'm pushing toward the high end specs, especially with regard to pushing ultra high nvme read and write speeds over pcie-4.0 using some kind of RAID configuration. If I can clone a VM in about a second, then for my purposes usability improves tremendously.
Also, it turns out that Trunas "jails" are apparently just the FreeBSD equivalent of linux or docker containers, so after realizing that I'm skeptical as to whether they offer any actual improvement in security vs what proxmox or linux offers.
-
OK, reporting back: it turned out to be a wild goose chase all because at time index 3:10 this guy:
https://youtu.be/uV61mVYsFM8appeared to demonstrate that he could clone a VM using Bhyve in about 1 second on a 10-year old laptop, whereas proxmox was taking him around 2 minutes to clone a VM. That's the claim that sent me off on this red herring.
Well, after digging into it, I've come to the conclusion that he wasn't truly cloning a VM but rather cloning a containerized VM. That option is also available in proxmox.
Either way, I'm hoping that when my Beast computer (above) is set up then ProxMox will be able to do rapid clone clones of either type, and that's why I'm pushing toward the high end specs, especially with regard to pushing ultra high nvme read and write speeds over pcie-4.0 using some kind of RAID configuration. If I can clone a VM in about a second, then for my purposes usability improves tremendously.
Also, it turns out that Trunas "jails" are apparently just the FreeBSD equivalent of linux or docker containers, so after realizing that I'm skeptical as to whether they offer any actual improvement in security vs what proxmox or linux offers.
-
@NeverDie may I ask, why do you need such speeds? How frequent are you going to clone VM's and how many of them are you going to have? Is this a home setup?
@monte said in Best PC platform for running Esxi/Docker at home?:
@NeverDie may I ask, why do you need such speeds? How frequent are you going to clone VM's and how many of them are you going to have? Is this a home setup?
I'm merely interesting in trying out something similar to Qubes OS--perhaps approximated using ProxMox--which on dated hardware I found was just too slow at spawning/cloning new VM's to be practical. I figure a one second delay for new spawns would be tolerable. Not sure where the exact cut-off of tolerability would be, but I figure one second probably wouldn't be too bothersome.
Also, sharpening the language of my earlier post, the TL;DR was that I'm now pretty confident the guy in the video was creating "linked clones" in Bhyve rather than "full clones," which is fine, but this wasn't clear in the video, which seems to have confusingly compared Bhyve linked-clone spawn times to ProxMox full-clone spawn times--which, of course, made Bhyve appear far better than it actually is.
I wasn't fully aware of "linked clones" previously, and they maybe are a good way to go, especially if you have a kind of "golden image" VM from which to spawn linked clones afterward. Proxmox can do linked clones too, not just Bhyve. Not entirely sure what the security ramifications are of using linked clones as opposed to full-clones though. I suppose full-clones would be the better of the two, but linked-clones might (?) be an acceptable trade-off in exchange for greatly reduced spawn times. Either way, admittedly, it may be overkill.
-
Closing-the-loop: https://looking-glass.io/ looks cool for tying it all together without sacrificing latency in user interaction. With pcie-5.0 deploying in the second half of this year, you'll get nvme's with even higher throughput. We're at one of those magic points in time where a lot of technology trends are coming together and reinforcing/leveraging each other. It's all pretty awesome.
-
Reporting back: being careful to setup everything right with the most current ProxMox on new hardware with a Ryzen 5700G and ZFS on root, it takes less than 0.1 second to create a linked clone of a virtual machine, as reported by Proxmox's task log. Size of the VM doesn't seem to matter. Same for snapshots.
So, at least for me, that settles the debate. Due to greater familiarity, and a more complete GUI than bhyve on Trunas, I've decided to stick with ProxMox.
As to why creating full-clone VM's takes so much longer than it seems like it should, I did manage to learn something new: on this fresh hardware, which uses nvme for storage, it turns out that creating a full clone of a VM is very much CPU bound rather than storage speed bound. Using htop to gain insight, I found that creating a full-clone pushes all 8 of the ryzen cores pretty nearly to maximum, even with automatic ZFS file compression and encryption both turned off. Running some tests, I found that the time taken to create a full clone is a function of both virtual disk size as well as how "full" that disk image is with the files inside it. So, for whatever reason, creating a full clone apparently involves more than just making a straight bit-for-bit copy of the VM's virtual disk image, and apparently that's why it ends up taking so long. Given this, I'm doubtful there's a workaround tthat would speed up the creation of full clones, but if anyone knows of one, or a different approach, then please do post a comment. Apparently the one thing that probably would help is using a CPU with more cores, like a Ryzen 9 5950x or a threadripper, since it' the CPU that's the chokepoint and the htop insight suggests that even more cores could be used in parallel to speed up the full-clone process. Perhaps the good news is that there's no need to build a super fast 20GBps+ nvme storage array, as faster storage alone isn't going to make a difference in the time it takes to create a full-clone.
Meanwhile I'll change my workflow to using more VM snapshots and fewer VM full-clones.
Lastly, being able to create a linked-clone in < 0.1 second means that it should be quite convenient to use "throwaway" linked-clone VM's for internet browsing for greater security against malware incursion. This is much faster than Qubes OS was able to spawn its disposable browser VM's. Getting to a 1 second full-clone of a VM might yet be possible using a very small, special-purpose browser distro. Judging from recent measurements, possibly sufficient would be a browser distro about one tenth the size of the linux lite distro plus maybe a more powerful CPU. BrowserLinux is just 96MB in size, so it might conceivably reach the objective even without a faster CPU. Unfortunately, it was last updated in 2014, so I'm guessing it's probably full of security holes....
-
Reporting back: being careful to setup everything right with the most current ProxMox on new hardware with a Ryzen 5700G and ZFS on root, it takes less than 0.1 second to create a linked clone of a virtual machine, as reported by Proxmox's task log. Size of the VM doesn't seem to matter. Same for snapshots.
So, at least for me, that settles the debate. Due to greater familiarity, and a more complete GUI than bhyve on Trunas, I've decided to stick with ProxMox.
As to why creating full-clone VM's takes so much longer than it seems like it should, I did manage to learn something new: on this fresh hardware, which uses nvme for storage, it turns out that creating a full clone of a VM is very much CPU bound rather than storage speed bound. Using htop to gain insight, I found that creating a full-clone pushes all 8 of the ryzen cores pretty nearly to maximum, even with automatic ZFS file compression and encryption both turned off. Running some tests, I found that the time taken to create a full clone is a function of both virtual disk size as well as how "full" that disk image is with the files inside it. So, for whatever reason, creating a full clone apparently involves more than just making a straight bit-for-bit copy of the VM's virtual disk image, and apparently that's why it ends up taking so long. Given this, I'm doubtful there's a workaround tthat would speed up the creation of full clones, but if anyone knows of one, or a different approach, then please do post a comment. Apparently the one thing that probably would help is using a CPU with more cores, like a Ryzen 9 5950x or a threadripper, since it' the CPU that's the chokepoint and the htop insight suggests that even more cores could be used in parallel to speed up the full-clone process. Perhaps the good news is that there's no need to build a super fast 20GBps+ nvme storage array, as faster storage alone isn't going to make a difference in the time it takes to create a full-clone.
Meanwhile I'll change my workflow to using more VM snapshots and fewer VM full-clones.
Lastly, being able to create a linked-clone in < 0.1 second means that it should be quite convenient to use "throwaway" linked-clone VM's for internet browsing for greater security against malware incursion. This is much faster than Qubes OS was able to spawn its disposable browser VM's. Getting to a 1 second full-clone of a VM might yet be possible using a very small, special-purpose browser distro. Judging from recent measurements, possibly sufficient would be a browser distro about one tenth the size of the linux lite distro plus maybe a more powerful CPU. BrowserLinux is just 96MB in size, so it might conceivably reach the objective even without a faster CPU. Unfortunately, it was last updated in 2014, so I'm guessing it's probably full of security holes....
@NeverDie make sure you enable the ssd option on the virtual disk in proxmox, and that the partitions are mounted with the discard option. That makes backups much smaller and faster (because deleted data doesn't need to be copied), so it will probably affect full cloning as well. You can also use the command line tool
fstrimto manually discard unused data. -
@NeverDie make sure you enable the ssd option on the virtual disk in proxmox, and that the partitions are mounted with the discard option. That makes backups much smaller and faster (because deleted data doesn't need to be copied), so it will probably affect full cloning as well. You can also use the command line tool
fstrimto manually discard unused data.@mfalkvidd said in Best PC platform for running Esxi/Docker at home?:
@NeverDie ...You can also use the command line tool
fstrimto manually discard unused data.Looking into this, I think checking the "Discard" box on the "Hard Disk" tab of the "Create a Virtual Machine" dialog box inside ProxMox may automatically accomplish the same thing. Also the ProxMox dataset that the VM is saved on needs to be set to thin-client. So, those two things plus the ssd emulation that you mentioned, and of course automatic file compression like lz4 needs to be enabled. So, in total, four things need to be set correctly for it to work optimally.
Originally I was concerned that because, if using ZFS as the file system, ProxMox only allows storing a VM as a pre-allocated "raw" file (rather than as a qcow2 file as ProxMox would if ProMox were using a linux ext16 file system instead of ZFS) that the file would take up enormous space even if the raw file (i.e. the VM's virtual disk) is mostly empty. So, I did the experiment, and it turns out that is true if the ProxMox dataset isn't configured as "thin-client" or if automatic file compression isn't turned on, but fortunately the true size of the raw file does indeed shrink down if those two conditions are enabled.
So, having proved that to myself, I'll pass on the tip: I now create a VM's disk to be as large as I can imagine it would ever need to grow, and then I let the thin-client mechanism maintain the true size of the virtual disk to be only as large as what is actually necessary. This way I don't have to worry that I created too small a VM disk, or that the VM will later outgrow the disk size that was originally allocated for it.
