Are there any benefits of running Proxmox and virtualizing everything, vs having a host OS and running Docker and libvirt to host VMs for services that need it? I know that Proxmox does some storage management etc, but it seems like I could get everything it does with a well-managed host OS + ZFS/btrfs and using virtualization tools
libvirt to host VMs for services that need it
What kind of app would require a VM ?
Home Assistant is a good example - its recommended installation is a VM
It’s recommended bare metals not vm. Home assistants often require the latest feature, and some of those features may not work well behind any kind of virtualisation (especially docker’s network virtualisation). Running it on a VM is definitely better than running it in an out dated environment (like debian) but it’s not perfect.
Running Home Assistant OS in a virtual machine using the provided VM images is a supported configuration.
From ADR-0015: Installation Method: Home Assistant OS:
Supported Hypervisors
The Home Assistant Operating System can be run on a Hypervisor and thus be run as a virtual machine. The following Hypervisors are supported:
- KVM/QEMU
- VirtualBox
- VMWare
- Xen
We will provide documentation for the following systems build on top of these technologies:
- Proxmox (KVM/QEMU based)
- Unraid (KVM/QEMU based)
- VirtualBox
- VMWare
Supported != To recommend
I use both Proxmox and libvirt on Debian. I don’t use clustering. For me, the biggest advantages of PVE are:
- Good VNC and serial console support integrated into the web interface. (Could probably get something similar with libvirt using Guacamole, but PVE makes it super easy).
- Good VM snapshot management. I’ve found libvirt snapshot management to be pretty limited and/or buggy, and I’ve had to resort to directly operating on qcow2 files.
On the other hand, there are a couple things I like more about libvirt:
- Good support for SR-IOV NICs. libvirt lets you create a pool of VFs and automatically assign a free VF to a VM. (It’s a little surprising to me that PVE doesn’t do better in this area.)
- Simpler with fewer moving parts to break.
I use libvirt for my most critical VMs (network infra like router, DNS, and DHCP). I strongly prefer PVE for anything where I’m going to be interacting with VMs regularly, like testing or lab setups.
This… It is ease vs control. If you have needs inside the box Proxmox has planned for, it is very easy. But if you want to step outside that box, you are in for a world of surprise. I install Proxmox for clients that do not have solid Linux chops. I run KVM/libvirt on my own stuff. And if you need a web front end, oVirt exists…
Proxmox simplifies storage, media, resource sharing, ZFS, Clustering, and Management.
It supports modern storage protocols so offloading your storage to a NAS is super easy.
I would say the biggest advantages are software package version control, stability, platform support, and compatibility. It makes VM management super simple without needing to be an expert in getting the entire KVM stack working properly.
vs having a host OS and running
docker[containers and VMs]I think this ^ is precisely what Proxmox is, its just purpose built for this purpose and has a lot of nice features and conveniences for this use case.
Proxmox is a host OS (customized Debian) built for virtualization (KVM/Qemu) and containers (LXC containers).
but it seems like I could get everything it does with a well-managed host OS + ZFS/btrfs and using virtualization tools
I think you can. It just depends whether you’d prefer to DIY it yourself or whether you’d like a purpose built solution that is already pre-configured for the task, both approaches are valid and can achieve much the same outcome. The nice thing about proxmox is its mostly ready to go and well configured out of the box (with the exception of not supporting encryption ootb)
Easy vs Painful
How important is it that the machine also host local graphical sessions? Asking because if I recall correctly, Promox doesn’t offer a GUI by default. If you’re going to use the machine for anything else, I feel like one would end up installing and configuring more things, in order to also have a graphical environment in Proxmox, than they would to install the preq’s for libvirt/KVM under Debian with a graphical install (et al.). Of course, there isn’t a snappy web-GUI if you go the libvirt/KVM route (by default), but virsh CLI and Virtual Machine Manager GUI, offer similar functionality.
Down vote with no explanation, hmmm