As in Proxmox you can create both virtual machines and Linux containers to host your chosen apps and services, you might be wondering what are the most important differences between the two, and which one is better for your particular use case. Here are all the things I think you should know before you make your choice!
VMs and CTs In a Nutshell – What Are They Exactly?
Virtual machines (VMs) are essentially computing environments which are almost fully isolated from their physical host (in our case the computer/server which runs Proxmox), and make use of a completely separate virtualized kernel with all the main system resources including RAM and file storage being assigned to them in specified amounts upon creation. Proxmox enables you to create and host virtual machines and manage them using its built in tools, and migrate them between clustered Proxmox machines without any downtime.
Linux containers, (or LXCs / CTs for short), are computing environments that unlike virtual machines, share the system resources with the host machine, including sharing the whole operating system kernel. The maximum level of isolation between the host and the container is decided upon container creation, as you can either create an unprivileged and privileged container type, as you will see in a short while. LXCs are also different from Docker containers, as they are classified as system containers and not application containers – they contain a full operating system less the kernel and some drivers, while the Docker containers are limited to their contained service environments.
Virtual Machine vs. a Linux Container (CT) – Which One Do You Need?
Here is a neat checklist I made to make it easier for you to choose between using a virtual machine and a Linux container in your particular use case.
You might want to consider using a virtual machine if:
- You need to run an operating system based on a non-Linux kernel (Proxmox shares the Ubuntu LTS kernel with its containers).
- Connected with the previous point – if you need to run non-Linux applications and services.
- Want to run a service open to the public and need the highest attainable level of isolation from your host machine.
- Need to be able to do quick migrations with virtually zero downtime between the members of a Proxmox cluster (VMs in Proxmox clusters support live migration, while the containers do not).
- You plan to run Docker (technically you can also run Docker on LXCs, however it is not advised according to the official Proxmox documentation).
- Want to assure full isolation of the virtualized environment from the host machine for maximum security out of the box.
- You want to run a networking service such as a firewall or a VPN and you don’t want to make changes to your host’s iptables.
- Don’t want an accidental kernel panic to bring down your whole host machine.
On the other hand, an LXC could be a better choice for you if:
- You’re short on RAM on your host machine – containers are more memory efficient than VMs.
- You don’t want to sacrifice storage space for a full-fledged OS virtualization (containers are much more lightweight).
- The services you want to run already have an easy-install LXC version in the open-source Proxmox VE Helper-Scripts directory and you want to make use of these.
- You need simple direct access to your host’s storage.
- Won’t need to live migrate your hosted services, and you don’t care about short downtime during regular service migrations.
- You know that you’re going to run a Linux-based operating system service.
Resource Management
As the VMs need to virtualize the whole operating system including the system kernel, they will always be tied to slower boot times and more computational overhead.
LXCs being much lighter overall will start faster, require less resources to run, and are much more memory efficient allowing you to run more services on a host machine with a lower amount of RAM on board.
When it comes to LXCs, they will also be much lighter than virtual machines in terms of the storage space needed on the host. A system container not having to double the kernel files, KVM and virtual I/O drivers among other things can weigh much less than a full-fledged VM, saving you a lot of disk space on your host machine, especially if you want to run each chosen service in a separate LXC.
Also it is important to note that if you want to use a system container to run an OS based on a kernel different from the kernel of the host (in our case with Proxmox, it could for instance be a Windows Server type OS), you’ll have to resort to using a virtual machine. The containers fully rely on the host’s own kernel and thus cannot run any operating systems based on another kernel type.
Host Isolation and Security
In general, virtual machines are always better secured than containers just because of their higher level of isolation from the host system. A VM runs and manages its own kernel and therefore if in a worst-case scenario a threat actor were to compromise an app or service running on a virtual machine, he would “only” be able to gain access to the VM’s root user, which is still completely isolated from your host machine running the hypervisor. Exploits making it possible to breach the constraints of a virtualized operating systems do exists, but they are extremely rare, especially when it comes to properly updated and secured VMs.
When it comes to containers, as I have already mentioned quite a few times, they share the operating system kernel with your host machine. That’s why if a threat actor successfully attempts to compromise a service running inside the container, he might get direct access to your host’s kernel, which is as you probably know a highly undesirable situation to say the least. When it comes to your LXC security, it is very much dependent on the type of LXC you’ll decide to create. There are two basic types of LXCs in Proxmox and they are privileged and unprivileged containers.
In unprivileged containers you, as a user, don’t have privileged access to the host’s hardware. By default, all your LXCs should be created as unprivileged. In privileged containers on the other hand, your root user on the container actually is the root user of your host’s system – a much more risky configuration if someone were to compromise a service running within one of your containers. If you don’t need to use a privileged container, don’t do it.
If you want maximum security on your LXC, make sure that it is properly updated alongside with your host OS, that it’s making use of appropriate custom firewall rules according to the best practices for your desired use case, and that it is run in the unprivileged mode.
Also, another thing that has to be mentioned here is that if for whatever reason kernel panic will be caused in your VM, naturally only the VM will crash and your host machine will remain unaffected. If that’s the case for a container, you’re looking at a system-wide crash on the host level potentially bringing your whole server down, as your containers directly share the kernel with your actual server OS, which in this case would be Proxmox.
Remember to always set routines to update your containers frequently, or do so manually every now and then. When it comes to Debian, you can use unattended-upgrades for that. Automation software like Ansible will also work for this purpose if you’re managing a larger amount of LXCs on your servers. Also, as the containers utilize the host machine’s kernel, always be sure to schedule regular updates for your main Proxmox OS installation.
OS Compatibility
While virtual machines can be used to run any kind of x86/x64 operating system (like for instance Windows, macOS, any flavor of Linux, etc.), LXCs can only run an OS that utilizes a Linux kernel.
This is because by their nature, containers make use of their host system kernel resources, and thus cannot be used with software which requires foreign kernel code to be utilized. Proxmox as an operating system is based on the Ubuntu LTS kernel.
Live Migrations and Backups
VMs in a Proxmox cluster will grant you both high availability and easy live migrations with virtually no downtime. If you have a few machines running Proxmox in a cluster, you will be able to essentially drag-and-drop your virtual machines between different hardware hosts and quickly migrate them that way.
While there are also pretty simple ways to move LXCs between two host machines running Proxmox even when they are not a part of a cluster (mainly using file backups) and the LXCs don’t really need a lot of time to start, these operations are usually connected with a certain amount of downtime, and are not instant like the VM migration within a properly set up cluster.
If you’re running a service which doesn’t really need to be up at all times and won’t suffer from a small amount of downtime you don’t really need to care about this that much. Remember that manually backing up and moving your VMs and LXCs for migration will always work, even between unclustered machines, if you don’t mind your services being inaccessible for the time the manual migration will take.
A Few LXCs, or One VM With a Few Docker Containers?
Two of the most popular approaches to self-host a larger number of services are as follows:
- Host each service in its own container (LXC).
- Create a VM with a Docker installation and run your services under Docker.
As I mentioned before, while it is possible to run a Docker instance within an LXC, it’s not really recommended by Proxmox, not endorsed on official forums, and it’s said to cause rather non-standard issues in some more exotic configurations. I, for one haven’t tried doing it this way, but I’ve already seen quite a few people experiment with it with varying degrees of success.
If you want to run Docker, it’s best to do so on a virtual machine, underneath your preferred flavor of Linux-based operating system. While it’s also said that virtual machines can have more overhead than LXCs, the differences aren’t really that noticeable in most simple applications.
If you’re wondering if it’s better to run one VM with a few Docker containers running on it, or a few containers managed within Proxmox, in most cases it’s a simple question of which technology are you more comfortable working with, what is your particular use case, and whether or not you want to be able to easily manage (and monitor) your hosted services directly from the Proxmox interface.
This last point is pretty important, as if you’re running your services on Docker containers concealed within a virtual machine, you won’t have an easy way of viewing the resource usage of each container from Proxmox itself. You naturally will have to do that from the level of your VM.
In the previously mentioned Proxmox VE Helper-Scripts directory you can find quite a few install scripts for services that can easily be run within LXCs, so if you like neat shortcuts, check if the app/service you want to run is listed in there. That’s really all from me, hope I was able to help!