Fry-IT

Linux Virtualization: OpenVZ vs. Vserver

By: Jan Kokoska, 7th of December 2007

7^th of December 2007

Article version: 1.1 (incorporated feedback from Herbert Poetzl)

Today I will compare relative merits of the two main players in opensource Linux OS virtualization (note the differences to para- or full virtualization explained previously), the venerable Linux-VServer project and a relative new-comer, the commercially backed OpenVZ.

The following is written from perspective of having run Linux OS virtualization in production for several years, powering all of Fry-IT server infrastructure since 2005.

Resource Management

It is crucial how contexts (in Linux-VServer's terminology) or virtual environments (VEs, is OpenVZ's talk) are separated in terms of resource allocation. We're talking about OS virtualization where everything runs on (and would fail with) one Linux kernel. Linux-VServer has traditional ulimit-based resource limits (such as total memory allocation, amount of shared memory, number of open files, etc), plus CPU buckets that set the ratios in which contexts will be allocated time quanta. Additionally, it is possible to switch on virtualization of resources such as memory, CPU and system load, akin to what OpenVZ has built in. These are very useful features and it's a pity that a sizeable part of Linux-VServer users might miss them by simply running the default configuration where these are disabled.

Lets make a digression and explain how to switch these on in Linux-VServer:

cat << EOF > /etc/vservers/<name>/flags
VIRT_MEM
VIRT_UPTIME
VIRT_CPU
VIRT_LOAD
EOF


This is just a basic change, see the above linked documentation for more options to set. Now we can compare like for like in resulting VM behaviour of Linux-VServer and OpenVZ.

Linux-VServer limits are set by modifying text files under /etc/vservers which applies on next VM start. Online change can be achieved through vlimit, though this tool doesn't appear to be as smooth to use as the one system that OpenVZ uses for both one-time and persistent resource limits modification (in either case applied with immediate effect), vzctl (with or without the --save, respectively).

OpenVZ provides a well-balanced set of custom resource limits that include kernel memory size, number of unswappable (locked) memory pages, shared memory pages, number of processes, number of TCP sockets, size of TCP send buffers and size of receive buffers, and about a dozen of other resource limits. It is in practice possible to throttle the virtual machine using these limits as if it in fact was on a slower, smaller machine than it really is, without affecting other virtual machines in any significant way. I believe the OpenVZ resource limiting system gives the system administrator slightly more power but I have managed to run hosts with 50 VMs on Linux-VServer without many practical issues, too. The OpenVZ limits have potential to prevent several classes of DOS attacks (TCP, IPC resources, kernel memory etc), that could come into focus as soon as we're unfortunate enough for any of these to occur on our systems.

However, OpenVZ resource limits can be confusing and the failure counts of hitting these limits needs to be monitored and the operator alerted in a timely manner to resolve the situation. Also, OpenVZ limits are quite a balancing act to get right in combination (however all the important relationships between them are sufficiently described in the documentation. Despite the noted balancing complexity, I would rank OpenVZ slightly higher on the topic of resource management.

Development and Maintenance Style

The Linux-VServer development process was described by Herbert Poetzl, the main developer, as mainly event-driven (i.e. they make Linux-VServer patch work on new kernels as soon as these come out, with the infrastructure that they provide, while keeping the patch lean). OpenVZ design, on the other hand, is clearly more roadmap-driven, they only fork the official kernel about once per year and assume responsibility of maintaining security patches on top of it. OpenVZ team contributes to profiling and security auditing of the kernel, fixing numerous bugs in the process, so OpenVZ kernel can be arguably sometimes more up-to-date than vanilla kernel itself. However, it is vastly more modified, so a peaceful coexistence with another patch, if such need arises (and it does), is far less likely than in the case of Linux-VServer. Even compiling OpenVZ with customized .config settings very frequently fails, so the simplest way out is to run their default configuration, or very close to it. OpenVZ team are however fast to patch these issues, so at least vanilla kernel configuration options should be eventually supportable in any reasonable combination. Applying patches from other parties is still a major problem though (e.g. swsusp2).

OpenVZ narrowly wins when it comes to code stability and maintainability of a large deployment. Linux-VServer eclipses OpenVZ in ease of configuration for custom kernels and availability for very latest kernel versions. For a laptop that needs to support hibernation via swsusp2 and OS virtualization for testing/development purposes, I would recommend Linux-VServer. But where servers are concerned, we're gradually migrating away from Linux-VServer and to OpenVZ.

To this effect, I wrote a vs2ovz migration procedure that takes a running Linux-VServer VM, an SSH-accessible physical host running OpenVZ and leaves the user with running OpenVZ VM with the same parameters as the original Linux-VServer one. This includes the IP, unique VEID derived from the IP, hostname, some selected resource limits that need to be preserved in the new system, and of course all the data. Two-pass rsync is performed to minimize the VM downtime during migration. If you're interested in using this, or other automation tools related to these two projects, write to me at "jan" at Fry-IT company.

Final Points

In the current development branch, OpenVZ supports live migration of the virtual machines. What a killer feature! (I still need to try it out, though.) Also the virtualized interfaces in OpenVZ that allow separate firewalls attached from inside the VEs and/or individual QoS queuing disciplines can come very handy in more sophisticated setups. Linux-VServer people might say this is inefficient and adds complexity, but I've found the overhead to be very low and the ability to at the very least have individual firewalls in each VM to be extremely useful.

Here's hoping that standard Linux kernel will soon include more OS virtualization features, after recent merge of IPC namespace virtualization (some kernel developers say network device virtualization will be the hardest to reach agreement on) and the basic features will be the same across the board. It certainly appears that virtualization is quickly becoming a commodity and the future developments might focus more on datacentre automation and similar tasks on top of reliable virtualized infrastructure.

As mentioned above, it is reasonably straightforward to convert virtual machines between the two virtualization technologies, even without any standards like those cropping up around hardware virtualization technologies recently. So it is possible to run development VMs on a laptop that can hibernate with swsusp2 and later copy/move this VM to production server that will be hosting it under OpenVZ. An opposite migration would be equally feasible.

The Score

A completely unscientific rating follows.

Linux-VServer loses one point in my eyes on resource management (OpenVZ's vzctl/vzquota simply set the standard high) and one on documentation (it's very hard to find the latest greatest way to do things, that initially lead to my writing version 1.0 of this article with lots of mistakes). Resource management consolidation, switching on useful features by default and centralized and end user focused documentation will further improve the already very successful project.

OpenVZ loses a point on lack of patch flexibility and option to combine with other kernel extensions, or even atypical configurations of standard kernel. Further features such as GUI management and centralized management console can be found in Parallels' commercial offering Virtuozzo. If I were rating Virtuozzo, I would subtract another point for delaying on support of my favourite platform, Ubuntu/Debian Linux, where 64-bit Ubuntu templates took a real effort to get working (despite Ubuntu being listed as officially supported platform and despite the hefty price tag on Virtuozzo solution), in fact it was much harder for me to work with than OpenVZ. However I'm sticking to comparing apples for apples in comparing the two opensource offerings and OpenVZ's packaging under Ubuntu/Debian is fortunately very good.

Community support from both projects is nothing short of amazing, helpful hints on how to get setups working, even patches to support my weird kernel option combinations, etc, are equally likely to come from either direction. In any case, and based on previous experience, I doubt many commercial vendors would be able to even match the level of dedication to support that both of the Linux OS virtualization projects offer consistently and for free.

Linux-VServer: 8/10,
OpenVZ: 9/10.

Comment