PVH
Some notes about PVH mode virtualisation. This is a combination of braindump and work in progress, regarding BitFolk's plans to switch to PVH mode. No part of this article should be considered a firm plan, and no timescales are binding.
What?
It's a different type of virtualisation available within Xen since v4.10. The three available types are:
- Paravirtualised mode (PV). The guest kernel is modified heavily to work with all its hardware virtualised. The first type of virtualisation that Xen offered many years ago, in order to work even on CPUs with no hardware virtualisation features.
- This is the current mode that all BitFolk VMs run under.
- Hardware virtualisation mode (HVM). Unmodified guest operating systems can be run, taking advantage of hardware virtualisation extensions in the host CPU and possible emulating all other required hardware typically through use of qemu. Paravirtualised device drivers can be used to improve performance of IO devices.
- Paravirtual+Hardware mode (PVH). Guests use CPU hardware support for memory operations and privileged CPU instructions and are able to use native instructions for most other things, requiring only slight modification of the guest OS. Paravirtual IO drivers are used for performance.
- BitFolk would like to move guests to this type of virtualisation.
Why PVH?
- PVH performs better than PV these days.
- PVH is more secure than PV. Mitigations against the various CPU side channel attacks of the last few years work better in PVH or HVM than in PV mode.
- PVH will soon be the only way to support 32-bit guests. Although it is hard to see why anyone should be running 32-bit guests in the year 2020, there is a significant legacy installation at BitFolk (some 40% of customer VMs).
- The next stable release of Xen will drop support for 32-bit PV guests. Anyone who hasn't upgraded to 64-bit will need to run under either PVH or HVM, and BitFolk does not intend to support HVM.
- The upstream Linux kernel removed support for 32-bit PV guests as of v5.9, so you can't actually run a mainline stable kernel as a 32-bit PV guest any more.
Why Not HVM?
Although it would be nice to be able to support unmodified guests (allowing, for example, FreeBSD, Haiku, Plan9 etc guests), BitFolk does not intend to support HVM in the near future.
There is significant extra complexity in running HVM. There have been a number of security vulnerabilities which only affect Xen running HVM guests. It also involves running a qemu process for each guest; there are more lines of code in qemu than in all of the rest of Xen. It is unclear what extra security burden that would involve.
It is possible that certain servers could be dedicated to only running HVM guests so that security issues would be somewhat partitioned and maintenance reboots for security issues would only affect these customers.
This is something that BitFolk will look into after PVH has been deployed.
How It Works Right Now
Since some time in 2017, BitFolk switched to booting guests using pvgrub2. This is the Grub bootloader compiled as a Xen PV kernel instead of a bootloader image.
The idea is that this Grub searches the customer's block devices for a grub.cfg file, in the context of the customer's guest, and then if it finds one it obeys that configuration file so as to boot how the customer's operating system intended.
The pvgrub2 bootloader needs to be the same architecture as the customer's eventual chosen kernel (32-bit/i386/i686 versus 64-bit/amd64/x86_64). So, there is the arch command in the Xen Shell for customers to toggle between the two different bootloaders. This is a source of some confusion from time to time.
All being well the customer has a Grub2 experience they are familiar with, they select a working kernel and initramfs and boot completes without issue.
What Needs To Change
In theory, Xen since v4.10 has fully supported PVH mode guests and the Linux kernel since v4.11 has fully supported being run as a PVH mode guest. In reality the situation is a lot more complicated.
From the Xen guest configuration point of view, it simply needs lines like…
type = "pv"
kernel = "/opt/grub/lib/grub-x86_64-xen.bin"
…changing to…
type = "pvh"
kernel = "/opt/grub/lib/pvhgrub.bin"
That tells Xen to boot an instance of Grub2 that is compiled as a PVH guest (henceforth referred to as pvhgrub), which should then work the same as in the pvgrub2 case. Unfortunately this may result in a guest not booting correctly.
The problem is that the hypervisor, the bootloader (pvhgrub), and the guest kernel (Linux) must all do things correctly.
Although PVH mode has theoretically been supported since Linux kernel 4.11, this has only worked for direct kernel booting, i.e. where the guest kernel exists as a file in the host server and is loaded directly by the hypervisor. That is not how BitFolk does things and to do things this way would be a backwards step, so much newer guest kernels than 4.11 are going to be required.
As a consequence there is going to be a significant period of time where legacy guests — so old they won't ever be receiving packaged kernel upgrades — will need to continue running under PV mode.
This also means that there will sadly come a time when:
- Xen does not support 32-bit PV mode any more;
- There are still 32-bit customers at BitFolk; and
- Those guest kernels will be too old to work in PVH mode
At that time, the only option for those customers will be to find some way to get a newer kernel installed on their old VM. It is usually reasonably easy to boot an old userland with a newer kernel.
Possibly there can be some consultancy offered to achieve this, but obviously BitFolk strongly suggests that no customer allow themselves to end up in this position in the first place.
The good news is that, when you do have a new enough hypervisor and guest kernel, after making the configuration change everything continues to work the same without needing anything changing in the guest. The guest VM just works better. Despite the avalanche of text in this document, it looks like a reasonably recent Linux distribution release should work in PVH mode without its owner even noticing anything has changed.
What About Installation?
The way that BitFolk customers currently install their guests is to download the netboot installer image that their Linux distribution provides and then boot it in PV mode. The install takes place and then the customer boots into their real operating system.
Most Linux distributions provide a kernel and initramfs that are intended to be booted over the network, start some installer process and then complete the installation over the network downloading packages from the distribution's mirrors as necessary.
On Debian and Ubuntu this software is the debian-installer, optionally using preseeding. On CentOS and Red Hat-derived distributions this is Anaconda, optionally using kickstart.
As these variant kernels may behave differently from the packaged kernels of the distribution, we'll need to test this as well. It wouldn't be the end of the world if we had to just do the install phase in PV mode, though it would mean that all new installs would have to be 64-bit.
Experimentation
In order to find out the scope of the problems we may be dealing with we'll have to do some experimentation.
Debian
| Release | Kernel Version | Installer works? | pvhgrub boot works? | Notes |
|---|---|---|---|---|
| 9.x (stretch/oldstable) | ||||
| 10.x (buster/stable) | 4.19.152-1 | Yes | Yes | Uneventful |