Ubuntu

Issues to be aware of when using Ubuntu at BitFolk.

Note that non-LTS releases will not be discussed here, so for example none of the 19.x releases will be covered.

20.04 (Focal Fossa) and beyond

No support for LZ4-compressed kernels at BitFolk

As of 19.10 Ubuntu switched to compressing its kernels with LZ4. At the time of writing (April 2020) BitFolk's hypervisor has a bug when trying to decompress LZ4 kernels which means that you cannot boot an LZ4-compressed kernel. It will report something like:

not a Xen kernel

…and then go back to the bootloader menu. This is because BitFolk's hypervisor failed to decompress it.

This bug has since been fixed, but the fix has not yet been deployed.

Resolution

BitFolk will need to deploy a new hypervisor in order to properly fix this. As that will require a reboot for everyone on each host BitFolk has decided not to schedule maintenance for this. As hosts with the fix become available customers who wish to use LZ4-compressed kernels will be moved to them. After the next security patch requires a reboot of all servers, LZ4-compressed kernels will be fully supported.

Workarounds

Decompress your kernel

Since the problem here is the type of compression used on the kernel image and not any actual functionality of the kernel, you can decompress your kernel image manually and then it will work without issue.

Requirements

• A working VM. If your VM doesn't currently boot you can select the previous (18.04) kernel and that will work well enough to complete this procedure.
• lz4 utility which is part of the lz4 package; normally comes with the base Ubuntu install.
• readelf utility which is part of the binutils package
• extract-vmlinux script available from Linus Torvalds

Procedure

### Download "extract-vmlinux" script and make it executable
$ curl https://raw.githubusercontent.com/torvalds/linux/master/scripts/extract-vmlinux
$ chmod +x extract-vmlinux
### extract-vmlinux outputs its kernel image to standard output.
### Also by default Ubuntu kernels are mode 0600 (only root can read), so sudo is necessary
$ sudo ./extract-vmlinux /boot/vmlinuz-5.4.0-26-generic | sudo tee /boot/vmlinuz-5.4.0-26-uncompressed >/dev/null
$ sudo chmod -c 0600 /boot/vmlinuz-5.4.0-26-uncompressed
mode of '/boot/vmlinuz-5.4.0-26-uncompressed' changed from 0644 (rw-r--r--) to 0600 (rw-------)
$ sudo update-grub
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.4.0-26-uncompressed
Found linux image: /boot/vmlinuz-5.4.0-26-generic
Found initrd image: /boot/initrd.img-5.4.0-26-generic
Found linux image: /boot/vmlinuz-4.15.0-96-generic
Found initrd image: /boot/initrd.img-4.15.0-96-generic
Found Ubuntu 20.04 LTS (20.04) on /dev/xvda1
done
$ ls -lh /boot
total 140M
-rw------- 1 root root 3.9M Apr  1 03:24 System.map-4.15.0-96-generic
-rw------- 1 root root 4.6M Apr 20 17:33 System.map-5.4.0-26-generic
-rw-r--r-- 1 root root 213K Apr  1 03:24 config-4.15.0-96-generic
-rw-r--r-- 1 root root 233K Apr 20 17:33 config-5.4.0-26-generic
drwxr-xr-x 5 root root 4.0K Apr 24 10:56 grub
-rw-r--r-- 1 root root  22M Apr 23 17:57 initrd.img-4.15.0-96-generic
-rw-r--r-- 1 root root  48M Apr 23 18:00 initrd.img-5.4.0-26-generic
-rw------- 1 root root 8.0M Apr  1 03:27 vmlinuz-4.15.0-96-generic
-rw------- 1 root root  12M Apr 20 17:41 vmlinuz-5.4.0-26-generic
-rw------- 1 root root  43M Apr 24 10:55 vmlinuz-5.4.0-26-uncompressed

Now halt your VM and boot it again, making sure to select the -uncompressed kernel from the Grub bootloader menu.

Of course you will have to remember to repeat this process any time there is a kernel update.

Recompile a kernel that doesn't use LZ4 compression

It should work to:

1. Take the regular Ubuntu kernel source package.
2. Adjust its kernel config so that it doesn't use LZ4 compression. Choosing any of XZ, Bzip2, Gzip or even no compression should work.
3. Rebuild it without any other changes.

That sounds a lot harder than the above option so it isn't yet documented here but if you do this please feel free to add your instructions here.

18.04 (Bionic Beaver) and beyond

The switch to netplan for configuring networking

By default in this release Ubuntu stopped using ifupdown (as configured in /etc/network/interfaces) to configure its networking, instead using netplan. netplan is supposed to ignore interfaces that are defined in /etc/network/interfaces but it doesn't seem to do that reliably, so those upgrading from earlier Ubuntu releases can experience networking problems. This is not a BitFolk bug.

Resolution

Migrate to netplan

BitFolk recommends that all Ubuntu users migrate their configuration to netplan before upgrading to 18.04. A typical netplan configuration (in /etc/netplan/01-netcfg.yaml) at BitFolk looks like this:

network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      addresses:
        - "85.119.82.225/21"
        - "85.119.83.73/32"
        - "2001:ba8:1f1:f004::2/64"
        - "2001:ba8:1f1:f004::22/128"
        gateway4: "85.119.80.1"
        gateway6: "2001:ba8:1f1:f004::1"

This assumes:

• A main IPv4 address of 85.119.82.225.
• An additional IPv4 address of 85.119.83.73.
• An assigned IPv6 network of 2001:ba8:1f1:f004::/64.
• An extra IPv6 address of 2001:ba8:1f1:f004::22/128 configured.

To update the config and then make it live:

$ sudo netplan generate
$ sudo netplan apply