Ubuntu
Issues to be aware of when using Ubuntu at BitFolk.
Note that non-LTS releases will not be discussed here, so for example none of the 19.x releases will be covered.
22.04 (Jammy Jellyfish) and beyond
Install process switched from Debian installer to cloud-init
Ubuntu switched away from the Debian installer as a supported install method in release 21.10. Ubuntu recommends using the official Ubuntu Cloud Image for installs on public clouds, so BitFolk has modified its Xen Shell to use that method for Ubuntu 22.04 and beyond.
Resolution
No action necessary. This is only to explain why the install process, which for meany years has been very similar between Debian and Ubuntu, is now unique to Ubuntu.
New self-installs will ask for the only information that is required, that being:
- hostname of new server
- password for the default user (ubuntu)
The official Ubuntu cloud image will then be copied to the customer's disk and then customised at first boot for use at BitFolk using the information provided. This is done using cloud-init.
At the moment it is not possible for customers to supply their own cloud-init data. Your cloud-init data is generated from a template using information held by BitFolk and the hostname and password that you supply. There is an open issue in BitFolk's tracker for allowing customers to supply their own cloud-init data. It may also be the case that more Linux distributions become available for install using cloud-init.
Upgrades from 20.04 to 22.04 can as usual be carried out using the usual do-release-upgrade command.
Kernel is compressed with zstd
Similarly to the issues with LZ4 on 20.04, kernels compressed with zstd can't be booted in Xen PV mode.
Resolution
Ubuntu 22.04 works fine in PVH mode, so should be run in that mode. This is the default for new installs so would only be noticeable if you were previously running some older distribution in PV mode and decided to reinstall. The Xen Shell will warn you that you should switch to PVH mode in this case.
20.04 (Focal Fossa) and beyond
No support for 32-bit
As of 20.04 Ubuntu no longer supports 32-bit x86, so those currently on 32-bit 18.04 will be unable to upgrade.
Resolution
Users will need to reinstall.
Or could try to cross-grade 32- to 64-bit, which is certainly possible but is completely unsupported by both Ubuntu and BitFolk and is arguably more work than just reinstalling.
Workarounds
There are no workarounds. 18.04 is in theory supported by Ubuntu until 2023 even on i686 but if upgrading further you will have to reinstall.
No support for LZ4-compressed kernels in Xen PV mode
In November 2020 BitFolk switched its default virtualisation mode from Xen PV to Xen PVH. Existing customers however did not have their virtualisation mode forcibly changed as there was a possibility of causing boot failure. Therefore the majority of customers are still running in PV mode.
As of 19.10 Ubuntu switched to compressing its kernels with LZ4. At the time of writing (November 2020) BitFolk's PV boot loader does not support LZ4 kernels. It will report something like:
not a Xen kernel
…and then go back to the bootloader menu. This is because the PV kernel part of the grub bootloader doesn't recognise LZ4 compression.
Resolution
BitFolk recommends all customers switch to PVH mode as soon as possible, and LZ4-compressed kernels work fine in PVH mode. The virtmode command of the Xen Shell is used to switch between virtualisation modes.
Workarounds
Since Ubuntu 20.04 works fine in PVH mode it is unclear why anyone would want to persist with trying to run it in PV mode, but if for some reason you do, here's how you could do it.
Decompress your kernel
Since the problem here is the type of compression used on the kernel image and not any actual functionality of the kernel, you can decompress your kernel image manually and then it will work without issue. This can be done in a kernel post-install hook script.
This workaround has already been applied to VMs ordered brand new or installed using the BitFolk installer. Only those upgrading from a previous version of Ubuntu need to do something like this.
Requirements
- A working VM. If your VM doesn't currently boot you can select the previous (18.04) kernel and that will work well enough to complete this procedure.
- lz4 utility which is part of the lz4 package; normally comes with the base Ubuntu install.
- readelf utility which is part of the binutils package.
- extract-vmlinux script available from the Linux kernel repository. Put in /usr/local/sbin/.
- BitFolk's kernel post-install hook. Put it in /etc/kernel/postinst.d/.
Procedure
$ sudo apt install binutils lz4
### Download "extract-vmlinux" script and make it executable
$ curl https://raw.githubusercontent.com/torvalds/linux/master/scripts/extract-vmlinux | sudo tee /usr/local/sbin/extract-vmlinux >/dev/null
$ sudo chmod -c +x /usr/local/sbin/extract-vmlinux
### Download the kernel post-install hook and make it executable
$ curl https://raw.githubusercontent.com/bitfolk/decompress-lz4-kernels/master/bitfolk-decompress-lz4-kernels | sudo tee /etc/kernel/postinst.d/bitfolk-decompress-lz4-kernels >/dev/null
$ sudo chmod -c +x /etc/kernel/postinst.d/bitfolk-decompress-lz4-kernels
### Verify that the current kernel is compressed (bzImage). Note that the kernel version number may be different to that shown below.
$ sudo file /boot/vmlinuz-5.4.0-29-generic
/boot/vmlinuz-5.4.0-29-generic: Linux kernel x86 boot executable bzImage, version 5.4.0-29-generic (buildd@lgw01-amd64-035) #33-Ubuntu SMP Wed Apr 29 14:32:27 UTC 2020, RO-rootFS, swap_dev 0xB, Normal VGA
### Force reinstall of the kernel package so the hook gets called. Note that the package name may be different due to a different version number.
$ sudo apt reinstall linux-image-5.4.0-29-generic
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 to upgrade, 0 to newly install, 1 reinstalled, 0 to remove and 0 not to upgrade.
Need to get 8,871 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://apt-cacher.lon.bitfolk.com/ubuntu/gb.archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-29-generic amd64 5.4.0-29.33 [8,871 kB]
Fetched 8,871 kB in 0s (25.8 MB/s)
(Reading database ... 50238 files and directories currently installed.)
Preparing to unpack .../linux-image-5.4.0-29-generic_5.4.0-29.33_amd64.deb ...
Unpacking linux-image-5.4.0-29-generic (5.4.0-29.33) over (5.4.0-29.33) ...
Setting up linux-image-5.4.0-29-generic (5.4.0-29.33) ...
Processing triggers for linux-image-5.4.0-29-generic (5.4.0-29.33) ...
/etc/kernel/postinst.d/bitfolk-decompress-lz4-kernels:
LZ4-compressed kernel /boot/vmlinuz-5.4.0-29-generic found; decompressing...
(This is a BitFolk-specific customisation. Please see https://tools.bitfolk.com/wiki/Ubuntu for more information.)
Successfully decompressed /boot/vmlinuz-5.4.0-29-generic.
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.4.0-29-generic
/etc/kernel/postinst.d/zz-update-grub:
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.4.0-29-generic
Found initrd image: /boot/initrd.img-5.4.0-29-generic
done
### Verify that the kernel image is no longer compressed (ELF 64-bit LSB executable).
$ sudo file /boot/vmlinuz-5.4.0-29-generic
/boot/vmlinuz-5.4.0-29-generic: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=e54a270642a361346d36d5be07edb8d2bfa3fe10, stripped
Now reboot.
Recompile a kernel that doesn't use LZ4 compression
It should work to:
- Take the regular Ubuntu kernel source package.
- Adjust its kernel config so that it doesn't use LZ4 compression. Choosing any of XZ, Bzip2, Gzip or even no compression should work.
- Rebuild it without any other changes.
That sounds a lot harder than the above option so it isn't yet documented here but if you do this please feel free to add your instructions here.
18.04 (Bionic Beaver) and beyond
The switch to netplan for configuring networking
By default in this release Ubuntu stopped using ifupdown (as configured in /etc/network/interfaces) to configure its networking, instead using netplan. netplan is supposed to ignore interfaces that are defined in /etc/network/interfaces but it doesn't seem to do that reliably, so those upgrading from earlier Ubuntu releases can experience networking problems. This is not a BitFolk bug.
Resolution
Migrate to netplan
BitFolk recommends that all Ubuntu users migrate their configuration to netplan before upgrading to 18.04. A typical netplan configuration (in /etc/netplan/01-netcfg.yaml) at BitFolk looks like this:
network:
version: 2
renderer: networkd
ethernets:
eth0:
addresses:
- "85.119.82.225/21"
- "85.119.83.73/32"
- "2001:ba8:1f1:f004::2/64"
- "2001:ba8:1f1:f004::22/128"
routes:
- to: default
via: "85.119.80.1"
- to: default
via: "2001:ba8:1f1:f004::1"
dhcp4: false
dhcp6: false
nameservers:
addresses:
- "85.119.80.232"
- "85.119.80.233"
- "2001:ba8:1f1:f205::53"
- "2001:ba8:1f1:f206::53"
This assumes:
- A main IPv4 address of 85.119.82.225.
- An additional IPv4 address of 85.119.83.73.
- An assigned IPv6 network of 2001:ba8:1f1:f004::/64.
- An extra IPv6 address of 2001:ba8:1f1:f004::22/128 configured.
To update the config and then make it live:
$ sudo netplan generate
$ sudo netplan apply