Full disk encryption

Full disk encryption (FDE) is the practice of encrypting the entirety of the VPS's storage such that even swap and temporary files are protected. On Linux it's typically achieved using LUKS and these days has very good installer support.

Rationale

Unless you encrypt your data it is always going to be possible for BitFolk to read it. While you should feel comfortable trusting BitFolk to have no desire to read your data, you cannot prevent BitFolk being legally compelled to read your data by the authorities without your knowledge. Also if BitFolk's systems were compromised then there would be nothing to stop the attacker from reading your data.

Do note that a determined attacker with root access at BitFolk can still fairly easily gain access to your data by reading all possible LUKS master keys from a memory dump and then trying all of them against a snapshot of your disk(s). Here is an example of someone doing this; it took them ~8.5 hours on their personal computer to break into the storage of a virtual machine with 1GiB of RAM and 5GiB of disk.

Challenges

There are some negatives to using full disk encryption on a BitFolk VPS.

• Your VPS can no longer boot unattended. Someone has to connect to the console and type the passphrase.
• The Xen console is still a bit of a weak link with regards to security; root at BitFolk can easily sniff it.
• Reduces the ability of BitFolk to help you out of any mishaps that you may have since BitFolk won't be able to manipulate your data from outside.
• Makes growing your storage a bit trickier since it's no longer just a case of letting BitFolk grow the partition and then you running resize2fs. See below.
• Makes shrinking extremely hazardous also. Perhaps best to move everything to new, smaller disk and then cancel larger one.
• Lose the passphrase, lose all the data.

Installing

BitFolk can't install your VPS for you if you wish to use FDE. As the idea is for BitFolk to be unable to read your data, it does not make sense for BitFolk to be the one choosing the passphrase and performing the install.

Fortunately, FDE support within modern Linux installers is very good. In the case of Debian and Ubuntu at least, it's easy to install a VPS with encrypted storage without ever leaving the installer. The rest of this article is going to assume that Debian or Ubuntu is in use.

Tips

• Choose the manual method of partitioning.

• Create a reasonably-sized /boot partition as xvda1.

  /boot is the only bit of your storage that BitFolk needs to read in order to boot your VPS. The only things that you will have stored in /boot are your kernels, initramfs and bootloader config. You'll never put any sensitive information in /boot.

  A kernel and initramfs will use up about 20MiB of storage but you will want a /boot of at least a couple of hundred MiB in size. Use the rest for an encrypted partition (xvda2).

• It's a good idea to use the encrypted device as a physical volume for LVM and then create one or more logical volumes inside that. It will make it a lot easier to resize things later.
• If you do use LVM, don't use all of the available space right away. Save some for adding additional logical volumes later when you know where the space will be needed. Growing is easier than shrinking.

What to do about xvdb?

BitFolk VPSes come with two disks: xvdba and xvdb. It is intended that you will use xvdb for swap. When using FDE you should also encrypt your swap, as any sensitive information from memory could end up on your swap device. The trouble is that this then results in you having two encryption passphrases to input during boot.

If it's just going to be used as swap then dm-crypt contains support for formatting with a random key at boot time. Here's how.

Install as normal

Perform a normal install using FDE only on xvda. Don't create a swap device. The installer will warn you that you don't have a swap device, but just continue.

Create a small filesystem on xvdb

Since dm-crypt is going to be overwriting this device on every boot, you want to be really, really certain that it's found the correct device. Doing this by device name is a very bad idea as devices could get rearranged one day. It's best to do it by UUID, but you need a constant filesystem to get a UUID out of. So, create a tiny ext2 filesystem at the start of xvdb.

root@debtest1:~# mkfs.ext2 -L cryptswap /dev/xvdb 1M
mke2fs 1.42.12 (29-Aug-2014)
Discarding device blocks: done                            
Creating filesystem with 1024 1k blocks and 128 inodes

Allocating group tables: done                            
Writing inode tables: done                            
Writing superblocks and filesystem accounting information: done

root@debtest1:~# blkid /dev/xvdb
/dev/xvdb: LABEL="cryptswap" UUID="3debde75-1007-40d7-9e97-58bcbc12e4da" TYPE="ext2"

Add an entry to crypttab for the rest of the disk

Now add an entry to crypttab that finds the small filesystem that was just made, skips over it and uses the rest of that disk. The offset parameter does that.

root@debtest1:~# echo "swap LABEL=cryptswap /dev/urandom swap,offset=2048,cipher=aes-xts-plain64,size=256" >> /etc/crypttab

Add an entry to fstab for the swap

On boot, dm-crypt should now create the /dev/mapper/crypt device, so add that to /etc/fstab as swap.

root@debtest1:~# echo "/dev/mapper/swap none	swap	sw		0	0" >> /etc/fstab

Reboot and confirm the swap worked

root@debtest1:~# free -m
             total       used       free     shared    buffers     cached
Mem:           996         93        903          4          9         38
-/+ buffers/cache:         44        951
Swap:         1022          0       1022
root@debtest1:~# swapon -s
Filename				Type		Size	Used	Priority
/dev/dm-2                              	partition	1047548	0	-1
root@debtest1:~# dmsetup info /dev/dm-2
Name:              swap
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        2
Event number:      0
Major, minor:      254, 2
Number of targets: 1
UUID: CRYPT-PLAIN-swap

Unlocking other devices by key

What about if you have additional disks added? If you encrypt these as well you'll have multiple passphrases to type during boot.

To make things easier you can have them unlocked by a key file that resides on your root (encrypted) filesystem. Here's how.

Here we will assume you have just add /dev/xvdc.

Create an encrypted device on xvdc

root@debtest1:~# cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha512 /dev/xvdc

WARNING!
========
This will overwrite data on /dev/xvdc irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase: 
Verify passphrase:

This asked for a passphrase and you should use a strong one as that passphrase will still be able to unlock the device. You won't be using the passphrase after you've got this working though.

Create a key file

Now put some random data into a key file somewhere that is inside xvda and will definitely be available soon in the boot process.

root@debtest1:~# dd if=/dev/urandom of=/root/xvdc_key bs=1024 count=4
4+0 records in
4+0 records out
4096 bytes (4.1 kB) copied, 0.000732261 s, 5.6 MB/s
root@debtest1:~# chmod -c 0400 /root/xvdb_key
mode of ‘/root/xvdc_key’ changed from 0644 (---r--r--) to 0400 (r--------)

Allow the new device to be unlocked with the key file

root@debtest1:~# cryptsetup luksAddKey /dev/xvdc /root/xvdc_key 
Enter any passphrase:

From this point on, cryptsetup will be able to unlock xvdc without a human needing to type the passphrase.

Add the new device to crypttab so it'll be set up every boot

You'll need the UUID of xvdc for this.

root@debtest1:~# blkid /dev/xvdc
/dev/xvdc: UUID="baddb6ec-3731-45c5-9a72-16f7b726d252" TYPE="crypto_LUKS"
root@debtest1:~# echo "xvdc_crypt UUID=baddb6ec-3731-45c5-9a72-16f7b726d252 /root/xvdc_key luks" >> /etc/crypttab

Reboot and check things out

You should have only been asked for one passphrase during boot (the one for xvda).

root@debtest1:~# dmsetup info /dev/dm-3
Name:              xvdc_crypt
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        2
Event number:      0
Major, minor:      254, 2
Number of targets: 1
UUID: CRYPT-LUKS1-baddb6ec373145c59a7216f7b726d252-xvdc_crypt

Growing a filesystem on an encrypted partition

With a normal unencrypted disk, growing is pretty simple. A quick recap:

1. You order extra storage from BitFolk.
2. BitFolk tells you when your disk has been grown.
3. Your VPS's kernel, if modern enough, already sees the device is larger. If your filesystem is directly on the device then you can just grow it using the usual tool (e.g. resize2fs).
4. If the device is partitioned and it's your root filesystem we're talking about then you may need to reboot to get it to see the new partition table. Then resize filesystem.

Things are a bit more involved with the encrypted setup.

Here we assume there is a 5GiB xvda with a small /boot as xvda1 and then an encrypted partition as xvda2. An order has been made to increase xvda to 10GiB and you've been told by BitFolk that this has now been grown, with xvda2 grown into the new space.

Your kernel will most likely have seen xvda grow, but it won't have been able to reread the partition table to see any change in xvda2. So, unfortunately, reboot is necessary.

If there will be a re-opening of the LUKS device then it doesn't normally need an explicit resize as it detects this when it's set up. After boot you can confirm the encrypted device is now ~10GiB in size (~200MiB was used for xvda1):

root@debtest1:~# cryptsetup status xvda2_crypt
/dev/mapper/xvda2_crypt is active and is in use.
  type:    LUKS1
  cipher:  aes-xts-plain64
  keysize: 512 bits
  device:  /dev/xvda2
  offset:  4096 sectors
  size:    20576256 sectors
  mode:    read/write
root@debtest1:~# echo $((20576256*512/1024/1024))
10047

The LVM physical volume hasn't been resized yet though; it's still occupying the first 5GiB of the partition:

root@debtest1:~# pvdisplay
  --- Physical volume ---
  PV Name               /dev/mapper/xvda2_crypt
  VG Name               debtest1
  PV Size               4.81 GiB / not usable 0   
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              1231
  Free PE               0
  Allocated PE          1231
  PV UUID               Sgn1SN-X2P8-GNtN-vaRj-O8LP-vBxP-askT2F

It's a simple matter to resize that with pvresize:

root@debtest1:~# pvresize /dev/mapper/xvda2_crypt
  Physical volume "/dev/mapper/xvda2_crypt" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized
root@debtest1:~# pvdisplay
  --- Physical volume ---
  PV Name               /dev/mapper/xvda2_crypt
  VG Name               debtest1
  PV Size               9.81 GiB / not usable 2.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              2511
  Free PE               1280
  Allocated PE          1231
  PV UUID               Sgn1SN-X2P8-GNtN-vaRj-O8LP-vBxP-askT2F

And then the logical volume that / is in, and finally the filesystem itself:

root@debtest1:~# lvresize -l100%VG /dev/debtest1/root
  Size of logical volume debtest1/root changed from 4.81 GiB (1231 extents) to 9.81 GiB (2511 extents).
  Logical volume root successfully resized
root@debtest1:~# resize2fs /dev/debtest1/root
resize2fs 1.42.12 (29-Aug-2014)
Filesystem at /dev/debtest1/root is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
[ 1324.193561] EXT4-fs (dm-1): resizing filesystem from 1260544 to 2571264 blocks
[ 1324.207357] EXT4-fs (dm-1): resized filesystem to 2571264
The filesystem on /dev/debtest1/root is now 2571264 (4k) blocks long.

root@debtest1:~# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/dm-1       9.6G  914M  8.3G  10% /

-r could have been used on the lvresize command line here to automatically invoke resize2fs but I thought I'd break it out for clarity.

Shrinking a filesystem on an encrypted partition

If you are brave enough to try this, please let us know how you got on by editing this article.

TODO

• Document how to setup initial LUKS passphrase entry by SSH. Xen console is insecure as it can easily be sniffed by root on BitFolk infrastructure. It is possible to set up early SSH with e.g. Dropbear: https://github.com/chadoe/luks-triple-unlock

Further information

• You've Got to Trust Your Vm Host
• Cryptsetup
• Details about how the auto-formatted encrypted swap works