Pop!_OS installation & configuration
All the servers and laptop have been installed with Pop!_OS. The reason being the special attention Pop!_OS is givng to Nvidia GPU support. We want good and flexible NVidia GPU support because we plan to use the CUDA libraries for machine learning applications. Pop!_OS is an Ubuntu derivative with a little more than just "re-theming".
Laptops are all configured using encrypted storage devices. This is a safety precaution as laptops are mobile devices and are regularly moved between locations. A stolen or lost laptop hence should not pose a security threat. Although the data on the laptop is non-confidential the mere fact that this is a Defence owned laptop will create an embarrassment. This we don't want even though there is no real security issue at stake here.
Server are supposed to be at a fixed secure location. Storage device encryption is not deemed necessary. If a server happens to be a laptop (yes this can be) we still implement storage level encryption.
Although strictly speaking not needed for laptops where possible we employ the btrfs filesystem. This has several advantages:
- Flexibility (expandability)
- Snapshots
Btrfs filesystems are by now considered mature and offer more flexibility than other enterprise grade filesystem such as zfs. For small deployments therefore btrfs is a better fit.
The possibility to create instant snapshots offer great way to safeguard against accidental loss through inadvertenty deleting file or other minor mishaps that can be annoying and cost substantial amounts of time to recover from.
The following guide https://mutschler.eu/linux/install-guides/pop-os-btrfs/ was used although adapted to our specific case, i.e. no virtual machine and depending on use case either encryption or not.
Encryption
Only laptops are going to be encrypted because they may in theory be transported and used elsewhere and could be stolen or lost. The embarrassment to have this happen to an unencrypted Defence laptop is reason enough to warrant encrypting the hard drive.
The mere form factor of a laptop does not preclude it from functioning as a server. There is however a small problem. When you are logged in remotely and you update/upgrade the operating system you may need to reboot the laptop. Due to the encryption you have to physically be present at the same location as the laptop to enter the pass-phrase that starts decryption on booting. Of course this is not possible when you are working remote. The answer to this problem is to create solution where an external USB flash drive is detected on booting the system and used to store/retrieve a key that enables decryption. The boot menu will still contain a back-up boot option that works with a physically entered pass-phrase. The default boot option will look for the USB flash drive with decryption key. A better way to achieve the same goal is to use a ubikey https://www.yubico.com/ It has some additional advantages like two factor authentication, etc. In case the USB flash drive with key is lost or stolen, one can simple remove the respective key from the keyslot on the encrypted device thus keeping it secure.
Implementation
As stated above one normally enters a passphrase to unlock the encrypted device and boot the system. The alternative is to use a key to do the job. This key is read automatically from a device and is used to decrypt the hard drive which contains the operating system and file system with user directories home.
Procedure outline
- Create an alternate backup systemd-boot boot option (for safety/emergency reasons).
- Create and save a suitable keyfile, either a binary or a character sequence.
- Add the keyfile to a key-slot on the encrypted device (the hard drive with the root and home filesystem)
- Prepare a USB device, obtain device partition ID and copy the keyfile to it.
- Create a script that is called at boot time and searches for the USB ID and copy it to /boot/.
- Edit /etc/crypttab to enable the keyfile.sh script and regenerate initrd.img
Finally Test by rebooting with USB flash drive inserted
1. Create a fallback boot option
To create a fallback boot option do the following:
cd /boot/efi/EFI/Pop_OS-eea7a2f5-b204-4de3-b605-3696ed323544 cp vmlinuz.efi vmlinuz-backup.efi cp initrd.img initrd.img-backup cd /boot/efi/loader/entries cp Pop_OS-current.conf Pop_OS-backup.conf
Then edit the file Pop_OS-backup.conf like shown below:
title Pop!_OS linux /EFI/Pop_OS-eea7a2f5-b204-4de3-b605-3696ed323544/vmlinuz-backup.efi initrd /EFI/Pop_OS-eea7a2f5-b204-4de3-b605-3696ed323544/initrd.img-backup options root=UUID=eea7a2f5-b204-4de3-b605-3696ed323544 ro quiet loglevel=0 systemd.show_status=false splash rootflags=subvol=@
Changing the title is optional. NB: it is important to reboot and test this option now!
2. Create the secret keyfile
The key can be a sequence of printable (visible) characters or any binary sequence of a certain length. Below examples of how to generate these:
binary sequence:
dd if=/dev/urandom of=/etc/luks-keys/mykey.bin bs=512 count=4
character sequence:
Either this one:
head -c 2048 /dev/urandom | base64 -w 0 > mykey.txt chmod -v 0400 mykey.txt
or slightly better, this one:
dd bs=256 count=8 if=/dev/random | base64 -w 0 > mykey.txt chmod -v 0400 mykey.txt
Either key is fine, but if you would like to be able to type it in via the keyboard, the latter option is preferred for obvious reasons. The key will reside on a USB flash drive. However this flash drive may be removed intentionally or unintentionally. Hence deploying the key based automated decryption method we still want to create a fallback option in case the key is not available. This fallback boot option uses an untainted kernel and initrd image (i.e. originals) to be used with the regular pass phrase that is to be entered at the prompt before booting the system.
3. Add key to keyslot on encrypted device
Be careful to choose the correct device for adding the key. Using parted /dev/nvme0n1 one can inspect the partition layout of the first device (in this case an SSD).
root@schumann:~# parted /dev/nvme0n1 GNU Parted 3.3 Using /dev/nvme0n1 Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) p Model: INTEL SSDPEKNW512G8H (nvme) Disk /dev/nvme0n1: 512GB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 538MB 537MB fat32 SYSTEM boot, esp 2 538MB 555MB 16.8MB Microsoft reserved partition msftres 3 555MB 105GB 105GB ntfs Basic data partition msftdata 5 105GB 512GB 406GB cryptdata 4 512GB 512GB 503MB ntfs Basic data partition hidden, diag (parted)
cryptsetup -v luksAddKey /dev/nvme0n1p5 mykey.txt
4. Store the keyfile on the USB flash drive
Upon boot the key needs to be retrieved from the USB flash drive so we need to store it there first. The USB flash drive is prepared in such a way as to obfuscate the fact that a key is present. The text below shows the partition layout of the USB flash drive. There is a small hidden 1MiB partition which we will use to hold the key. One could also decide to obfuscate a little more by not making a true partition but simply start the first partition at an offset of 1MiB beyond the partition table and store the key at a known (predetermined) offset within that 1MiB space. However this we consider potentially error prone and needs to be tested well before deciding to adopt this approach.
root@schumann:~# parted /dev/sda GNU Parted 3.3 Using /dev/sda Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) p Model: USB BAR (scsi) Disk /dev/sda: 65.4MB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 2097kB 1049kB KEY hidden 2 2097kB 65.4MB 63.3MB fat16 SILVER msftdata (parted)
To be able to detect the USB flash drive when inserted at boot time we need to know it's device ID.
root@schumann:~# ls -l /dev/disk/by-id | grep sda2 lrwxrwxrwx 1 root root 10 Apr 21 10:07 usb-USB_USB_BAR-0:0-part2 -> ../../sda2
We also need to know (confirm) the size of the key file.
root@schumann:~# ls -l total 4 drwxr-xr-x 1 root root 32 Apr 21 09:38 bin drwxr-xr-x 1 root root 28 Apr 19 12:06 docs -rw-r--r-- 1 root root 2732 Apr 21 10:45 mykey.txt
root@schumann:~# dd if=mykey.txt of=/dev/disk/by-id/usb-USB_USB_BAR-0:0-part1 bs=1 count=2732 2732+0 records in 2732+0 records out 2732 bytes (2.7 kB, 2.7 KiB) copied, 0.00868143 s, 315 kB/s
5. Create the keyfile script to get key to stdout
The script below will wait for our USB device to become available and read the key from the respective number of bytes (2732) on partition 1. The script keyfile.sh is (for now) stored in the /bin/ directory.
for TRY in {1..60}
do
if ! [ -e /dev/disk/by-id/usb-USB_USB_BAR-0:0-part2 ]; then
# echo $TRY
sleep 1
fi
done # for TRY
if [ -e /dev/disk/by-id/usb-USB_USB_BAR-0:0-part1 ]; then
dd if=/dev/disk/by-id/usb-USB_USB_BAR-0:0-part1 bs=1 skip=0 count=2732 2>/dev/null
fi
Let's see if the key that is read by the script is identical to the original one in our home directory.
{ [[ "$(~/bin/keyfile.sh)" == "$(cat ~/mykey.txt)" ]] && echo 'SUCCESS.'; } || echo 'FAILURE: NOT THE SAME.';
Make sure the keyfile.sh script is executable (should already be).
chmod +x keyfile.sh chmod -v 0500 keyfile.sh cp keyfile.sh /boot/
6. Edit /etc/crypttab to enable the key script and regenerate initrd.img
Now modify /etc/crypttab from:
cryptdata UUID=bacc09ab-6e12-4365-be8f-d7b09ae0c0a5 none luks
to:
cryptdata UUID=bacc09ab-6e12-4365-be8f-d7b09ae0c0a5 /dev/disk/by-id/usb-USB_USB_BAR-0:0-part1 luks,keyscript=/boot/keyfile.sh
and update initrd.img
update-initramfs -u