Technical Documentation - Pop!_OS installation & configuration

Page / - last modified by Jan Rhebergen on 2022/01/24 15:57

Pop!_OS installation & configuration

last modified by Jan Rhebergen

on 2022/01/24 15:57

Table of Contents

Encryption

Implementation

Procedure outline

1. Create a fallback boot option

2. Create the secret keyfile

3. Add key to keyslot on encrypted device

4. Store the keyfile on the USB flash drive

5. Create the keyfile script to get key to stdout

6. Edit /etc/crypttab to enable the key script and regenerate initrd.img

All the servers and laptop have been installed with Pop!_OS. The reason being the special attention Pop!_OS is givng to Nvidia GPU support. We want good and flexible NVidia GPU support because we plan to use the CUDA libraries for machine learning applications. Pop!_OS is an Ubuntu derivative with a little more than just "re-theming".

Laptops are all configured using encrypted storage devices. This is a safety precaution as laptops are mobile devices and are regularly moved between locations. A stolen or lost laptop hence should not pose a security threat. Although the data on the laptop is non-confidential the mere fact that this is a Defence owned laptop will create an embarrassment. This we don't want even though there is no real security issue at stake here.

Server are supposed to be at a fixed secure location. Storage device encryption is not deemed necessary. If a server happens to be a laptop (yes this can be) we still implement storage level encryption.

Although strictly speaking not needed for laptops where possible we employ the btrfs filesystem. This has several advantages:

Flexibility (expandability)

Snapshots

Btrfs filesystems are by now considered mature and offer more flexibility than other enterprise grade filesystem such as zfs. For small deployments therefore btrfs is a better fit.

The possibility to create instant snapshots offer great way to safeguard against accidental loss through inadvertenty deleting file or other minor mishaps that can be annoying and cost substantial amounts of time to recover from.

The following guide https://mutschler.eu/linux/install-guides/pop-os-btrfs/ was used although adapted to our specific case, i.e. no virtual machine and depending on use case either encryption or not.

Encryption

Only laptops are going to be encrypted because they may in theory be transported and used elsewhere and could be stolen or lost. The embarrassment to have this happen to an unencrypted Defence laptop is reason enough to warrant encrypting the hard drive.

The mere form factor of a laptop does not preclude it from functioning as a server. There is however a small problem. When you are logged in remotely and you update/upgrade the operating system you may need to reboot the laptop. Due to the encryption you have to physically be present at the same location as the laptop to enter the pass-phrase that starts decryption on booting. Of course this is not possible when you are working remote. The answer to this problem is to create solution where an external USB flash drive is detected on booting the system and used to store/retrieve a key that enables decryption. The boot menu will still contain a back-up boot option that works with a physically entered pass-phrase. The default boot option will look for the USB flash drive with decryption key. A better way to achieve the same goal is to use a ubikey https://www.yubico.com/ It has some additional advantages like two factor authentication, etc. In case the USB flash drive with key is lost or stolen, one can simple remove the respective key from the keyslot on the encrypted device thus keeping it secure.

Implementation

As stated above one normally enters a passphrase to unlock the encrypted device and boot the system. The alternative is to use a key to do the job. This key is read automatically from a device and is used to decrypt the hard drive which contains the operating system and file system with user directories home.

Procedure outline

Create an alternate backup systemd-boot boot option (for safety/emergency reasons).

Create and save a suitable keyfile, either a binary or a character sequence.

Add the keyfile to a key-slot on the encrypted device (the hard drive with the root and home filesystem)

Prepare a USB device, obtain device partition ID and copy the keyfile to it.

Create a script that is called at boot time and searches for the USB ID and copy it to /boot/.

Edit /etc/crypttab to enable the keyfile.sh script and regenerate initrd.img

Finally Test by rebooting with USB flash drive inserted

1. Create a fallback boot option

To create a fallback boot option do the following:

cd /boot/efi/EFI/Pop_OS-eea7a2f5-b204-4de3-b605-3696ed323544

cp vmlinuz.efi vmlinuz-backup.efi

cp initrd.img initrd.img-backup

cd /boot/efi/loader/entries

cp Pop_OS-current.conf Pop_OS-backup.conf

Then edit the file Pop_OS-backup.conf like shown below:

title Pop!_OS

linux /EFI/Pop_OS-eea7a2f5-b204-4de3-b605-3696ed323544/vmlinuz-backup.efi

initrd /EFI/Pop_OS-eea7a2f5-b204-4de3-b605-3696ed323544/initrd.img-backup

options root=UUID=eea7a2f5-b204-4de3-b605-3696ed323544 ro quiet loglevel=0 systemd.show_status=false splash rootflags=subvol=@

Changing the title is optional. NB: it is important to reboot and test this option now!

2. Create the secret keyfile

The key can be a sequence of printable (visible) characters or any binary sequence of a certain length. Below examples of how to generate these:

binary sequence:

dd if=/dev/urandom of=/etc/luks-keys/mykey.bin bs=512 count=4

character sequence:

Either this one:

head -c 2048 /dev/urandom | base64 -w 0 > mykey.txt

chmod -v 0400 mykey.txt

or slightly better, this one:

dd bs=256 count=8 if=/dev/random | base64 -w 0 > mykey.txt

chmod -v 0400 mykey.txt

Either key is fine, but if you would like to be able to type it in via the keyboard, the latter option is preferred for obvious reasons. The key will reside on a USB flash drive. However this flash drive may be removed intentionally or unintentionally. Hence deploying the key based automated decryption method we still want to create a fallback option in case the key is not available. This fallback boot option uses an untainted kernel and initrd image (i.e. originals) to be used with the regular pass phrase that is to be entered at the prompt before booting the system.

3. Add key to keyslot on encrypted device

Be careful to choose the correct device for adding the key. Using parted /dev/nvme0n1 one can inspect the partition layout of the first device (in this case an SSD).

root@schumann:~# parted /dev/nvme0n1

GNU Parted 3.3

Using /dev/nvme0n1

Welcome to GNU Parted! Type 'help' to view a list of commands.

(parted) p

Model: INTEL SSDPEKNW512G8H (nvme)

Disk /dev/nvme0n1: 512GB

Sector size (logical/physical): 512B/512B

Partition Table: gpt

Disk Flags:

Number Start End Size File system Name Flags

1 1049kB 538MB 537MB fat32 SYSTEM boot, esp

2 538MB 555MB 16.8MB Microsoft reserved partition msftres

3 555MB 105GB 105GB ntfs Basic data partition msftdata

5 105GB 512GB 406GB cryptdata

4 512GB 512GB 503MB ntfs Basic data partition hidden, diag

(parted)

cryptsetup -v luksAddKey /dev/nvme0n1p5 mykey.txt

4. Store the keyfile on the USB flash drive

Upon boot the key needs to be retrieved from the USB flash drive so we need to store it there first. The USB flash drive is prepared in such a way as to obfuscate the fact that a key is present. The text below shows the partition layout of the USB flash drive. There is a small hidden 1MiB partition which we will use to hold the key. One could also decide to obfuscate a little more by not making a true partition but simply start the first partition at an offset of 1MiB beyond the partition table and store the key at a known (predetermined) offset within that 1MiB space. However this we consider potentially error prone and needs to be tested well before deciding to adopt this approach.

root@schumann:~# parted /dev/sda

GNU Parted 3.3

Using /dev/sda

Welcome to GNU Parted! Type 'help' to view a list of commands.

(parted) p

Model: USB BAR (scsi)

Disk /dev/sda: 65.4MB

Sector size (logical/physical): 512B/512B

Partition Table: gpt

Disk Flags:

Number Start End Size File system Name Flags

1 1049kB 2097kB 1049kB KEY hidden

2 2097kB 65.4MB 63.3MB fat16 SILVER msftdata

(parted)

To be able to detect the USB flash drive when inserted at boot time we need to know it's device ID.

root@schumann:~# ls -l /dev/disk/by-id | grep sda2

lrwxrwxrwx 1 root root 10 Apr 21 10:07 usb-USB_USB_BAR-0:0-part2 -> ../../sda2

We also need to know (confirm) the size of the key file.

root@schumann:~# ls -l

total 4

drwxr-xr-x 1 root root 32 Apr 21 09:38 bin

drwxr-xr-x 1 root root 28 Apr 19 12:06 docs

-rw-r--r-- 1 root root 2732 Apr 21 10:45 mykey.txt

root@schumann:~# dd if=mykey.txt of=/dev/disk/by-id/usb-USB_USB_BAR-0:0-part1 bs=1 count=2732

2732+0 records in

2732+0 records out

2732 bytes (2.7 kB, 2.7 KiB) copied, 0.00868143 s, 315 kB/s

5. Create the keyfile script to get key to stdout

The script below will wait for our USB device to become available and read the key from the respective number of bytes (2732) on partition 1. The script keyfile.sh is (for now) stored in the /bin/ directory.

for TRY in {1..60}

do

if ! [ -e /dev/disk/by-id/usb-USB_USB_BAR-0:0-part2 ]; then

# echo $TRY

sleep 1

fi

done # for TRY

if [ -e /dev/disk/by-id/usb-USB_USB_BAR-0:0-part1 ]; then

dd if=/dev/disk/by-id/usb-USB_USB_BAR-0:0-part1 bs=1 skip=0 count=2732 2>/dev/null

fi

Let's see if the key that is read by the script is identical to the original one in our home directory.

{ [[ "$(~/bin/keyfile.sh)" == "$(cat ~/mykey.txt)" ]] && echo 'SUCCESS.'; } || echo 'FAILURE: NOT THE SAME.';

Make sure the keyfile.sh script is executable (should already be).

chmod +x keyfile.sh

chmod -v 0500 keyfile.sh

cp keyfile.sh /boot/

6. Edit /etc/crypttab to enable the key script and regenerate initrd.img

Now modify /etc/crypttab from:

cryptdata UUID=bacc09ab-6e12-4365-be8f-d7b09ae0c0a5 none luks

to:

cryptdata UUID=bacc09ab-6e12-4365-be8f-d7b09ae0c0a5 /dev/disk/by-id/usb-USB_USB_BAR-0:0-part1 luks,keyscript=/boot/keyfile.sh

and update initrd.img

update-initramfs -u