Technical Documentation - Pop!_OS installation & configuration 

Page / - last modified by Jan Rhebergen on 2022/01/24 15:57 

Pop!_OS installation & configuration 
last modified by Jan Rhebergen
on 2022/01/24 15:57

Table of Contents
Encryption
Implementation
Procedure outline
1. Create a fallback boot option
2. Create the secret keyfile
3. Add key to keyslot on encrypted device
4. Store the keyfile on the USB flash drive
5. Create the keyfile script to get key to stdout
6. Edit /etc/crypttab to enable the key script and regenerate initrd.img

All the servers and laptop have been installed with Pop!_OS. The reason being the special attention Pop!_OS is givng to Nvidia GPU support. We want good and flexible NVidia GPU support because we plan to use the CUDA libraries for machine learning applications. Pop!_OS is an Ubuntu derivative with a little more than just "re-theming".
Laptops are all configured using encrypted storage devices. This is a safety precaution as laptops are mobile devices and are regularly moved between locations. A stolen or lost laptop hence should not pose a security threat. Although the data on the laptop is non-confidential the mere fact that this is a Defence owned laptop will create an embarrassment. This we don't want even though there is no real security issue at stake here.
Server are supposed to be at a fixed secure location. Storage device encryption is not deemed necessary. If a server happens to be a laptop (yes this can be) we still implement storage level encryption.
Although strictly speaking not needed for laptops where possible we employ the btrfs filesystem. This has several advantages:
•	Flexibility (expandability)
•	Snapshots
Btrfs filesystems are by now considered mature and offer more flexibility than other enterprise grade filesystem such as zfs. For small deployments therefore btrfs is a better fit.
The possibility to create instant snapshots offer great way to safeguard against accidental loss through inadvertenty deleting file or other minor mishaps that can be annoying and cost substantial amounts of time to recover from.
The following guide https://mutschler.eu/linux/install-guides/pop-os-btrfs/ was used although adapted to our specific case, i.e. no virtual machine and depending on use case either encryption or not.
Encryption
Only laptops are going to be encrypted because they may in theory be transported and used elsewhere and could  be stolen or lost. The embarrassment to have this happen to an unencrypted Defence laptop is reason enough to warrant encrypting the hard drive.
The mere form factor of a laptop does not preclude it from functioning as a server. There is however a small problem. When you are logged in remotely and you update/upgrade the operating system you may need to reboot the laptop. Due to the encryption you have to physically be present at the same location as the laptop to enter the pass-phrase that starts decryption on booting. Of course this is not possible when you are working remote. The answer to this problem is to create solution where an external USB flash drive is detected on booting the system and used to store/retrieve a key that enables decryption. The boot menu will still contain a back-up boot option that works with a physically entered pass-phrase. The default boot option will look for the USB flash drive with decryption key. A better way to achieve the same goal is to use a ubikey https://www.yubico.com/ It has some additional advantages like two factor authentication, etc. In case the USB flash drive with key is lost or stolen, one can simple remove the respective key from the keyslot on the encrypted device thus keeping it secure.
Implementation
As stated above one normally enters a passphrase to unlock the encrypted device and boot the system. The alternative is to use a key to do the job. This key is read automatically from a device and is used to decrypt the hard drive which contains the operating system and file system with user directories home. 
Procedure outline
 1.	Create an alternate backup systemd-boot boot option (for safety/emergency reasons).
 2.	Create and save a suitable keyfile, either a binary or a character sequence.
 3.	Add the keyfile to a key-slot on the encrypted device (the hard drive with the root and home filesystem)
 4.	Prepare a USB device, obtain device partition ID and copy the keyfile to it.
 5.	Create a script that is called at boot time and searches for the USB ID and copy it to /boot/.
 6.	Edit /etc/crypttab to enable the keyfile.sh script and regenerate initrd.img
Finally Test by rebooting with USB flash drive inserted
1. Create a fallback boot option
To create a fallback boot option do the following:
cd /boot/efi/EFI/Pop_OS-eea7a2f5-b204-4de3-b605-3696ed323544
cp vmlinuz.efi vmlinuz-backup.efi
cp initrd.img initrd.img-backup

cd /boot/efi/loader/entries
cp Pop_OS-current.conf Pop_OS-backup.conf 
Then edit the file Pop_OS-backup.conf like shown below:
title Pop!_OS
linux /EFI/Pop_OS-eea7a2f5-b204-4de3-b605-3696ed323544/vmlinuz-backup.efi
initrd /EFI/Pop_OS-eea7a2f5-b204-4de3-b605-3696ed323544/initrd.img-backup
options root=UUID=eea7a2f5-b204-4de3-b605-3696ed323544 ro quiet loglevel=0 systemd.show_status=false splash rootflags=subvol=@
Changing the title is optional. NB: it is important to reboot and test this option now!
2. Create the secret keyfile
The key can be a sequence of printable (visible) characters or any binary sequence of a certain length. Below examples of how to generate these:
binary sequence:
dd if=/dev/urandom of=/etc/luks-keys/mykey.bin bs=512 count=4
character sequence:
Either this one:
head -c 2048 /dev/urandom | base64 -w 0 > mykey.txt
chmod -v 0400 mykey.txt
or slightly better, this one:
dd bs=256 count=8 if=/dev/random | base64 -w 0 > mykey.txt
chmod -v 0400 mykey.txt
Either key is fine, but if you would like to be able to type it in via the keyboard, the latter option is preferred for obvious reasons. The key will reside on a USB flash drive. However this flash drive may be removed intentionally or unintentionally. Hence deploying the key based automated decryption method we still want to create a fallback option in case the key is not available. This fallback boot option uses an untainted kernel and initrd image (i.e. originals) to be used with the regular pass phrase that is to be entered at the prompt before booting the system.
3. Add key to keyslot on encrypted device
Be careful to choose the correct device for adding the key. Using parted /dev/nvme0n1 one can inspect the partition layout of the first device (in this case an SSD).
root@schumann:~# parted /dev/nvme0n1
GNU Parted 3.3
Using /dev/nvme0n1
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p                                                                
Model: INTEL SSDPEKNW512G8H (nvme)
Disk /dev/nvme0n1: 512GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End    Size    File system  Name                          Flags
 1      1049kB  538MB  537MB   fat32        SYSTEM                        boot, esp
 2      538MB   555MB  16.8MB               Microsoft reserved partition  msftres
 3      555MB   105GB  105GB   ntfs         Basic data partition          msftdata
 5      105GB   512GB  406GB                cryptdata
 4      512GB   512GB  503MB   ntfs         Basic data partition          hidden, diag

(parted)                                                                  

cryptsetup -v luksAddKey /dev/nvme0n1p5 mykey.txt
4. Store the keyfile on the USB flash drive
Upon boot the key needs to be retrieved from the USB flash drive so we need to store it there first. The USB flash drive is prepared in such a way as to obfuscate the fact that a key is present. The text below shows the partition layout of the USB flash drive. There is a small hidden 1MiB partition which we will use to hold the key. One could also decide to obfuscate a little more by not making a true partition but simply start the first partition at an offset of 1MiB beyond the partition table and store the key at a known (predetermined) offset within that 1MiB space. However this we consider potentially error prone and needs to be tested well before deciding to adopt this approach. 
root@schumann:~# parted /dev/sda
GNU Parted 3.3
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p                                                                
Model:  USB BAR (scsi)
Disk /dev/sda: 65.4MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name    Flags
 1      1049kB  2097kB  1049kB               KEY     hidden
 2      2097kB  65.4MB  63.3MB  fat16        SILVER  msftdata

(parted)  
To be able to detect the USB flash drive when inserted at boot time we need to know it's device ID. 
root@schumann:~# ls -l /dev/disk/by-id | grep sda2
lrwxrwxrwx 1 root root 10 Apr 21 10:07 usb-USB_USB_BAR-0:0-part2 -> ../../sda2
We also need to know (confirm) the size of the key file.
root@schumann:~# ls -l
total 4
drwxr-xr-x 1 root root   32 Apr 21 09:38 bin
drwxr-xr-x 1 root root   28 Apr 19 12:06 docs
-rw-r--r-- 1 root root 2732 Apr 21 10:45 mykey.txt
root@schumann:~# dd if=mykey.txt of=/dev/disk/by-id/usb-USB_USB_BAR-0:0-part1  bs=1 count=2732
2732+0 records in
2732+0 records out
2732 bytes (2.7 kB, 2.7 KiB) copied, 0.00868143 s, 315 kB/s
5. Create the keyfile script to get key to stdout
The script below will wait for our USB device to become available and read the key from the respective number of bytes (2732) on partition 1. The script keyfile.sh is (for now) stored in the /bin/ directory.
for TRY in {1..60}
do
  if ! [ -e /dev/disk/by-id/usb-USB_USB_BAR-0:0-part2 ]; then
#    echo $TRY
    sleep 1
  fi
done # for TRY
if [ -e /dev/disk/by-id/usb-USB_USB_BAR-0:0-part1 ]; then
  dd if=/dev/disk/by-id/usb-USB_USB_BAR-0:0-part1 bs=1 skip=0 count=2732 2>/dev/null
fi
Let's see if the key that is read by the script is identical to the original one in our home directory.
{ [[ "$(~/bin/keyfile.sh)" == "$(cat ~/mykey.txt)" ]] && echo 'SUCCESS.'; } || echo 'FAILURE: NOT THE SAME.';
Make sure the keyfile.sh script is executable (should already be).
chmod +x keyfile.sh
chmod -v 0500 keyfile.sh
cp keyfile.sh /boot/
6. Edit /etc/crypttab to enable the key script and regenerate initrd.img
Now modify /etc/crypttab from:
cryptdata UUID=bacc09ab-6e12-4365-be8f-d7b09ae0c0a5 none luks
to:
cryptdata UUID=bacc09ab-6e12-4365-be8f-d7b09ae0c0a5 /dev/disk/by-id/usb-USB_USB_BAR-0:0-part1 luks,keyscript=/boot/keyfile.sh
and update initrd.img
update-initramfs -u