Pop!_OS installation & configuration

All the servers and laptop have been installed with Pop!_OS. The reason being the special attention Pop!_OS is givng to Nvidia GPU support. We want good and flexible NVidia GPU support because we plan to use the CUDA libraries for machine learning applications. Pop!_OS is an Ubuntu derivative with a little more than just "re-theming".

2

3

Laptops are all configured using encrypted storage devices. This is a safety precaution as laptops are mobile devices and are regularly moved between locations. A stolen or lost laptop hence should not pose a security threat. Although the data on the laptop is non-confidential the mere fact that this is a Defence owned laptop will create an embarrassment. This we don't want even though there is no real security issue at stake here.

4

5

Server are supposed to be at a fixed secure location. Storage device encryption is not deemed necessary. If a server happens to be a laptop (yes this can be) we still implement storage level encryption.

6

7

Although strictly speaking not needed for laptops where possible we employ the ##btrfs## filesystem. This has several advantages:

8

9

* Flexibility (expandability)

10

* Snapshots

11

12

Btrfs filesystems are by now considered mature and offer more flexibility than other enterprise grade filesystem such as ##zfs##. For small deployments therefore ##btrfs## is a better fit.

13

14

The possibility to create instant snapshots offer great way to safeguard against accidental loss through inadvertenty deleting file or other minor mishaps that can be annoying and cost substantial amounts of time to recover from.

15

16

The following guide [[https:~~/~~/mutschler.eu/linux/install-guides/pop-os-btrfs/>>https://mutschler.eu/linux/install-guides/pop-os-btrfs/]] was used although adapted to our specific case, i.e. no virtual machine and depending on use case either encryption or not.

= Encryption =

Only laptops are going to be encrypted because they may in theory be transported and used elsewhere and could be stolen or lost. The embarrassment to have this happen to an unencrypted Defence laptop is reason enough to warrant encrypting the hard drive.

21

22

The mere form factor of a laptop does not preclude it from functioning as a server. There is however a small problem. When you are logged in remotely and you update/upgrade the operating system you may need to reboot the laptop. Due to the encryption you have to physically be present at the same location as the laptop to enter the pass-phrase that starts decryption on booting. Of course this is not possible when you are working remote. The answer to this problem is to create solution where an external USB flash drive is detected on booting the system and used to store/retrieve a key that enables decryption. The boot menu will still contain a back-up boot option that works with a physically entered pass-phrase. The default boot option will look for the USB flash drive with decryption key. A better way to achieve the same goal is to use a ##ubikey## [[https:~~/~~/www.yubico.com/>>https://www.yubico.com/]] It has some additional advantages like two factor authentication, etc. In case the USB flash drive with key is lost or stolen, one can simple remove the respective key from the keyslot on the encrypted device thus keeping it secure.

== Implementation ==

As stated above one normally enters a passphrase to unlock the encrypted device and boot the system. The alternative is to use a key to do the job. This key is read automatically from a device and is used to decrypt the hard drive which contains the operating system and file system with user directories home.

27

28

=== Procedure outline ===

29

30

1. Create an alternate backup systemd-boot boot option (for safety/emergency reasons).

31

1. Create and save a suitable keyfile, either a binary or a character sequence.

32

1. Add the keyfile to a key-slot on the encrypted device (the hard drive with the root and home filesystem)

33

1. Prepare a USB device, obtain device partition ID and copy the keyfile to it.

34

1. Create a script that is called at boot time and searches for the USB ID and copy it to## /boot/##.

35

1. Edit ##/etc/crypttab ##to enable the ##keyfile.sh## script and regenerate ##initrd.img##

36

37

Finally Test by rebooting with USB flash drive inserted

38

39

=== 1. Create a fallback boot option ===

40

41

To create a fallback boot option do the following:

42

43

(% class="box" %)

44

{{{cd /boot/efi/EFI/Pop_OS-eea7a2f5-b204-4de3-b605-3696ed323544

45

cp vmlinuz.efi vmlinuz-backup.efi

46

cp initrd.img initrd.img-backup

47

48

cd /boot/efi/loader/entries

49

cp Pop_OS-current.conf Pop_OS-backup.conf

50

}}}

51

52

Then edit the file Pop_OS-backup.conf like shown below:

(% class="box" %)

{{{title Pop!_OS

linux /EFI/Pop_OS-eea7a2f5-b204-4de3-b605-3696ed323544/vmlinuz-backup.efi

57

initrd /EFI/Pop_OS-eea7a2f5-b204-4de3-b605-3696ed323544/initrd.img-backup

58

options root=UUID=eea7a2f5-b204-4de3-b605-3696ed323544 ro quiet loglevel=0 systemd.show_status=false splash rootflags=subvol=@

59

}}}

60

61

Changing the title is optional. **NB: it is important to reboot and test this option now!**

62

63

=== 2. Create the secret keyfile ===

64

65

The key can be a sequence of printable (visible) characters or any binary sequence of a certain length. Below examples of how to generate these:

**binary sequence:**

(% class="box" %)

{{{dd if=/dev/urandom of=/etc/luks-keys/mykey.bin bs=512 count=4}}}

71

72

**character sequence:**

Either this one:

(% class="box" %)

{{{head -c 2048 /dev/urandom | base64 -w 0 > mykey.txt

78

chmod -v 0400 mykey.txt

79

}}}

80

81

or slightly better, this one:

82

83

(% class="box" %)

84

{{{dd bs=256 count=8 if=/dev/random | base64 -w 0 > mykey.txt

85

chmod -v 0400 mykey.txt

86

}}}

87

88

Either key is fine, but if you would like to be able to type it in via the keyboard, the latter option is preferred for obvious reasons. The key will reside on a USB flash drive. However this flash drive may be removed intentionally or unintentionally. Hence deploying the key based automated decryption method we still want to create a fallback option in case the key is not available. This fallback boot option uses an untainted kernel and initrd image (i.e. originals) to be used with the regular pass phrase that is to be entered at the prompt before booting the system.

89

90

=== 3. Add key to keyslot on encrypted device ===

91

92

Be careful to choose the correct device for adding the key. Using parted## /dev/nvme0n1 ##one can inspect the partition layout of the first device (in this case an SSD).

93

94

(% class="box" %)

95

{{{root@schumann:~# parted /dev/nvme0n1

96

GNU Parted 3.3

97

Using /dev/nvme0n1

98

Welcome to GNU Parted! Type 'help' to view a list of commands.

99

(parted) p

100

Model: INTEL SSDPEKNW512G8H (nvme)

101

Disk /dev/nvme0n1: 512GB

102

Sector size (logical/physical): 512B/512B

Partition Table: gpt

Disk Flags:

Number Start End Size File system Name Flags

107

1 1049kB 538MB 537MB fat32 SYSTEM boot, esp

108

2 538MB 555MB 16.8MB Microsoft reserved partition msftres

109

3 555MB 105GB 105GB ntfs Basic data partition msftdata

110

5 105GB 512GB 406GB cryptdata

111

4 512GB 512GB 503MB ntfs Basic data partition hidden, diag

(parted)

}}}

(% class="box" %)

{{{cryptsetup -v luksAddKey /dev/nvme0n1p5 mykey.txt

119

}}}

120

121

=== 4. Store the keyfile on the USB flash drive ===

122

123

Upon boot the key needs to be retrieved from the USB flash drive so we need to store it there first. The USB flash drive is prepared in such a way as to obfuscate the fact that a key is present. The text below shows the partition layout of the USB flash drive. There is a small hidden 1MiB partition which we will use to hold the key. One could also decide to obfuscate a little more by not making a true partition but simply start the first partition at an offset of 1MiB beyond the partition table and store the key at a known (predetermined) offset within that 1MiB space. However this we consider potentially error prone and needs to be tested well before deciding to adopt this approach.

124

125

(% class="box" %)

126

{{{root@schumann:~# parted /dev/sda

127

GNU Parted 3.3

128

Using /dev/sda

129

Welcome to GNU Parted! Type 'help' to view a list of commands.

130

(parted) p

131

Model: USB BAR (scsi)

132

Disk /dev/sda: 65.4MB

133

Sector size (logical/physical): 512B/512B

Partition Table: gpt

Disk Flags:

Number Start End Size File system Name Flags

138

1 1049kB 2097kB 1049kB KEY hidden

139

2 2097kB 65.4MB 63.3MB fat16 SILVER msftdata

(parted)

}}}

To be able to detect the USB flash drive when inserted at boot time we need to know it's device ID.

145

146

(% class="box" %)

147

{{{root@schumann:~# ls -l /dev/disk/by-id | grep sda2

148

lrwxrwxrwx 1 root root 10 Apr 21 10:07 usb-USB_USB_BAR-0:0-part2 -> ../../sda2

149

}}}

150

151

We also need to know (confirm) the size of the key file.

152

153

(% class="box" %)

154

{{{root@schumann:~# ls -l

155

total 4

156

drwxr-xr-x 1 root root 32 Apr 21 09:38 bin

157

drwxr-xr-x 1 root root 28 Apr 19 12:06 docs

158

-rw-r--r-- 1 root root 2732 Apr 21 10:45 mykey.txt

}}}

(% class="box" %)

{{{root@schumann:~# dd if=mykey.txt of=/dev/disk/by-id/usb-USB_USB_BAR-0:0-part1 bs=1 count=2732

163

2732+0 records in

164

2732+0 records out

165

2732 bytes (2.7 kB, 2.7 KiB) copied, 0.00868143 s, 315 kB/s

166

}}}

167

168

=== 5. Create the ##keyfile## script to get key to ##stdout## ===

169

170

The script below will wait for our USB device to become available and read the key from the respective number of bytes (2732) on partition 1. The script ##keyfile.sh## is (for now) stored in the ##/bin/## directory.

171

172

(% class="box" %)

173

{{{for TRY in {1..60}

174

do

175

if ! [ -e /dev/disk/by-id/usb-USB_USB_BAR-0:0-part2 ]; then

176

# echo $TRY

177