Regenerate Dracut initramfs images from a live USB

Thanks to an incompatability between dracut and systemd version 256, I was left with an unbootable system. I booted a live USB system, downgraded systemd back to 255, but I had to regenerate the initramfs images to be compatible with my old systemd.

While using systemd-nspawn is convenient for modifying the system from a live usb, generating dracut initramfs images through it, resulted in missing hardware support (for example, no nvme module) due to the abstracted hardware in the container. The solution is to revert to the traditional chroot and generate the images from there:

$ sudo /usr/lib/systemd/systemd-cryptsetup attach root /dev/nvme0n1p3
$ sudo mount /dev/mapper/root /mnt/
$ sudo mount /dev/nvme0n1p2 /mnt/boot/
$ sudo mount --bind /dev/ /mnt/dev/
$ sudo mount --bind /proc/ /mnt/proc/
$ sudo mount --bind /sys/ /mnt/sys/
$ sudo chroot /mnt/
# dracut -f --regenerate-all

Automating DNS Configurations for F5 VPN Tunnel using Systemd-resolved and NetworkManager-dispatcher

F5 VPN does not play well with split DNS configuration using systemd-resolved because it insists on trying to rewrite /etc/resolv.conf. The workaround is to make resolv.conf immutable, and configure the DNS settings for the tunnel manually. systemd-resolved does not have a mechanism for persistant per-interface configuration, and it relies on NetworkManager to configure each connection correctly. F5 VPN is not compatible with NetworkManager, and does not make it easy to configure it this way.

NetworkManager-dispatcher allows you to run scripts based on network events. In our case, we will use it to automatically add DNS configurations when the F5 VPN tunnel tun0 is up, and thus provide persistent configuration.

Here is the script:

#!/bin/bash

INTERFACE=$1
STATUS=$2

case "$STATUS" in
    'up')
        if [ "$INTERFACE" = "tun0" ]; then
            # Add your search domains here
            SEARCH_DOMAINS="~example.corp ~example.local"

            resolvectl domain "$INTERFACE" $SEARCH_DOMAINS
            resolvectl dns $INTERFACE 192.168.100.20 192.168.100.22
            resolvectl dnsovertls tun0 no
        fi
        ;;
esac

The script checks if the interface is tun0 and if the current action is up. If so, it uses resolvectl to configure search domains and local DNS servers. Lastly, DNS over TLS is disabled, as the corporate DNS servers do not support them.

To make this script work, install in the /etc/NetworkManager/dispatcher.d/ directory with the name f5-vpn. Make sure it’s executable and only writable by root. NetworkManager-dispatcher will run this script whenever a network interface goes up, automatically setting the DNS configurations for F5 VPN tunnel.

Split DNS using systemd-resolved

Many corporate environments have internal DNS servers that are required to resolve internal resources. However, you might prefer a different DNS server for external resources, for example 1.1.1.1 or 8.8.8.8. This allows you to use more secure DNS features like DNS over TLS (DoT). The solution is to set up systemd-resolved as your DNS resolver, and configure it for split DNS resolving.

Starting with systemd 251, Debian ships systemd-resolved as a separate package. If it isn’t installed, go ahead and install it.

$ sudo apt install systemd-resolved
$ sudo systemctl enable --now systemd-resolved.service

Create the following configuration file under /etc/systemd/resolved.conf.d/99-split.conf:

[Resolve]
DNS=1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com

Domains=~.
DNSOverTLS=opportunistic

Domains=~. gives priority to the global DNS (1.1.1.1 in our case) over the link-local DNS configurations which are pushed through DHCP (like internal DNS servers).

DNSOverTLS=opportunistic defaults to DNS over TLS but allows fallback to regular DNS. This is useful when corporate DNS doesn’t support DNS over TLS and you still want to resolve corporate internal domains.

Restart systemd-resolved to reload the configuration:

$ sudo systemctl restart systemd-resolved

The final step is to redirect programs relying on /etc/resolv.conf (possibly through the glibc API) to the systemd-resolved resolver. The recommended way according to the systemd-resolved man page is to symlink it to /run/systemd/resolv/stup-resolv.conf.

$ sudo ln -rsf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

F5 VPN

F5 VPN doesn’t play well with the above configuration. First, F5 VPN tries to overwrite the DNS configuration in /etc/resolv.conf, by removing the existing file and replacing it with its own (pushing corporate DNS server configuration through it). The solution is to prevent F5 VPN from deleting the /etc/resolv.conf, by setting it to immutable. However, we cannot chattr +i a symlink. We have to resort to copying the configuration statically, and then protect it.

$ sudo cp /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
$ sudo chattr +i /etc/resolv.conf

Finally, because now F5 VPN can’t update the DNS configuration, we would have to manually configure the corporate DNS servers and the search domains.

$ sudo resolvectl dns tun0 192.168.100.20 192.168.100.22
$ sudo resolvectl domain tun0 ~example.corp ~example.local

Update: See Automating DNS Configurations for F5 VPN Tunnel using Systemd-resolved and NetworkManager-dispatcher for a script that automates the configuration.

Unlock LUKS volume with a YubiKey

Update: The dracut configuration has been updated and now udev consistently recognizes the YubiKey in the initramfs.

Unlocking LUKS encrypted drives with a YubiKey has been supported since systemd 248. In Debian, systemd>=250 is required, as the feature has not been enabled in prior versions. This tutorial is geared towards Yubikeys, but it should work with slight modifications with any other FIDO2 token.

YubiKey series 5 and later should support the hmac-secret extension. You can make sure your Yubikey supports the needed hmac-secret extension by querying it with ykman:

$ ykman --diagnose 2>&1 | grep hmac-secret

Backup your LUKS header

In case you mess anything up, you would need a backup of your LUKS header. Remember to save your backup to some external storage, so you can actually access it if anything goes sideways.

# cryptsetup luksHeaderBackup /dev/nvme0n1p3 --header-backup-file /media/guyru/E474-2D80/luks_backup.bin

Set FIDO2 PIN

We would like to set a FIDO2 PIN for the Yubikey, so unlocking the encrypted drive would require both the physical Yubikey and the PIN. You can set the PIN using:

$ ykman fido access change-pin

Enroll the Yubikey

Start by verifying that systemd-cryptenroll can see and can use your YubiKey:

$ systemd-cryptenroll --fido2-device=list
PATH         MANUFACTURER PRODUCT
/dev/hidraw0 Yubico       YubiKey FIDO+CCID

Now, enroll the Yubikey, replacing /dev/nvme0n1p3 with the block device of the LUKS encrypted drive.

$ sudo systemd-cryptenroll /dev/nvme0n1p3 --fido2-device=auto  --fido2-with-client-pin=yes
🔐 Please enter current passphrase for disk /dev/nvme0n1p3: (no echo)
Initializing FIDO2 credential on security token.
👆 (Hint: This might require confirmation of user presence on security token.)
🔐 Please enter security token PIN: (no echo)
Generating secret key on FIDO2 security token.
👆 In order to allow secret key generation, please confirm presence on security token.
New FIDO2 token enrolled as key slot 0.

Modify /etc/crypttab

We need to modify /etc/crypttab in order to tell cryptsetup to unlock the device using the YubiKey. Add fido2-device=auto in the options field of the crypttab entry for your device. For example:

nvme0n1p3_crypt UUID=307a6bef-5599-4963-8ce0-d9e999026c1a none luks,discard,fido2-device=auto

Switch to dracut

Debian’s default initramfs generator, update-initramfs of the initramfs-tools is using the old cryptsetup for mounting encrypted drives. However, cryptsetup doesn’t recognize the fido2-device option. Running update-initramfs will fail with the following error:

$ sudo update-initramfs -u
update-initramfs: Generating /boot/initrd.img-5.15.0-3-amd64
cryptsetup: WARNING: nvme0n1p3_crypt: ignoring unknown option 'fido2-device'

This is unfortunate. The simplest solution is to switch to dracut, a more modern initramfs generator, which among other things relies on systemd to activate encrypted volumes. This solves the issue of the unknown fido2-device.

Before installing dracut, I would highly recommend creating a copy of the existing initramfs in the boot partition in case something goes wrong.

$ sudo apt install dracut

Dracut includes systemd-cryptsetup by default. systemd-cryptsetup depends on libfido for unlocking devices using FIDO2 tokens. At least in Debian, systemd-cryptsetup dynamically loads libfido2.so (as opposed to being dynamically linked), which causes dracut not to have libfido2.so in the initramfs. This causes systemd-cryptsetup to issue the following error upon boot:

FIDO2 tokens not supported on this build. 

We fix it by manually adding libfido2.so to the initramfs. Of course, we also need to include libfido2’s dependencies as well. Dracut has a mechanism for automatically adding dependencies for executables, but it doesn’t work on libraries. As a workaround, instead of adding libfido2 directly, we will add an executable that depends on libfido2, which will add libfido2 and its dependencies to the initramfs. We will usefido2-token from the fido2-tools package for this trick.

$ sudo apt install fido2-tools
$ cat << EOF | sudo tee /etc/dracut.conf.d/11-fido2.conf
## Spaces in the quotes are critical.
# install_optional_items+=" /usr/lib/x86_64-linux-gnu/libfido2.so.* "

## Ugly workround because the line above doesn't fetch
## dependencies of libfido2.so
install_items+=" /usr/bin/fido2-token "

# Required detecting the fido2 key
install_items+=" /usr/lib/udev/rules.d/60-fido-id.rules /usr/lib/udev/fido_id "
EOF

Now, recreate the initramfs images:

$ sudo dracut -f

Last remarks

At this point, we are done. Reboot you’re machine and it will prompt you for your YubiKey and allow you to unlock your LUKS encrypted root patition with it. If you don’t have your YubiKey, it will give the following prompt:

Security token not present for unlocking volume root (nvme0n1p3_crypt), please plug it in.

After around 30 seconds, it would time out and display the following message:

Timed out waiting for security device, aborting security device based authentication attempt.

Afterwards, it would allow you to unlock the partition using a password (or a recovery key).

In case you run into any trouble, append rd.break=initqueue to the kernel command line, and dracut will enter a shell before attempting to mount the partitions. You can manually mount the drive using the following command:

# /usr/lib/systemd/systemd-cryptsetup attach root /dev/nvme0n1p3

Exit the emergency shell, and the system will continue its normal boot.

Autostart rclone mount using systemd

Create the following file under ~/.config/systemd/user/rclone-dropbox.service:

[Unit]
Description=Dropbox (rclone)
AssertPathIsDirectory=%h/Dropbox
# Make sure we have network enabled
After=network.target
[Service]
Type=simple
ExecStart=/usr/bin/rclone mount --vfs-cache-mode full Dropbox: Dropbox
# Perform lazy unmount
ExecStop=/usr/bin/fusermount -zu %h/Dropbox
# Restart the service whenever rclone exists with non-zero exit code
Restart=on-failure
RestartSec=15
[Install]
# Autostart after reboot
WantedBy=default.target

Reload the user services, enable and start the new service:

$ systemctl --user daemon-reload
$ systemctl --user enable --now rclone-dropbox