aldengolab/ubuntu-secureboot-pxe-netboot

Ubuntu Secure Boot PXE Netboot

Problem

Building a PXE netboot server for Ubuntu with UEFI Secure Boot requires assembling components from multiple sources with specific, undocumented compatibility requirements. Getting any piece wrong produces silent failures or misleading errors.

Context / Trigger Conditions

• Building automated bare-metal provisioning for Ubuntu servers
• UEFI Secure Boot must remain enabled
• Need fully unattended installs via autoinstall (cloud-init)
• Post-install bootstrapping (Tailscale, k3s, etc.) via first-boot scripts

Architecture

UEFI Firmware (PXE client)
  -> DHCP proxy (dnsmasq) provides boot filename + TFTP server
  -> TFTP: BOOTx64.EFI (signed shim)
  -> TFTP: grubx64.efi (signed NETBOOT GRUB -- not local-boot GRUB)
  -> TFTP: grub/grub.cfg (GRUB menu)
  -> HTTP: ubuntu-live-server.iso (full ISO for live filesystem)
  -> HTTP: autoinstall/user-data (cloud-init autoinstall config)

Building the Boot Chain

1. Extract Assets from the Ubuntu ISO

The ISO contains everything needed, but you must select the right binaries:

| Asset | ISO Path | Purpose | |-------|----------|---------| | Signed shim | EFI/boot/bootx64.efi | First-stage bootloader, validates GRUB signature | | Signed netboot GRUB | pool/main/g/grub2-signed/grub-efi-amd64-signed_*.deb -> grubnetx64.efi.signed | GRUB with tftp/http/efinet modules | | Kernel | casper/vmlinuz | Linux kernel | | Initrd | casper/initrd | Initial ramdisk | | Live filesystem | The ISO itself | Contains filesystem.squashfs for the installer |

Critical: Do NOT use EFI/boot/grubx64.efi -- this is the local-boot GRUB without network protocol modules. It will load via UEFI firmware's PXE stack but cannot fetch files over TFTP/HTTP on its own.

Extract the netboot GRUB from the deb in the ISO's package pool:

GRUB_DEB=$(bsdtar -tf "$ISO_FILE" | grep 'pool/main/g/grub2-signed/grub-efi-amd64-signed_.*\.deb$')
bsdtar -xf "$ISO_FILE" -C /tmp "$GRUB_DEB"
dpkg-deb -x "/tmp/${GRUB_DEB}" /tmp/grub-signed
cp /tmp/grub-signed/usr/lib/grub/x86_64-efi-signed/grubnetx64.efi.signed "${DEST}/grubx64.efi"

2. TFTP Root Layout

The netboot GRUB binary uses prefix /grub (not /boot/grub):

/srv/tftp/
  BOOTx64.EFI              # Signed shim (renamed to uppercase)
  grubx64.efi              # Signed NETBOOT GRUB (grubnetx64.efi.signed)
  grub/grub.cfg            # GRUB config ($prefix/grub.cfg)

Verification: dnsmasq TFTP logs will show GRUB requesting grub/x86_64-efi/command.lst -- the directory before the filename confirms the prefix.

3. GRUB Configuration

set timeout=5
set default=0

menuentry "Boot from local disk" {
    exit
}

menuentry "Ubuntu Autoinstall" {
    linux (http,SERVER_IP:PORT)/ubuntu/vmlinuz \
        url=http://SERVER_IP:PORT/ubuntu/ubuntu-live-server.iso \
        autoinstall ds=nocloud-net\;seedfrom=http://SERVER_IP:PORT/autoinstall/ \
        ip=dhcp ---
    initrd (http,SERVER_IP:PORT)/ubuntu/initrd
}

Key parameters:

• url= -- points to the full ISO; the installer downloads and mounts it for filesystem.squashfs (just vmlinuz+initrd is not enough)
• ip=dhcp -- ensures networking is up before the ISO download
• ds=nocloud-net;seedfrom= -- tells cloud-init where to fetch autoinstall config
• Default "Boot from local disk" entry prevents boot loops on installed nodes (see: pxe-grub-persistent-server-pattern skill)

4. Autoinstall Configuration

Required fields for fully unattended install:

autoinstall:
  version: 1
  interactive-sections: []    # REQUIRED -- suppresses OOBE wizard
  locale: en_US.UTF-8
  keyboard:
    layout: us
  storage:
    layout:
      name: direct
  identity:
    hostname: ...
    username: ...
    password: ...             # SHA-512 hash ($6$...)
  ssh:
    install-server: true
    authorized-keys:
      - ...
  network:
    version: 2
    ethernets:
      all-en:
        match:
          name: en*
        dhcp4: true
  packages:
    - jq                      # or any packages needed by late-commands
  late-commands:
    - ...                     # post-install bootstrapping

Without interactive-sections: [], Ubuntu 24.04 shows a first-boot setup wizard even with a complete config, which interrupts late-commands execution.

5. Late-Commands: Chroot vs Installer Environment

curtin in-target -- runs commands inside a chroot of the installed system. The chroot does NOT have:

• /sys/firmware/efi/efivars -- efibootmgr fails silently
• Hardware/firmware access
• Running services

Rule of thumb:

• Software installation, file writes, systemd enable -> curtin in-target
• Firmware manipulation, EFI changes -> run directly (no curtin in-target)

6. First-Boot Scripts with curl|sh Installers

When writing scripts that use curl | sh install patterns (k3s, Tailscale, etc.), export environment variables instead of inline assignment:

# UNRELIABLE -- vars may not pass through pipe
curl -sfL https://example.com/install.sh | MY_VAR="value" sh -

# RELIABLE -- export ensures vars reach the piped script
export MY_VAR="value"
curl -sfL https://example.com/install.sh | sh -

For k3s specifically, INSTALL_K3S_EXEC="agent" must be exported to avoid installing as a server node.

dnsmasq Proxy DHCP Configuration

port=0                                          # Disable DNS
dhcp-range=192.168.0.0,proxy                    # Proxy mode -- no IP allocation
dhcp-match=set:efi-x86_64,option:client-arch,7
dhcp-match=set:efi-x86_64,option:client-arch,9
pxe-service=tag:efi-x86_64,x86-64_EFI,"PXE Boot",BOOTx64.EFI
dhcp-boot=tag:efi-x86_64,BOOTx64.EFI,,SERVER_IP
enable-tftp
tftp-root=/srv/tftp

Verification Checklist

1. dnsmasq TFTP logs show sent grubx64.efi (not just BOOTx64.EFI)
2. TFTP logs show GRUB requesting grub/grub.cfg (confirms correct prefix)
3. Installer shows "Downloading and mounting ISO" (confirms url= works)
4. No OOBE wizard appears (confirms interactive-sections)
5. Post-install artifacts exist: scripts, systemd units, config files
6. Services install in correct mode (e.g., k3s-agent not k3s server)

Troubleshooting

When things go wrong, read references/troubleshooting.md for symptom-based diagnostics:

• GRUB drops to interactive prompt
• TFTP transfer fails
• "Unable to find a medium containing a live file system"
• Autoinstall shows interactive OOBE wizard
• Late-commands artifacts missing after install
• First-boot script can't reach provisioning API
• k3s installs as server instead of agent
• Flannel subnet.env missing

Notes

• revocations.efi not found in TFTP logs is expected (shim SBAT check, non-fatal)
• grubnetx64.efi.signed is Canonical-signed -- shim trusts it with Secure Boot
• Provisioning nonces/tokens in-memory on the server are lost on pod/process restart -- design for re-imaging if first-boot bootstrapping fails
• Password hashes for autoinstall use random salts (non-deterministic output) but the plaintext password is what you type at login -- store the hash, remember the password