aldengolab/pxe-grub-persistent-server-pattern

PXE GRUB Persistent Server Pattern

Problem

PXE servers typically require manual lifecycle management -- scale up to provision, scale down after. If left running, installed nodes re-enter the installer on reboot because UEFI boot order has network boot first.

Context / Trigger Conditions

• Designing PXE infrastructure for a datacenter, homelab, or cloud bare-metal
• PXE server is managed by GitOps (ArgoCD, Flux) with selfHeal/reconciliation
• UEFI firmware on target nodes has PXE as first boot option
• Want zero-touch operation after initial setup

Solution

The Pattern: GRUB Default-to-Exit

Make the GRUB menu default to an exit entry that returns control to UEFI firmware, which then boots from the next entry (local disk):

set timeout=5
set default=0

menuentry "Boot from local disk" {
    exit
}

menuentry "Install Ubuntu" {
    linux /vmlinuz ...
    initrd /initrd
}

How It Works

Installed node reboots
  -> UEFI: PXE boot (first in order)
  -> dnsmasq: responds with BOOTx64.EFI
  -> shim -> GRUB -> shows menu
  -> 5s timeout -> "Boot from local disk" (default)
  -> GRUB exit -> UEFI: next boot entry -> local disk
  -> Ubuntu boots normally

For new nodes, an operator selects "Install Ubuntu" during the 5s window.

Why Not efibootmgr?

efibootmgr modifies UEFI NVRAM to change boot order. In practice:

• Behavior varies across firmware implementations
• curtin in-target chroot doesn't mount /sys/firmware/efi/efivars
• Running outside chroot in autoinstall late-commands is fragile
• Some firmware ignores or resets programmatic boot order changes

The GRUB exit pattern works at the bootloader level, which is firmware-agnostic.

GitOps Integration

With this pattern, the PXE server can run permanently:

• Set replicas: 1 in Helm values (not 0)
• ArgoCD selfHeal won't disrupt provisioning by scaling down
• No manual kubectl scale needed before/after provisioning
• First-boot scripts on new nodes can always reach the provisioning API

Verification

• Installed node: PXE boots -> GRUB menu -> 5s -> boots locally (no intervention)
• New node: PXE boots -> GRUB menu -> operator selects install -> installer starts

Troubleshooting

When things go wrong, read references/troubleshooting.md for symptom-based diagnostics:

• Installed nodes re-enter installer despite default-to-exit
• Node boots locally but adds 5s delay on every reboot
• GitOps (ArgoCD) keeps scaling PXE server to 0
• New node can't select "Install" in time

Notes

• Adjust timeout for your environment (5s is good for console access, increase for remote KVM/IPMI where there's more latency)
• This pattern works with any GRUB-based PXE chain, not just Ubuntu
• For fully automated (no-operator) provisioning, extend the GRUB config to query a provisioning API via GRUB's http module and auto-select based on MAC status
• Alternative: dnsmasq MAC-based filtering (only respond to known-new MACs), but requires maintaining an allowlist and doesn't solve the boot-order problem