overthinkos/vm-deploy-target

vm-deploy-target

Implementation notes

• The pod deploy target is PodDeployTarget (ov/deploy_target_pod.go); ledger target keying uses pod:<name>.
• vmNameFromDeployName strips the vm: prefix. The dispatch upstream (deploy_add_cmd.go) rewrites a plain deploy key like eval-arch-vm to vm:<vm-name> before resolving via ResolveTarget → VmUnifiedTarget.Add / .Del, so internal VM code always sees the prefixed form.
• UnifiedDeployTarget / LifecycleTarget interfaces (ov/deploy_target_unified.go) + the ResolveTarget dispatcher (ov/unified_targets.go) provide the full lifecycle contract (Add / Del / Test / Update / Start / Stop / Status / Logs / Shell / Rebuild).
• Disposability for ov update <vm> reads from the DeploymentNode with target: vm + matching vm: (see rebuild.go::vmDisposableFromDeployments); it is NOT a VmSpec field.

VmDeployTarget brings ov deploy add vm:<name> online: the same InstallPlan IR that drives pod builds and host deploys now runs inside a VM over SSH. Shell bodies that LocalDeployTarget would exec via local sudo bash -s are instead exec'd via ssh guest 'sudo bash -s' through an SSHExecutor. Ledger writes land on the guest filesystem under the guest user's ~/.config/overthink/installed/; teardown runs in the guest via SSH as well.

VmDeployTarget is the 4th DeployTarget interface implementer — after OCITarget (build-mode Containerfile emission), PodDeployTarget (podman quadlet), and LocalDeployTarget (local filesystem). K8sDeployTarget is the 5th. See /ov-internals:install-plan for the shared IR.

Source files

| File | Contents | |---|---| | ov/deploy_target_vm.go | VmDeployTarget struct + Emit flow | | ov/deploy_executor.go | DeployExecutor interface (RunShell, Scp, Close) + ShellExecutor — local shell exec (reused by LocalDeployTarget for the builder-image step) | | ov/deploy_executor_ssh.go | SSHExecutor — ssh client with passt-friendly timeouts + WaitForSSH + WaitForCloudInit | | ov/deploy_add_cmd_vm.go | VM-only deploy helpers (deployNestedPodsInGuest, buildVmReverseRunner, vmNameFromDeployName); ov deploy add/del vm:<name> dispatches through ResolveTarget → VmUnifiedTarget.Add / .Del | | ov/vm_create_spec.go | VmCreateCmd.runVmSpecCreate — prereq: VM must be created before deploy |

DeployExecutor interface

type DeployExecutor interface {
    RunShell(ctx context.Context, script string, opts ShellOpts) (ExecResult, error)
    Scp(ctx context.Context, src io.Reader, dst string, mode os.FileMode) error
    Close() error
}

Two implementations:

• ShellExecutor — bash -c <script> / file copy. Used by LocalDeployTarget for container-builder invocations and by the dry-run path of any target.
• SSHExecutor — ssh/scp via golang.org/x/crypto/ssh. Used exclusively by VmDeployTarget. Carries Host/Port/User/KeyPath + maintains a persistent connection across multiple shell invocations.

Name choice: the interface is DeployExecutor (not Executor) to avoid a clash with the Executor type in testrun.go; likewise deployShellQuote (not shellQuote) avoids a clash in wl.go.

VmDeployTarget.Emit flow

Five preflight steps before walking plans:

1. Wait for SSH. SSHExecutor.WaitForSSH(ctx, 120) — polls net.Dial to host:port with exponential backoff. 120s timeout accommodates cold-boot VMs where cloud-init is provisioning sshd.
2. Wait for cloud-init (cloud_image sources only). SSHExecutor.WaitForCloudInit polls cloud-init status --wait until status is done. Bootc guests skip this step unless the cloud-init layer is present.
3. EnsureOvInGuest. Runs the VmOvInstall.Strategy state machine (see /ov-internals:cloud-init-renderer).
4. Ensure guest ledger dir exists. ssh -- mkdir -p ~/.config/overthink/installed/{deploys,layers}.
5. Resolve the guest home once (t.Exec.ResolveHome(ctx, "")) and cache it on t.guestHome. Every home-bearing step field is resolved against THIS home, not the host operator's — see below.
6. Walk plans. Same batched (Scope, Venue) logic as LocalDeployTarget, but with sudo bash -s wrapped in ssh. See /ov-local:local-deploy for the grouping rules.
7. Write the env.d-sourcing managed block into the guest's detected login-shell init (EnsureManagedBlockVia) so the env.d files actually get sourced at login.

Guest-home resolution (deploy-time {{.Home}})

Home-bearing step fields — ShellHookStep env values + path_append, ShellSnippetStep snippet/destination, FileStep.Dest — are compiled with the deferred {{.Home}} token (HomeToken), NOT a baked compile-time home. Each target resolves the token at emit via InstallPlan.ResolveHome(home): img.Home for OCI/pod-overlay, the host home for LocalDeployTarget, and the GUEST home (t.guestHome) for VmDeployTarget. This is why a target: vm deploy writes /home/<guest-user>/.config/overthink/env.d/<layer>.env whose contents point at /home/<guest-user>/… rather than the host operator's home. cmd: task bodies are left untouched — ~/$HOME there shell-expand at runtime on the guest as the deploy user, already correct. See /ov-internals:install-plan "Deferred home resolution".

env.d-sourcing managed block (guest login shell)

VmDeployTarget calls EnsureManagedBlockVia(ctx, t.Exec, shell, t.guestHome, opts) after the plan loop — the SAME executor-based writer LocalDeployTarget uses (shell_profile.go; the os-based EnsureManagedBlock is a thin wrapper over it). Without this block the per-layer env.d files exist but are never sourced, so PATH never picks up ~/.npm-global/bin etc. The shell is detected from the GUEST /etc/passwd via detectGuestShell (getent), because the guest's interactive default may differ from the operator's (CachyOS ships fish) — writing bash syntax to ~/.profile when the guest runs fish would never load.

Cross-host builders (npm / pixi / cargo / aur)

execBuilder runs every builder on the HOST (podman) and ships the result into the guest — guests never need a container runtime:

• aur → builds .pkg.tar.zst in a host staging dir, scp's them in, pacman -U.
• npm / pixi / cargo (execHomeArtifactBuilder) → bind-mounts a host staging dir AS the guest home path so npm shebangs / cargo rpaths / pixi activation scripts bake the path the guest will actually use, runs the same renderBuilderScript body as the local path, then tars the produced home subdirs (~/.npm-global, ~/.pixi, ~/.cargo; caches excluded), scp's the tarball in, and extracts it into the guest $HOME as the guest user so ownership + baked paths are correct. The builder image resolves via resolveBuilderImage (--builder-image → compiled BuilderStep.BuilderImage → BuilderImageResolver). Unknown builders honor --skip-incompatible.

This is what makes the full ov-cachyos stack — including the npm-builder AI CLIs (claude-code, codex, gemini, oracle, forgecode) — install on a VM.

RebootStep — the one step only this target executes

When a layer declares reboot: true, BuildDeployPlan appends a trailing RebootStep. VmDeployTarget.execReboot is the sole executor that acts on it (OCI/pod/k8s skip; LocalDeployTarget skips + warns — it never reboots the operator host). It records the guest's /proc/sys/kernel/random/boot_id, fires (sleep 1; systemctl reboot) & so the ssh session closes cleanly, then polls until SSH answers AND the boot_id has changed — deterministic, not a fixed sleep, so the still-up pre-reboot sshd can't be mistaken for "back up". This is what lets a kernel-module layer (e.g. the CachyOS nvidia-driver layer) load its module on a clean boot mid-deploy. See /ov-internals:install-plan RebootStep.

Host→guest image transfer (ov vm cp-image)

ov vm cp-image <vm> <ref> [--as <tag>] [--rootless] (and the reusable TransferImageToGuest helper) stream a host-built image into a running guest's podman storage via podman save | ssh podman load (NO intermediate tarball — the guest /tmp tmpfs is too small for a multi-GB image), idempotent (skips an intact present image, re-streams a torn-overlay one — a name-only check would wrongly skip a corrupt image) and offline (no registry). --rootless selects the storage, and ALL of the load / integrity-probe / tag steps follow it consistently (via the podmanCmd(rootless) helper):

• default → the guest's ROOT podman (sudo podman), for a sudo podman run --device nvidia.com/gpu=all consumer that needs /dev/nvidia* via root.
• --rootless → the SSH user's ROOTLESS podman (podman, no sudo; the tag runs via RunUser, not RunSystem). This is what deployNestedPodsInGuest uses: the nested pod comes up via the guest user's own ov deploy from-image (a --user quadlet) which reads the USER's storage, so the image MUST land there — a root-loaded image would be invisible to it.

Nested pod-in-VM — persistent in-guest quadlet (deployNestedPodsInGuest)

A target: vm deploy whose nested: map has target: pod children brings each child up as a PERSISTENT in-guest quadlet — the nested-pod-in-VM capability. VmUnifiedTarget.Add constructs the VmDeployTarget and calls deployNestedPodsInGuest AFTER VmDeployTarget.Emit (so the guest's own layers, including any kernel-driver reboot + the boot-time nvidia-ctk cdi generate, are already applied). For each child it:

1. ov image build <child.Image> on the HOST (the guest needs no project).
2. ov vm cp-image <vm> <child.Image> --as localhost/ov-<childKey>:latest --rootless — into the guest USER's rootless podman.
3. over SSH as the guest user: loginctl enable-linger (so the --user quadlet auto-starts at boot and survives reboot), then export XDG_RUNTIME_DIR=/run/user/$(id -u) (so systemctl --user reaches the lingering user bus over the non-login SSH session — same requirement as VmDeployTarget's own user services), then the guest's own project-free ov deploy from-image localhost/ov-<childKey>:latest <childKey> — which generates + starts the quadlet from the image's baked OCI labels (ports, services, GPU device auto-detected in the guest; rootless GPU via CDI — /dev/nvidia* are world-rw and the CDI spec is world-readable).

Idempotent (cp-image skips an intact image; from-image re-applies on ov update). The dispatch routes a VM-root deploy node-only (its pod children deploy in-guest here, never via a host tree walk). The existing ov eval live <vm>.<child> multi-hop chain reaches the running nested pod unchanged. ov vm cp-image is the host→guest delivery for it.

VmDeployState persistence

type VmDeployState struct {
    InstanceID              string                  // stable UUIDv4 cloud-init instance-id, pinned across re-renders
    DiskPath                string                  // absolute path to the qcow2 (may be a CoW overlay on a cached base)
    SeedIso                 string                  // NoCloud cidata ISO path (empty for bootc with injection disabled)
    SshPort                 int                     // host port forwarded to the guest's :22
    SshUser                 string                  // guest account VmDeployTarget SSHes in as
    Backend                 string                  // "qemu" or "libvirt", pinned at first apply
    KeyInjectionResolved    *VmKeyInjectionResolved // resolved SSH key-injection plan
    OvInstallStrategy       string                  // how ov is installed into the guest
    CloudInitRenderedDigest string                  // digest of the rendered cloud-init (re-render detection)
    Snapshots               []VmSnapshotState       // libvirt snapshot ledger
    Ephemeral               *EphemeralRuntime       // transient run-state for an ephemeral VM
}

Persisted in ~/.config/ov/deploy.yml as the vm_state: field on the VM's deploy entry (DeploymentNode.VmState). Each ov vm build / ov vm create / ov deploy add vm:<name> iteration updates the relevant fields. ov deploy del vm:<name> preserves the state (so re-adding picks up InstanceID etc.) unless --purge is passed.

SSH key idempotency

generateSSHKeypair in ov/vm.go checks for <vmStateDir>/id_ed25519.pub before creating. Rebuilding a VM doesn't regenerate the keypair. First ov vm build writes the keypair; subsequent calls leave it untouched — so iterated rebuilds keep a stable pubkey and SSH stays valid.

CLI dispatch: ResolveTarget → VmUnifiedTarget.Add / .Del

ov deploy add vm:<name> resolves via deploy_add_cmd.go::dispatchNode → ResolveTarget → VmUnifiedTarget.Add when the deploy name starts with vm: (or target: vm is set in deploy.yml):

ov deploy add vm:arch ripgrep           # apply ripgrep layer in the guest
ov deploy add vm:arch fedora-coder \    # apply full fedora-coder layer set
    --add-layer team-extras \
    --add-layer github.com/team/configs/layers/sshkeys
ov deploy del vm:arch                   # reverse all applied layers in the guest

Prereq: VM must exist (ov vm create arch first). VmUnifiedTarget.Add does NOT auto-provision the VM — keeps the "provision" step explicit. If the VM is undefined, the dispatch returns a clean error pointing at ov vm create.

passt backend + SSH port forwarding

When the VM's network uses libvirt user-mode + <backend type='passt'/> + <portForward> (see /ov-internals:libvirt-renderer), SSHExecutor connects to 127.0.0.1:<host-port>. The portForward maps that through passt into the guest's :22. The indirection is invisible to SSHExecutor — it sees a normal TCP connect.

Cross-References

• /ov-internals:install-plan — InstallPlan IR (the 4 DeployTarget implementers and the 9 step kinds)
• /ov-internals:vm-spec — VmSpec consumed by VmDeployTarget
• /ov-internals:libvirt-renderer — renders domain XML; portForward + passt backend
• /ov-internals:cloud-init-renderer — EnsureOvInGuest lives there
• /ov-core:deploy — ov deploy add vm:<name> command + deploy.yml schema
• /ov-local:local-deploy — parallel target (LocalDeployTarget); ReverseOps model also used on VM target
• /ov-vm:vm — VM lifecycle; creates the target Emit runs against
• /ov-vm:arch — canonical worked example — VmDeployState persistence; ssh_key idempotency live-test