bolivian-peru/flight-recorder

Flight Recorder — Server Black Box

Continuous state capture for post-incident forensics.

Snapshot Capture (run every 5 minutes via heartbeat)

Capture a lightweight system snapshot:

system_health()
shell_exec({ command: "ps aux --sort=-%mem | head -10" })
shell_exec({ command: "ss -s" })  # connection summary

Store as a compact memory entry:

memory_store({
  summary: "Flight recorder: CPU 12%, RAM 4.1G, load 0.3, 142 connections",
  detail: "Top processes: postgres 1.2G, openclaw 380M, node 210M. Net: 142 established, 12 listening.",
  category: "system.config",
  tags: "flight-recorder,snapshot,telemetry"
})

Post-Incident Analysis

When investigating "what happened at 3 AM":

1. Pull flight recorder snapshots

memory_recall({ query: "flight-recorder snapshot", timeframe: "24h", max_results: 50 })

2. Pull journal logs for the incident window

journal_logs({ since: "2 hours ago", priority: "warning", lines: 100 })

3. Pull audit ledger

event_log({ limit: 50 })

4. Reconstruct the incident

Present a forensic timeline:

Incident Report — Server Crash at 03:12 UTC

Timeline:
  02:00  Flight recorder: CPU 15%, RAM 5.2G (normal)
  02:30  Flight recorder: CPU 23%, RAM 5.8G (rising)
  02:47  Flight recorder: CPU 89%, RAM 7.1G (critical)
  02:47  OOM killer triggered (killed: postgres worker)
  02:48  PostgreSQL: 3 connections terminated
  02:49  App health check failed → Caddy returned 502
  02:51  systemd restarted PostgreSQL automatically
  02:52  All services recovered
  02:55  Flight recorder: CPU 12%, RAM 4.1G (normal)

Root cause: PostgreSQL autovacuum on users table (12M rows)
during traffic spike. Memory peaked at 7.4/7.6 GB.

Recommendations:
  1. Add 2GB swap (NixOS config change)
  2. Schedule autovacuum for low-traffic hours
  3. Consider 16GB RAM plan ($6/mo more on Hetzner)

All data sourced from flight recorder snapshots,
journal logs, and audit ledger. Chain verified.

Data Retention

Flight recorder snapshots are compact (~200 bytes each). At 5-minute intervals: 288/day, ~56KB/day, ~1.7MB/month. Keep 30 days of snapshots. Memory recall handles the rest.