SethGammon/telemetry

/telemetry — Telemetry Hub

When to Use

• "What does Citadel track?" / "What telemetry does it have?"
• "What did this session cost?" / "How much have I spent?"
• "How do I turn off the cost alerts?" / "Can I disable telemetry?"
• "Show me hook activity" / "What hooks fired?"
• "What trust level am I at?"
• Directly: /telemetry

Routed here by /do for: "telemetry", "what did this cost", "session stats", "session cost", "how much did that cost", "what hooks fired", "trust level", "show me telemetry", "cost breakdown", "spending".

Commands

| Command | Behavior | |---|---| | /telemetry | Full hub — stats + command directory + settings | | /telemetry --costs | Cost section only: session, today, all-time, by campaign | | /telemetry --hooks | Hook activity only: last 20 fires with timing and outcomes | | /telemetry --verify | Telemetry and artifact integrity check: verify hashes/signatures, flag tampered records, report legacy records | | /telemetry --config | Show current telemetry settings from harness.json | | /telemetry off | Disable session summary, reduce hook verbosity | | /telemetry on | Re-enable all telemetry | | /telemetry --threshold N | Set cost alert threshold step (e.g. --threshold 10 = alert every $10) |

Protocol

Step 1: COLLECT DATA

Read the following in parallel. All are optional — treat missing files as zero/empty.

Live session cost:

• Run node scripts/session-tokens.js --today 2>/dev/null — captures real token data
• If unavailable, read .planning/telemetry/cost-tracker-state.json for burn rate
• Real cost is always preferred over estimated. Mark clearly: $X.XX vs $X.XX (est)

Historical costs:

• Run node scripts/session-tokens.js --all 2>/dev/null for all-time real totals
• Read last 20 lines of .planning/telemetry/session-costs.jsonl for recent sessions
• For each entry: prefer real_cost > override_cost > estimated_cost

Hook activity:

• Read last 20 lines of .planning/telemetry/hook-timing.jsonl
• For each event: "timing" entry: extract hook, duration_ms, timestamp
• For each event: "counter" entry: extract hook, metric
• Check .planning/telemetry/hook-errors.jsonl (last 20 lines) for recent blocks

Trust level:

• Read .claude/harness.json → trust object
• Compute: novice (sessions < 5), familiar (5-19), trusted (20+ with 2+ campaigns)
• If trust.override set, use that

Settings:

• Read .claude/harness.json → telemetry object
• Show current values with defaults if missing

Step 2: RENDER HUB

Output this format. Omit a section only if the data source is completely unavailable.

=== Citadel Telemetry ===

CURRENT SESSION
  Cost:       $X.XX [real] | $X.XX (est)
  Duration:   N min | $X.XX/min burn rate
  Tokens:     NNK input | NK output | NK cache read | NK cache write
  Messages:   N
  Agents:     N spawned
  Hooks fired: N (today)

TODAY
  $X.XX across N sessions
  Most expensive: {slug or "unattached"} — $X.XX

ALL TIME
  $X.XX across N sessions, N campaigns
  Cache savings: ~$X.XX (cache reads vs full input price)

BY CAMPAIGN (recent 5)
  {slug}: $X.XX — N sessions
  _unattached: $X.XX — N sessions

HOOK ACTIVITY (last 10 fires)
  {relative time} | {hook} | {duration_ms}ms | {outcome}
  (no hook timing recorded yet)

TRUST LEVEL
  Level:    {novice | familiar | trusted}
  Sessions: N completed
  Campaigns: N completed
  (novice = 0-4 sessions | familiar = 5-19 | trusted = 20+ with 2+ campaigns)

TELEMETRY SETTINGS
  Enabled:          {true | false}
  Session summary:  {auto | always | off}   ← the [session] line at session end
  Cost alerts:      {on | off}  at thresholds: {list or "default ($5,$15,$30...)"}
  Hook timing:      {on | off}
  Audit log:        {on | off}
  — or, when harness.json is absent —
  (harness.json not found — defaults active)
  → Run /do setup to unlock cost tracking, configure thresholds, and register your install.

COMMAND DIRECTORY
  /telemetry                            This screen
  /telemetry --costs                    Cost breakdown only
  /telemetry --hooks                    Hook activity only
  /telemetry --verify                   Telemetry/artifact integrity check (hash/signature verification)
  /cost                                 Deep cost exploration by session/campaign/week
  /dashboard                            Full harness state (campaigns, fleet, all costs)

  node scripts/session-tokens.js --today   Today's sessions with exact token counts
  node scripts/session-tokens.js --all     All-time totals (real data, not estimates)

  cat .planning/telemetry/session-costs.jsonl   Raw session cost log
  cat .planning/telemetry/hook-timing.jsonl     Raw hook execution log
  cat .planning/telemetry/audit.jsonl           Raw tool call audit log

CONTROLS
  /telemetry off                        Disable session summary + reduce verbosity
  /telemetry on                         Re-enable
  /telemetry --threshold N              Alert every $N (writes to harness.json)
  /telemetry --config                   Edit settings interactively

Step 3: SUB-COMMAND HANDLING

/telemetry off: Set telemetry.sessionSummary = "off" and telemetry.costAlerts = false in harness.json. Output: "Telemetry summary disabled. Hook safety checks remain active." Safety hooks (protect-files, circuit-breaker, external-action-gate) are never disabled.

/telemetry on: Set telemetry.sessionSummary = "auto" and telemetry.costAlerts = true. Output: "Telemetry re-enabled."

/telemetry --threshold N: Validate N is positive. Generate [N, N*2, N*5, N*10, N*20, N*50, N*100] (capped at 500). Write to harness.json under policy.costTracker.thresholds.

/telemetry --verify: Run the project verifier:

node scripts/verify-telemetry-integrity.js

The verifier scans .planning/telemetry/*.jsonl and .planning/artifacts/*.jsonl. Display verified, signed, legacy, tampered, invalid, and signature-warning counts. Use --strict-legacy only when old unsigned records should fail the check.

Output format:

=== Telemetry Integrity ===

file.jsonl
  Total records:    N
  Verified (hash):  N
  Verified (signed): N
  Legacy (no hash): N
  TAMPERED:         N
  Invalid JSON:     N
  Signature warnings: N

Status: CLEAN or FAILED

If any tampered records: list each with timestamp, event, and both the stored and expected hash (first 16 chars each). Tampering can indicate log corruption, manual edits, or a bug — not necessarily malicious.

If only legacy records (no tampered): note "Legacy records were written before telemetry integrity hashing was added. New telemetry and artifact records are hashed automatically."

/telemetry --config: Show current settings with the node -e "..." command to change each — don't auto-apply.

Step 4: ACCURACY BADGES

Always mark data source clearly:

• [real] — data from Claude Code's native session JSONL (exact)
• (est) — estimated from the fallback model ($1 base + $0.50/agent + $0.10/min)
• (override) — manually entered by the user

Never blend real and estimated in the same total without flagging it.

What Telemetry Covers

Covered: session cost (real token data), duration/burn rate/message count, agent spawn count, hook timing and outcomes, campaign cost attribution, trust level.

Not covered (by design): per-tool-call cost, per-subagent cost isolation, real-time streaming token count.

Safety hooks always on (cannot be disabled): protect-files, external-action-gate, circuit-breaker, quality-gate.

Quality Gates

• Never show raw JSONL to the user — always parse and format
• Cost totals must be labeled with their source (real / est / mixed)
• /telemetry off must NOT disable safety hooks — make this explicit in output
• Relative timestamps required — no raw ISO strings in output
• If all data sources are missing, show the empty-state version with setup hint

Fringe Cases

• .planning/telemetry/ missing: Show empty state with "Run /do setup to initialize telemetry."
• session-tokens.js unavailable: Fall back to session-costs.jsonl; mark (est).
• harness.json missing: Show "(harness.json not found — defaults active)" and "→ Run /do setup to unlock cost tracking."
• telemetry.enabled: false: Show banner "Telemetry is disabled. Run /telemetry on to re-enable."
• --verify with missing files: Report "No telemetry or artifact JSONL files found." Not an error.
• --verify when scripts/verify-telemetry-integrity.js is unavailable: Report that the verifier is missing and show the raw file paths to inspect; do not claim hash verification ran.

Contextual Gates

Disclosure: Read-only by default. --threshold, off, on, --config write harness.json. Reversibility: amber — harness.json writes; undo with git checkout .claude/harness.json. Trust gates: Any — no restrictions.

Exit Protocol

/telemetry does not produce a HANDOFF block. It is a read-only observability tool (except for --threshold, off, on, --config which write harness.json). After displaying output, wait for the next user command.