gtapps/smoke-test

Smoke Test

Validate the post-hatch setup. Produces a structured report with PASS/WARN/FAIL per check.

Plan

1. Initialize counters

Track passed, warnings, failures counts. Collect output lines for the final report.

2. Config validation

• Read .claude-code-hermit/config.json
  • If missing or invalid JSON: FAIL config.json missing or invalid
  • If valid: PASS config.json parsed
• Read ${CLAUDE_PLUGIN_ROOT}/.claude-plugin/plugin.json to get the current plugin version
• Compare config._hermit_versions["claude-code-hermit"] with the plugin version
  • Match: PASS version matches plugin (X.Y.Z)
  • Mismatch: WARN config version X.Y.Z != plugin X.Y.Z — run /claude-code-hermit:hermit-evolve
• Check required top-level keys exist: agent_name, channels, env, heartbeat
  • Each missing key: FAIL missing required key: <key>

3. OPERATOR.md sanity

• Check if .claude-code-hermit/OPERATOR.md exists
  • Missing: WARN OPERATOR.md missing — run hatch or create manually
  • Empty: WARN OPERATOR.md is empty — fill in project context
• If exists and non-empty:
  • Check for {{ placeholder patterns (NOT <!-- which are legitimate HTML comments)
    • Found: WARN OPERATOR.md contains unfilled {{ placeholders
    • Not found: PASS OPERATOR.md readable

4. Scheduled check references

• Read scheduled_checks array from config.json
• For each entry:
  • Extract the plugin and skill name from the skill field (formats accepted: /plugin:skill-name or plugin:skill-name)
  • Look up <plugin>:<skill-name> in the harness's available-skills list (the system-reminder enumerating loaded skills is authoritative — do NOT grep the plugin cache; cross-marketplace layouts make path checks unreliable)
  • Found: PASS scheduled_checks[N]: <plugin>:<skill-name> loaded
  • Not found: WARN scheduled_checks[N]: <plugin>:<skill-name> not loaded — install the plugin or remove the entry

4b. Verify the tz-shift helper

Run: bun ${CLAUDE_PLUGIN_ROOT}/scripts/routines.ts tz-shift "0 4 * * *" "UTC"

• Exit 0 and stdout is 0 4 * * *: PASS routines.ts tz-shift helper available
• Exit non-zero or wrong stdout: FAIL routines.ts tz-shift helper missing or broken — run /hermit-evolve

5. Routine validation (static)

• Read routines array from config.json
• For each routine:
  • Check required keys (id, schedule, skill, enabled): FAIL per missing key
  • If time or days keys are present: FAIL routine "<id>" uses legacy "time"/"days" — migrate to "schedule" (5-field cron)
  • Validate schedule is a 5-field cron expression (space-separated, 5 fields: min hour dom month dow): FAIL if malformed
  • Validate skill contains : (plugin:skill-name format): FAIL with expected format
  • PASS routine "<id>" valid (<schedule>, <skill>)
• Check for duplicate routine IDs: WARN per duplicate

6. State file validation

Template defaults: ${CLAUDE_PLUGIN_ROOT}/state-templates/ — alert-state.json.template, micro-proposals.json.template.

For each file: parse JSON. If missing or unparseable, rewrite from the corresponding template. If parseable but wrong shape, repair in place — backfill missing keys from template defaults, overwrite keys that exist with the wrong type, preserve all other existing data. Emit one WARN per file that needed repair (noting what was fixed), or PASS if valid.

• .claude-code-hermit/state/alert-state.json — object with: alerts (object), self_eval (object), total_ticks (number), last_digest_date (any, presence required)

  • For each missing or wrong-type key: overwrite with template default (e.g. alerts: {}, self_eval: {}, total_ticks: 0, last_digest_date: null) → WARN alert-state.json repaired: <keys>

• .claude-code-hermit/state/micro-proposals.json — object with pending key (array)

  • Missing pending key (and active key present): migrate — read active value; if non-null, set pending: [<active value>]; if null, set pending: []; remove active key → WARN micro-proposals.json migrated: active → pending
  • Missing pending key (and no active key): add "pending": [] → WARN micro-proposals.json missing pending key — backfilled
  • pending is not an array: overwrite with [] → WARN micro-proposals.json pending is not an array — reset

7. Channel test (optional)

• Check if any channels are configured and enabled in config.json
• If no channels: skip silently
• If channels configured:
  • For each enabled channel, attempt to send a test message via the channel's MCP tool:
    • Discord: use the Discord MCP send tool
    • Telegram: use the Telegram MCP send tool
  • Message content: "Smoke test — channel is working." (include agent name if configured)
  • Success: PASS <channel> test message sent
  • Failure: FAIL <channel> test message failed: <error>
  • If the MCP tool is not available (plugin not installed): WARN <channel> plugin not available for test
• For each enabled channel that also sets maintainer_channel_id, send one probe to the maintainer chat via bun ${CLAUDE_PLUGIN_ROOT}/scripts/channel-send.ts .claude-code-hermit --tier maintainer - (short line on stdin, e.g. "Smoke test: maintainer channel is working."):
  • Sent: PASS <channel> maintainer-chat test message sent
  • Failure: FAIL <channel> maintainer-chat test failed: <error> (a typo'd maintainer_channel_id fails here rather than silently on every future alert)
  • Channels with no maintainer_channel_id: skip the maintainer probe silently.

7b. Push notification test (optional)

• Read push_notifications from config.json
• If false or absent: skip silently
• If true: call PushNotification(message="Smoke test: push fallback is wired.", status="proactive")
  • Call accepted, push sent: PASS PushNotification test dispatched (visible: desktop if daemon running, mobile if Remote Control paired)
  • Call accepted, push not delivered: WARN PushNotification accepted but not delivered. Push will silently no-op until a notification daemon is running or Remote Control is paired.
  • Call errored or schema-rejected: FAIL PushNotification test failed: <error> (likely a model/schema bug, surface for inspection)

8. Print report

Output each check result as exactly one line:

PASS  config.json parsed and version matches plugin (0.3.4)
WARN  OPERATOR.md contains unfilled {{ placeholders
WARN  scheduled_checks[0]: claude-code-setup:claude-automation-recommender not loaded — install the plugin or remove the entry
PASS  routine "heartbeat-restart" valid (0 4 * * *, claude-code-hermit:heartbeat start)
PASS  routine "morning" valid (0 9 * * *, claude-code-hermit:reflect)
FAIL  routine "bad" uses legacy "time" field — migrate to "schedule" (5-field cron)

Summary line at end:

Smoke test: 4 passed, 1 warning, 1 failed

For each FAIL, include a remediation hint on the same line or the line after.