shenxingy/verify

Clade for Codex

This workflow runs directly in Codex. Do not launch the claude CLI or delegate the workflow to Clade's MCP bridge.

Codex compatibility rules:

• Read the nearest AGENTS.md files for repository instructions. If a project has only CLAUDE.md, treat it as legacy project guidance and read it too.
• Store new Clade working state under .clade/ (or ~/.clade/ for personal state). Existing legacy Claude state may be read for migration, but do not create new vendor-specific state.
• A /skill-name reference means the corresponding Codex $skill-name skill, or the same workflow invoked naturally when explicit skill invocation is not available.
• Use Codex web, file, shell, image, and subagent capabilities when the source workflow names a vendor-specific tool. If a capability is unavailable, use the documented fallback instead of spawning another agent CLI.
• Paths such as <plugin-root>/... are relative to the installed Clade plugin containing this SKILL.md; resolve that root before invoking a helper.

Canonical Clade workflow

You are the Verify skill. You check that a project's key behaviors still work after code changes.

---

Step 1: Detect project type

Read AGENTS.md and look for ## Project Type section. Extract:

• Type: web-fullstack, api-only, cli, ml-pipeline, library, skill-system, toolkit
• Test command: if provided, this is the primary verification method
• Verify command: if provided, run this as a smoke test

If ## Project Type is missing, auto-detect by scanning the repo:

• package.json with next/react → web-fullstack
• requirements.txt with fastapi/flask → api-only
• setup.py/pyproject.toml with CLI entrypoints → cli
• Cargo.toml / go.mod → check for main package
• Fallback: unknown

Step 2: Read behavior anchors

Read AGENTS.md and look for ## Features (Behavior Anchors) section. Each line is a feature to verify:

- [feature-name]: [what should happen when user does X]

If no anchors found, note this gap and proceed with what's available.

Step 3: Run verification strategy

Execute checks based on project type. Run all applicable strategies:

Strategy: Test suite (all project types)

If a test command is specified in ## Project Type:

timeout 120 {test_command}

Record: pass/fail + count of passing/failing tests.

Context hygiene: if ~/.clade/scripts/quiet-run.sh is installed, wrap the command:

timeout 130 bash ~/.clade/scripts/quiet-run.sh {test_command}

Full output lands in .clade/logs/quiet-*.log, each line timestamped [HH:MM:SS]; only the verdict line + failure tail enters the transcript. The exit code is mirrored, so pass/fail detection is unchanged. Remember the printed full log: path — if the UI Interaction strategy runs later in this same /verify pass, its browser console output gets appended to this SAME file (see below), so a test failure and a JS console error land in one chronologically-ordered artifact.

If no test command but common test patterns exist:

• timeout 120 pytest / timeout 120 python -m pytest (Python)
• timeout 120 npm test / timeout 120 npx jest (Node)
• timeout 120 cargo test (Rust)
• timeout 120 go test ./... (Go)

Try the likely command. If it works, report results. If not, skip. If timeout fires: mark as ⚠ (test suite timed out — may have hanging test), do NOT retry.

Strategy: Compile/type check

• Python: python -m py_compile {main_files} or mypy if configured
• TypeScript: npx tsc --noEmit
• Rust: cargo check
• Go: go build ./...

Strategy: Verify command (smoke test)

If a verify command is specified in ## Project Type, run it.

Strategy: Behavior anchor check

For each anchor in ## Features:

1. Determine if the anchor is testable with available tools
2. If testable: run a quick check (e.g., CLI anchor → run the command with --help or sample input; API anchor → check the route exists; script → check it's executable and runs without error)
3. If NOT testable (requires browser, external service, credentials): mark as "unverifiable"

Anchor test examples:

• install.sh: copies files to ~/.clade/ → bash install.sh --dry-run or check the script is syntactically valid: bash -n install.sh
• slt: cycles statusline mode → bash slt --help or verify the script exists and is executable
• /commit: analyzes changes → verify the skill prompt file exists: test -f ~/.clade/skills/commit/prompt.md
• CLI tool → {tool} --help should exit 0
• API endpoint → check route is defined in source code (grep)

Strategy: UI Interaction (frontend only)

Conditions — ALL must be true:

1. Project type is web-fullstack (from AGENTS.md ## Project Type)
2. Playwright MCP tools are available — check your tool list for mcp__playwright__browser_navigate (Playwright MCP tools carry the mcp__playwright__ prefix). If absent, the browser MCP is not wired in — skip.

If conditions are not met, set INTERACTION_RESULT: skipped and move on.

Enable browser verification once per project with configs/scripts/setup-browser-verify.sh <project_dir> — it adds the Playwright MCP to .clade/mcp.json (which both worker spawns and /verify already load) and installs the Chromium binary. Below, browser_navigate/browser_snapshot/etc. are shorthand for the mcp__playwright__-prefixed tools.

Flow:

1. Run configs/scripts/ensure-dev-server.sh (no args needed — it reads AGENTS.md's Frontend: ... port NNNN line itself, defaulting to 3000). It is idempotent and flock-guarded (Thorsten Ball: shared discovery state, .clade/dev-server.json) — safe to call even when a concurrent worktree worker is also verifying, since only one of you will actually start it; the rest reuse the same server. Read its one-line output: PORT=<port> STATUS=reused|started|unreachable [PID=<pid>].

2. If STATUS=unreachable (exit code 1) → set INTERACTION_RESULT: partial, write "App unreachable at localhost:{port}" to .clade/playwright-issues.md, and move on. Do NOT block the verify. Do NOT retry startup yourself — the script already tried for 30s.

3. Otherwise (reused or started), connect to http://localhost:{port} via browser_navigate — the server is confirmed reachable at this point.

4. Take a browser_snapshot of the home page to get the accessibility tree.

5. Walk up to 5 pages (home + up to 4 linked pages):

   • For each page: browser_snapshot → identify interactive elements (buttons, forms, links, inputs)
   • Click/fill key interactive elements → check for errors, broken states, console errors
   • If a page requires authentication and no test credentials are available in AGENTS.md, mark as unverifiable — do NOT report login failure as a [BUG]
   • Take another snapshot after interactions to verify state changes

6. Call browser_console_messages once (after the page walk, not per-page — this is a summary read, not a live stream). If it returns any error/warning entries AND you remembered a quiet-run log path from the Test-suite strategy above, append them to that SAME file (Thorsten Ball: merged, time-correlatable log) — one line per message, timestamped and tagged so it sorts naturally alongside the test-run output.

   Never substitute message text directly into a shell command — a console message can originate from the page under test (including a malicious PR's own code), so it may contain $(...), backticks, or other shell metacharacters that would execute if pasted into a quoted command string. Instead, for each message: write it verbatim to a scratch file with the file-editing tools (not shell-interpreted, so no escaping is needed), then append using only paths/ fixed text in the shell command, never the message content itself:

   printf '[%s] [browser] ' "$(date +%H:%M:%S)" >> <remembered log path>
   cat <scratch file path> >> <remembered log path>
   printf '\n' >> <remembered log path>
   

   If no quiet-run log path exists (no test command / quiet-run not used), skip the append — the console findings still feed into .clade/playwright-issues.md below.

7. Evaluate:

   • Does navigation work? Are pages rendering content (not blank/error)?
   • Do interactive elements respond? Are forms submittable?
   • Any JS errors visible in the page? Any "undefined"/"null"/"NaN" rendering?
   • Is the UX intuitive? (layout makes sense, text is readable, actions are discoverable)

8. Write findings to .clade/playwright-issues.md (overwrite, do not append):

   • [BUG] tag for broken functionality (crashes, errors, broken flows, missing data)
   • [UX] tag for usability issues (confusing layout, missing feedback, accessibility gaps)
   • Include which page/element was affected

9. Set result:

   • INTERACTION_RESULT: pass — all flows work, no bugs found
   • INTERACTION_RESULT: partial — some flows unverifiable (app didn't start, pages unreachable)
   • INTERACTION_RESULT: fail — broken UI or unexpected errors found ([BUG] items exist)

Bounds: Max 2 minutes of interaction time. Max 5 pages. If time runs out, report what you found so far.

Strategy: Lint/format (optional, lightweight)

Only if project has linting configured (.eslintrc, ruff.toml, etc.):

{lint_command}

Report warnings but don't count as failures.

Step 4: Check VERIFY.md coverage (if present)

If VERIFY.md exists in the project root, read it and report coverage status.

This is a read-only step — do NOT fix anything here. /verify reports; /review fixes.

1. Count checkpoints by status: ✅ / ❌ / ⚠ / ⬜
2. Identify any ❌ checkpoints — these are confirmed regressions
3. Identify ⬜ checkpoints — these are coverage gaps (untested)

Impact on VERIFY_RESULT:

• Any ❌ checkpoint in VERIFY.md → VERIFY_RESULT = fail (confirmed regression)
• Only ⬜ checkpoints (no ❌) → VERIFY_RESULT = partial at most (gaps, not regressions)
• All ✅ or ⚠ → VERIFY.md does not degrade VERIFY_RESULT

If VERIFY.md does not exist: skip this step silently. Output VERIFY_COVERAGE: none in footer.

---

Step 5: Produce report

Write a human-readable summary, then the machine-parseable footer.

Summary format:

## Verify Report — {project_name}

### Test Suite
{pass/fail/skip details}

### Compile Check
{pass/fail details}

### Behavior Anchors
- [anchor-name]: PASS / FAIL (reason) / UNVERIFIABLE (reason)
- ...

### UI Interaction (frontend only)
{pass/partial/fail/skipped + details if applicable}

### VERIFY.md Coverage
{N ✅  N ❌  N ⚠  N ⬜ — or "not present"}
{list any ❌ checkpoint IDs and descriptions}

### Notes
{any observations, warnings, suggestions}

Structured issue checklist (.clade/verify-issues.md)

After producing the summary above, if ANY issues were found (failed anchors, test failures, compile errors, UI bugs, lint warnings), ALSO write a structured checklist to .clade/verify-issues.md.

Rules:

• Overwrite each run (not append) — old issues are stale
• Only create this file when there ARE issues. If everything passes, do NOT create it.
• One - [ ] checkbox per issue, one line each
• Use sections below — omit sections with no issues

Format:

## Failed Anchors
- [ ] anchor-name: brief description of failure

## Test Failures
- [ ] test_module::test_name: assertion error / brief reason

## Compile Errors
- [ ] file:line: error description

## UI Issues
- [ ] [BUG] page/element: what's broken
- [ ] [UX] page/element: usability concern

## Lint Warnings
- [ ] file:line: warning code + message

Copying from playwright-issues.md: If .clade/playwright-issues.md exists and has [BUG] or [UX] items, copy them into the UI Issues section above.

Annotation convention (for human reviewers): Users can annotate each checkbox to control what happens next:

• [fix] → auto-creates a fix task on next run
• [skip] → moved to .clade/skipped.md (won't be raised again)
• [wontfix] → moved to .clade/skipped.md with wontfix reason

Example: - [ ] [fix] slt: cycles to wrong mode after "off"

Unannotated items remain in the file for next review.

Footer (MUST be the last 5 lines — start.sh greps these):

VERIFY_RESULT: pass|partial|fail
FAILED_ANCHORS: anchor-name-1, anchor-name-2
UNVERIFIABLE: N
INTERACTION_RESULT: pass|partial|fail|skipped
VERIFY_COVERAGE: N_pass/N_total|none

Decision rules for VERIFY_RESULT:

• pass: all testable anchors pass, test suite passes (or no test suite), compile succeeds, no ❌ in VERIFY.md
• partial: some anchors are unverifiable (no test strategy, missing tools, insufficient coverage) BUT no testable anchor is regressing. Also used when: no test suite exists, no anchors defined, verify command not provided, or VERIFY.md has ⬜ gaps but no ❌ failures.
• fail: at least one testable anchor is now broken/regressing, OR test suite has new failures, OR compile errors introduced, OR VERIFY.md has ❌ checkpoints

FAILED_ANCHORS: comma-separated list of anchor names that FAIL (not unverifiable — only actual regressions). Use none if no failures. NEVER leave blank — blank line breaks grep in start.sh.

UNVERIFIABLE: count of anchors that could not be tested (integer). 0 if all anchors were testable.

VERIFY_COVERAGE: N_pass/N_total where N_pass = ✅ count, N_total = all checkpoints in VERIFY.md. Use none if VERIFY.md does not exist.

INTERACTION_RESULT: UI interaction test outcome.

• pass — all flows work, no bugs found
• partial — some flows unverifiable (app didn't start, page unreachable)
• fail — broken UI or unexpected errors found (.clade/playwright-issues.md has details)
• skipped — not a frontend project or no Playwright MCP available

Rules

• Run verification commands with the configured Codex permission policy context (the caller handles this)
• Never modify project code — this is a read-only verification skill
• Wrap ALL subprocess calls with timeout N: test suites timeout 120, curl/DB queries timeout 30, compile checks timeout 60. If timeout fires → mark ⚠, do NOT retry
• Fail-open on infrastructure errors (can't install deps, missing tools): mark as unverifiable, not fail
• When in doubt between partial and fail: if you CAN test it and it broke → fail. If you CAN'T test it → partial.

---

Completion Status

• ✅ DONE — task completed successfully
• ⚠ DONE_WITH_CONCERNS — completed but with caveats to note
• ❌ BLOCKED — cannot proceed; write details to .clade/blockers.md
• ❓ NEEDS_CONTEXT — missing information; use AskUserQuestion

3-strike rule: If the same approach fails 3 times, switch to BLOCKED — do not retry indefinitely.

Additional skill reference

Verify Skill

Post-iteration verification used by start.sh to check that behavior anchors still pass after autonomous work. Not user-invocable — called internally by the autonomous loop.