vibeeval/factcheck-guard

FactCheck Guard — Runtime Claim Verification

Prevents false claims about the codebase. Grep results are hypotheses, not proof. Every assertion about "what the code does" must be verified by reading the actual code.

Why This Exists

An 80% false claim rate was observed when agents trusted grep results without reading files. The pattern repeats: grep finds a string, agent asserts behavior, behavior is wrong. This skill converts the claim-verification guidelines into an actionable protocol.

Confidence Markers (MANDATORY)

Every factual claim about the codebase MUST carry one of these markers:

| Marker | Meaning | Requirements | |--------|---------|-------------| | [VERIFIED] | Read the file, traced the logic | Must cite file:line | | [INFERRED] | Based on grep or file name | Must verify before asserting to others | | [UNCERTAIN] | Not yet checked | Must investigate before making the claim |

Never promote an [INFERRED] claim to a statement of fact. Never let [UNCERTAIN] appear in a response to the user — investigate first.

---

Verification Protocol

Level 1: Existence Claims

When asserting "X exists in the codebase":

Step 1 — Search (produces [INFERRED])

Grep/Glob for the pattern.
Result: "X appears at path/to/file.ts:42" [INFERRED]

Step 2 — Read the file (required to reach [VERIFIED])

Read path/to/file.ts, locate line 42.
Confirm the match is actual code, not a comment or string literal.

Step 3 — Check context

Is the code in a live code path, or inside a dead branch / disabled block?
Is it imported and actually used?
Does the function name match what you expected, or is it a different concept?

Step 4 — State the claim with citation

"Error handling exists at src/api/handler.ts:67 — try/catch wraps the database call" [VERIFIED]

---

Level 2: Absence Claims

When asserting "X does NOT exist" — absence claims are the most dangerous. A single missed search term invalidates the entire claim.

Step 1 — Primary search

Grep for the primary term. No results found.
Status: [INFERRED ABSENT] — not yet VERIFIED

Step 2 — Alternative terms (minimum 3)

Different naming conventions, abbreviations, synonyms.

Example: Searching for "rate limiting"
Try: "rateLimit", "rate_limit", "throttle", "throttling", "requestLimit", "limiter"
If any match: investigate that file before concluding

Step 3 — Check likely locations manually

Where would X logically live if it existed?
Read those files directly — grep may miss dynamically constructed patterns.

Example: Checking for auth middleware
Read: src/middleware/, app/middleware.ts, server.ts, app.ts
Even if grep found nothing, scan these files for auth-related logic.

Step 4 — Only then claim VERIFIED ABSENT

"Rate limiting does not exist. Searched: rateLimit, rate_limit, throttle, throttling,
requestLimit, limiter. Checked: src/middleware/index.ts, server.ts, app.ts. None found." [VERIFIED ABSENT]

---

Level 3: Behavior Claims

When asserting "the system does X" — the most subtle failure mode. Finding a function name does not tell you what the function actually does.

Step 1 — Locate the function

Grep/Glob finds the function name.
Status: [INFERRED] — you know it exists, not what it does

Step 2 — Read the full function body

Read the file. Read the entire function, not just the signature.
Look for: branches, early returns, side effects, what it actually returns.

Step 3 — Trace the logic path

Follow the execution path relevant to your claim.
If the function calls other functions, read those too (one level deep minimum).
Check: what happens in the error path? What are the edge cases?

Step 4 — State behavior with evidence

"processPayment() validates card number via Luhn check (payment.ts:34),
then calls Stripe API (payment.ts:41), and on failure rolls back the DB
transaction (payment.ts:58) — it does NOT charge the card on validation failure." [VERIFIED]

---

Common False Claim Patterns

| Pattern | Why It Fails | Correct Approach | |---------|-------------|-----------------| | grep found "try.*catch" | Could be in comment, string, or test mock | Read the file, confirm it wraps real code | | grep found nothing | Wrong term, wrong directory, different naming | Try 3+ alternative terms | | File named auth.ts exists | Doesn't mean auth is implemented | Read the file | | Type definition has a field | Doesn't mean the field is set or used | Grep for field assignment and usage | | Function is called somewhere | Doesn't mean it does what the name implies | Read the function body | | Found in tests directory | Test mocks are not real implementations | Confirm the actual source file | | grep -L returned files | Those files lack the pattern — but pattern might exist under a different name | Try alternative terms before concluding | | Found in commented-out block | Not active code | Check if block is enabled |

---

Two-Pass Audit Protocol

For system audits, security reviews, and architecture reviews:

Pass 1: Generate Hypotheses

Go through each area systematically. Produce hypothesis statements only:

"Input validation might be missing from the /api/upload route" [INFERRED]
"The auth token might not be refreshed on expiry" [INFERRED]
"Database queries might not use parameterized statements" [INFERRED]

Do not state these as facts. Do not include them in the user-facing summary.

Pass 2: Verify Each Hypothesis

For every [INFERRED] hypothesis:

Hypothesis: "Input validation missing from /api/upload" [INFERRED]

Action: Read src/app/api/upload/route.ts
Found: Zod schema validation at line 18, applied before file processing.
Result: "Input validation exists at api/upload/route.ts:18" [VERIFIED]
Hypothesis was wrong.

Hypothesis: "Database queries might not be parameterized" [INFERRED]

Action: Search for raw SQL patterns (string interpolation in queries)
Grep: `db.query(\`.*\${`, `db.execute(\`.*\${`
Read: src/lib/db.ts, src/repositories/*.ts
Found: All queries use prepared statements. One raw query at analytics.ts:89.
Result: "Raw SQL injection risk at src/analytics.ts:89" [VERIFIED]

Only after Pass 2 are claims safe to report.

---

Quick Checklist

Before making any codebase claim, confirm:

• [ ] Have I read the actual file, not just grep results?
• [ ] Is the code in a live path (not commented out, not test-only)?
• [ ] For absence claims: did I try 3+ alternative search terms?
• [ ] For absence claims: did I manually check the likely locations?
• [ ] For behavior claims: did I read the full function, not just the name?
• [ ] Does my claim include a file:line citation?
• [ ] Is my marker accurate: [VERIFIED] / [INFERRED] / [UNCERTAIN]?

---

Integration Points

claim-verification.md — the rule file provides guidelines; this skill provides the action protocol.

Agents that must use this skill:

• code-reviewer: before asserting "the code lacks X"
• security-reviewer: before asserting "vulnerability exists/doesn't exist"
• architect: before asserting "pattern is/isn't used"
• sleuth: before asserting "root cause is X"
• coroner: before asserting "same bug exists elsewhere"

Hook integration (optional): After every Grep tool result, the epistemic-reminder hook injects a warning. That warning is this skill's Level 1 trigger — search results are [INFERRED], not [VERIFIED].

---

Examples: Wrong vs Right

Existence claim — WRONG

Grep found "authMiddleware" in 3 files.
Claim: "Auth middleware is applied to all routes."

Existence claim — RIGHT

Grep found "authMiddleware" in 3 files.
Read server.ts: middleware applied only to /api/* routes, not /public/*.
Read admin/router.ts: authMiddleware missing from /admin/health endpoint.
Claim: "Auth middleware applied to /api/* (server.ts:23) but missing from
/admin/health (admin/router.ts:8)." [VERIFIED]

Absence claim — WRONG

Grep "rate limit" returned no results.
Claim: "No rate limiting implemented."

Absence claim — RIGHT

Grep "rate limit" → no results.
Grep "rateLimit", "throttle", "limiter", "requestLimit" → no results.
Read: src/middleware/index.ts, server.ts, next.config.js.
Confirmed none contain request throttling logic.
Claim: "No rate limiting found after searching 6 term variants and
reading 3 likely files." [VERIFIED ABSENT]