ashaykubal/mock-detection

Mock Detection

Prompt template for deep mock appropriateness analysis using call graph tracing. Designed for a Sonnet sub-agent to detect T1-T4 violations and track violation scope.

---

When to Use This Skill

This is an internal skill loaded by the orchestrator during Test Audit pipeline.

| Context | Action | |---------|--------| | /test-audit invoked | Orchestrator loads this skill for Stage 2 | | Test Audit pipeline triggered by hook | Orchestrator loads this skill for Stage 2 | | Need deep mock analysis | Load directly as prompt template for Sonnet | | Files flagged by test-classification | Analyze only needs_deep_analysis: true files |

DO NOT use for:

• Direct user invocation (not user-invocable)
• Surface-level classification (use test-classification skill)
• Full audit synthesis (use test-audit skill)

---

Role in Test Audit Pipeline

This skill provides the second stage prompt template:

test-audit (P0.8) orchestrates:
  Stage 1: test-classification (Haiku) → classification YAML
  Stage 2: mock-detection (Sonnet) → violations YAML    ← THIS SKILL
  Stage 3: synthesis (Sonnet) → audit report

The orchestrator loads this skill and constructs a 4-part prompt for a general-purpose Sonnet sub-agent.

---

4-Part Prompt Template

GOAL

Analyze flagged test files for T1-T4 violations using mock appropriateness rubric and call graph analysis. Track the full scope of each violation for test effectiveness calculation.

CONSTRAINTS

• Do NOT modify any files
• Only analyze files with needs_deep_analysis: true from classification
• Use call graph analysis to detect broken integration chains
• Track violation scope (all affected lines, not just violation line)
• Provide full context for each violation (line, snippet, reason, fix)
• Complete within 50 tool calls

CONTEXT

Classification output: {classification_yaml_path}

Files to analyze: List of files with needs_deep_analysis: true

Mock appropriateness rubric: See "Mock Appropriateness Rubric" section below

T1-T4 detection patterns: See "T1-T4 Detection Patterns" section below

Violation scope tracking: See "Violation Scope Tracking" section below

Extended stub/fake patterns: See references/stub-patterns.md — Meszaros taxonomy, class hierarchy detection, factory function classification

False positive prevention: See references/false-positive-prevention.md — Two-tier allowlist (Universal Safe / Context-Dependent) and decision tree. Consult BEFORE flagging borderline patterns.

OUTPUT

Write violations to: logs/mock-detection-{YYYYMMDD-HHMMSS}.yaml

Write diagnostics to: logs/diagnostics/mock-detection-{YYYYMMDD-HHMMSS}.yaml

Use the schema specified in "Output Schema" section below.

---

Mock Appropriateness Rubric

Determine whether mocks are appropriate based on test type:

| Test Type | Expected Mocks (OK) | Inappropriate Mocks (VIOLATION) | |-----------|---------------------|--------------------------------| | Unit | External deps (DB, HTTP, fs) to isolate unit | Mocking function/module under test (T1) | | Integration | Unrelated systems only | Mocking integration boundaries (T3), broken chain (T3+) | | E2E | Almost never | Any mock breaking end-to-end flow |

Mixed-Type Files (MANDATORY)

Test files commonly contain multiple test types in different describe blocks (e.g., unit tests at top, integration tests at bottom). You MUST evaluate mock appropriateness per describe block/section, not per file. A jest.fn() that is safe in a unit test section is a T3 violation in an integration test section of the same file.

Classification signals (language-agnostic — apply to TypeScript, Python, Java, Go, Ruby, etc.):

• Block/suite name containing keywords: integration, e2e, end-to-end, acceptance, system
• Preceding comments or section headers: // INTEGRATION TESTS, # E2E, /* system tests */
• Setup patterns within the block (real DB connections = integration, browser launch = e2e)

If AST integration-mock metadata is available (from npx tsx skills/test-audit/scripts/integration-mock-detector.ts), use it as ground truth for section boundaries and mock locations. Validate AST leads and add any the AST missed.

BINDING: AST classification is final. When the AST script classifies a section as integration or e2e, that classification is NOT subject to LLM override. You MUST evaluate mocks in that section against integration/e2e rules — even if you believe the section is "actually" a unit test. Dismissing an AST T3 lead by re-classifying the section as a different test type is a rule violation. If you believe the section is mislabeled, note it as advisory — but still flag T3 violations against the classified type.

See references/false-positive-prevention.md § "Worked Example: Mixed-Type File" for a concrete demonstration.

Key Principle

A mock is inappropriate when it defeats the purpose of the test:

• Unit test claims to test function X but mocks function X → T1
• Integration test claims to verify HTTP calls but mocks HTTP → T3
• Test verifies function was called but not what it produced → T2

---

T1-T4 Detection Patterns

T1: Mocking System Under Test

Severity: Critical Priority: P0 (False Confidence)

Detection patterns:

• jest.spyOn(ModuleUnderTest, 'functionBeingTested')
• jest.mock('./file-being-tested')
• vi.mock() on the module imported in the test's subject
• Mock intercepts the exact function the test claims to verify

Call graph check: Trace from test assertion back to setup. If mock sits between "action" and "assertion" for the claimed behavior, it's T1.

T2: Verifying Calls Not Results

Severity: High Priority: P1 (Incomplete Verification)

Detection patterns:

• expect(fn).toHaveBeenCalled() without result assertion
• expect(fn).toHaveBeenCalledWith(...) without verifying the effect
• verify(mock).someMethod() without outcome check

Call graph check: After the toHaveBeenCalled assertion, is there a result/state assertion for the same operation?

T3: Mock at Integration Boundary

Severity: Critical Priority: P1 (Incomplete Verification)

Detection patterns:

• In .integration.* file: jest.mock('node-fetch'), jest.mock('fs'), jest.mock('http')
• In integration test: Mocking the very boundary being integrated

Call graph check: Does the test claim to verify "integration with X" while mocking X?

T3+: Broken Integration Chain

Severity: Critical Priority: P0 (False Confidence)

Detection patterns:

• mockData used where real function output should flow
• Data manually constructed instead of flowing from upstream function
• Integration test with hardcoded intermediate values
• Property access on manually-constructed objects used in downstream calls (e.g., mockOrder.id passed into new objects)

Class hierarchy signals (see references/stub-patterns.md for full taxonomy):

• Classes named Fake*/Stub*/Mock*/InMemory*/Test* that implement interfaces or extend base classes
• Manual stub objects without naming conventions (all methods are no-ops or return hardcoded values)
• Factory functions prefixed with buildMock* or createFake*

Call graph check: Trace data flow. If Component A should output to Component B, but test injects mockAOutput into B, the chain is broken.

T4: No Test Execution Verification

Severity: Medium Priority: P2 (Pattern Issues)

Detection patterns:

• Test file exists but npx jest (or your project test runner) not run
• Tests pass in isolation but skip in suite
• Manual notation: "Requires V3 manual verification"

Note: T4 is primarily a process check. Flag for manual review.

---

Violation Scope Tracking

Track the full scope of each violation - not just the violation line, but all lines affected by it. This enables accurate test effectiveness calculation.

Scope Calculation Rules

| Violation Type | Scope Definition | |----------------|------------------| | T1 (Mock SUT) | All lines that use the mock: assertions depending on mock, calls using mock return value | | T2 (Call-only) | The assertion line itself (single line) | | T3 (Mock boundary) | All lines using the mocked boundary (similar to T1) | | T3+ (Broken chain) | All lines using the incorrect/mocked data downstream |

Example

// Line 15: Mock setup (violation line)
const mockSpawn = jest.spyOn(child_process, 'spawn')
  .mockReturnValue(mockProcess);

// Lines 20-95: All use mockSpawn results
const proxy = startProxy();      // Line 20 - uses mock
await proxy.waitForReady();      // Line 21 - uses mock
expect(proxy.port).toBe(8080);   // Line 25 - assertion on mock
// ... more lines using mock ...
expect(proxy.isRunning()).toBe(true);  // Line 95 - still mock

// violation_scope: [15, 95]
// affected_lines: 80

---

Call Graph Analysis Approach

For each flagged file, perform systematic analysis:

Step 1: Identify Test Claims

What does each test claim to verify? Look at:

• Test name/description (it('starts proxy correctly', ...))
• Assertion statements (what is being expected?)

Step 2: Trace Data Flow

How does data flow from setup → action → assertion?

• Where is input created?
• What transforms the input?
• What does the assertion check?

Step 3: Locate Mock Interception

Where do mocks intercept this flow?

• Is mock between action and the claimed verification?
• Does mock replace real behavior the test should exercise?

Step 4: Evaluate Appropriateness

Does the mock defeat the test's purpose?

• If test claims "proxy starts" but mocks spawn → T1
• If test claims "API integration" but mocks fetch → T3

---

Output Schema

metadata:
  skill: mock-detection
  timestamp: "{ISO-8601}"
  classification_source: logs/test-classification-{YYYYMMDD-HHMMSS}.yaml
  model: sonnet
  files_analyzed: 5

violations:
  - file: tests/proxy.test.ts
    line: 15
    violation_scope: [15, 95]
    affected_lines: 80
    rule: T1
    severity: critical
    priority: P0
    pattern: "jest.spyOn(child_process, 'spawn')"
    code_snippet: |
      const mockSpawn = jest.spyOn(child_process, 'spawn')
        .mockReturnValue(mockProcess);
    reason: |
      Test claims to verify "proxy starts correctly" but mocks spawn().
      This provides false confidence - mock always succeeds.
      Lines 15-95 all use this mock, making them ineffective.
    suggested_fix: |
      Replace mock with real spawn. Use port check to verify proxy started.

  - file: tests/api.integration.ts
    line: 8
    violation_scope: [8, 45]
    affected_lines: 37
    rule: T3
    severity: critical
    priority: P1
    pattern: "jest.mock('node-fetch')"
    code_snippet: |
      jest.mock('node-fetch');
      // ... later in test
      const response = await fetchUserData(userId);
    reason: |
      Integration test should verify real HTTP communication.
      Mocking fetch defeats the purpose of integration testing.
    suggested_fix: |
      Remove jest.mock('node-fetch'). Use test server or MSW.

  - file: tests/workflow.integration.ts
    line: 42
    violation_scope: [42, 78]
    affected_lines: 36
    rule: T3+
    severity: critical
    priority: P0
    pattern: "Broken integration chain"
    code_snippet: |
      const result = await processOrder(mockOrderData);
      // mockOrderData should come from createOrder() output
    reason: |
      Test uses mockOrderData instead of real createOrder() output.
      This breaks the integration chain - no real integration tested.
    suggested_fix: |
      Replace mockOrderData with: const order = await createOrder(input);

  - file: tests/config.test.ts
    line: 42
    violation_scope: [42, 42]
    affected_lines: 1
    rule: T2
    severity: high
    priority: P1
    pattern: "expect(db.save).toHaveBeenCalled()"
    code_snippet: |
      await saveConfig(newConfig);
      expect(db.save).toHaveBeenCalled();
    reason: |
      Verifies db.save was called but not what was saved.
      Call verification without result verification is incomplete.
    suggested_fix: |
      Add result verification: expect(saved.value).toBe(newConfig.value);

totals:
  critical: 3
  high: 1
  medium: 0
  low: 0
  total_affected_lines: 154

file_summaries:
  # For each file, compute affected_lines as the UNION of all violation_scope ranges
  # (merge overlapping/identical ranges). Do NOT sum individual affected_lines values.
  # Example: two violations both scoped to [228, 269] = 42 affected lines, not 84.
  - file: tests/proxy.test.ts
    verification_lines: 95
    affected_lines: 80
    test_effectiveness: 16%
  - file: tests/api.integration.ts
    verification_lines: 55
    affected_lines: 37
    test_effectiveness: 33%
  - file: tests/workflow.integration.ts
    verification_lines: 50
    affected_lines: 36
    test_effectiveness: 28%
  - file: tests/config.test.ts
    verification_lines: 40
    affected_lines: 1
    test_effectiveness: 98%

summary: |
  Analyzed 5 flagged files. Found 4 violations affecting 154 lines.
  3 files below 95% test effectiveness threshold.
  P0 violations (false confidence): proxy.test.ts, workflow.integration.ts
  P1 violations (incomplete): api.integration.ts, config.test.ts

---

Diagnostic Output

Write diagnostic output to logs/diagnostics/mock-detection-{YYYYMMDD-HHMMSS}.yaml:

diagnostic:
  skill: mock-detection
  timestamp: "{ISO-8601}"
  model: sonnet

execution:
  tool_calls: 35
  files_analyzed: 5
  analysis_depth: "call graph tracing"

decisions:
  - file: tests/proxy.test.ts
    decision: T1_violation
    call_graph_analysis: |
      Test claims: "proxy starts correctly"
      Action: startProxy() calls child_process.spawn()
      Mock: jest.spyOn intercepts spawn()
      Result: Assertion verifies mock behavior, not real spawn
    confidence: high

  - file: tests/config.test.ts
    decision: T2_violation
    call_graph_analysis: |
      Assertion: toHaveBeenCalled() on db.save
      Missing: No assertion on saved data value
      Scope: Single assertion line (minimal impact)
    confidence: high

errors: []

---

Priority Classification

P0: False Confidence

Tests that pass but provide no real assurance:

• T1: Mock hides real failures - test always passes regardless of SUT behavior
• T3+: Broken chain means integration is never actually tested

P1: Incomplete Verification

Tests that run real code but don't fully verify:

• T2: Call happened but effect not verified
• T3: Integration boundary mocked (partial integration)

P2: Pattern Issues

Style and organization issues:

• Minor mock patterns
• Test structure recommendations

---

Integration Notes

Orchestrator Usage

The orchestrator (P0.8) constructs the full prompt by:

1. Loading this skill content
2. Including classification YAML path in CONTEXT
3. Spawning: Task(subagent_type="general-purpose", model="sonnet", prompt=...)
4. Reading output from logs/mock-detection-{YYYYMMDD-HHMMSS}.yaml

Upstream Input

From P0.6 (test-classification):

• needs_deep_analysis: true file list
• verification_lines count per file
• mock_indicators as analysis starting points

Downstream Output

To P0.8 (test-audit synthesis):

• Violation details with scope tracking
• affected_lines per file
• Pre-calculated test_effectiveness per file

---

Batching for Scale

When processing many flagged files (>10), the orchestrator must batch detection to avoid context limits.

Batching Instructions

IF flagged_file_count > 10:
    Split flagged files into batches of 10-15
    FOR each batch:
        Spawn Sonnet sub-agent with batch file list
        Include verification_lines from classification for each file
        Collect violations YAML for batch
    Merge all batch results into single detection output
ELSE:
    Process all flagged files in single sub-agent call

Batch Merge Strategy

When merging batch results:

1. Combine all violations arrays
2. Recalculate totals across all batches
3. Combine all file_summaries
4. Preserve individual violation details exactly

Parallel Execution

For optimal performance, spawn batch sub-agents in parallel:

Task(subagent_type="general-purpose", model="sonnet", prompt=batch1_prompt, run_in_background=true)
Task(subagent_type="general-purpose", model="sonnet", prompt=batch2_prompt, run_in_background=true)
...

Read all outputs after completion, then merge.

---

References

| Document | Purpose | |----------|---------| | references/stub-patterns.md | Meszaros test double taxonomy, class hierarchy detection, factory function classification | | references/false-positive-prevention.md | Two-tier allowlist (Universal Safe / Context-Dependent), decision tree for violation evaluation |

Related Skills

• test-classification (P0.6) - Surface classification (upstream)
• test-audit (P0.8) - Orchestration and synthesis (downstream)
• pipeline-templates (P0.3) - Test Audit pipeline definition