renatocaliari/stelow-product-testing-ai-code

AI-Aware Testing Strategy

Based on empirical research:

• AgentAssay (2026): Non-deterministic agent testing framework
• MSR 2026: Over-mocking anti-patterns in AI-generated tests
• Veracode 2025: 45% of AI code contains vulnerabilities
• CodeRabbit 2025: AI code has 1.7x more bugs than human code
• CoderEval (2023): 43.1% of AI code is less robust

Standalone awareness: when inside stelow, triggered automatically by product_type in spec-product.md frontmatter. When standalone, invoke directly with a spec-product.md path. Appetite defaults to Core if not found — documented in output. All test-breadth tables and quality baselines work identically in both modes.

Activation

• Trigger: product_type: software or product_type: hybrid in spec-product.md frontmatter
• Phase: Phase 11 (Tech Planning)
• Prerequisite: approved spec-product.md with scope defined

Step 2: Read Appetite and Product Context

Read appetite from spec-product.md before generating test scopes. When running standalone, appetite defaults to Core if not found in frontmatter — the skill documents this assumption in the output.

Appetite controls test breadth, not quality baseline.

| Appetite | Test breadth | |----------|-------------| | Lean | Behavior/E2E (1 happy path) + smoke tests + critical-path unit tests. Add integration only when an external seam is in scope. | | Core | Behavior/E2E (happy path + variations) + unit tests for main logic + integration tests for DB/API/external seams. | | Complete | Behavior/E2E (full coverage + edge cases) + unit + integration + security tests/scans. |

Quality baseline applies to every appetite: build/test/lint/typecheck always run when available, and a11y checks run whenever UI files exist. Appetite changes exploration breadth, not whether quality gates exist.

Then determine the product context:

Before generating testing strategy, determine the product context:

| Context | Description | Testing Approach | |---------|-------------|-----------------| | Greenfield | New product, no existing code | TDD-first, appetite-specific coverage targets, clean slate | | Brownfield | Existing product with features | TDD for critical paths, test-after for existing code, regression focus | | Hybrid | Adding features to existing product | Separate new from existing, protect invariants |

Based on context from setup or spec-product.md:

• greenfield: TDD recommendation, appetite-specific coverage targets
• brownfield: TDD for critical paths only, test-after + regression for existing code
• hybrid: Add test-regression scopes for existing functionality

Input Detection (Standalone Mode)

When called outside the workflow with no pre-existing spec-product.md:

Input:
  ├── User provided a spec-product*.md path?
  │   └→ Read product_type, appetite, scope from frontmatter
  ├── User provided a description of the project?
  │   └→ Extract: language, product type, risk level
  └── No structured input?
      └→ Default: product_type=software, appetite=Core, context=brownfield

Graceful degradation: The skill works standalone. If product_type or appetite cannot be determined from frontmatter, sensible defaults are used (product_type=software, appetite=Core, context=brownfield). Every step documents its assumptions and flags them in the output.

Input

From Tech Planning context (or standalone):

• spec-product.md (frontmatter with product_type, appetite)
• spec-tech.md (scopes to add test-* types, if available)
• Tech stack detection from project files

Output

| Artifact | Path | |----------|------| | testing-strategy.md | .stelow/{date}/{_dir}/plans/ |

---

Process

Step 1: Detect Tech Stack

Auto-detect testing frameworks from project files:

| Language | Detection File | Unit Framework | Coverage Tool | Security Tool | |----------|--------------|---------------|--------------|---------------| | Python | requirements.txt, pyproject.toml | pytest | pytest-cov | Bandit | | JavaScript/TypeScript | package.json | Vitest, Jest | c8 / V8 coverage | ESLint + SAST | | Go | go.mod | testing | go test -cover | Gosec | | Rust | Cargo.toml | cargo test | cargo-tarpaulin | cargo-audit | | Java | pom.xml, build.gradle | JUnit | JaCoCo | SpotBugs |

Step 3: Classify Scope Risk

Based on spec-tech.md scopes:

| Risk Level | Examples | Testing Depth | |------------|----------|----------------| | Critical | Payment, auth, data persistence, security | High coverage + negative cases + security gates | | Standard | CRUD, UI, API endpoints | Core paths + obvious edge cases | | Experimental | Prototypes, new features | Smoke tests + regression hooks |

Step 4: Generate Appetite-Specific Test Targets

From research: coverage alone is insufficient. A test suite can execute every line but still miss behavioral bugs, security gaps, and non-deterministic agent failures. Appetite selects breadth; risk selects which paths inside that breadth are mandatory.

| Appetite | Path Type | Testing Depth | |-----------|-----------|-------------| | Lean | Critical path | E2E/behavior (1 happy path) + smoke + unit for the one negative case | | Lean | Standard/experimental | E2E/behavior (1 happy path); no broad integration suite | | Core | Critical path | E2E/behavior + unit tests + negative cases + integration seams | | Core | Standard features | E2E/behavior (key variations) + unit tests + integration for external seams | | Complete | Critical path | E2E/behavior (full) + unit + integration + security gates | | Complete | Complex flows | E2E/behavior for multi-step UI or agent workflows |

Step 5: Define Test Scope Types

For each IN scope in spec-product.md, add corresponding test scopes. Appetite controls breadth:

Lean:

| Code Type | Test Type | When to Use | TDD? | |----------|-----------|-------------|------| | User-facing flows | test-behavior | 1 E2E test for happy path | No — browser/e2e | | Critical business logic | test-unit | Happy path + one negative case | Yes for deterministic logic | | External APIs | test-integration | Only if external seam is in Lean IN scope | No — test-after | | Security-sensitive | test-security | Only if auth/payment/data is in Lean IN scope | Automated SAST |

Core:

| Code Type | Test Type | When to Use | TDD? | |----------|-----------|-------------|------| | User-facing flows | test-behavior | E2E for happy path + key variations | No — browser/e2e | | Agent workflows | test-behavior | Multi-step agents | Multi-run validation | | Business logic | test-unit | Main flows + obvious edge cases | Yes — critical paths | | External APIs | test-integration | DB, APIs, queues | No — test-after | | Security-sensitive | test-security | Auth, payment, data | Automated SAST |

Complete:

| Code Type | Test Type | When to Use | TDD? | |----------|-----------|-------------|------| | User-facing flows | test-behavior | Full E2E coverage + edge cases | No — browser/e2e | | Complex UI/user flows | test-behavior | Forms, modals, multi-step flows | Browser/e2e | | Agent workflows | test-behavior | Multi-step agents | Multi-run validation | | Business logic | test-unit | Full edge mapping | Yes — critical paths | | External APIs | test-integration | All external seams | No — test-after with contract checks | | Security-sensitive | test-security | Auth, payment, data, permissions | Automated SAST + targeted tests |

Brownfield/Hybrid (existing code):

| Code Type | Test Type | When to Use | Context | |----------|-----------|-------------|---------| | Protect existing | test-regression | Detect regressions | Brownfield/Hybrid | | Document behavior | test-characterization | Golden tests | Brownfield | | Replay past tasks | test-simulation | Verify consistency | Brownfield | | Impact analysis | test-impact | TDAD-style | Brownfield |

Based on MSR 2026 research (agents use mocks 36% vs 26% humans):

Anti-patterns to flag:

• ❌ Mock count > 3 per test
• ❌ Mocks for simple objects (use real objects instead)
• ❌ 100% coverage target (coverage ≠ test quality)
• ❌ Snapshot tests for non-UI components
• ❌ Single-run validation (agents are non-deterministic)
• ❌ Same AI for both code AND test generation (circular validation)

Step 6: Evaluate Mutation Testing Fit

Mutation testing evaluates whether a test suite would notice a regression — not just whether it executed a line. Research shows LLM-generated tests cluster around the same blind spots as the code they test (Test Homogenization Trap, AgentPatterns 2026). Mutation forces tests to prove they'd catch a defect.

Evaluate whether mutation testing adds value for this project:

# Heuristic: mutation testing fit score (0-10)
SCORE=0

# +1 if language is compiled (Go, Rust, Java, TypeScript with strict config)
+2 if language is Go, Rust, or Java

# +1 if there are critical scopes (auth, payment, data persistence)
+2 if @critical scopes exist

# +2 if project has >5 scopes or appetite=Complete
+3 if appetite=Complete or scope count > 5

# -1 if appetite=Lean (prototype/validation)
-2 if appetite=Lean

# -1 if pure CRUD with no critical paths (no payment, auth, data persistence)
-2 if pure CRUD and no @critical scopes

# Score >= 5: recommend mutation testing
# Score 2-4: recommend only for specific critical modules
# Score < 2: skip (document why)

| Context | Mutation Testing Recommendation | |---------|-------------------------------| | Production app with critical paths (auth, payment, data) | ✅ Recommend: [TYPE] test-mutation with target 50%, scoped to critical modules, run nightly | | CRUD web app, REST API without sensitive paths | 🟡 Optional: [TYPE] test-mutation with target 40%, only if existing test suite is mature | | Prototype, validation, appetite=Lean | ❌ Skip — cost (CI minutes, equivalent-mutant triage) > regression risk | | Service/no-code product | ❌ Skip — no code to test | | Skill/CLI/workflow package (e.g., stelow itself) | ❌ Skip — integration + smoke tests sufficient for low regression risk |

If recommending, add to testing strategy:

mutation_testing:
  recommended: true
  rationale: "Project has {critical_count} critical scopes in {language}; mutation testing
    validates test suite can detect regressions in business logic."
  scope: "critical modules only, not full suite"
  target: "50% starter"
  tool: "Stryker (JS/TS), mutmut (Python), PIT (Java), go-mutate (Go)"
  schedule: "nightly, not per-PR — mutation runs are too slow for gate gating without
    selective mutation (Zenseact 2023, Google 2021)"

Validated loop (AgentPatterns 2026, MUTGEN 2025):

1. Generate tests (AI)
2. Run mutation tool
3. Feed surviving mutants into prompt
4. Generate additional tests targeting survivors
5. Repeat (plateau ~4 iterations)

Anti-patterns for mutation testing with AI code:

• ❌ Same model that wrote code generates tests AND judges equivalent mutants (circular bias)
• ❌ Full-suite mutation as per-PR gate (hours of CI, Zenseact 2023)
• ❌ High targets on first pass (start at 50%, not 70%)
• ❌ No equivalent-mutant filter (>50% survival = noise, Facebook 2020)

Step 7: Create CI/CD Gates

GATES:
  critical_path_coverage:
    condition: "missing required tests"
    action: BLOCK
    rationale: "Critical paths need executable regression checks"
  
  security_findings:
    condition: "> 0 on critical paths"
    action: BLOCK
    rationale: "45% of AI code contains vulnerabilities (Veracode 2025)"
  
  flaky_rate:
    condition: "> 5%"
    action: WARN
    rationale: "Agents generate non-deterministic tests"

---

TDD Guidance (Research-Based)

Empirical findings on TDD with AI agents:

| Research | Key Finding | |----------|-------------| | LLM4TDD (2023) | Including test cases alongside problem statements enhances code generation and increases success rate on benchmarks like MBPP and HumanEval | | TDD-with-AI-Agents (2026) | TDD reduces bugs by 40-80% compared to test-after for AI workflows | | AgentPatterns.ai | Tests as specifications constrain AI behavior — "the test is a contract the agent cannot fake" | | QASkills (2026) | Without TDD, AI agents write tests that validate their own broken logic |

When to Use TDD (Based on Research)

| Code Type | TDD Recommended? | Rationale | |-----------|-----------------|-----------| | Critical business logic | ✅ Yes — recommended | TDD provides design feedback, validates understanding, and constrains AI output | | Security-sensitive | ⚠️ TDD + automated gates | Write tests first, then run SAST continuously (45% vulnerability rate) | | External APIs | ❌ No — test-after | Over-mocking is anti-pattern; use real dependencies | | Agent workflows | ❌ No — behavioral testing | Non-deterministic — needs multi-run validation | | Standard features | ⚠️ Optional | Use TDD for clarity; risk-based tests for standard paths |

Brownfield Testing (Existing Products)

When evolving an existing product (research-based):

| Aspect | Strategy | Rationale | |--------|----------|----------| | Existing tests | Adapt, don't replace | High coverage = regression focus; Low coverage = characterization tests | | New features | TDD for critical, test-after for standard | Protect existing, innovate safely | | Existing invariants | Regression + simulation/replay testing | AI agents can break invariants without detection | | Technical debt | Risk-aware testing targets | Higher depth for risky areas |

---

test-regression: Protect Existing Functionality

Purpose: Run existing test suite to detect regressions when AI modifies code.

When to use: Any scope that touches existing code in brownfield/hybrid context.

Steps:

# 1. Identify affected tests before changes
find . -path ./node_modules -prune -o \n  -name "*.test.*" -print -o \n  -name "*.spec.*" -print | xargs rg -l "module_name" > affected_tests.txt

# 2. Run baseline (before changes)
npm test -- --testPathPattern="$(cat affected_tests.txt | tr '\n' '|')" > baseline_results.json

# 3. After scope changes, rerun same tests
npm test -- --testPathPattern="$(cat affected_tests.txt | tr '\n' '|')" > post_change_results.json

# 4. Compare: any new failures = regression
diff baseline_results.json post_change_results.json

CI/CD Gate:

regression:
  condition: "baseline_failures != post_change_failures"
  action: BLOCK
  rationale: "6.08% regression rate in vanilla agent runs (TDAD paper)"

---

test-characterization: Document Existing Behavior

Purpose: Create golden tests that capture current behavior before AI changes.

When to use: Before modifying complex existing modules (no or few tests).

Steps:

# 1. Identify target module
TARGET_MODULE="src/auth/session.ts"

# 2. Generate characterization tests (capture existing behavior)
# Use AI to generate tests that pass with current implementation
npx vitest create test --filter "$TARGET_MODULE" --type characterization

# 3. Run and confirm all pass (baseline)
npm test -- --grep "$TARGET_MODULE"

# 4. Only then proceed with changes
# These tests become the regression guard

Output: *.characterization.test.ts files that document current behavior.

Key principle: Tests should PASS initially — they document what the code currently does, not what it should do.

---

test-simulation: Replay Past Tasks

Purpose: Replay successful agent tasks from history to verify consistent behavior.

When to use: After AI completes similar tasks — verify it behaves the same way.

Steps:

# 1. Record task execution (from git history or logs)
task_id="fix-login-2026-05-15"
echo "Task: $task_id" > simulation_input.txt
cat commit_message.txt >> simulation_input.txt
cat changed_files.txt >> simulation_input.txt

# 2. Replay with different agent configuration
# Compare output to original successful run
agent --config "$AGENT_CONFIG" --replay simulation_input.txt > replay_output.txt

# 3. Diff against expected (original successful output)
diff expected_output.txt replay_output.txt

# 4. If diff > threshold → behavioral regression

Tool integration: AgentPatterns.ai recommends replay testing for agent verification.

CI/CD Gate:

simulation:
  condition: "diff > tolerance_threshold"
  action: WARN
  rationale: "Behavioral drift from baseline"

---

test-impact: TDAD-Style Impact Analysis

Purpose: Graph-based analysis to identify which tests to run before AI commits.

When to use: Before ANY scope execution in brownfield/hybrid context.

Steps:

# 1. Build code-to-test dependency graph
# Python example with pytest:
pytest --co -q | awk '{print $1}' | while read test; do
  deps=$(rg -o "import.*from.*['\"]\([^'\"]*\)['\"]\|require\(['\"]\([^'\"]*\)['\"]\)" "$test" | cut -d: -f2 | sort -u)
  echo "$test: $deps"
done > code_test_graph.json

# 2. For proposed change, query affected tests
TARGET_FILES="src/auth/ src/payment/"
cat code_test_graph.json | jq -r '.[] | select(.code_files | split(",") | inside($target)) | .test_file' \
  --arg target "$TARGET_FILES"

# 3. Run impact subset before changes (baseline)
npm test -- --testPathPattern="affected_tests" > impact_baseline.txt

# 4. After scope changes, run same tests
# Any new failures → scope not complete until fixed

Alternative (simpler) using madge:

# Generate dependency graph
npx madge --image dependencies.svg --format dot .

# Find tests affecting modified modules
npx madge --inverse src/payment/

CI/CD Gate:

impact:
  condition: "new_failures_in_impact_set > 0"
  action: BLOCK
  rationale: "TDAD reduced regressions by 70%"

---

Additional test scopes for Brownfield:

Greenfield Testing (New Products)

When building a new product:

| Aspect | Strategy | Rationale | |--------|----------|----------| | TDD adoption | Full recommended | No legacy constraints, clean architecture | | Coverage/risk targets | Appetite-specific: Lean critical path only; Core main logic + external seams; Complete full edge mapping + security | Establish quality baseline from day one | | Coverage | Define target upfront | 80% baseline, higher for critical | | Technical debt | None yet | Focus on clean patterns, not remediation |

Hybrid (Feature Addition)

When adding features to existing product:

| Aspect | Strategy | Rationale | |--------|----------|----------| | New code | TDD for critical, test-after for standard | Same as greenfield | | Existing code | Regression + protection | Same as brownfield | | Integration points | Extra verification | Ensure new doesn't break old | | Agent behavior | Behavioral + regression | Non-deterministic risk |

Anti-patterns for Brownfield/Hybrid:

• ❌ AI refactoring without test protection
• ❌ AI modifying existing code without characterization tests
• ❌ Over-mocking existing integrations
• ❌ Ignoring technical debt in scope planning

TDD Cycle for AI Agents

1. RED: Write failing test (human or AI with explicit constraints)
2. GREEN: AI implements only enough to pass test
3. REFACTOR: Clean up with tests still passing

Key difference from human TDD:
- AI must see failing test BEFORE implementation
- Tests must be written independently of implementation
- Human validates test quality via critical-path coverage and negative cases

---

Risk-Based Test Feedback Loop

Use this loop when tests miss important behavior or regressions appear after AI changes:

1. Identify the missed behavior or regression
2. Add a focused test that captures the invariant
3. Rerun the affected test suite
4. Feed the invariant back into future test scopes

Test quality signals:

• Critical-path coverage
• Negative-case coverage
• Security gate results
• Flaky-rate monitoring

---

Output Format

testing-strategy.md

---
version: 1
product_type: software
generated_by: stelow-product-testing-ai-code
generated_at: {YYYY-MM-DD}
---

# Testing Strategy for {product_name}

## Tech Stack
- Language: {language}
- Unit: {framework}
- Coverage: {tool}
- Security: {tool}

## Coverage and Risk Targets
| Path Type | Required Evidence |
|-----------|-------------------|
| Critical | Branch coverage, negative cases, security gates |
| Standard | Core paths and obvious edge cases |
| Experimental | Smoke tests and regression hooks |

## Test Scopes
| Scope | Type | Required Evidence |
|-------|------|----------------|
| {feature-name} | test-unit | Critical path + negative cases |
| {feature-name} | test-integration | External seams covered |
| {feature-name} | test-security | SAST clean on critical paths |

## CI/CD Gates
- BLOCK: missing required critical-path tests
- BLOCK: security_findings > 0 on critical paths
- WARN: flaky_rate > 5%

## Anti-Patterns
- ❌ > 3 mocks per test
- ❌ Mocks for simple objects
- ❌ 100% coverage target
- ❌ Same AI for code AND tests

---

Success Criteria

• [ ] testing-strategy.md generated
• [ ] test-* scopes added to spec-tech.md
• [ ] Coverage and risk targets calibrated per risk level
• [ ] CI/CD gates documented
• [ ] Anti-patterns checklist included

---

Fallback

If tech stack detection fails:

• Python → pytest + pytest-cov + Bandit
• JavaScript/TypeScript → Vitest + c8/V8 coverage + ESLint
• Go → testing + go test -cover + Gosec
• Rust → cargo test + cargo-tarpaulin + cargo-audit

---

Related Skills

• stelow-product-tech-planning: Produces scopes to test
• stelow-product-scope-executor: Executes test scopes
• stelow-product-plan-critique: Can review testing strategy