openclaw/1kalin/afrexai-qa-engine

QA & Test Engineering Command Center

Complete quality assurance system — from test strategy to automation frameworks, coverage analysis, and release readiness. Works for any stack, any team size.

When to Use

• Planning test strategy for a new feature or project
• Writing unit, integration, or E2E tests
• Reviewing test quality and coverage gaps
• Setting up test automation and CI/CD quality gates
• Performance testing and load analysis
• Security testing checklist
• Bug triage and defect management
• Release readiness assessment

---

Phase 1: Test Strategy

Strategy Brief

Before writing any tests, define the strategy:

# test-strategy.yaml
project: "[name]"
scope: "[feature/module/full product]"
risk_level: high | medium | low
stack:
  language: "[TypeScript/Python/Java/Go]"
  framework: "[React/Express/Django/Spring]"
  test_runner: "[Jest/Vitest/pytest/JUnit/Go test]"
  e2e_tool: "[Playwright/Cypress/Selenium]"

# What are we testing?
test_scope:
  - area: "[e.g., Auth module]"
    risk: high
    test_types: [unit, integration, e2e]
    priority: 1
  - area: "[e.g., Settings page]"
    risk: low
    test_types: [unit]
    priority: 3

# What's NOT in scope (and why)
exclusions:
  - "[e.g., Third-party widget — covered by vendor]"

# Quality targets
targets:
  line_coverage: 80
  branch_coverage: 70
  critical_path_coverage: 100
  max_flaky_rate: 2%
  max_test_duration_unit: 10ms
  max_test_duration_integration: 500ms
  max_test_duration_e2e: 30s

Risk-Based Test Allocation

Not everything needs the same testing depth. Use the risk matrix:

| Risk Level | Unit Tests | Integration | E2E | Manual/Exploratory | |-----------|-----------|-------------|-----|-------------------| | Critical (payments, auth, data loss) | 95%+ coverage | Full API coverage | Happy + error paths | Exploratory session | | High (core features, user-facing) | 85%+ coverage | Key integrations | Happy path | Spot check | | Medium (secondary features) | 70%+ coverage | Critical paths only | Smoke only | On release | | Low (admin, internal tools) | 50%+ coverage | None | None | None |

Test Pyramid

Follow the pyramid — not the ice cream cone:

         /  E2E  \          ← Few (5-10%) — slow, expensive, brittle
        / Integr. \         ← Some (15-25%) — API contracts, DB queries
       /   Unit    \        ← Many (65-80%) — fast, isolated, cheap

Anti-pattern: Ice cream cone (mostly E2E, few unit tests) = slow CI, flaky builds, expensive maintenance.

Decision rule: Can this be tested at a lower level? → Test it there.

---

Phase 2: Unit Testing

Anatomy of a Good Unit Test

Every unit test follows AAA (Arrange-Act-Assert):

1. ARRANGE — Set up test data, mocks, state
2. ACT     — Call the function/method under test
3. ASSERT  — Verify the output matches expectations

Unit Test Checklist (per function)

For each function/method, verify:

• [ ] Happy path — expected input → expected output
• [ ] Edge cases — empty input, null/undefined, zero, max values
• [ ] Boundary values — off-by-one, min-1, max+1
• [ ] Error handling — invalid input → correct error thrown
• [ ] Return types — correct type, shape, structure
• [ ] Side effects — does it modify state it shouldn't?
• [ ] Idempotency — calling twice gives same result?

What to Mock (and What NOT to Mock)

Mock these:

• External APIs (HTTP calls, third-party services)
• Database queries (in unit tests only)
• File system operations
• Date/time (use fake timers)
• Random number generators
• Environment variables

DO NOT mock these:

• The function under test itself
• Pure utility functions (test them directly)
• Data transformations
• Simple value objects

Mock rule of thumb: If removing the mock would make the test hit the network, file system, or database → mock it. Otherwise → don't.

Test Naming Convention

Use the pattern: [unit] [scenario] [expected result]

Examples:

• calculateTotal returns 0 for empty cart
• validateEmail throws for missing @ symbol
• parseDate handles ISO 8601 with timezone offset

Coverage Analysis

Metrics that matter: | Metric | Target | Why | |--------|--------|-----| | Line coverage | 80%+ | Basic completeness | | Branch coverage | 70%+ | Catches missed if/else paths | | Function coverage | 90%+ | Ensures all functions are tested | | Critical path coverage | 100% | Business-critical code fully verified |

Coverage traps to avoid:

• 100% line coverage ≠ good tests (assertions matter more than lines hit)
• Coverage on generated code inflates numbers
• Trivial getters/setters pad coverage without value
• Coverage should INCREASE over time, never decrease

---

Phase 3: Integration Testing

What Integration Tests Cover

Integration tests verify that components work TOGETHER:

• API endpoint → middleware → handler → database → response
• Service A calls Service B and handles the response
• Message queue producer → consumer → side effect
• Auth flow: login → token → authenticated request

Integration Test Patterns

Pattern 1: API Contract Testing

1. Start test server (or use supertest/httptest)
2. Send HTTP request with specific payload
3. Assert: status code, response body shape, headers
4. Assert: database state changed correctly
5. Assert: side effects triggered (emails, events)

Pattern 2: Database Integration

1. Start test database (SQLite in-memory or test container)
2. Run migrations
3. Seed test data
4. Execute query/operation
5. Assert: data matches expectations
6. Teardown (truncate or rollback transaction)

Pattern 3: External Service

1. Record real API response (VCR/nock/wiremock)
2. Replay recorded response in tests
3. Assert: your code handles the response correctly
4. Also test: timeout, 500 error, malformed response

Integration Test Checklist

• [ ] Happy path — full flow works end-to-end
• [ ] Auth — unauthenticated returns 401, wrong role returns 403
• [ ] Validation — bad payload returns 400 with error details
• [ ] Not found — missing resource returns 404
• [ ] Conflict — duplicate create returns 409
• [ ] Rate limiting — excessive requests return 429
• [ ] Database constraints — unique violations, foreign keys
• [ ] Concurrency — two simultaneous writes don't corrupt data
• [ ] Timeout handling — external service timeout → graceful fallback

---

Phase 4: End-to-End (E2E) Testing

E2E Strategy

E2E tests verify complete user journeys. They're expensive — be strategic:

Test these E2E:

• User registration → email verification → first login
• Purchase flow → payment → confirmation
• Critical business workflows (the ones that make money)
• Cross-browser/device smoke tests

DON'T test these E2E:

• Individual form validations (unit test)
• API error handling (integration test)
• Edge cases (lower-level tests)
• Visual styling (visual regression tools)

E2E Test Template

test_name: "[User journey name]"
preconditions:
  - "[User is logged in]"
  - "[Product exists in catalog]"
steps:
  - action: "Navigate to /products"
    verify: "Product list is visible"
  - action: "Click 'Add to Cart' on Product A"
    verify: "Cart badge shows 1"
  - action: "Click 'Checkout'"
    verify: "Checkout form displayed"
  - action: "Fill payment details and submit"
    verify: "Order confirmation page with order ID"
postconditions:
  - "Order exists in database with status 'paid'"
  - "Confirmation email sent"
max_duration: 30s

Flaky Test Management

Flaky tests are the #1 CI killer. Handle them:

Flaky Test Triage:

1. Identify — Track test pass rates over 10+ runs
2. Classify — Why is it flaky?
   • Timing/race condition → Add explicit waits, not sleep()
   • Test data dependency → Isolate test data per run
   • External service → Mock it or use test container
   • Browser rendering → Use visibility checks, not delays
3. Quarantine — Move to @flaky suite, run separately
4. Fix or delete — Flaky test unfixed for 2 weeks → delete it

Flaky rate target: < 2% of total test runs

---

Phase 5: Performance Testing

Performance Test Types

| Type | Purpose | When | |------|---------|------| | Load test | Normal traffic handling | Before every release | | Stress test | Find breaking point | Quarterly or before scaling | | Spike test | Sudden traffic burst | Before marketing campaigns | | Soak test | Memory leaks over time | Monthly or after major changes | | Capacity test | Max users/throughput | Planning infrastructure |

Performance Test Plan

test_name: "[API/Page] Load Test"
target: "[URL or endpoint]"
baseline:
  p50_response: "[current p50 ms]"
  p95_response: "[current p95 ms]"
  p99_response: "[current p99 ms]"
  error_rate: "[current %]"

scenarios:
  - name: "Normal load"
    vus: 50          # virtual users
    duration: 5m
    ramp_up: 30s
    thresholds:
      p95_response: "< 500ms"
      error_rate: "< 1%"

  - name: "Peak load"
    vus: 200
    duration: 10m
    ramp_up: 1m
    thresholds:
      p95_response: "< 2000ms"
      error_rate: "< 5%"

  - name: "Stress test"
    vus: 500
    duration: 5m
    ramp_up: 2m
    # Find the breaking point — no thresholds, observe

Performance Metrics Dashboard

Track these per endpoint:

| Metric | Green | Yellow | Red | |--------|-------|--------|-----| | p50 response | < 200ms | 200-500ms | > 500ms | | p95 response | < 500ms | 500ms-2s | > 2s | | p99 response | < 1s | 1-5s | > 5s | | Error rate | < 0.1% | 0.1-1% | > 1% | | Throughput | > baseline | 80-100% baseline | < 80% | | CPU usage | < 60% | 60-80% | > 80% | | Memory usage | < 70% | 70-85% | > 85% | | DB query time | < 50ms avg | 50-200ms | > 200ms |

Common Performance Fixes

| Symptom | Likely Cause | Fix | |---------|-------------|-----| | Slow API response | N+1 queries | Batch/join queries | | Memory climbing | Object retention | Profile heap, fix leaks | | Timeout spikes | Connection pool exhaustion | Increase pool, add queuing | | Slow page load | Large bundle | Code split, lazy load | | DB bottleneck | Missing index | Add index on WHERE/JOIN columns | | High CPU | Synchronous compute | Move to worker/queue |

---

Phase 6: Security Testing

Security Test Checklist

Run through these for every feature/release:

Authentication & Authorization:

• [ ] Passwords hashed with bcrypt/argon2 (not MD5/SHA1)
• [ ] Session tokens are random, sufficient length (128+ bits)
• [ ] JWT tokens have short expiry (15 min access, 7 day refresh)
• [ ] Failed login rate limiting (5 attempts → lockout)
• [ ] Password reset tokens expire (1 hour max)
• [ ] Role-based access enforced server-side (not just UI)
• [ ] Can't access other users' data by changing IDs in URL

Input Validation:

• [ ] SQL injection — parameterized queries everywhere
• [ ] XSS — output encoding, CSP headers
• [ ] CSRF — tokens on state-changing requests
• [ ] Path traversal — validate file paths, no ../
• [ ] Command injection — never pass user input to shell
• [ ] File upload — validate type, size, scan for malware
• [ ] JSON/XML parsing — depth limits, entity expansion disabled

Data Protection:

• [ ] HTTPS everywhere (HSTS header)
• [ ] Sensitive data encrypted at rest
• [ ] PII not logged (mask in log output)
• [ ] API keys not in client-side code
• [ ] CORS configured correctly (not *)
• [ ] Security headers set (X-Frame-Options, X-Content-Type-Options)

Infrastructure:

• [ ] Dependencies scanned for CVEs (npm audit / pip audit)
• [ ] Docker images scanned (Trivy/Snyk)
• [ ] Secrets not in code/env files (use vault)
• [ ] Error messages don't leak internals
• [ ] Admin endpoints behind VPN/IP allowlist

OWASP Top 10 Quick Reference

| # | Vulnerability | Test For | |---|--------------|----------| | A01 | Broken Access Control | Access other users' resources, bypass role checks | | A02 | Cryptographic Failures | Weak hashing, plaintext secrets, expired certs | | A03 | Injection | SQL, XSS, command, LDAP injection | | A04 | Insecure Design | Business logic flaws, missing rate limits | | A05 | Security Misconfiguration | Default creds, verbose errors, open ports | | A06 | Vulnerable Components | Outdated deps with known CVEs | | A07 | Authentication Failures | Brute force, weak passwords, session fixation | | A08 | Data Integrity Failures | Unsigned updates, CI/CD pipeline injection | | A09 | Logging Failures | Missing audit logs, no alerting on breaches | | A10 | SSRF | Internal network access via user-controlled URLs |

---

Phase 7: Bug Triage & Defect Management

Bug Report Template

bug_id: "[auto or manual]"
title: "[Short description of the bug]"
severity: P0-critical | P1-high | P2-medium | P3-low
reporter: "[name]"
date: "[YYYY-MM-DD]"

environment:
  os: "[OS + version]"
  browser: "[Browser + version]"
  app_version: "[version/commit]"
  
steps_to_reproduce:
  1. "[Step 1]"
  2. "[Step 2]"
  3. "[Step 3]"

expected_result: "[What should happen]"
actual_result: "[What actually happens]"
frequency: "always | intermittent | once"
screenshots: "[links]"
logs: "[relevant log output]"

Severity Classification

| Level | Definition | SLA | Example | |-------|-----------|-----|---------| | P0 Critical | System down, data loss, security breach | Fix in 4 hours | Payment processing broken | | P1 High | Major feature broken, no workaround | Fix in 24 hours | Users can't login | | P2 Medium | Feature broken with workaround | Fix this sprint | Search returns wrong results sometimes | | P3 Low | Minor issue, cosmetic | Fix when convenient | Button alignment off by 2px |

Bug Triage Process (Weekly)

1. Review all new bugs (unassigned)
2. For each bug:
   a. Reproduce — can you trigger it?
   b. Classify severity (P0-P3)
   c. Estimate fix effort (S/M/L)
   d. Assign to owner + sprint
   e. Link to related bugs/stories
3. Review P0/P1 bugs from last week — are they fixed?
4. Close bugs that can't be reproduced (after 2 attempts)
5. Update metrics dashboard

Bug Metrics Dashboard

Track weekly:

| Metric | Formula | Target | |--------|---------|--------| | Bug escape rate | Bugs found in prod / total bugs | < 10% | | Mean time to fix (P0) | Avg hours from report to deploy | < 8 hours | | Mean time to fix (P1) | Avg hours from report to deploy | < 48 hours | | Bug reopen rate | Reopened bugs / closed bugs | < 5% | | Test escape analysis | Bugs that SHOULD have been caught | Track & reduce | | Open bug count | Total open by severity | Trending down |

---

Phase 8: Release Readiness

Release Checklist

Before shipping to production:

Code Quality:

• [ ] All unit tests passing
• [ ] All integration tests passing
• [ ] E2E smoke suite passing
• [ ] No new lint warnings/errors
• [ ] Code reviewed and approved
• [ ] No known P0/P1 bugs open for this release

Coverage & Quality Gates:

• [ ] Line coverage ≥ target (80%)
• [ ] Branch coverage ≥ target (70%)
• [ ] No coverage decrease from last release
• [ ] Mutation testing score ≥ 60% (if applicable)

Performance:

• [ ] Load test passed (within thresholds)
• [ ] No performance regressions vs baseline
• [ ] Bundle size within budget

Security:

• [ ] Dependency audit clean (no critical/high CVEs)
• [ ] Security checklist completed
• [ ] Secrets rotated if needed

Operational Readiness:

• [ ] Monitoring/alerts configured for new features
• [ ] Rollback plan documented
• [ ] Feature flags in place for risky changes
• [ ] Database migration tested and reversible
• [ ] Runbook updated

Release Readiness Score

Score 0-100 across 5 dimensions:

| Dimension | Weight | Scoring | |-----------|--------|---------| | Test coverage | 25% | 100 if targets met, -10 per gap area | | Bug status | 25% | 100 if 0 P0/P1, -20 per open P0, -10 per P1 | | Performance | 20% | 100 if all green, -15 per yellow, -30 per red | | Security | 20% | 100 if clean, -25 per critical, -15 per high | | Operational | 10% | 100 if checklist complete, -20 per missing item |

Ship threshold: ≥ 80 overall, no dimension below 60

---

Phase 9: CI/CD Quality Gates

Pipeline Quality Gates

Configure these gates in your CI pipeline:

# Quality gate configuration
gates:
  - name: "Lint"
    stage: pre-commit
    command: "npm run lint"
    blocking: true
    
  - name: "Unit Tests"
    stage: commit
    command: "npm test -- --coverage"
    blocking: true
    thresholds:
      pass_rate: 100%
      coverage_line: 80%
      coverage_branch: 70%
      
  - name: "Integration Tests"
    stage: merge
    command: "npm run test:integration"
    blocking: true
    thresholds:
      pass_rate: 100%
      
  - name: "Security Scan"
    stage: merge
    command: "npm audit --audit-level=high"
    blocking: true
    
  - name: "E2E Smoke"
    stage: staging
    command: "npm run test:e2e:smoke"
    blocking: true
    thresholds:
      pass_rate: 100%
      
  - name: "Performance"
    stage: staging
    command: "npm run test:perf"
    blocking: false  # Alert only
    thresholds:
      p95_regression: 20%

Test Automation Maturity Model

Rate your team 1-5:

| Level | Description | Characteristics | |-------|------------|-----------------| | 1 — Manual | All testing is manual | No automation, long release cycles | | 2 — Reactive | Some unit tests, no CI | Tests written after bugs, not before | | 3 — Structured | Test pyramid, CI pipeline | Unit + integration, automated on push | | 4 — Proactive | Full automation, quality gates | E2E + perf + security in pipeline, TDD | | 5 — Optimized | Self-healing, predictive | Flaky auto-quarantine, AI-assisted testing, continuous deployment |

---

Phase 10: Test Maintenance

Weekly Test Health Review

review_date: "[YYYY-MM-DD]"

metrics:
  total_tests: 0
  pass_rate_7d: "0%"
  flaky_tests: 0
  flaky_rate: "0%"
  avg_suite_duration: "0s"
  coverage_line: "0%"
  coverage_branch: "0%"
  
actions:
  quarantined: []     # Tests moved to flaky suite
  deleted: []         # Tests removed (obsolete/unfixable)
  fixed: []           # Flaky tests fixed this week
  added: []           # New tests added
  
trends:
  coverage_delta: "+0%"     # vs last week
  flaky_delta: "+0"         # vs last week
  duration_delta: "+0s"     # vs last week
  
notes: ""

Test Maintenance Rules

1. No commented-out tests — delete or fix, never comment
2. No skipped tests > 2 weeks — fix or remove
3. No test duplication — each behavior tested once at the right level
4. Test names must be readable — someone new should understand what broke
5. Shared test utilities — common setup in fixtures/factories, not copy-pasted
6. Test data isolation — each test creates its own data, cleans up after
7. No magic numbers — use named constants in assertions
8. Assertion messages — custom messages on complex assertions

Common Test Anti-Patterns

| Anti-Pattern | Problem | Fix | |-------------|---------|-----| | Sleeping tests | sleep(2000) instead of waiting | Use explicit waits/polling | | Test interdependence | Test B relies on Test A's state | Isolate — each test sets up its own state | | Assertionless tests | Test runs code but doesn't assert | Add meaningful assertions | | Brittle selectors | CSS selectors that break on redesign | Use data-testid or aria roles | | God test | One test verifying 20 things | Split into focused tests | | Mock overload | Everything mocked, nothing real tested | Only mock external boundaries | | Hardcoded data | Tests break when seed data changes | Use factories/builders | | Ignoring test output | "It passed, ship it" | Review WHY it passed — is the assertion meaningful? |

---

Quick Reference: Natural Language Commands

Tell the agent:

• "Create test strategy for [feature]" → Generates strategy brief
• "Write unit tests for [function/file]" → AAA-structured tests with edge cases
• "Review test coverage for [module]" → Gap analysis + recommendations
• "Write integration tests for [API endpoint]" → Full HTTP test suite
• "Plan E2E tests for [user journey]" → E2E test template
• "Run security checklist for [feature]" → OWASP-based security review
• "Triage these bugs: [list]" → Severity classification + assignment
• "Release readiness check" → Full readiness score + blockers
• "Performance test plan for [endpoint]" → Load/stress test configuration
• "Fix flaky test [name]" → Root cause analysis + fix strategy