vibeeval/test-strategy

Test Strategy

Test Pyramid Ratio Guidance

        /\
       /e2e\         ~10% — critical user flows only
      /------\
     /  integ  \     ~20% — API contracts, DB interactions
    /------------\
   /    unit      \  ~70% — pure logic, transformations, edge cases
  /--------------\

Keep the pyramid right-side-up. Inverting it (too many e2e) leads to slow, flaky CI.

Decision Matrix: Task Type → Test Type

| Task Type | Test Type | Tool | |-----------|-----------|------| | Pure function / utility | Unit | Jest / Vitest | | API endpoint | Integration | Supertest / httpx | | Critical user flow | E2E | Playwright | | Data transformation | Property-based | fast-check / Hypothesis | | React/Vue component | Component | Testing Library | | CLI command | Integration | execa + assertions | | Database query | Integration (real DB) | jest + pg / pytest | | Cron job / scheduler | Unit (mocked time) | Jest fakeTimers |

Mock vs Real Dependency Decision Tree

Is it an external API (Stripe, Sendgrid, etc.)?
  → YES: Always mock. Use recorded fixtures or MSW.

Is it a database?
  → Unit test context: mock (in-memory store or jest.fn())
  → Integration test context: real DB (test container or local)

Is it the file system?
  → Mock with memfs or tmp dir, then clean up.

Is it time / Date.now()?
  → Always mock. Use Jest fakeTimers or freezegun (Python).

Is it a third-party SDK wrapper you wrote?
  → Skip testing the wrapper itself, test your code's behavior.

Coverage Targets by Project Type

| Project Type | Branch Coverage | Notes | |-------------|----------------|-------| | Published library | 90%+ | Every exported function needs tests | | Production app | 80%+ | Focus on critical paths | | Internal tool | 70%+ | Happy path + main error cases | | Prototype / spike | Skip | Throw it away anyway | | Generated code | Skip | Don't test codegen output |

Test Naming Conventions

Jest / Vitest (describe + it)

describe('calculateDiscount', () => {
  it('returns 10% for gold members', () => { ... })
  it('returns 0% when cart is empty', () => { ... })
  it('throws when discount rate exceeds 100', () => { ... })
})

Given-When-Then (BDD style)

describe('OrderService', () => {
  describe('given a confirmed order', () => {
    describe('when the user cancels', () => {
      it('then it transitions to CANCELLED state', () => { ... })
      it('then it sends a cancellation email', () => { ... })
    })
  })
})

When NOT to Test

• Generated code (Prisma client, GraphQL types, protobuf outputs)
• Third-party SDK wrappers with zero custom logic
• Trivial getters/setters (getEmail() { return this.email })
• Config files
• Framework boilerplate (Next.js _app.tsx, Express server bootstrap)

Test Isolation Strategies

Transaction rollback (PostgreSQL)

beforeEach(async () => {
  await db.query('BEGIN')
})

afterEach(async () => {
  await db.query('ROLLBACK')
})

Cleanup hooks

afterEach(() => {
  jest.clearAllMocks()        // clear call counts
  jest.resetAllMocks()        // reset return values
  jest.restoreAllMocks()      // restore spied originals
})

Test containers (real DB, isolated)

import { PostgreSqlContainer } from '@testcontainers/postgresql'

let container: StartedPostgreSqlContainer

beforeAll(async () => {
  container = await new PostgreSqlContainer().start()
  process.env.DATABASE_URL = container.getConnectionUri()
})

afterAll(async () => {
  await container.stop()
})

Flaky Test Triage

When a test is flaky (passes/fails non-deterministically):

1. Check for shared mutable state (global variables, singleton caches)
2. Check for missing await on async calls
3. Check for time-dependent assertions (setTimeout, Date.now())
4. Check for ordering dependencies (tests relying on previous test state)
5. Add --runInBand to isolate and confirm

Mutation Testing (Stryker)

Mutation testing verifies that your tests actually catch bugs:

npx stryker run

// stryker.config.json
{
  "mutator": { "excludedMutations": ["StringLiteral"] },
  "thresholds": { "high": 80, "low": 60, "break": 50 },
  "reporters": ["html", "progress"]
}

Mutation score < 60% means tests pass without catching real logic errors. Focus on the surviving mutants — each one is an untested code path.

Test ROI Analysis

High ROI (write these first):

• Business logic with branching conditions
• Error handling paths
• Data validation functions
• State machine transitions

Low ROI (write last or skip):

• Simple CRUD with no custom logic
• Pass-through adapters
• Logging statements
• UI cosmetic details