vibeeval/mutation-testing

Mutation Testing

Nedir?

Mutation testing, test suite'inin kalitesini olcen bir tekniktir. Kaynak kodda kucuk degisiklikler (mutasyonlar) yapilir ve testlerin bu degisiklikleri yakalayip yakalamadigina bakilir.

• Mutant: Kaynak kodda yapilan kucuk degisiklik
• Killed: Test suite mutant'i yakaladi (test fail etti)
• Survived: Test suite mutant'i yakalayamadi (testler hala geciyor)
• Kill Ratio: Killed / Total mutants (yuzde olarak)

Code coverage "kodun ne kadari calistiriliyor?" sorusunu yanitlar. Mutation testing "testler gercekten bir seyi kontrol ediyor mu?" sorusunu yanitlar.

%100 code coverage'a sahip ama assertion'i olmayan testler mutation testing'de FAIL alir.

Tool Setup

Stryker (JavaScript / TypeScript)

# Install
npm install --save-dev @stryker-mutator/core
npx stryker init

# Jest runner
npm install --save-dev @stryker-mutator/jest-runner

# Vitest runner
npm install --save-dev @stryker-mutator/vitest-runner

# TypeScript support
npm install --save-dev @stryker-mutator/typescript-checker

Config (stryker.config.mjs):

/** @type {import('@stryker-mutator/api/core').PartialStrykerOptions} */
export default {
  mutate: [
    'src/**/*.ts',
    '!src/**/*.test.ts',
    '!src/**/*.spec.ts',
    '!src/**/*.d.ts',
    '!src/**/index.ts'
  ],
  testRunner: 'jest',
  checkers: ['typescript'],
  reporters: ['html', 'clear-text', 'progress', 'json'],
  coverageAnalysis: 'perTest',
  thresholds: {
    high: 80,
    low: 60,
    break: null  // Set to 60 to fail CI on low kill ratio
  },
  timeoutMS: 60000,
  concurrency: 4
};

Run:

npx stryker run
# Report: reports/mutation/mutation.html

mutmut (Python)

pip install mutmut

Config (pyproject.toml):

[tool.mutmut]
paths_to_mutate = "src/"
tests_dir = "tests/"
runner = "python -m pytest -x --tb=short -q"
dict_synonyms = "Struct,NamedStruct"

Run:

# Full run
mutmut run

# Results
mutmut results

# Show specific mutant
mutmut show 42

# HTML report
mutmut html

go-mutesting (Go)

go install github.com/zimmski/go-mutesting/cmd/go-mutesting@latest

Run:

# Full run
go-mutesting ./...

# Specific package
go-mutesting ./pkg/calculator/...

# With score threshold
go-mutesting --score 0.8 ./...

Mutation Operatorleri

Arithmetic Mutations

a + b  ->  a - b, a * b, a / b
a * b  ->  a / b, a + b
a++    ->  a--

Neyi test eder: Matematiksel hesaplamalarin dogrulugu

Conditional Boundary Mutations

a > b   ->  a >= b
a < b   ->  a <= b
a >= b  ->  a > b
a <= b  ->  a < b

Neyi test eder: Boundary condition'lar, off-by-one hatalari

Boolean Mutations

true    ->  false
a && b  ->  a || b
a || b  ->  a && b
!a      ->  a

Neyi test eder: Boolean logic, branch coverage

Negation Mutations

if (condition)   ->  if (!condition)
while (x > 0)    ->  while (x <= 0)

Neyi test eder: Kontrol akisinin dogrulugu

Return Value Mutations

return x       ->  return 0
return true    ->  return false
return "hello" ->  return ""
return obj     ->  return null

Neyi test eder: Return value assertion'lari

String Mutations

"hello"  ->  ""
"hello"  ->  "Stryker was here!"

Neyi test eder: String handling, empty string kontrolu

Statement Removal

doSomething();  ->  (removed)
x = calculate() ->  (removed)

Neyi test eder: Side effect'lerin test edilip edilmedigi

Kill Ratio Hedefleri

| Seviye | Kill Ratio | Anlami | |--------|-----------|--------| | Mukemmel | 90%+ | Test suite cok guclu | | Iyi | 80-89% | Kabul edilebilir, kucuk iyilestirmeler | | Orta | 60-79% | Ciddi iyilestirme gerekli | | Zayif | < 60% | Test suite guvenilemez |

Hedef: Her projede minimum %80 kill ratio

Survived Mutant Analizi

Bir mutant survive ettiyse su adimlari takip et:

1. Mutant'i Anla

Dosya: src/calculator.ts:15
Original:  if (balance > 0) { ... }
Mutant:    if (balance >= 0) { ... }
Durum:     SURVIVED

2. Neden Survive Etti?

• Hicbir test balance === 0 durumunu test etmiyor
• Boundary condition icin test eksik

3. Test Yaz

it('should handle zero balance', () => {
  const result = processBalance(0);
  expect(result).toBe('no_funds'); // Bu test mutant'i oldurur
});

4. Tekrar Calistir

npx stryker run --mutate "src/calculator.ts"

Test Iyilestirme Pattern'leri

Pattern 1: Boundary Testing

Survived mutant > -> >= ise:

// Her boundary icin 3 test yaz: altinda, ustunde, tam sinirda
it('rejects when below minimum', () => expect(validate(-1)).toBe(false));
it('rejects at exact minimum', () => expect(validate(0)).toBe(false));
it('accepts above minimum', () => expect(validate(1)).toBe(true));

Pattern 2: Return Value Assertion

Survived mutant return x -> return 0 ise:

// Testlerde return value'yu MUTLAKA assert et
const result = calculate(5, 3);
expect(result).toBe(8); // Spesifik deger kontrolu

Pattern 3: Boolean Logic

Survived mutant && -> || ise:

// Her boolean kombinasyonu test et
it('fails when only A is true', () => expect(check(true, false)).toBe(false));
it('fails when only B is true', () => expect(check(false, true)).toBe(false));
it('passes when both are true', () => expect(check(true, true)).toBe(true));
it('fails when both are false', () => expect(check(false, false)).toBe(false));

Pattern 4: Side Effect Testing

Survived mutant statement removal ise:

// Side effect'leri de test et
calculate(5);
expect(mockLogger.info).toHaveBeenCalledWith('Calculated: 5');
expect(mockMetrics.increment).toHaveBeenCalledWith('calculations');

Pattern 5: Negation Testing

Survived mutant !x -> x ise:

// Her iki yolu da test et
it('handles truthy input', () => expect(process(true)).toBe('A'));
it('handles falsy input', () => expect(process(false)).toBe('B'));

CI/CD Entegrasyonu

GitHub Actions

name: Mutation Testing
on:
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 2 * * 0'  # Haftalik tam tarama

jobs:
  mutation-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - run: npx stryker run
      - uses: actions/upload-artifact@v4
        with:
          name: mutation-report
          path: reports/mutation/
      - name: Check kill ratio
        run: |
          SCORE=$(cat reports/mutation/mutation.json | jq '.schemaVersion' -r)
          # Custom threshold check script

GitLab CI

mutation-test:
  stage: test
  script:
    - npm ci
    - npx stryker run
  artifacts:
    paths:
      - reports/mutation/
    expire_in: 7 days
  only:
    - merge_requests
  allow_failure: true  # Ilk baslarken, sonra kaldir

Incremental CI (PR'larda)

PR'larda sadece degisen dosyalari mutate et:

- name: Get changed files
  id: changed
  run: |
    FILES=$(git diff --name-only origin/main...HEAD -- '*.ts' | grep -v test | tr '\n' ',')
    echo "files=$FILES" >> $GITHUB_OUTPUT
- name: Run incremental mutation
  if: steps.changed.outputs.files != ''
  run: npx stryker run --mutate "${{ steps.changed.outputs.files }}"

Performance Optimization

1. Incremental Mutation Testing

Sadece degisen dosyalari mutate et:

# Stryker
npx stryker run --mutate "src/changed-file.ts"

# mutmut
mutmut run --paths-to-mutate src/changed_module/

2. Per-Test Coverage Analysis

Stryker'da coverageAnalysis: 'perTest' kullan. Her mutant sadece ilgili testlerle calistirilir.

3. Timeout Ayari

Sonsuz donguye giren mutant'lar icin makul timeout:

timeoutMS: 60000,     // 60 saniye max
timeoutFactor: 1.5    // Normal surenin 1.5 kati

4. Concurrency

CPU sayisina gore paralel calistir:

concurrency: 4  // veya os.cpus().length - 1

5. Incremental Mode (Stryker)

Onceki sonuclari cache'le:

incremental: true,
incrementalFile: 'reports/stryker-incremental.json'

Common Pitfalls

1. Equivalent Mutants

Bazi mutasyonlar kodun davranisini degistirmez:

// Original
const i = 0;

// Mutant (equivalent - davranis ayni)
const i = -0;

Cozum: Equivalent mutant'lari rapordan cikar, survived olarak sayma.

2. Infinite Loop Mutants

while (true) veya for(;;) gibi durumlar: Cozum: Timeout ayarini dogru yap, timeout mutant'larini "killed" say.

3. Cok Uzun Calisma Suresi

Buyuk codebase'lerde saatlerce surebilir: Cozum: Incremental mode, per-test coverage, parallelism kullan.

4. Test Isolation Sorunlari

Mutant, baska testleri de etkiler: Cozum: Testlerin bagimsiz oldugunu dogrula, shared state kullanma.

5. Flaky Test False Positives

Flaky testler mutant'lari yanlis killed gosterebilir: Cozum: Once flaky testleri duzelt, sonra mutation test calistir.

6. Config Dosyalari / Constants

Config dosyalarini mutate etmenin anlami yok: Cozum: mutate pattern'indan config, constants, types dosyalarini haric tut.

Triggers

Bu skill su durumlarda aktive olur:

• "mutation test" dendiginde
• "test quality" soruldugunda
• "test effectiveness" degerlendirmesi istendiginde
• "kill ratio" soruldugunda
• "survived mutant" analizi gerektiginde
• Test suite guvenirligi sorgulandiginda