jmagly/flaky-fix

Flaky Fix Skill

Purpose

Analyze flaky test patterns and suggest or auto-apply fixes. Based on FlakyFix research showing LLMs can automatically repair flaky tests with targeted prompts.

Research Foundation

| Finding | Source | Reference | |---------|--------|-----------| | LLM Auto-repair | FlakyFix (2023) | arXiv:2307.00012 - 70%+ success rate | | Flaky Taxonomy | Google (2016) | Flaky Tests Study | | Pattern-based Fixes | FlaKat (2024) | arXiv:2403.01003 |

When This Skill Applies

• After flaky-detect identifies flaky tests
• User asks to "fix flaky test" or "make test reliable"
• CI is failing intermittently on specific tests
• Test marked as flaky needs repair

Trigger Phrases

| Natural Language | Action | |------------------|--------| | "Fix this flaky test" | Analyze and suggest fix | | "Make this test reliable" | Apply deterministic patterns | | "Why is this test flaky?" | Root cause analysis + fix | | "Auto-fix flaky tests" | Batch fix safe patterns | | "Remove timing dependency" | Specific timing fix |

Fix Patterns by Category

1. Timing Issues (45% of flaky tests)

Problem: Uses Real Time

// FLAKY: Time-dependent
it('should expire after 1 hour', () => {
  const token = createToken();
  expect(token.expiresAt).toBeGreaterThan(Date.now());
});

Fix: Mock Time

// FIXED: Mocked time
it('should expire after 1 hour', () => {
  const fixedTime = new Date('2024-01-01T00:00:00Z');
  vi.setSystemTime(fixedTime);

  const token = createToken();

  expect(token.expiresAt).toBe(fixedTime.getTime() + 3600000);
  vi.useRealTimers();
});

Problem: Explicit Sleep/Delay

// FLAKY: Arbitrary delay
it('should complete async operation', async () => {
  startAsyncOperation();
  await sleep(100);  // Race condition!
  expect(result).toBeDefined();
});

Fix: Proper Async Handling

// FIXED: Wait for actual completion
it('should complete async operation', async () => {
  const result = await startAsyncOperation();
  expect(result).toBeDefined();
});

// Or use waitFor for DOM
it('should show loading state', async () => {
  render(<Component />);
  await waitFor(() => {
    expect(screen.getByText('Loaded')).toBeInTheDocument();
  });
});

2. Async Issues (25% of flaky tests)

Problem: Missing Await

// FLAKY: Promise not awaited
it('should fetch data', () => {
  const promise = fetchData();
  promise.then(data => {
    expect(data).toBeDefined();  // May not run before test ends
  });
});

Fix: Proper Async/Await

// FIXED: Awaited promise
it('should fetch data', async () => {
  const data = await fetchData();
  expect(data).toBeDefined();
});

Problem: Race Condition

// FLAKY: Order not guaranteed
it('should process items', async () => {
  const results = [];
  items.forEach(async item => {
    results.push(await process(item));
  });
  expect(results.length).toBe(3);  // Race!
});

Fix: Sequential or Parallel Await

// FIXED: Guaranteed order
it('should process items', async () => {
  const results = await Promise.all(
    items.map(item => process(item))
  );
  expect(results.length).toBe(3);
});

3. Test Order Dependencies (20% of flaky tests)

Problem: Shared State

// FLAKY: Shared state between tests
let counter = 0;

it('should increment', () => {
  counter++;
  expect(counter).toBe(1);  // Fails if other test runs first
});

Fix: Test Isolation

// FIXED: Isolated state
describe('counter', () => {
  let counter;

  beforeEach(() => {
    counter = 0;  // Fresh state each test
  });

  it('should increment', () => {
    counter++;
    expect(counter).toBe(1);
  });
});

4. Non-deterministic Values (10% of flaky tests)

Problem: Random/UUID Values

// FLAKY: Random ID
it('should create user with ID', () => {
  const user = createUser();
  expect(user.id).toBe('expected-id');  // Random!
});

Fix: Mock Random Generation

// FIXED: Deterministic ID
it('should create user with ID', () => {
  vi.mock('uuid', () => ({
    v4: () => 'test-uuid-1234'
  }));

  const user = createUser();
  expect(user.id).toBe('test-uuid-1234');
});

5. Environment Dependencies (15% of flaky tests)

Problem: Network Calls

// FLAKY: Real network
it('should fetch from API', async () => {
  const data = await fetch('https://api.example.com/data');
  expect(data).toBeDefined();  // Network failures!
});

Fix: Mock Network

// FIXED: Mocked network
it('should fetch from API', async () => {
  vi.mock('node-fetch', () => ({
    default: vi.fn().mockResolvedValue({
      json: () => ({ success: true })
    })
  }));

  const data = await fetchFromApi();
  expect(data.success).toBe(true);
});

Auto-Fix Rules

Safe to Auto-Fix (Apply Automatically)

| Pattern | Detection | Fix | |---------|-----------|-----| | Date.now() in assertion | Regex | Wrap with vi.setSystemTime() | | Missing await on async | AST analysis | Add await keyword | | setTimeout in test | Regex | Replace with vi.advanceTimersByTime() | | Math.random() | Regex | Mock with deterministic value |

Requires Review (Suggest Only)

| Pattern | Why Review Needed | |---------|-------------------| | Shared test state | May require architectural changes | | Database fixtures | Needs isolation strategy | | External service calls | Mock design decision | | Complex async flows | Multiple fix approaches |

Output Format

## Flaky Test Fix Report

### Test: `test/api/login.test.ts:45`

**Root Cause**: Timing - uses `Date.now()` in assertion
**Confidence**: HIGH (pattern match)
**Auto-fixable**: YES

#### Original Code
```typescript
it('should create token with expiry', () => {
  const token = createToken();
  expect(token.expiresAt).toBeGreaterThan(Date.now());
});

Suggested Fix

it('should create token with expiry', () => {
  const now = new Date('2024-01-01T12:00:00Z');
  vi.setSystemTime(now);

  const token = createToken();

  expect(token.expiresAt).toBe(now.getTime() + TOKEN_LIFETIME);
  vi.useRealTimers();
});

Changes Summary

• Added: vi.setSystemTime() for deterministic time
• Added: vi.useRealTimers() cleanup
• Changed: Assertion to exact value match

Verification

Run 10x to confirm fix:

for i in {1..10}; do npm test -- test/api/login.test.ts:45; done

Batch Fix Summary

| Test | Category | Auto-Fixed | Status | |------|----------|-----------|--------| | login.test.ts:45 | Timing | Yes | ✅ Fixed | | user.test.ts:23 | Async | Yes | ✅ Fixed | | db.test.ts:67 | State | No | Suggested | | api.test.ts:12 | Network | No | Suggested |

Auto-fixed: 2 tests Manual review: 2 tests Estimated stability improvement: +1.5%

## Integration Points

- Works with `flaky-detect` for test identification
- Reports to Test Engineer for complex fixes
- Feeds into CI stability metrics
- Updates `.aiwg/testing/flaky-fixes.md`

## Script Reference

### flaky_fixer.py
Analyze and fix flaky tests:
```bash
python scripts/flaky_fixer.py --test test/api/login.test.ts --auto-fix

batch_fix.py

Fix multiple flaky tests:

python scripts/batch_fix.py --input flaky-report.json --safe-only

References

• @$AIWG_ROOT/agentic/code/addons/testing-quality/README.md — Testing quality addon overview
• @$AIWG_ROOT/agentic/code/addons/aiwg-utils/rules/human-authorization.md — Authorization before auto-applying fixes
• @$AIWG_ROOT/agentic/code/frameworks/sdlc-complete/README.md — SDLC framework context
• @$AIWG_ROOT/docs/cli-reference.md — CLI reference