thkt/shake

/shake - Flaky Test Check

Shake the target tests until latent flakiness falls out, then classify and root-cause it. A green test is not assumed stable; it is shaken and smell-scanned before it earns that verdict.

Input

• $ARGUMENTS holds a test path, file, or suite filter.
• Empty $ARGUMENTS targets tests touched in the working tree (git diff --name-only + git diff main...HEAD --name-only, test files only).

Routing

| $ARGUMENTS | Target set | | -------------------- | ------------------------------------------------- | | path / file / filter | That test path or suite | | empty | Changed test files in the working tree and branch |

Scope Invariant

A flaky fix changes how a test runs, never what it asserts. Pinning a clock, seeding randomness, isolating state, or mocking I/O is in scope. Loosening an assertion, widening a tolerance, or skipping a case to dodge the flake is out of scope and forbidden (see Anti-gaming).

Detection

Run both paths. Dynamic shake is the verdict source; static smell explains why and catches green tests that have not deviated yet.

Dynamic shake

Run the target ≥10 times per dimension. A dimension is the same result 10/10 = no signal from it; 1+ deviation = confirmed-flaky. Vary one dimension at a time so the trigger is attributable.

| Dimension | What it exposes | How | | ----------- | --------------------------- | ----------------------------------------------------------------- | | Repeat | Unseeded randomness, timing | Run the same command ≥10 times | | Order | Cross-test shared state | Randomize test order with a varied seed each run | | Parallelism | Races on shared resources | Toggle between serial and max-worker execution | | Seed | Direct PRNG dependence | Re-run with a different explicit seed where the suite accepts one |

Command flags by ecosystem. Detect from the project manifest.

| Ecosystem | Serialize | Randomize order | Repeat | | ----------------- | ----------------------- | ---------------------------------------------- | ----------------------------- | | Rust (nextest) | --test-threads=1 | nextest runs randomized; vary --test-threads | shell loop ×10 | | Rust (cargo test) | -- --test-threads=1 | reorder via --test-threads | shell loop ×10 | | Jest | --runInBand | --shuffle (seeded) | shell loop ×10 or --shuffle | | Vitest | --no-file-parallelism | --sequence.shuffle | shell loop ×10 |

Static smell

Scan the target test and its setup with ugrep. Each hit on a currently-passing test is reported as latent-flaky with a fix direction.

| Smell | Pattern to grep | Why flaky | | --------------- | --------------------------------------------------------------------- | ---------------------------------- | | Wall-clock | Date.now, new Date(, Instant::now, SystemTime::, time.Now | Result depends on run instant | | Unseeded random | Math.random, thread_rng, rand::random, faker | Input varies per run | | Fixed sleep | setTimeout, thread::sleep, tokio::time::sleep, sleep( | Races the system under timing load | | Shared state | static mut, module-level let/global, singleton, no per-test reset | Order leaks state between tests | | Fixed-path I/O | /tmp/<fixed>, fixed port, real network, env read | Collides under parallelism |

Classification

| Verdict | Criteria | | --------------- | ----------------------------------------------- | | confirmed-flaky | ≥1 deviating run across any shake dimension | | latent-flaky | 10/10 stable so far, but ≥1 static smell hit | | stable | 10/10 stable across all dimensions and no smell |

Fix direction

Match the trigger to its root-cause fix. Retries are not a fix.

| Trigger | Root-cause fix | | -------------- | ------------------------------------------------------------------------ | | Wall-clock | Inject a clock; fake timers (jest.useFakeTimers, tokio::time::pause) | | Random | Pin the seed; assert on derived invariants, not raw draws | | Fixed sleep | Await the actual condition; fake the clock | | Shared state | Fresh fixture per test; reset in teardown; remove globals | | Order | Make each test self-contained; keep randomized order on | | Fixed-path I/O | Unique temp dir/port per test; mock the external boundary |

Anti-gaming

Forbidden routes to a green suite. These convert a flaky test into a silent gap, which is worse than the flake.

| Forbidden | Instead | | ------------------------------------------------- | --------------------------------------------------- | | Delete, skip, or .only away the flaky test | Fix the trigger so it passes under shake | | Loosen the assertion or widen a tolerance to pass | Keep the assertion; remove the nondeterminism | | Replace a real assertion with an always-pass stub | Assert on a stable derived value | | Add a blanket retry and call it fixed | Retry only as documented last resort, flagged flaky | | Edit the shake command or count to force a pass | Report the residual flake as a blocker |

Verification

| Check | Required | | --------------------------------------------------------------- | ------------------------- | | Every target test classified | Yes | | confirmed-flaky: root-cause fix applied or deferred with reason | Yes | | latent-flaky: smell + fix direction reported | Yes | | Post-fix re-shake ≥10 runs consistent | Yes (for each fixed test) | | Assertions unchanged (Scope Invariant held) | Yes |

Stop point

Stop when every target carries a verdict, each confirmed-flaky test is fixed-and-re-shaken or explicitly deferred, and the report lists verdicts, triggers, and fix directions. If a flake survives 3 distinct fix attempts, stop and report it as a blocker rather than retrying or weakening the test.