jaktestowac/tracking-quality-trends

Tracking Quality Trends

Use this skill when quality gets reported as a number, and nobody can say whether it is better or worse than last time.

A snapshot is almost useless on its own. "Coverage is 71%" prompts an argument about whether 71 is good. "Coverage on changed lines has moved 62 → 71 over three releases, limit 75, goal 85" prompts a decision. Direction is the finding; the absolute value is context.

The second thing this catches is regression in the quality system - a threshold lowered, a job made non-blocking, a waiver renewed for the fourth time. Those never show up in a snapshot, because a snapshot reports what is measured, not what stopped being measured.

When to Use

• quality reporting is a series of unconnected numbers
• a metric is quoted with no baseline and no target
• a team needs to show a quarter of improvement, or explain a quarter without it
• a contract has been re-derived and the question is what moved
• gates are being quietly weakened and nobody has noticed
• a release decision needs direction, not just current state

Operating Principles

• Limit / current / goal, always three numbers. Limit is the threshold that triggers action; current is measured; goal is the target. A metric with only a current value cannot be acted on.
• Direction and magnitude, not just the delta. "−3pp" needs "inside the noise band" or "third consecutive fall" beside it to mean anything.
• Archive the run, do not recompute history. Store each reading with its date, commit, and how it was measured. Recomputed history changes when the method changes, and then the trend is fiction.
• A method change breaks the series. Say so, and start a new one rather than pretending the numbers are comparable.
• Track the quality system, not only the code. Thresholds, gate blocking-ness, waiver counts and ages, suppression counts. A repo whose coverage rose while its threshold fell has got worse.
• Fewer metrics, honestly measured. Six metrics a team acts on beat twenty nobody reads. Every metric needs a named owner and an action if it crosses its limit.
• Every metric carries its caveat. Coverage does not prove correctness; a rising pass rate can mean weaker tests. Report the caveat inline, not in a footnote.
• Never trend a metric that can be gamed without saying so. Coverage, test count, and defect count all move under pressure without quality moving.

Workflow

Phase 1: Fix the metric set

Start from what the contract already implies, and cap it at six to eight. From ./resources/trend-metrics.md:

| Metric | Limit / goal example | Caveat to print | | --- | --- | --- | | Diff coverage | ≥ 75% / ≥ 85% | proves execution, not assertion quality | | Repo coverage direction | must not decrease | a big denominator absorbs new gaps | | Flake rate | < 1% / < 0.3% | only measurable if retries are recorded | | Suite duration (p95) | < 10 min / < 5 min | shortcuts appear when this rises | | Contracted practices PRESENT | - / all MUST | PRESENT means configured and enforced | | Blockers open | 0 / 0 | a MUST practice missing | | Waivers: count, expired, oldest | 0 expired | count alone hides age | | Suppressions total | trending down | includes disables, ignores, skips | | Escaped defects | trending down | depends on consistent triage | | Eval pass rate per capability | no regressions | per capability, never aggregated |

Each metric needs an owner and a stated action at the limit. A metric with neither is a dashboard decoration.

Phase 2: Archive each run

.qa/
  quality-contract.md          # current
  trends.md                    # the report
  history/
    2026-05-02/{contract.md,metrics.json}
    2026-06-13/{contract.md,metrics.json}
    2026-08-21/{contract.md,metrics.json}

Each metrics.json records the value, the date, the commit, and how it was measured - the tool, the version, and the scope. That last field is what lets a future reader tell a real improvement from a method change.

Phase 3: Diff against the previous run

Per metric: previous, current, delta, direction, and position against limit and goal. Then, separately, the structural diff nobody else reports:

• practices that moved into PRESENT - real wins, name them
• practices that regressed out of PRESENT - the most important line in the report
• practices newly N/A or deferred - usually a contract edit, so check it was deliberate
• thresholds that changed, in either direction
• gates that became non-blocking
• waivers added, expired, renewed
• suppressions added and removed

A threshold quietly lowered from 75 to 60 will otherwise appear as a coverage improvement. Reading the config diff alongside the metric diff is what catches it, and it is the single highest-value habit in this skill.

Phase 4: Establish the noise band

Before reporting a movement as a finding, know what movement is normal. Flake rate and suite duration move on their own; coverage moves with the size of the release; eval scores move with model variance.

Rule: a movement inside the noise band is not a finding, and a movement in the same direction three periods running is a finding regardless of size. Slow drift is what a threshold-based alert never catches.

Phase 5: Report

Use ./resources/trend-report-template.md. Written to .qa/trends.md, with:

1. Direction summary - improving, flat, or degrading, with the two or three metrics driving it
2. Metric table - limit, previous, current, goal, direction, and the caveat
3. Structural changes - practices gained and lost, thresholds and gates changed
4. Governance - waiver and suppression movement
5. Findings - regressions, three-period drifts, limits crossed
6. Series breaks - where a method changed and comparison stops being valid

Then the honest closing paragraph: what this report cannot see. Untracked metrics, unmeasured practices, and anything where the tooling changed.

Cadence

Tie it to something that already happens - a release, a sprint boundary, a monthly review. A report with no cadence gets written once and admired.

Two rules: re-derive the contract on the same cadence, and read the previous report before writing the new one. A trend report that does not reference its predecessor's findings is a snapshot with a date on it.

Common Failure Modes

• Hand-maintained numbers in prose. "We have 803 tests" in a README, wrong three months later. If it is worth tracking, generate it.
• Recomputing history. New tool, new method, retroactively applied - the trend now shows a change that never happened.
• Trending only code metrics. Missing the lowered threshold and the disabled job, which are the changes that made the code metrics look better.
• Reporting deltas without noise bands. Every run has a finding, so nobody reads the findings.
• Twenty metrics. Nobody acts on any of them.
• Aggregating what should be split. One eval score across capabilities; one coverage number across a monorepo.
• Celebrating a rising pass rate. It rises when tests get weaker, too. Pair it with flake rate and assertion quality.
• No owner per metric. Nothing happens when a limit is crossed.

Resource Map

• ./resources/trend-metrics.md - the metric set with limits, goals, how to measure each in a JS/TS repo, caveats, and gaming risks
• ./resources/trend-report-template.md - the report structure, the history layout, the metrics.json shape, and worked findings

Related Skills

• analyzing-quality-metrics - metric definitions, anti-metrics, and how to interpret each honestly; this skill adds history and direction
• deriving-a-quality-contract - supplies the practice list, the thresholds, and the gap matrix each run diffs
• governing-quality-waivers - waiver count, age and expiry are tracked metrics here
• verifying-change-coverage - the source of the diff-coverage reading
• assessing-comprehension-debt - its band is a trended, advisory metric
• testing-llm-features - eval pass rate per capability, trended against a baseline
• assessing-release-readiness - consumes direction, not just current state

Definition of Done

This skill is complete when:

• the metric set is six to eight metrics, each with a limit, a goal, an owner, and a stated action at the limit
• each run is archived with its value, date, commit, and measurement method
• the report shows direction per metric, not only current values
• structural changes are reported: practices gained and lost, thresholds changed, gates made non-blocking
• movements are judged against a noise band, and three-period drifts are findings regardless of size
• each metric's caveat is printed with it, not footnoted
• series breaks caused by method changes are stated, and comparison stops there
• the report names what it cannot see