posthog/triaging-visual-review-runs

Triaging visual review runs

Visual Review is PostHog's screenshot-regression product: CI captures storybook + playwright screenshots, diffs them against committed baseline hashes, and gates the PR until a human approves the visible changes. A PR with visual changes carries a visual-review GitHub status check that stays red until each diffed snapshot is approved or tolerated in the VR UI.

This skill teaches an agent how to answer the questions a human reviewer would actually ask, by chaining the VR MCP tools — instead of reaching for gh pr view and tab-hopping to the VR web UI. The read tools cover status / scope / history / triage. Two are reversible DB-only triage marks (approve-create, tolerate-create); one ships the change (finalize-create) — it commits the baseline and greens the gate, and only that one needs explicit per-run human confirmation.

When this skill applies

Trigger this skill on any of:

• A PR number, branch name, or commit SHA paired with words like visual review, VR, snapshot, screenshot, storybook diff, playwright snapshot, baseline, approve, tolerated, quarantine.
• Questions about why a PR is blocked, what visually changed, or whether a diff is real.
• "Is my run done?" / "What's left to review?" / "Has this story flaked recently?"
• A failing visual-review GitHub check or a PR comment from the posthog-bot mentioning visual review.

When the user asks for the rendered diff image itself, the VR web UI is faster — direct them there. This skill is for everything around the diff: status, scope, history, triage.

Tools

Read tools (safe to call freely):

| Tool | Purpose | | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | posthog:visual-review-runs-list | List runs, filter by pr_number / commit_sha / branch / review_state. Start here. | | posthog:visual-review-runs-retrieve | Full detail for a single run (status, summary counts, supersession). | | posthog:visual-review-runs-snapshots-list | Per-snapshot results inside a run: identifier, result, diff %, classification, baseline + current artifact URLs. | | posthog:visual-review-runs-snapshot-history-list | A single story's last N runs across master/PRs — the flake check. | | posthog:visual-review-runs-counts-retrieve | Aggregate counts for queue triage (how many runs in needs_review, etc.). | | posthog:visual-review-runs-tolerated-hashes-list | Hashes the team has explicitly accepted as "known flake / acceptable variation". | | posthog:visual-review-repos-list | Repos (one per GitHub repo) — usually only one matters; useful for filtering. | | posthog:visual-review-repos-retrieve | Repo metadata: baseline file paths, PR-comment configuration. |

Triage tools (reversible, DB-only — they record a review decision but do NOT change the baseline or the gate):

| Tool | Purpose | | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | posthog:visual-review-runs-approve-create | Mark changed / new snapshots reviewed (approved) in the DB. Does NOT commit or green the gate — ship via finalize. | | posthog:visual-review-runs-tolerate-create | Mark a single changed snapshot as a known tolerated alternate. Does NOT change the baseline — use for benign variants. |

Ship tool (irreversible, outward-facing — requires explicit per-run human confirmation; see the gate):

| Tool | Purpose | | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | posthog:visual-review-runs-finalize-create | Commit the approved baseline to the PR branch and green the GitHub visual-review check. This ships the change. |

Mark-reviewed call shape (approve-create):

• id (required) — the run UUID. It's the route parameter, so the call fails without it.
• snapshots: [{identifier, new_hash}] — new_hash is the content_hash of each snapshot's current_artifact. This only records the review in the DB; nothing is committed and the gate stays red until you finalize.

Toleration call shape — both fields are required:

• id (required) — the run UUID. It's the route parameter, so the call fails without it.
• snapshot_id (required) — the UUID of the individual snapshot to tolerate (from visual-review-runs-snapshots-list). This identifies which snapshot inside the run; it does not replace the run id.

Finalize call shape (finalize-create) — the all-or-nothing ship action:

• id (required) — the run UUID.
• approve_all: true — approve every still-pending changed/new snapshot before finalizing (tolerated ones are left alone). Use when you've verified every remaining diff is intended.
• Omit approve_all (default false) to finalize a run you've already reviewed snapshot-by-snapshot. Finalize is all-or-nothing: it fails with 409 not_fully_resolved (and lists what's left) unless every changed/new snapshot is approved, tolerated, or quarantined.
• It commits exactly the snapshots approved in the DB — tolerated snapshots keep their baseline and are never overwritten. When a baseline commit is pushed, its SHA comes back on the run's metadata.baseline_commit_sha. It's absent when nothing needed committing (everything resolved by toleration/quarantine — the gate still greens) or when the commit was skipped: no PR, or a 409 sha_mismatch because the PR has newer commits (that one leaves the gate red — re-run CI on the latest commit and finalize again).

If finalize fails with 409 stale_run, the run has been superseded — visual-review-runs-list { pr_number } and finalize the newest one. A successful finalize often kicks off a fresh CI run, which is normal.

The finalize gate

Finalize is the one irreversible, outward-facing action in this skill: it rewrites the baseline committed to the PR and greens the merge gate. Treat it like pushing to someone's branch — never automatic.

Before any finalize-create call, all of these must hold:

1. You verified the diffs. You pulled the current (and, for changed, baseline) PNGs and looked at them, ran the flake check on anything suspect, and reached a per-snapshot verdict. Metadata alone is never enough.
2. You presented the verdict and waited. Show the user, per snapshot, what changed and your recommendation, then stop.
3. The user explicitly approved this run. A broad "get the gate green" / "fix the PR" is permission to investigate and recommend — NOT to finalize. When the task implies finalizing but the human hasn't said it for this specific run, ask.

approve-create and tolerate-create are reversible triage and don't need this gate — but they don't ship anything either. The moment you're about to finalize-create and can't point to a specific human "yes" for this run, stop and ask.

Vocabulary cheat sheet

These appear in tool output and matter for interpretation:

• Run review_state: needs_review (open, awaiting human), clean (zero diffs), processing (CI still uploading), stale (a newer run on the same PR has superseded this one — check superseded_by_id).
• Run run_type: storybook (component snapshots) or playwright (full-page e2e snapshots).
• Snapshot result: unchanged, changed (real diff), new (no baseline yet), removed.
• Snapshot classification_reason: tolerated_hash (matches a known-tolerated hash, no action needed), below_threshold (under the noise floor), exact (byte-identical), "" (real diff requiring review).
• Snapshot review_state: pending or approved.
• Run summary: total / changed / new / removed / unchanged / unresolved / tolerated_matched — unresolved is what's actually blocking review.

Workflows

"What's the VR status of this PR?"

The single most common job. Map a PR number to its run state in two calls.

1. posthog:visual-review-runs-list { pr_number: <n>, limit: 5 } — sort by created_at desc, take the latest non-stale one.
2. If the run has summary.changed > 0 or summary.unresolved > 0, drill in: posthog:visual-review-runs-snapshots-list { id: <run_id> } and report the changed snapshots.

Report back: PR number, run UUID, review_state, summary counts, and the _posthogUrl deep link so the user can click straight to the diff viewer.

"Is the diff real or unrelated?"

The most useful judgment a code-aware agent can add. Combine three signals: scope match, flake history, and the actual rendered images. The agent should look at the screenshots — not just describe metadata.

1. Scope check — git diff master...HEAD --stat (or against the PR's base branch) → list of touched paths. Cross-reference with posthog:visual-review-runs-snapshots-list { id } filtered to result: changed → story identifiers. Stories are namespaced like <area>-<scene>--<story>--<theme>; e.g. scenes-app-settings-user--settings-user-profile--dark maps to frontend/src/scenes/settings/user/.... Use this to translate story id → likely source path.

2. Visual inspection — for each changed snapshot, the tool result contains current_artifact.download_url and baseline_artifact.download_url. These are pre-signed S3 URLs to PNG files; pull them and look:

   curl -s -o /tmp/vr-baseline.png "<baseline_artifact.download_url>"
   curl -s -o /tmp/vr-current.png "<current_artifact.download_url>"
   

   Then Read both files (the Read tool renders images visually) and compare. Things to call out:

   • The actual visible delta (text changed, button moved, layout shift, color drift, missing element).
   • Whether the change is consistent with the diff_pixel_count and diff_percentage in the metadata (e.g. 54% diff but the images look near-identical → screenshot framing changed, not the UI).
   • Whether the baseline and current have different dimensions (width / height fields). Mismatched dimensions usually mean the story rendered to a different viewport or didn't fully render before screenshot — a flake signal, not a regression.

3. Flake history — run the flake check below for any story that looks suspect.

4. Verdict — combine all three:

   • Scope plausible + visible regression matches the code change → real diff, recommend approval.
   • Scope mismatch + dimensions mismatch + frequent prior changes → flake, recommend tolerating the hash.
   • Scope plausible + visible regression looks unintended → push a fix; do not approve.

Always include a one-line description of what you saw in the images — the user uses this to decide whether to trust your verdict without opening the VR UI themselves.

Flake check: "Has this story been changing?"

Once you have a suspect snapshot identifier:

posthog:visual-review-runs-snapshot-history-list { id: <snapshot_id> } → returns prior outcomes for the same story.

Verdicts:

• Mostly unchanged and this run's diff is the outlier → likely a real regression caused by this PR.
• Frequent changed across unrelated branches/master → flaky story; recommend tolerating the hash via the UI.
• Recent removed or large-jump dimension change → baseline likely stale; recommend re-baselining on master.

Triaging the queue

When the user is doing housekeeping rather than asking about a specific PR:

1. posthog:visual-review-runs-counts-retrieve → total queue size.
2. posthog:visual-review-runs-list { review_state: needs_review, limit: 50 } (paginate if needed).
3. Group by branch author or run_type to surface clusters (e.g., "12 PRs blocked on the same shared component change" usually means a single underlying root cause to address).
4. Prefer surfacing runs whose summary.changed > 0 over runs that are only new — new means no baseline yet, which is usually trivial to approve; changed is the real review work.

Output expectations

For PR-status questions, lead with the verdict in one line, then 2-4 bullets of supporting context. Always include the _posthogUrl deep link to the run — humans need to see the rendered images to make the call, the agent can only describe the metadata.

For triage / aggregate questions, a short table beats prose. Group by what the user is going to act on.

What NOT to do

• Do not approve or tolerate without explicit user confirmation. The verdict is yours to recommend; the decision to ship belongs to the user. Once they say "approve those" / "tolerate that", call the tool.
• Do not assume the failing GitHub check on a PR is unrelated to VR — if a visual-review check is red on a PR you're working on, that's the trigger to run this skill.
• Do not declare a verdict from metadata alone when result: changed. Pull the baseline and current PNGs and look at them; metadata can only say "something changed", not whether the change is intended.