nicolas-codemate/qa-run

QA Run Skill

Execute the test plan and report failures as tickets in the project's tracker (YouTrack or GitHub, based on .claude/ticket-config.json).

This skill runs scenarios. It does not write or update them — that's /qa-sync.

Arguments

| Flag | Default | Effect | |---|---|---| | --flow <name> | all flows | Run only docs/qa/<name>.md | | --scenario <id> | all in scope | Run a single scenario by id | | --no-tickets | off | Run scenarios but do not create tickets on failure (dry mode) | | --max-failures <N> | unlimited | Stop after N failed scenarios (avoids ticket flooding on broken envs) | | --skip-preflight | off | Skip the application reachability check (use when targeting an external URL the sandbox can't ping) | | --no-grouping | off | Disable root-cause bucketing — create one ticket per FAILED scenario (legacy behavior, see Step 5c) |

Workflow

Step 1 — Pre-flight: configuration

Verify the prerequisites. Abort early with a clear message if anything is missing.

1. Playwright MCP available. Look up mcp__playwright__* tools in the current session. If none, tell the user:

   Playwright MCP n'est pas configuré. Installez-le avec :
   
     claude mcp add playwright npx -- @playwright/mcp@latest
   
   Puis relancez /qa-run.
   

   Stop the skill.

2. Project config. Read .claude/ticket-config.json. If absent → Lancez /init-project puis /qa-sync avant /qa-run. and stop. If present but no qa section → Lancez /qa-sync pour bootstrapper la QA. and stop.

3. .env.qa. If at least one scenario uses auth: user|admin, check for .env.qa at the project root. If missing → ask the user whether to abort or run with anonymous-only scenarios.

4. Test plan present. docs/qa/ must contain at least one *.md file other than README.md.

Step 2 — Pre-flight: application reachability

Skip this step when --skip-preflight is set.

BASE_URL=$(jq -r '.qa.base_url' .claude/ticket-config.json)
HTTP_CODE=$(curl -sS -o /dev/null -w "%{http_code}" --max-time 5 "$BASE_URL" || echo "000")

Treat as down when HTTP_CODE is 000 (timeout/connection refused), or starts with 5. Treat 200-499 as up (a 4xx on the home page is rare but means the server is responding).

If down, follow references/preflight.md §Application down — it covers the Docker compose prompt, the readiness polling, and the no-compose fallback.

Step 2b — Pre-run hook

Read .claude/ticket-config.json key qa.pre_run (string, optional). When set, this command resets fixtures / seeds the QA environment between runs, and may print QA_* variables on stdout that get folded into .env.qa.local.

PRE_RUN=$(jq -r '.qa.pre_run // empty' .claude/ticket-config.json)

Unset → skip silently. Set → follow references/preflight.md §Pre-run hook for the execution, the abort-on-failure rule, and the QA_* extraction contract.

Skip this sub-step when --skip-preflight is set — same intent as Step 2: "I know what I'm doing, just run scenarios".

Step 3 — Parse the test plan

Iterate over docs/qa/*.md (excluding README.md). For each file, extract every fenced ```yaml ... ``` block whose top-level keys include id: and start: — that's a scenario.

Build a flat list of scenarios with their source file and the section heading they appear under.

Apply filters:

• If --scenario <id>: keep only that ID. Abort if not found.
• If --flow <name>: keep only scenarios whose source file is docs/qa/<name>.md. Abort if file missing or has no scenarios.

Pre-filter: fully manual scenarios. After applying --scenario / --flow filters, inspect each remaining scenario's steps: list. A scenario is fully manual when its steps: list is non-empty AND every entry in steps: uses the check_manual: action key. For each such scenario:

• Set status to SKIPPED.
• Set reason to scenario fully manual (all steps are check_manual) (fixed string — used by Step 7 to group these under a single "Skipped scenarios" heading).
• Remove it from the active execution list.
• Do not open the browser for it — it is never passed to Step 5 (execution loop).

Hybrid scenarios (at least one non-check_manual: action in steps:) and scenarios with an empty or absent steps: are not affected. The pre-filtered SKIPPED scenarios are kept in memory and surfaced in Step 7's "Skipped scenarios" subsection exactly like precondition-based and auth-based SKIPPED outcomes.

The check_manual: action is a step action only — never an assertion — per ~/.claude/skills/qa-sync/references/scenario-format.md §Step actions. That is why this pre-filter inspects steps: only and not expect:.

If the resulting list is empty, print No scenarios match the filters. and stop.

Step 4 — Authentication setup

If any scenario uses auth: user, auth: admin, auth: user_fresh, or auth: user_unverified, the runner handles authentication per references/playwright-runner.md §Auth handling. This step pre-warms the cache for user and admin only:

1. For each user / admin profile referenced in the filtered plan, look for a scenario with id: AUTH-PROFILE-USER (or AUTH-PROFILE-ADMIN).
2. If present, run it; on success, capture the resulting storage state (cookies + localStorage) into an in-memory cache keyed by profile, per references/playwright-runner.md §Auth handling step 2.
3. If absent, fall back to a simple form login: navigate to /login, fill ${QA_USER_EMAIL} and ${QA_USER_PWD} from .env.qa, submit, and apply the same capture as step 2. If this fails, log a warning and SKIP auth-protected scenarios with reason auth setup failed.

user_fresh and user_unverified are not pre-warmed here: they re-login (with their fixture-specific credentials, see §Auth handling step 5) just before each scenario that uses them, and their state is never cached.

The captured storage is reused for every same-profile scenario via injection (no re-login). Re-login is triggered only on profile change, on qa.session_revalidation_endpoint returning 401, or on capture failure. The full procedure - including the injection, revalidation, and fallback rules - lives in references/playwright-runner.md §Auth handling.

Step 5 — Execute scenarios

For each scenario, execute the scenario blocks in this order: precondition → setup → start → steps → expect → teardown. Between steps and expect the runner performs an internal drain of the network and console captures (sub-step 5 below); this drain is a runtime step of the runner, not a scenario block — authors of docs/qa/*.md cannot write a drain: key. The auth profile setup (Step 4) has already run before this step — preconditions need an authenticated session for ${QA_USER_ID}-style references.

1. Open a browser context. If auth: matches the current profile and a captured storage state exists for it, inject the cached cookies + localStorage (per references/playwright-runner.md §Auth handling step 3b) and run the optional session revalidation if qa.session_revalidation_endpoint is set. On profile change, on revalidation 401, or when no cache exists, clear the context and re-login (steps 3a / 4 of §Auth handling) before recapturing.
2. Start a console + network capture for the duration of the scenario.
3. Preconditions (if precondition: block present): evaluate each guard via the browser_evaluate templates documented in references/playwright-runner.md §Precondition mapping. The schema and SKIPPED reason format are canonical in ~/.claude/skills/qa-sync/references/scenario-format.md §Preconditions.
   • On the first failing guard: mark the scenario SKIPPED with reason = <formatted guard string> (e.g., precondition GET /api/game-sessions/abc failed: status 404). Do NOT execute setup, start, steps, expect, or teardown, since setup never ran (the internal drain is also skipped — nothing to drain). Skip to the next scenario.
   • On all guards passing: continue to step 4.
4. Execute setup, then navigate to start, then each step.
5. Drain network and console (internal runtime step), before evaluating any expect: assertion. Call mcp__playwright__browser_network_requests and mcp__playwright__browser_console_messages exactly once and store the raw lists on the scenario record. They are reused by the assertion sub-step below, by Step 5c (root-cause bucketing), and by Step 6 — Issue creation (<console_block> / <network_block> in the ticket body). Filtering rules — same-host, drop static assets, drop OPTIONS for the network drain; type: "error" (or level: "error" on MCP builds that expose level instead) for the console — are documented in references/playwright-runner.md §status_max and §console_clean.
6. Evaluate each expect: assertion. Map actions to Playwright MCP calls via references/playwright-runner.md. The new status_max: and console_clean: assertions consume the drains from sub-step 5 and FAIL with normalized <observed> lines (Réponse <METHOD> <path> → <status> dépasse status_max=<n>, Erreur console: <first_error_message>).
7. Track outcome: PASSED, FAILED (which step/assertion), or SKIPPED (precondition failure, check_manual:, or auth setup failure — see Edge cases).
8. Run teardown regardless of pass/fail (best-effort, swallow errors with a warning). Exception: if the scenario was SKIPPED due to a precondition failure, teardown is also skipped (per step 3 above).

Stop the loop early if --max-failures is reached. Note: SKIPPED scenarios do not count toward --max-failures (only FAILED scenarios do).

Step 5c — Bucket FAILED scenarios by root cause

Skip this step entirely if --no-grouping is set, or if --no-tickets is set (no tickets means no buckets to merge), or if there are fewer than 2 FAILED scenarios.

When N scenarios share the same root cause (stale fixture, missing endpoint, missing selector), opening N near-identical tickets pollutes the tracker. Group failures before Step 6 so the tracker sees one parent ticket per root cause instead of N siblings.

Follow references/failure-bucketing.md: it defines the signal priority that derives a bucket key, the confidence rules, the interactive-vs-auto merge policy, and the bucket record handed to Step 6.

Step 6 — Issue creation on failure

Skip this step entirely if --no-tickets is set.

Iterate over the buckets produced by Step 5c (or the flat FAILED list when Step 5c was skipped — --no-grouping or fewer than 2 failures; --no-tickets already short-circuited above). For each bucket:

• Singleton bucket (size 1, or any bucket with merge_decision = split): use the per-scenario flow below — title [QA] <id> failed at step <N>, dedup key [QA] <id>, body from references/issue-templates.md §Body — GitHub or §Body — YouTrack.
• Parent bucket (size >= 2 with merge_decision = parent): use the parent flow below — title [QA] <flow_basename> — <signature_summary> (<N> scenarios), dedup key [QA] <flow_basename> — <signature_summary>, body from references/issue-templates.md §Parent bucket. <flow_basename> is the source filename without extension (e.g., checkout for docs/qa/checkout.md).

6a. Duplicate detection (default behavior)

Before creating any new ticket, search the tracker for an open issue whose title starts with the bucket's dedup key.

| Bucket type | Dedup key | Rationale | |---|---|---| | Singleton | [QA] <id> | Same scenario failing across runs is the same bug. The trailing step number can differ between runs — match by ID prefix. | | Parent | [QA] <flow_basename> — <signature_summary> | Same root cause across runs is the same bug. The scenario count and member list change between runs — match by flow + signature. |

• GitHub:

  gh issue list --state open --search "<dedup_key> in:title" --json number,title,url --repo "<github.repo>"
  

  Match any result whose title starts with <dedup_key>.

• YouTrack: mcp__youtrack__search_issues with query project: <prefix> summary: "<dedup_key>" State: Open (or equivalent unresolved state).

If a matching open issue exists:

1. Do not create a new ticket.

2. Post a comment using the appropriate template in references/issue-templates.md:

   • Singleton bucket → §Duplicate detection — comment template
   • Parent bucket → §Parent bucket — comment template (includes the full member list of the current run)

   Comment via:

   • GitHub: gh issue comment <number> --body "<comment>"
   • YouTrack: the appropriate mcp__youtrack__*_comment tool (or update_issue adding to the comments collection if no dedicated tool is available).

3. Track the comment in the run report under the existing ticket's URL — do not list it as a "new ticket".

If no match is found, proceed to 6b.

6b. Create the ticket

Submission via the project's tracker (read .claude/ticket-config.json):

• YouTrack: mcp__youtrack__create_issue with project = qa.youtrack_project_prefix or youtrack.project_prefix, summary = title, description = body, type = "Bug". Add a label/tag from qa.label (default qa).
• GitHub: gh issue create --title "<title>" --body "<body>" --label "<qa.label>,bug" --repo "<github.repo>".

Body placeholders <console_block> and <network_block> are filled from the per-scenario drains captured during Step 5's internal drain sub-step (between steps and expect). Formatting and caps follow references/issue-templates.md §Console block formatting (warn+error, 30-line cap) and §Network block formatting (>= 400 only, 20-row cap, sorted by status desc then URL). Empty sections are omitted per the template's "omit if empty" rule, but the drain itself always runs so the diagnostic context is captured even when an unrelated assertion is the headline failure.

Capture the returned issue URL/number for the final report. Track per-bucket whether the result is a singleton ticket, a parent ticket, or a duplicate-comment hit (Step 7 splits these).

Step 7 — Final report

Print a summary:

## QA run summary

| Flow | Scenarios | Passed | Failed | Skipped |
|------|-----------|--------|--------|---------|
| auth | 3 | 3 | 0 | 0 |
| checkout | 5 | 1 | 4 | 0 |

**Total**: 8 / 8 ran (4 passed, 4 failed across 2 root causes).

### Failure buckets

- **checkout — `POST /api/coupons → 500`** (3 scenarios)
  - CHECKOUT-DISCOUNT-01 (step 3)
  - CHECKOUT-DISCOUNT-02 (step 3)
  - CHECKOUT-PAYMENT-01 (step 5)
- **checkout — selector `button[data-test=apply-coupon]` introuvable** (1 scenario)
  - CHECKOUT-COUPON-01 (step 2)

### Skipped scenarios

- **precondition GET /api/game-sessions/abc failed: status 404** (2 scenarios)
  - CHECKOUT-DISCOUNT-01
  - CHECKOUT-DISCOUNT-02
- **auth setup failed** (1 scenario)
  - LOGIN-SUCCESS-01

### Tickets

**Parent tickets opened** (root cause shared by N scenarios):
- [PROJ-142](https://...) — checkout `POST /api/coupons → 500` (3 scenarios: CHECKOUT-DISCOUNT-01, CHECKOUT-DISCOUNT-02, CHECKOUT-PAYMENT-01)

**Tickets opened**:
- [PROJ-143](https://...) — CHECKOUT-COUPON-01 failed at step 2

**Comments added** (existing tickets still failing):
- [PROJ-118](https://...) — checkout `GET /api/billing/profile → 401` still failing (2 scenarios)

Run `/qa-run --scenario <id>` to re-test a specific scenario after a fix.

Failure buckets subsection

Lists every Step 5c bucket with at least 2 members — singletons are already named in "Tickets opened" below. Build and sort it per references/failure-bucketing.md §Rendering. Omit the heading entirely when no multi-member bucket remains.

Skipped scenarios grouping

The "Skipped scenarios" subsection groups every SKIPPED scenario by its reason field. Build it as follows:

1. Collect all scenarios whose final status is SKIPPED. Each carries a reason string (set in Step 5 — precondition failure reasons match the format documented in ~/.claude/skills/qa-sync/references/scenario-format.md §Preconditions; auth setup failures use auth setup failed; check_manual: uses the manual description).
2. Group by exact reason string equality.
3. Sort groups by descending count, then alphabetically by reason for ties.
4. Within each group, list scenario IDs alphabetically.
5. If no scenario is SKIPPED, omit the entire ### Skipped scenarios subsection (do not print an empty heading).

Skipped scenarios do NOT open or comment on tickets — see references/issue-templates.md §When NOT to open a ticket. This subsection is the only visibility surface for skipped causes.

If no failures: end with Tout est vert. Rien à signaler..

Patterns reused (no duplication)

• Ticket source detection: same logic as ~/.claude/skills/fetch-ticket/SKILL.md. Read .claude/ticket-config.json, fall back to git remote.
• Issue creation: same primitives as ~/.claude/skills/create-ticket/SKILL.md (gh CLI for GitHub, mcp__youtrack__create_issue for YouTrack).
• AskUserQuestion patterns: same shape as ~/.claude/skills/init-project/SKILL.md.
• Bucketing pattern: Step 5c reuses the group-by-key + sort-by-descending-count idea from the existing "Skipped scenarios grouping" (Step 7); the only difference is the key derivation (root cause signature vs. exact reason string).

Edge cases

• Auth setup fails: continue with anonymous scenarios; mark all auth: user|admin scenarios as SKIPPED with reason auth setup failed.
• Single scenario explodes (browser crash, MCP timeout): log the trace, mark the scenario FAILED, continue.
• --max-failures reached: stop, print partial summary, mention --skip-preflight/--scenario for re-runs.

What this skill does NOT do

• Does not modify scenarios. Use /qa-sync for that.
• Does not commit anything.
• Does not deploy or restart the application beyond docker compose up -d during preflight.
• Does not perform visual regression. Use visual-verify agent for design-vs-render checks.
• Does not group failures across different docs/qa/<flow>.md files. A 401 in auth.md and a 401 in cards.md produce separate buckets — a fix in one flow does not auto-fix the other.