AZANIR/qa-playswag

PlaySwag — API Coverage Analyzer

Analyze an OpenAPI spec against your test suite and generate an HTML report with coverage gaps + ready-made QA automation tasks.

Usage: /playswag [spec-file] [tests-dir] [--js|--ts|--py]

All arguments are optional — missing ones will be asked interactively.

---

Step 1 — Gather Required Inputs

Spec file

If $ARGUMENTS contains a path to a .json, .yaml, or .yml file — use it as SPEC_FILE.

Otherwise, ask the user:

"What is the path to your OpenAPI/Swagger spec file? (JSON or YAML)"

Verify the file exists:

test -f "$SPEC_FILE" && echo "found" || echo "not found"

If not found, show the error and ask again.

---

Tests directory

If $ARGUMENTS contains a second path argument — use it as TESTS_DIR.

Otherwise, auto-detect by checking (relative to spec file's parent dir, then cwd): tests/ → test/ → e2e/ → __tests__/ → specs/ → src/tests/ → . (cwd)

If none of these exist, ask the user:

"Where are your test files? (provide directory path, or press Enter to use current directory)"

---

Language / Runner

Determine LANG from:

1. --py flag in $ARGUMENTS → LANG=py
2. --ts flag in $ARGUMENTS → LANG=ts
3. --js flag in $ARGUMENTS → LANG=js
4. Auto-detect: check if node is available → LANG=js (recommended default)
5. Auto-detect: check if python3 is available → LANG=py
6. If unclear, ask the user:

"Which runner should I use to analyze?"

• JavaScript (node analyze.js) — runs anywhere with Node.js installed, no extra packages needed ✅
• TypeScript (npx tsx analyze.ts) — requires tsx
• Python (python3 analyze.py) — requires Python 3

---

Step 2 — Locate Scripts

Find skill scripts dir. Try in order:

# Project-local (preferred)
SCRIPTS=".claude/skills/qa-playswag/scripts"

# Global install
SCRIPTS="$HOME/.claude/skills/qa-playswag/scripts"

Use whichever exists:

[ -f "$SCRIPTS/analyze.js" ] && echo "found" || echo "not found"

---

Step 3 — Run the Analyzer

JavaScript (LANG=js) — PRIMARY, recommended

analyze.js is pure CommonJS — runs with any Node.js 14+, zero dependencies needed for JSON specs. For YAML specs it auto-tries js-yaml, yaml, then falls back to python3.

node "$SCRIPTS/analyze.js" "$SPEC_FILE" "$TESTS_DIR" [options]

TypeScript (LANG=ts) — try runners in order:

npx --yes tsx "$SCRIPTS/analyze.ts" "$SPEC_FILE" "$TESTS_DIR" [options]

If tsx fails, try ts-node or compile+run. If all fail → auto-fallback to analyze.js.

Python (LANG=py):

python3 "$SCRIPTS/analyze.py" "$SPEC_FILE" "$TESTS_DIR" [options]

If PyYAML missing and spec is YAML — pip3 install pyyaml first. If Python fails → auto-fallback to analyze.js (if node available).

Multi-spec & URL support

# Multiple spec files (merge endpoints, dedup by method+path)
node "$SCRIPTS/analyze.js" spec1.yaml spec2.yaml -- "$TESTS_DIR"

# Spec from URL (fetched and cached in /tmp for 5 min)
node "$SCRIPTS/analyze.js" https://api.example.com/openapi.json "$TESTS_DIR"

CLI Options (all runtimes)

| Flag | Description | |------|-------------| | --fail-under <pct> | Exit 1 if endpoint coverage < pct% (CI quality gate) | | --output <dir>, -o <dir> | Output directory (default: ./playswag-report) | | --format <list> | Comma-separated: html, json, tasks, badge, junit (default: all) | | --json-only | Shorthand for --format json | | --include <patterns> | Only analyze matching paths (comma-sep, wildcard *) | | --exclude <patterns> | Skip matching paths | | --include-tags <tags> | Only analyze endpoints with these tags | | --exclude-tags <tags> | Skip endpoints with these tags | | --history | Append to playswag-history.json and show delta |

Exit Codes

| Code | Meaning | |------|---------| | 0 | Success (and coverage >= threshold, if specified) | | 1 | Coverage below --fail-under threshold | | 2 | Fatal error (spec not found, parse failure) |

CI/CD Example (GitHub Actions)

- name: API coverage gate
  run: node .cursor/skills/qa-playswag/scripts/analyze.js openapi.yaml tests/ --fail-under 80 --json-only

---

Step 4 — Show Results

After the script runs, read ./playswag-report/summary.json and display:

## PlaySwag Coverage Report

**Spec:** {spec-file}
**Tests:** {tests-dir}

| Metric                    | Value      |
|---------------------------|------------|
| 📊 Endpoint Coverage      | XX% (A/B)  |
| ✗ Uncovered endpoints     | N          |
| 🗑 Deprecated still tested | N          |
| 📋 QA Tasks created       | N          |
| ⚠ Unmatched test calls    | N          |

### Uncovered Endpoints (N) — by priority:

🔴 HIGH (POST/PUT/DELETE/auth):
  - POST /api/users — Create user [users]
  - DELETE /api/orders/{id} — Cancel order [orders]

🟡 MEDIUM (GET):
  - GET /api/reports — List reports [reports]

### Files:
→ HTML Report: ./playswag-report/report.html
→ QA Tasks:    ./playswag-report/tasks.md
→ Badge:       ./playswag-report/playswag-badge.svg
→ Summary:     ./playswag-report/summary.json

Open: open ./playswag-report/report.html

---

Step 5 — Error Handling

| Error | What to do | |-------|-----------| | Spec file not found | Ask user to confirm path | | JSON parse error | Show first 5 lines of file, ask user to fix | | YAML parse error | Suggest npm i js-yaml or pip3 install pyyaml | | node not found | Use Python script | | python3 not found | Inform user, suggest brew install python3 | | No test files found | Run anyway (0% coverage), inform user | | Script error | Show stderr, offer manual analysis via Read+Grep |

Manual fallback (if all scripts fail):

1. Read spec with Read tool
2. Scan tests with Grep for URL/request patterns
3. Print text summary in chat
4. Write tasks.md manually

---

Notes

Supported spec formats

| Format | Version | Features | |--------|---------|----------| | OpenAPI 2.0 (Swagger) | JSON/YAML | basePath, definitions, $ref resolution | | OpenAPI 3.0 | JSON/YAML | servers[].url, components.schemas, $ref resolution | | OpenAPI 3.1 | JSON/YAML | Same as 3.0 |

Test files detected

.spec.ts/js, .test.ts/js, .e2e.ts/js, .spec.tsx/jsx, test_*.py, *_test.py

HTTP clients detected

Playwright request, axios, fetch, supertest (request(app)), got, ky, httpx, requests, cy.request (Cypress), Node.js http/https, RestAssured (given().when()), aiohttp session, .request('METHOD', path)

Template literal URLs (/api/users/${id}) are partially matched by extracting the static prefix.

Output files

| File | Description | |------|-------------| | report.html | Interactive HTML report with filters, search, tag coverage, copy/export/print | | summary.json | Machine-readable coverage data with coverageByTag, status code and parameter metrics | | tasks.md | Markdown QA automation tasks sorted by priority | | playswag-badge.svg | Shields.io-style SVG badge | | playswag-junit.xml | JUnit XML for CI (uncovered = failure) | | playswag-history.json | Coverage history for trend tracking (--history) |

Coverage dimensions

| Dimension | Method | |-----------|--------| | Endpoint | Regex-matched API calls in tests vs spec paths | | Status code | Assertion patterns (.expect(200), toHaveStatus(), assert status_code ==) | | Parameter | Name-match scan (param names from spec appearing in test files) | | By tag | Per-tag breakdown (spec tags field) |

QA task example

From tasks.md:

### TASK-001: Cover `POST /api/users`

| Field | Value |
|-------|-------|
| **Priority** | High |
| **Endpoint** | `POST /api/users` |
| **Auth required** | Yes |

**Acceptance Criteria:**
- [ ] Happy path -> `201`
- [ ] Invalid input -> 400/422
- [ ] Unauthenticated -> 401

Known limitations

• Static analysis only — regex-based; cannot detect dynamic URL construction or runtime-generated paths beyond template literal prefixes.
• Status code coverage is assertion-based (scans for .expect(200) etc.), not runtime-verified.
• Parameter coverage uses name-matching heuristic — a param name appearing in a test file is counted as "used", even if it's unrelated.
• No auth flow verification — authRequired flag comes from spec security definitions, but actual auth testing is not verified.
• YAML requires either js-yaml/yaml npm package, PyYAML, or python3 with PyYAML as fallback.

vs MichalFidor/playswag (npm)

| | npm playswag | This skill | |---|----------------|------------| | Approach | Runtime (Playwright intercepts HTTP) | Static (regex scan of test files) | | Spec | $ref, servers | Resolved $ref; basePath / servers[].url; multi-spec merge | | Languages | TypeScript tests | JS / TS / Python (same canonical regex set) | | Output | HTML | HTML + JSON + Markdown + SVG + JUnit XML | | Coverage | Endpoint only | Endpoint + status code + parameter + by-tag | | CI | — | --fail-under, JUnit XML, --history delta |

Analyzer behavior

• basePath (Swagger 2.0) and servers[].url path (OAS 3.x) are collected and used when matching test URLs to spec paths (direct match, strip base, or base + template).
• $ref in the spec is resolved (JSON Pointer #/...) before parsing operations, so request body fields in QA tasks are populated from components.schemas / definitions.
• Multi-spec (-- separator): endpoints are merged and deduplicated by method:path.
• URL specs: fetched via curl/wget/node http, cached in /tmp for 5 minutes.
• YAML fallback (when no js-yaml in Node): the temp spec path is passed via PLAYSWAG_YAML_FILE to Python — not embedded in the shell string.
• analyze.js is the safest runner — no build step, no extra packages, works in any Node.js 14+ project.