etanhey/qa-video

/qa-video — Video-Based QA Pipeline

Record your screen while narrating. The pipeline extracts your speech, identifies what you were looking at, and compiles structured QA findings ready to hand off to an implementing agent.

How It Works

Screen Recording (.mov)
  → ffmpeg audio extraction
    → whisper-cli transcription (SRT + TXT)
      → LLM reads SRT, identifies QA-relevant segments
        → Frame extraction at hotspot timestamps + regular intervals
          → Claude Vision reads frames + correlates with transcript
            → Structured QA findings document
              → Agent handoff (Codex/Claude worker via cmux)

The Cardinal Rule: Narrate Before You Act

Tell the user BEFORE every QA recording session:

Narrate your intentions BEFORE clicking. Say "I'm about to click Spin Rare on Sarah" → click → describe what happened. This aligns speech timestamps with actions, making the pipeline 3x more accurate at identifying what you were pointing at.

The 2-5 second gap between clicking and narrating is the #1 accuracy killer. Coaching the user to narrate-first is more impactful than any technical fix.

---

Workflow Detection & Routing

Read the user's request and route to the right workflow:

| User says | Route to | |-----------|----------| | "let's do QA", "test this", "QA round" | workflows/record.md — Pre-QA checklist + recording setup | | "process this video", "I recorded QA", path to .mov file | workflows/process.md — Stalker pipeline processing | | "send fixes to Codex", "hand off findings" | workflows/handoff.md — Agent handoff pattern | | "next round", "retest", "QA round N" | workflows/iterate.md — Multi-round QA cycle | | "set up click capture", "qa-record" | references/click-capture.md — CGEventTap + qa-record.sh |

If ambiguous: Default to the full cycle — checklist → record → process → handoff.

---

Key Design Decisions (learned from real usage)

1. LLM reads the SRT directly — no automated hotspot detection (sox/ImageMagick). Claude reading the transcript is a better hotspot detector than volume spikes or frame diffs. The automated signals (from the original Twitch stalker pipeline) are unnecessary for QA narration.

2. Regular interval + hotspot frames — Extract one frame every 30 seconds for full visual coverage, PLUS frames at each identified hotspot. Hotspot-only extraction misses visual bugs described after the fact.

3. Before/after context frames — At each hotspot, extract frames at -5s and +5s in addition to the hotspot timestamp. The user may explain an issue THEN show it, or show it THEN explain.

4. Whisper model: ggml-small — Fast on Apple Silicon (~14s for 7min video), accurate enough for English QA narration. The ggml-large-v3 is better but 5x slower — not worth it for QA.

5. Findings live in the PROJECT repo — docs/qa-session-YYYY-MM-DD/ in the project being tested, not in the orchestrator. Each round gets a suffix: qa-findings-round2.md.

6. BrainLayer storage is mandatory — After every video processing run, brain_store the findings summary with tags ["qa", "<project>", "round-N"].

7. QA is iterative — Expect 3-6 rounds per feature. The skill supports multi-round workflows with proper round numbering and delta tracking (what was fixed vs. what persists).

---

Prerequisites

| Tool | Check | Install | |------|-------|---------| | ffmpeg | which ffmpeg | brew install ffmpeg | | whisper-cli | which whisper-cli | brew install whisper-cpp | | whisper model | ls ~/.cache/whisper/ggml-small.bin | whisper-cli --download-model small |

Optional (click capture — Phase 2):

| Tool | Check | Install | |------|-------|---------| | pyobjc | python3 -c "import Quartz" | pip3 install pyobjc-framework-Quartz pyobjc-framework-ApplicationServices pyobjc-framework-Cocoa | | qa_click_logger.py | ls ~/Gits/orchestrator/scripts/qa/qa_click_logger.py | Already exists in orchestrator repo | | qa-record.sh | ls ~/Gits/orchestrator/scripts/qa/qa-record.sh | Already exists in orchestrator repo |

---

Quick Reference

Start a recording session:

# With click capture (recommended):
bash ~/Gits/orchestrator/scripts/qa/qa-record.sh ~/Gits/<project>/docs/

# Manual (just screen recording):
# Cmd+Shift+5 → Record Selected Portion → narrate while testing

Process a video:

VIDEO="/path/to/recording.mov"
WORKDIR="~/Gits/<project>/docs/qa-session-$(date +%Y-%m-%d)"
mkdir -p "$WORKDIR/frames"

# 1. Extract audio
ffmpeg -i "$VIDEO" -vn -acodec pcm_s16le -ar 16000 -ac 1 "$WORKDIR/audio.wav"

# 2. Transcribe
whisper-cli -m ~/.cache/whisper/ggml-small.bin -f "$WORKDIR/audio.wav" \
  --output-srt --output-txt -of "$WORKDIR/transcript" -l auto

# 3. Read transcript.srt → identify hotspot timestamps
# 4. Extract frames (hotspots + every 30s)
# 5. Read frames with Claude Vision
# 6. Compile findings doc

For the full step-by-step, load workflows/process.md.