benjaminshoemaker/browser-verification

Browser Verification Skill

Verify UI-related acceptance criteria using MCP browser tooling with evidence (snapshots, logs, or screenshots).

Inputs

For each criterion, use metadata from Verify::

• route (e.g., /login)
• selector (e.g., [data-testid="login-form"])
• expected state (visible, text, attribute, etc.)
• optional actions (click, type, hover)
• optional viewport

Workflow Overview

Copy this checklist and track progress:

Browser Verification Progress:
- [ ] Step 1: Load configuration
- [ ] Step 1.5: Resolve base URL (preview or localhost)
- [ ] Step 1.6: HTTP-first evaluation (skip browser if possible)
- [ ] Step 2: Select browser tool (with fallback chain)
- [ ] Step 3: Authenticate (if required)
- [ ] Step 4: Navigate and validate
- [ ] Step 5: Capture evidence
- [ ] Step 6: Handle errors and recover
- [ ] Step 7: Report results

Step 1: Load Configuration

Read .claude/verification-config.json for:

Dev Server:

• devServer.command
• devServer.url
• devServer.startupSeconds

Authentication:

• auth.strategy — none, env, or storage-state
• auth.loginRoute — URL path for login form
• auth.credentials.usernameVar — Env var name for username
• auth.credentials.passwordVar — Env var name for password
• auth.storageState — Path to save/load session state

Browser:

• browser.tool — auto, executeautomation, browsermcp, playwright, chromedevtools
• browser.headless — Run headless (default: true)
• browser.timeout — Default timeout in ms (default: 30000)
• browser.navigationTimeout — Navigation timeout in ms (default: 60000)

Deployment:

• deployment.enabled — Whether to use preview deployments
• deployment.useForBrowserVerification — Use preview URL for browser tests
• deployment.fallbackToLocal — Fall back to localhost if no preview
• deployment.waitForDeployment — Wait for deployment to be ready
• deployment.deploymentTimeout — Max seconds to wait

If config is missing or incomplete, ask the human to run /configure-verification.

Verify the config file was read successfully by confirming all required fields are present. If any field is null or missing, log which fields are absent before proceeding.

Step 1.5: Resolve Base URL

Determine the base URL for browser verification based on deployment configuration.

Decision Logic

IF deployment.enabled AND deployment.useForBrowserVerification:
  1. Invoke vercel-preview skill to get preview URL
  2. IF URL found:
       BASE_URL = preview URL
       Skip dev server startup (not needed)
       Log: "Using Vercel preview: {URL}"
  3. ELSE IF fallbackToLocal:
       BASE_URL = devServer.url
       Log: "WARNING: No preview URL found, falling back to localhost"
       Ensure dev server is running
  4. ELSE (no fallback):
       BLOCK with error: "No preview deployment and fallback disabled"
ELSE:
  BASE_URL = devServer.url (current behavior)
  Ensure dev server is running

Preview URL Resolution

When deployment is enabled, invoke the vercel-preview skill:

1. Get current git branch
2. Query Vercel for ready deployments matching branch
3. If waitForDeployment enabled and deployment building, wait up to timeout
4. Return URL or null
5. Verify the resolved URL is reachable with curl -sf {URL} -o /dev/null before proceeding. If unreachable, log a warning and fall back to localhost.

Output

BASE URL RESOLUTION
===================
Mode: Vercel Preview | Local Dev Server | Fallback to Local
URL: {resolved URL}
Branch: {git branch} (if preview)
Status: Ready | Building | Not Found (if preview)

Using BASE_URL

All subsequent steps use BASE_URL instead of hardcoded devServer.url:

• Step 1.6 (HTTP-First): Use BASE_URL for curl checks
• Step 4 (Navigate): Navigate to BASE_URL + route
• Step 7 (Output): Report which URL was used
• Manual fallback instructions: Use BASE_URL in all generated URLs

IMPORTANT: When browser verification falls back to manual, all URLs in the generated instructions MUST use BASE_URL. This ensures manual verification guides point to the correct environment (preview URL when deployment enabled).

Step 1.6: HTTP-First Evaluation

Before launching browser tools, evaluate if the criterion can be satisfied with HTTP. This optimization skips browser overhead for criteria that don't require DOM inspection.

Criteria Eligible for HTTP-First

| Criterion Pattern | HTTP Check | Skip Browser If... | |-------------------|------------|-------------------| | "Page loads at {url}" | curl -sf {url} | HTTP 200 returned | | "API returns {status}" | curl -o /dev/null -w "%{http_code}" {url} | Status matches | | "Endpoint responds with {data}" | curl -s {url} \| jq/grep | Data found | | "Service health check" | curl -sf {url}/health | Health endpoint OK | | "Redirect to {url}" | curl -sI {url} \| grep Location | Location header matches |

HTTP-First Execution

# Generic page accessibility check
curl -sf "{BASE_URL}{route}" -o /dev/null && echo "HTTP_PASS" || echo "HTTP_FAIL"

# Response content check
curl -s "{BASE_URL}{route}" | grep -q "{expected_text}" && echo "HTTP_PASS" || echo "HTTP_FAIL"

# Status code check
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" "{url}")
[ "$HTTP_CODE" = "{expected}" ] && echo "HTTP_PASS" || echo "HTTP_FAIL"

Decision Logic

If HTTP_PASS and criterion does NOT require:

• Specific DOM element verification (selector, data-testid, visible)
• Visual appearance check (screenshot, layout, style)
• User interaction simulation (click, type, hover)
• Console log inspection (console, errors, warnings)
• Network timing analysis (performance, timing)

Then:

• Mark criterion as PASS
• Skip browser verification entirely
• Note method as "HTTP-first" in evidence
• Log: "Verified via HTTP (browser tools not required)"

Otherwise:

• Continue to Step 2 (browser tool selection)
• HTTP result can inform browser verification (e.g., server is reachable)

HTTP-First Benefits

• Speed: curl is ~10x faster than browser launch
• Reliability: No browser MCP dependency issues
• Simplicity: Works even if no browser tools configured
• Resources: Lower memory and CPU usage

Step 2: Tool Selection (Automatic with Fallback)

Automatically detect and use the best available browser tool. Try tools in order until one responds:

1. ExecuteAutomation Playwright MCP (primary)
   - Package: @executeautomation/playwright-mcp-server
   - Test: Check for mcp__playwright__* or mcp__executeautomation__* tools
   - Why primary: Most stable, actively maintained (312+ commits), 143 device presets
   - Avoids @playwright/mcp@latest beta instability issues

2. Browser MCP (if extension installed)
   - Source: browsermcp.io browser extension
   - Test: Check for mcp__browsermcp__* tools
   - Advantage: Uses existing browser profile (stays logged in, avoids bot detection)
   - Requires: User has Browser MCP extension installed

3. Microsoft Playwright MCP (pinned version)
   - Package: @anthropic-ai/mcp-server-playwright (pinned, NOT @latest)
   - Test: Check for mcp__playwright__* tools (if not already found)
   - Why fallback: Official but @latest includes unstable betas
   - Use pinned version only

4. Chrome DevTools MCP (basic fallback)
   - Test: Call mcp__chrome-devtools__list_pages
   - Often pre-installed with Claude Code
   - Limited: Not designed for automation, less stable for complex workflows
   - Use for: Simple navigation, screenshots, basic interactions

5. Manual Verification (last resort) — SOFT BLOCK
   - If all tools fail, do NOT silently continue
   - Display warning and prompt user:
     ```
     ⚠️  NO BROWSER TOOLS AVAILABLE

     Attempted to detect browser MCP tools but none responded:
     - ExecuteAutomation Playwright: {status}
     - Browser MCP: {status}
     - Microsoft Playwright: {status}
     - Chrome DevTools: {status}

     Browser criteria cannot be verified automatically.

     Options:
     1. Continue anyway (criteria become manual verification)
     2. Stop and configure browser tools first

     To enable automated browser verification, add to ~/.mcp.json:
     {
       "mcpServers": {
         "executeautomation-playwright": {
           "command": "npx",
           "args": ["-y", "@executeautomation/playwright-mcp-server"]
         }
       }
     }
     ```
   - Use AskUserQuestion to let user choose:
     - "Continue with manual verification" → Mark criteria as BLOCKED, list for human
     - "Stop to configure tools" → Halt and provide detailed setup instructions

Log tool selection:

Browser Tool: ExecuteAutomation Playwright MCP (auto-detected)
Fallback chain: Browser MCP → Microsoft Playwright → Chrome DevTools → Manual (soft block)

Verify the selected tool responds by issuing a lightweight probe (e.g., browser_snapshot or list_pages). If the probe fails, move to the next tool in the chain immediately.

Step 3: Authentication

Skip if auth.strategy is none.

Strategy: storage-state

1. Check if auth.storageState file exists and is recent (< 24h)
2. If exists, load session state into browser
3. Navigate to a protected route to verify session validity
4. If session invalid, proceed to login flow

Strategy: env (or storage-state with invalid session)

1. Read credential env var names from config (NOT the values)
2. Navigate to auth.loginRoute
3. Identify login form fields (username/email, password)
4. Fill credentials using browser tool's form fill
   • Pass env var names to the tool, which reads values from environment
   • Agent NEVER sees plaintext credentials
5. Submit the form
6. Wait for successful login indicator (redirect, logged-in element)
7. Save session to auth.storageState for reuse
8. Verify the saved session file exists and is non-empty before proceeding

Auth Failure Handling

If login fails after 2 attempts:

• Mark criteria as BLOCKED (not FAIL)
• Report: "Authentication failed. Check credentials in .env.verification"
• Do NOT expose credential values in error messages

Step 4: Navigate and Validate

For each browser criterion:

1. Start from base URL (BASE_URL — preview or localhost per Step 1.5)
2. Navigate to the criterion's route
3. Wait for content to load:
   • Use wait-for-text or wait-for-selector
   • Respect browser.navigationTimeout
4. Perform optional actions in order (click, type, hover)
5. Validate expected state:
   • visibility
   • text content
   • attribute values
   • layout position
6. Capture console errors and network failures when relevant

Step 5: Evidence Capture

Capture evidence appropriate to the criterion type using the selected browser MCP tool (not Bash commands):

| Type | Evidence | |------|----------| | DOM | Accessibility snapshot, selector details | | VISUAL | Screenshot | | CONSOLE | Console log excerpt | | NETWORK | Request/response summary | | PERFORMANCE | Timing metrics (if supported) | | ACCESSIBILITY | ARIA/semantic checks and notes |

Save evidence under .claude/verification/ with stable names:

• browser-{task-id}-{criterion-id}.png
• browser-{task-id}-{criterion-id}.json

After saving each evidence file, verify it was written by checking that the file exists and is non-empty (e.g., ls -la .claude/verification/browser-{task-id}-{criterion-id}.*). If the file is missing or zero-length, retry the capture once before marking evidence as unavailable.

Step 6: Error Recovery

Detect failures:

• Tool returns "undefined" or "Error calling tool"
• Connection timeout
• Server not responding

Recovery procedure:

1. Log the error with context:
   "Tool {name} returned: {error}"

2. If transient error (timeout, connection):
   - Wait 2 seconds
   - Retry current operation (max 2 retries)

3. If persistent error or tool unavailable:
   - Attempt tool restart (kill and re-init MCP)
   - If restart succeeds, retry current criterion

4. If restart fails:
   - Try next tool in fallback chain
   - Log: "Switching from {current} to {fallback}"

5. If all tools exhausted:
   - Mark remaining criteria as BLOCKED
   - Report: "Browser tools unavailable after trying:
     - ExecuteAutomation Playwright: {error}
     - Browser MCP: {error or 'not installed'}
     - Microsoft Playwright: {error}
     - Chrome DevTools: {error}
     Manual verification required."

Session degradation detection:

Some tools work initially then fail after extended use. Monitor for:

• Increasing response times
• Intermittent "undefined" errors
• Loss of page state

If detected, proactively switch to fallback before complete failure.

Step 7: Output Format

Return a structured result for each criterion:

BROWSER VERIFICATION RESULT
---------------------------
Instruction ID: [ID]
Status: PASS | FAIL | BLOCKED
Type: DOM | VISUAL | CONSOLE | NETWORK | PERFORMANCE | ACCESSIBILITY
Target: Vercel Preview (https://...) | Local Dev Server (http://localhost:...)
URL: [full URL tested] | Viewport: [width]x[height]

Finding: [What was observed]
Expected: [What was expected]

Evidence:
- {path}

Suggested Fix: [If FAIL]

Failure Handling Summary

| Condition | Action | Status | |-----------|--------|--------| | Dev server unreachable | Report failing command | BLOCKED | | Selector missing | Capture screenshot | FAIL | | Auth fails | Check credentials | BLOCKED | | Tool unavailable | Try fallback chain | Continue or BLOCKED | | Tool returns undefined | Retry then switch | Continue or BLOCKED | | All tools fail | Require manual verification | BLOCKED |

Error Handling

| Situation | Action | |-----------|--------| | Config file .claude/verification-config.json missing or unparseable | Ask the user to run /configure-verification and halt verification | | Dev server unreachable after startup timeout | Report the failing command and its stderr output; mark criteria as BLOCKED | | Browser MCP tool returns "undefined" or connection error | Retry once after 2 seconds, then fall to the next tool in the fallback chain | | Authentication fails after 2 login attempts | Mark criteria as BLOCKED with message; do NOT expose credential values | | Evidence file write produces zero-byte file | Retry capture once; if still empty, note "evidence unavailable" in report |

Tool-Specific Notes

See TOOL_NOTES.md for detailed information on each browser MCP tool option:

• ExecuteAutomation Playwright MCP (recommended primary)
• Browser MCP (best for auth-heavy apps)
• Microsoft Playwright MCP (use pinned version)
• Chrome DevTools MCP (basic fallback)
• Browserbase + Stagehand (cloud option)

---

REMINDER: All URLs in manual fallback instructions MUST use BASE_URL variable, not hardcoded URLs.