markus41/plugins/claude-code-expert/skills/computer-use

Computer Use & GUI Automation

Computer use lets Claude interact with GUIs: click buttons, fill forms, take screenshots, and navigate native apps. This is powerful but expensive and slow — use it only when a more precise tool doesn't exist.

Tool Selection Priority

Before reaching for computer use, exhaust these options first:

| Task | Prefer This | Over Computer Use | |------|------------|-------------------| | API endpoint testing | Bash + curl | Clicking through UI | | Database inspection | MCP postgres/sqlite | Navigating admin UI | | File operations | Read/Write/Edit | Drag-and-drop UI | | Web scraping | Firecrawl MCP | Screenshot + parse | | Browser automation | Playwright MCP | Computer use click | | CI status | GitHub API / gh CLI | Browser navigation | | Log inspection | Bash + grep | Terminal screenshot |

Rule: If you can express the task as a shell command or API call, do that. Computer use is the fallback for GUI-only workflows.

---

When Computer Use Is the Right Choice

1. Native App Validation

Testing a desktop app that has no API or CLI interface.

# Example: Validate Electron app UI after a build
Take a screenshot of the app after launch.
Click the "New Project" button.
Verify the dialog opens with the correct fields.
Fill in project name: "Test Project 2026"
Click Create and verify the project appears in the list.

2. Visual Regression Checks

Detecting layout regressions that unit tests can't catch.

# Workflow:
1. Take baseline screenshot of the current UI state
2. Apply the change
3. Take comparison screenshot
4. Highlight pixel differences > 1%
5. Human reviews diff

3. GUI-Only Admin Tools

Admin panels, legacy enterprise software, and embedded UIs with no API.

# Example: Generate a report from a legacy admin panel
Navigate to: http://admin.internal/reports
Click: "Export" → "CSV" → "Last 30 days"
Wait for download
Move file to: /tmp/report-{date}.csv

4. Local Simulator Flows

Mobile simulator or desktop app testing that requires visual interaction.

# Example: iOS simulator validation
Launch: xcrun simctl launch booted com.example.MyApp
Take screenshot
Verify: "Welcome" text is visible in the header
Tap: "Get Started" button (coordinates or element description)
Verify: onboarding screen loads

---

Result Verification

Computer use output is inherently visual and unstructured. Always verify results with a structured check after GUI actions:

Verification Pattern

After each GUI action:
1. Take a screenshot
2. Verify the expected visual state (specific text, element position, color)
3. If verification fails: log "FAIL: {what was expected vs. what was seen}"
4. If unsure: take another screenshot from a wider viewport

At the end:
- List each action and its verification result
- Count: {N} actions taken, {M} verified OK, {K} failed

Confidence Levels

| Confidence | Verification | Action | |------------|-------------|--------| | HIGH | Text matches exactly / element found by ID | Proceed | | MEDIUM | Visual match but element found by position | Log and proceed | | LOW | Can't find element / ambiguous screenshot | Stop, report to human |

---

Safety Guardrails

Computer use can cause irreversible actions (delete files, send emails, submit forms). Apply these guardrails:

Never Without Confirmation

• Form submissions in production environments
• Delete or "Archive" actions
• Payment or billing interactions
• Sending emails or messages
• Anything involving real user data

Screenshot Audit Trail

Keep screenshots of:

• State before any action
• State after each major action
• Final state

Dry-Run First

For complex GUI flows, describe the steps and ask for confirmation before executing:

Before I click "Submit", here's what will happen:
- Form data: {summary}
- This action cannot be undone
- Proceeding? (yes/no)

---

Computer Use vs. Playwright MCP

For web UIs, Playwright MCP is almost always better than computer use:

| | Playwright MCP | Computer Use | |--|---------------|-------------| | Reliability | High (DOM-based) | Medium (pixel-based) | | Speed | Fast | Slow (screenshot per action) | | Testability | Scriptable, repeatable | Hard to reproduce exactly | | Cost | Low | High (vision model per screenshot) | | Works on | Web browsers | Any visual surface |

Use Playwright MCP for: Web app testing, scraping, form automation on websites.

Use Computer Use for: Native desktop apps, embedded UIs, legacy apps with no API.

---

Cost Awareness

Computer use is expensive:

• Each screenshot = vision model inference (high token cost)
• A 10-step GUI flow = 10+ vision inferences
• Compare: a 10-step shell script = near-zero cost

Estimate before using: If a GUI flow has N steps, expect N × (screenshot tokens + generation tokens). For flows > 20 steps, consider whether a shell/API approach exists.

---

Claude Desktop Requirement

Computer use requires the Claude Desktop app (not CLI or Web). The Desktop app has the screen capture and input simulation capabilities that CLI lacks.

CLI:     ❌ Computer use not available
Web:     ❌ Computer use not available
Desktop: ✅ Computer use available