renatocaliari/agent-browser

Browser Automation with agent_browser

This skill uses the native agent_browser tool (from pi-agent-browser-native), not bash. All browser operations use structured JSON arguments instead of shell commands.

The upstream CLI agent-browser (Rust) is still the engine. Templates in templates/ are shell scripts for CI/terminal use. For LLM-driven automation, always use the native tool.

Tool Shapes

The agent_browser tool accepts exactly one of these mutually exclusive modes per call:

| Mode | When to use | Example | |------|-------------|---------| | args | Direct CLI args — any upstream command | { "args": ["open", "https://example.com"] } | | semanticAction | Find shorthand — click/fill/select by locator | { "semanticAction": { "action": "click", "locator": "text", "value": "Submit" } } | | job | Constrained multi-step flow (schema-safe) | { "job": { "steps": [...] } } | | qa | Lightweight smoke test preset | { "qa": { "url": "...", "expectedText": "..." } } | | batch | Arbitrary multi-step flow (raw JSON stdin) | { "args": ["batch"], "stdin": "..." } |

All modes support sessionMode: "auto" (default, reuses implicit session) or "fresh" (rotates to new session).

Core Workflow

1. Navigate → agent_browser({args: ["open", <url>]})
2. Snapshot → agent_browser({args: ["snapshot", "-i"]})  (get @e1, @e2 refs)
3. Interact → agent_browser({args: ["click", "@e1"]}) or semanticAction
4. Re-snapshot → after navigation/DOM changes

// Open and inspect
{ "args": ["open", "https://example.com/form"] }
{ "args": ["snapshot", "-i"] }

// Interact
{ "args": ["fill", "@e1", "[email protected]"] }
{ "args": ["fill", "@e2", "password123"] }
{ "args": ["click", "@e3"] }
{ "args": ["wait", "--load", "networkidle"] }
{ "args": ["snapshot", "-i"] }

Common Mistakes

❌ Passing an array to stdin instead of a JSON string

The stdin parameter must always be a string containing JSON — never a raw array or object. This is the most common error when using batch.

Wrong (will fail with Validation failed):

{ "args": ["batch"], "stdin": [["open", "https://example.com"], ["snapshot", "-i"]] }

Right (string-escaped JSON):

{ "args": ["batch"], "stdin": "[[\"open\",\"https://example.com\"],[\"snapshot\",\"-i\"]]" }

Safest — use JSON.stringify() (when composing tool calls programmatically):

const steps = [["open", "https://example.com"], ["snapshot", "-i"]];
const toolCall = { args: ["batch"], stdin: JSON.stringify(steps) };

This also applies to eval --stdin: the stdin value must be a string.

❌ Forgetting to re-snapshot after navigation

Refs (@e1, @e2, etc.) point to elements on the current page. After any click, form submit, or scroll that changes the DOM, all refs are invalidated. See the Ref Lifecycle section for details.

Wrong:

{ "args": ["click", "@e3"] }
{ "args": ["click", "@e1"] }  // @e1 is stale — page changed

Right:

{ "args": ["click", "@e3"] }
{ "args": ["snapshot", "-i"] }
{ "args": ["click", "@e1"] }  // @e1 from fresh snapshot

❌ Using data-show for DaisyUI modals

DaisyUI modals use the modal-open class — not the show attribute. Use data-class with Datastar signals instead of data-show or JavaScript for modal visibility.

Wrong:

<dialog class="modal" data-show="$openModal">  <!-- will not work -->

Right:

<dialog class="modal" data-class="{'modal-open': $openModal}">

Chaining Multiple Commands

Use batch, job, or qa to run multiple steps in a single tool call. This is more efficient than separate calls.

batch (any upstream command)

{ "args": ["batch"], "stdin": "[[\"open\",\"https://example.com\"],[\"wait\",\"--load\",\"networkidle\"],[\"snapshot\",\"-i\"]]" }

job (constrained schema — safer)

Supported steps: open, click, fill, wait, assertText, assertUrl, waitForDownload, screenshot

{
  "job": {
    "steps": [
      { "action": "open", "url": "https://example.com/form" },
      { "action": "fill", "selector": "@e1", "text": "[email protected]" },
      { "action": "fill", "selector": "@e2", "text": "password123" },
      { "action": "click", "selector": "@e3" },
      { "action": "wait", "milliseconds": 1000 },
      { "action": "screenshot", "path": "result.png" }
    ]
  }
}

qa (smoke test)

{ "qa": { "url": "https://example.com", "expectedText": "Example Domain", "screenshotPath": "qa.png" } }

Essential Commands (via args)

Navigation

{ "args": ["open", "<url>"] }
{ "args": ["back"] }
{ "args": ["forward"] }
{ "args": ["reload"] }

Snapshot

{ "args": ["snapshot", "-i"] }              // Interactive elements only (recommended)
{ "args": ["snapshot"] }                     // Full accessibility tree
{ "args": ["snapshot", "-i", "--urls"] }     // Interactive + link hrefs
{ "args": ["snapshot", "-s", "#selector"] }  // Scoped to CSS selector

Interaction

{ "args": ["click", "@e1"] }
{ "args": ["fill", "@e2", "value"] }
{ "args": ["type", "@e2", "value"] }         // Type without clearing
{ "args": ["select", "@e3", "option-value"] }
{ "args": ["check", "@e4"] }
{ "args": ["press", "Enter"] }
{ "args": ["hover", "@e5"] }
{ "args": ["scroll", "down", "500"] }
{ "args": ["scrollintoview", "@e1"] }

Semantic locators (via semanticAction or find)

// Using semanticAction shorthand
{ "semanticAction": { "action": "click", "locator": "text", "value": "Sign In" } }
{ "semanticAction": { "action": "fill", "locator": "label", "value": "Email", "text": "[email protected]" } }
{ "semanticAction": { "action": "click", "locator": "role", "value": "button", "name": "Submit" } }
{ "semanticAction": { "action": "select", "locator": "label", "value": "Country", "text": "Brazil" } }

// Using find via args (equivalent)
{ "args": ["find", "text", "Sign In", "click"] }
{ "args": ["find", "role", "button", "click", "--name", "Close"] }

Wait

{ "args": ["wait", "--load", "networkidle"] }
{ "args": ["wait", "@e1"] }                                  // Wait for element
{ "args": ["wait", "--url", "**/dashboard"] }                 // Wait for URL
{ "args": ["wait", "--text", "Welcome"] }                     // Wait for text
{ "args": ["wait", "2000"] }                                  // Fixed ms
{ "args": ["wait", "--download", "./report.pdf"] }            // Wait for download

Capture

{ "args": ["screenshot"] }                                    // Temp path
{ "args": ["screenshot", "--full", "page.png"] }              // Full page
{ "args": ["screenshot", "--annotate", "annotated.png"] }     // Annotated with refs
{ "args": ["pdf", "page.pdf"] }

Data extraction

{ "args": ["get", "title"] }
{ "args": ["get", "url"] }
{ "args": ["get", "text", "@e1"] }
{ "args": ["eval", "--stdin"], "stdin": "document.title" }
{ "args": ["eval", "--stdin"], "stdin": "JSON.stringify(Array.from(document.querySelectorAll('a')).map(a => a.href))" }

Diff

{ "args": ["diff", "snapshot"] }                              // Compare vs last snapshot
{ "args": ["diff", "screenshot", "--baseline", "before.png"] } // Visual diff
{ "args": ["diff", "url", "<url1>", "<url2>"] }               // Compare pages

Keyboard

{ "args": ["keyboard", "type", "text with keystrokes"] }      // At current focus
{ "args": ["keyboard", "inserttext", "text without events"] } // Insert without key events

Downloads

{ "args": ["download", "@e1", "./file.pdf"] }                 // Click & download
{ "args": ["wait", "--download", "./output.zip"] }            // Wait for pending download

Batch download

{ "args": ["batch"], "stdin": "[[\"click\",\"@export\"],[\"wait\",\"--download\",\"report.csv\"]]" }

Tabs

{ "args": ["tab", "list"] }
{ "args": ["tab", "t2"] }                                     // Switch by id
{ "args": ["tab", "new", "https://example.com"] }
{ "args": ["tab", "close"] }

State & Auth

{ "args": ["auth", "save", "my-login", "--password-stdin"], "stdin": "<password>" }
{ "args": ["auth", "login", "my-login"] }
{ "args": ["state", "save", "./auth.json"] }
{ "args": ["state", "load", "./auth.json"], "sessionMode": "fresh" }

Session info

{ "args": ["session"] }
{ "args": ["session", "list"] }

React & Performance

// Requires --enable react-devtools at launch
{ "args": ["--enable", "react-devtools", "open", "https://example.com"], "sessionMode": "fresh" }
{ "args": ["react", "tree"] }
{ "args": ["react", "inspect", "<fiberId>"] }
{ "args": ["vitals", "https://example.com"] }
{ "args": ["pushstate", "/dashboard"] }                       // SPA navigation

Console & Errors

{ "args": ["console"] }
{ "args": ["errors"] }
{ "args": ["console", "--clear"] }
{ "args": ["errors", "--clear"] }

Record & Trace (for bugs)

{ "args": ["record", "start", "./issue-001-repro.webm"] }
{ "args": ["record", "stop"] }
{ "args": ["trace", "start"] }
{ "args": ["trace", "stop", "./trace.json"] }

Session Management

The extension manages sessions automatically:

• Default: sessionMode: "auto" — reuses the implicit session across calls
• Fresh: sessionMode: "fresh" — rotates to a new session (for profile/auth changes)
• Explicit: Add --session <name> in args for named sessions

// Normal browsing — auto session
{ "args": ["open", "https://example.com"] }

// Fresh session (e.g., after login)
{ "args": ["--profile", "Default", "open", "https://mail.google.com"], "sessionMode": "fresh" }

// Named session for parallel browsing
{ "args": ["--session", "site1", "open", "https://site-a.com"] }
{ "args": ["--session", "site2", "open", "https://site-b.com"] }

Cleanup is automatic (on pi quit or idle timeout). Explicit close:

{ "args": ["close"] }
{ "args": ["close", "--all"] }

Ref Lifecycle (Important)

Refs (@e1, @e2, etc.) are invalidated when the page changes. Always re-snapshot after:

• Clicking links/buttons that navigate
• Form submissions
• Dynamic content (dropdowns, modals, scroll)

{ "args": ["click", "@e5"] }           // Navigated
{ "args": ["snapshot", "-i"] }         // MUST re-snapshot
{ "args": ["click", "@e1"] }           // Use new refs

The tool provides automatic recovery guidance via details.nextActions when stale refs are detected.

Annotated Screenshots

Use --annotate for pages with unlabeled icon buttons, canvas/charts, or when you need spatial reasoning. Each label [N] maps to ref @eN.

{ "args": ["screenshot", "--annotate", "page.png"] }
// Output includes legend: [1] @e1 button "Submit", [2] @e2 link "Home"
{ "args": ["click", "@e1"] }

Configuration

Create agent-browser.json in the project root for persistent settings:

{
  "headed": true,
  "proxy": "http://localhost:8080",
  "profile": "./browser-data"
}

Use env vars for quick overrides: AGENT_BROWSER_COLOR_SCHEME=dark, AGENT_BROWSER_DEFAULT_TIMEOUT=45000.

Connect to Existing Chrome

{ "args": ["--auto-connect", "open", "https://example.com"], "sessionMode": "fresh" }
{ "args": ["--cdp", "9222", "snapshot"], "sessionMode": "fresh" }

Deep-Dive Documentation (CLI Templates)

| Reference | When to Use | |-----------|-------------| | references/commands.md | Full command reference with all options | | references/snapshot-refs.md | Ref lifecycle, invalidation rules | | references/session-management.md | Parallel sessions, scraping | | references/authentication.md | Login flows, OAuth, 2FA, state reuse | | references/video-recording.md | Recording workflows | | references/profiling.md | DevTools profiling | | references/proxy-support.md | Proxy config, geo-testing |

Ready-to-Use Shell Templates (CI / Terminal)

These are shell scripts for direct use outside pi:

| Template | Description | |----------|-------------| | templates/form-automation.sh | Form filling | | templates/authenticated-session.sh | Login workflow | | templates/capture-workflow.sh | Content extraction |

# Run directly in terminal (not through agent_browser)
./templates/form-automation.sh https://example.com/form

npm Documentation

For complete upstream CLI reference, see the agent-browser docs.

The pi-agent-browser-native extension docs (TOOL_CONTRACT.md, COMMAND_REFERENCE.md) are in the package at /opt/homebrew/lib/node_modules/pi-agent-browser-native/docs/.