pinchtab/pinchtab-opt

PinchTab Optimization Loop

Run blind subagents against 108 browser automation steps (47 groups) to measure how well an AI agent can drive PinchTab without hand-held selectors.

Path Resolution

All paths below are relative to the project root (git root). Resolve it first:

PROJECT_ROOT=$(git rev-parse --show-toplevel)
TOOLS_DIR="$PROJECT_ROOT/tests/tools"

The subagents must run with $TOOLS_DIR as their working directory because ./scripts/pt and ./scripts/runner live there.

Prerequisites

Stop any native PinchTab server that might occupy port 9867, then ensure Docker services are running:

# Kill native PinchTab server if running — it binds the same port as Docker services.
pkill -f 'pinchtab server' 2>/dev/null
pkill -f 'pinchtab.*serve' 2>/dev/null
lsof -ti:9867 2>/dev/null | xargs kill 2>/dev/null
sleep 1

Verify Docker health:

$TOOLS_DIR/scripts/pt health

If unhealthy, start the services:

docker compose -f "$TOOLS_DIR/docker-compose.yml" up -d --build

Wait a few seconds and re-check health.

Execution

0. Create per-agent report files

Before spawning agents, create isolated report files so concurrent writes don't corrupt a shared file:

RESULTS_DIR="$TOOLS_DIR/../benchmark/results"
TIMESTAMP=$(date -u +%Y%m%d_%H%M%S)
mkdir -p "$RESULTS_DIR"

for agent in A B C; do
  cat > "$RESULTS_DIR/agent${agent}_${TIMESTAMP}.json" <<SEED
{
  "benchmark": {"type": "pinchtab", "timestamp": "${TIMESTAMP}", "agent": "${agent}"},
  "totals": {"steps_answered": 0},
  "steps": []
}
SEED
done

Save the three file paths — you'll pass one to each subagent.

1. Spawn 3 parallel subagents

Use the Agent tool with run_in_background: true. Split the 45 groups into three batches:

• Batch A: groups 0-14 (45 steps)
• Batch B: groups 15-29 (30 steps)
• Batch C: groups 30-46 (33 steps)

Each subagent gets the same prompt template — only the group range and {REPORT_FILE} change. Replace {START}, {END}, {START_PAD}, {END_PAD}, {PROJECT_ROOT}, and {REPORT_FILE} with actual values:

You are running PinchTab optimization tasks. Your job is to execute groups {START} through {END}.

CRITICAL: Your working directory MUST be {PROJECT_ROOT}/tests/tools for all commands. Prefix every shell command with `cd {PROJECT_ROOT}/tests/tools && `.

Your report file is: {REPORT_FILE}
Use `--report-file {REPORT_FILE}` on every `./scripts/runner step-end` call.

Start by reading these files to understand your tools and tasks:
1. Read `{PROJECT_ROOT}/tests/optimization/subagent-context.md` — environment, wrapper, and recording format.
2. Read `{PROJECT_ROOT}/skills/pinchtab/SKILL.md` — full PinchTab command reference.
3. Read each group file from `{PROJECT_ROOT}/tests/optimization/group-{START_PAD}.md` through `{PROJECT_ROOT}/tests/optimization/group-{END_PAD}.md`.

DO NOT read `{PROJECT_ROOT}/tests/tools/scripts/baseline.sh` or any file under `{PROJECT_ROOT}/tests/benchmark/`.

After reading the above files, execute each step in each group sequentially:
- Always cd to {PROJECT_ROOT}/tests/tools before running commands.
- Use `./scripts/pt` as the wrapper for all PinchTab commands.
- After each step, record the result with `./scripts/runner step-end --report-file {REPORT_FILE} <group> <step> answer "<observation>" pass "notes"` (or fail if it didn't work).
- Use your judgment to figure out the right PinchTab commands from the skill doc. The group files describe WHAT to do, not HOW.

Work through every step in groups {START}-{END}. Do not skip any.

2. Monitor progress

While agents run, periodically count step-end recordings in each agent's output file:

grep -c "step-end" <output_file>

Expected totals: Batch A ~45, Batch B ~30, Batch C ~33 = 108 total.

3. Collect and summarize

Once all 3 agents complete, run these steps in order. The Agent tool returns each subagent's output file path — save all three as TRANSCRIPT_A, TRANSCRIPT_B, TRANSCRIPT_C.

SKILL_DIR=~/.claude/skills/pinchtab-opt
MERGED="$RESULTS_DIR/merged_${TIMESTAMP}.json"

# Merge the three agent reports into one JSON (strip non-JSON header lines)
cd "$TOOLS_DIR" && \
  ./scripts/runner opt merge-reports \
    "$RESULTS_DIR/agentA_${TIMESTAMP}.json" \
    "$RESULTS_DIR/agentB_${TIMESTAMP}.json" \
    "$RESULTS_DIR/agentC_${TIMESTAMP}.json" \
  2>/dev/null | grep -v '^Loaded\|^Merged' > "$MERGED"

# Inject token usage from the subagent JSONL transcripts
./scripts/runner opt inject-usage \
  -r "$MERGED" \
  "$TRANSCRIPT_A" "$TRANSCRIPT_B" "$TRANSCRIPT_C"

# Print the final comparison table — present this output to the user as-is
./scripts/runner opt summarize \
  -r "$MERGED" \
  -b "$SKILL_DIR/baseline-ref.json" \
  "$TRANSCRIPT_A" "$TRANSCRIPT_B" "$TRANSCRIPT_C"

The --baseline / -b flag loads stored reference timing and ops from baseline-ref.json so the Baseline column is fully populated. The transcripts enable the Browser ops and Ops/step rows.

Reference Numbers

• Baseline: 108/108 steps, 272 ops, 49s total, 0.5s/step (stored in baseline-ref.json)
• Expected agent range: 250-400 browser ops, 2.5-4 ops/step
• Group count: 47 groups, 108 total steps

File Locations (relative to project root)

| Path | Purpose | |------|---------| | tests/optimization/subagent-context.md | Subagent instructions (env, wrapper, recording) | | tests/optimization/index.md | Group listing | | tests/optimization/group-00.md .. group-46.md | Task descriptions | | skills/pinchtab/SKILL.md | PinchTab command reference (read by subagent) | | tests/tools/scripts/pt | PinchTab wrapper (CWD must be tests/tools) | | tests/tools/scripts/runner | Step recorder (CWD must be tests/tools) | | tests/tools/scripts/baseline.sh | Baseline (subagent must NOT read this) | | ~/.claude/skills/pinchtab-opt/baseline-ref.json | Stored baseline timing/ops reference for table |