rounak/phoneagent

PhoneAgent

Use this workflow to drive iOS or Android UI through PhoneAgent's JSON-RPC bridge.

All shell commands below assume you are in the repo root:

cd "$(git rev-parse --show-toplevel)"

Start the RPC bridge

1. Choose a platform bridge (both listen on 127.0.0.1:45678 by default).

# iOS (XCTest-hosted bridge)
./.agents/skills/phoneagent/scripts/start_rpc_bridge_local.sh

# Android (adb bridge; emulator or physical device)
./.agents/skills/phoneagent/scripts/start_android_rpc_bridge_local.sh

Notes:

• start_rpc_bridge_local.sh is interactive and will show a numbered list of iOS devices/simulators. Enter the number for the destination you want.
• start_rpc_bridge_local.sh starts a localhost-only forwarder.
• On Xcode "Connect via network", it uses the CoreDevice tunnel automatically (no extra deps).
• For USB fallback forwarding, install pymobiledevice3 into a local venv: python3 -m venv .venv && ./.venv/bin/python -m pip install -U pip && ./.venv/bin/python -m pip install pymobiledevice3
• start_android_rpc_bridge_local.sh uses adb; if multiple devices are connected it prompts for the serial.

2. Keep the bridge process running.
3. Wait for PHONEAGENT_RPC_READY ... in logs before sending RPC calls.
4. Confirm socket readiness before first RPC:

./.agents/skills/phoneagent/scripts/rpc.py get-tree >/dev/null && echo rpc-ready

Resolve host and port

1. Always use 127.0.0.1:45678 as the RPC endpoint (or rpc.py --port <port> if customized).

Notes:

• Both bridges are localhost-only.
• iOS physical-device flow uses a localhost forwarder.
• If you need to forward manually, first get a device UDID via xcrun devicectl list devices, then run: python3 ./.agents/skills/phoneagent/scripts/forward_rpc_localhost.py --udid <UDID> (binds 127.0.0.1:45678)

Send RPC calls

Use the helper CLI:

# iOS bundle identifier
./.agents/skills/phoneagent/scripts/rpc.py open-app com.apple.Preferences

# Android package name
./.agents/skills/phoneagent/scripts/rpc.py open-app com.android.settings
./.agents/skills/phoneagent/scripts/rpc.py get-tree | head

# Use coordinates copied from the tree (XCUI frame string).
./.agents/skills/phoneagent/scripts/rpc.py enter-text \
  --coordinate '{{33.0, 861.0}, {364.0, 38.0}}' \
  --text 'Display'

./.agents/skills/phoneagent/scripts/rpc.py tap-element \
  --coordinate '{{37.7, 969.7}, {199.7, 29.0}}'

Core operating loop

1. Call get_tree.
2. Identify the best target element in the tree (label/identifier) and copy its frame coordinate string.
3. Prefer coordinate-based actions (tap_element / enter_text).
4. Use the returned tree from the action response to verify the UI changed as expected.
5. Repeat until complete.
6. When the task is complete, always capture a screenshot for the user:
   • Prefer get_context and write result.screenshot_base64 to a PNG (or use ./.agents/skills/phoneagent/scripts/rpc.py get-screen-image, which writes PNG files to /tmp/phoneagent-artifacts).
   • Include the PNG path in your final message so the user can open it.

Use swipe to reveal off-screen content, then use the returned tree (or call get_tree if needed). Use one request at a time per server. Do not fire concurrent batches. Split long keyboard input into chunks; do not send giant enter_text payloads in one call.

RPC method reference

All RPC requests are newline-delimited JSON objects with this shape:

{"id":1,"method":"<method>","params":{...}}

All success responses look like:

{"id":1,"result":{...}}

get_tree

• Does: Returns the accessibility tree of the currently focused app.
• Params: none.
• Returns: {"tree": "<string>"}

Example:

{"id":1,"method":"get_tree","params":{}}

get_screen_image

• Does: Captures the current screen as a base64-encoded PNG plus image dimensions (when available).
• Params: none.
• Returns: {"screenshot_base64":"<base64>","metadata":{"width":<number>,"height":<number>}}

Example:

{"id":2,"method":"get_screen_image","params":{}}

get_context

• Does: Convenience method that returns both the current accessibility tree and the current screen image.
• Params: none.
• Returns: {"tree":"<string>","screenshot_base64":"<base64>","metadata":{"width":<number>,"height":<number>}}

Example:

{"id":3,"method":"get_context","params":{}}

open_app

• Does: Brings the specified app to the foreground (and makes it the focused app for subsequent calls).
• Params: bundle_identifier (string, required).
  • iOS: pass bundle identifier (example com.apple.Preferences).
  • Android: pass package name (example com.android.settings).
• Returns: {"bundle_identifier":"<string>", "tree":"<string>"} (Android also includes package_name).

Example:

{"id":4,"method":"open_app","params":{"bundle_identifier":"com.apple.Preferences"}}

tap

• Does: Taps an absolute point in the current app.
• Params: x (number, required), y (number, required). Coordinates are in absolute screen points as reported by the tree.
• Returns: {"tree":"<string>"}

Example:

{"id":5,"method":"tap","params":{"x":120,"y":300}}

tap_element

• Does: Taps the center of an element using its XCUI frame string from the accessibility tree.
• Params:
• coordinate (string, required). Must look like {{x, y}, {w, h}} (copied from the tree).
• count (integer, optional; default 1). Use 2 for double-tap.
• longPress (boolean, optional; default false). When true, performs a long-press gesture.
• Returns: {"coordinate":"<string>", "count":<number>, "longPress":<bool>, "tree":"<string>"}

Example:

{"id":6,"method":"tap_element","params":{"coordinate":"{{20.0, 165.0}, {390.0, 90.0}}","count":1,"longPress":false}}

enter_text

• Does: Taps the center of the target element (to focus it), waits briefly for the keyboard, then types the provided text followed by a newline (Return).
• Params:
• coordinate (string, required). Must look like {{x, y}, {w, h}} (copied from the tree).
• text (string, required).
• Returns: {"coordinate":"<string>", "tree":"<string>"}

Example:

{"id":7,"method":"enter_text","params":{"coordinate":"{{33.0, 861.0}, {364.0, 38.0}}","text":"hello"}}

scroll

• Does: Scrolls by dragging from a starting point by the provided deltas.
• Params: x (number, required), y (number, required), distanceX (number, required), distanceY (number, required).
• Returns: {"tree":"<string>"}

Example:

{"id":8,"method":"scroll","params":{"x":215,"y":760,"distanceX":0,"distanceY":-460}}

swipe

• Does: Swipes in a direction starting from a given point (implemented as a bounded drag gesture).
• Params: x (number, required), y (number, required), direction (string, required; one of up, down, left, right).
• Returns: {"tree":"<string>"}

Example:

{"id":9,"method":"swipe","params":{"x":215,"y":760,"direction":"up"}}

stop

• Does: Stops the RPC server test (ends the xcodebuild test session).
• Params: none.
• Returns: {}

Example:

{"id":10,"method":"stop","params":{}}

iOS app bundle IDs

• Settings: com.apple.Preferences
• Camera: com.apple.camera
• Photos: com.apple.mobileslideshow
• Messages: com.apple.MobileSMS
• Home Screen: com.apple.springboard

Android package names

• Settings: com.android.settings
• Camera (AOSP): com.android.camera2
• Photos (Google): com.google.android.apps.photos
• Messages (Google): com.google.android.apps.messaging
• Home Screen: launcher package varies by emulator/device

Recovery playbook

1. If RPC hangs after open_app, restart the test-hosted server and retry with a known-good bundle id.
2. If taps fail due stale UI, call get_tree again and recalculate target.
3. If iOS bridge becomes unresponsive, stop/restart xcodebuild test and resume from latest verified app state.
4. If Android bridge becomes unresponsive, restart adb (adb kill-server && adb start-server), relaunch the bridge, and retry.

End session

1. Send stop only when the task is complete.
2. If stop is not sent, terminate the xcodebuild session manually.