adhikjoshi/macpilot-ui-inspector

MacPilot UI Inspector

Use MacPilot's accessibility (AX) commands to inspect, find, and interact with UI elements in any macOS application. This skill enables precise programmatic control of app interfaces.

When to Use

Use this skill when the user asks to:

• Find a specific button, text field, checkbox, or UI element in an app
• Read the value or state of a UI control
• Click a button or interact with a control by its label
• Inspect the full UI hierarchy/accessibility tree of an app
• Set values in text fields or toggle checkboxes programmatically
• Discover what keyboard shortcuts an app supports
• Identify elements at specific screen coordinates

Commands

List UI Elements

macpilot ui list --app "Safari" --json                    # All elements
macpilot ui list --app "Safari" --role AXButton --json    # Only buttons
macpilot ui list --app "Safari" --depth 5 --json          # Deeper scan
macpilot ui list --app "Safari" --hierarchy --json        # With hierarchy

Find Elements by Text

macpilot ui find "Save" --app "TextEdit" --json           # Find by label
macpilot ui find "Save" --role AXButton --json            # Filter by role
macpilot ui find "Save" --exact --json                    # Exact match only

Find Element with Coordinates

macpilot ui find-text "Submit" --app "Safari" --json
# Returns: position {x, y}, size {w, h} - use for clicking

Click UI Element by Label

macpilot ui click "Save" --app "TextEdit" --json
macpilot ui click "Cancel" --role AXButton --json

Get/Set Values

macpilot ui get-value "Search" --app "Safari" --json          # Read field value
macpilot ui set-value "Search" "query text" --app "Safari"    # Set field value
macpilot ui set-value "Dark Mode" "1" --role AXCheckBox       # Toggle checkbox

Set Focus

macpilot ui set-focus "Search" --app "Safari" --json    # Focus an element

Scroll Within Elements

macpilot ui scroll "content" down 5 --app "Safari"     # Scroll element

Inspect Element Attributes

macpilot ui attributes "Save" --app "TextEdit" --json   # All AX attributes

Elements at Coordinates

macpilot ui elements-at 500 300 --json                  # What's at x=500 y=300
macpilot ui elements-at 500 300 --radius 50 --json      # Search wider area

Accessibility Tree

macpilot ui tree --app "Finder" --json                  # Full AX tree
macpilot ui tree --app "Finder" --depth 3 --json        # Limit depth

Keyboard Shortcuts

macpilot ui shortcuts --app "Safari" --json             # All shortcuts
macpilot ui shortcuts --app "Safari" --menu File --json # Menu-specific

Wait for Elements

macpilot wait element "Download Complete" --app "Safari" --timeout 30 --json

Common AX Roles

| Role | Description | |------|-------------| | AXButton | Buttons (push, toggle) | | AXTextField | Text input fields | | AXTextArea | Multi-line text areas | | AXStaticText | Labels and display text | | AXCheckBox | Checkboxes and toggles | | AXRadioButton | Radio buttons | | AXPopUpButton | Dropdown menus | | AXComboBox | Combo boxes | | AXTable | Tables and lists | | AXRow | Table/list rows | | AXMenuItem | Menu items | | AXToolbar | Toolbars | | AXScrollArea | Scrollable regions | | AXWindow | Windows | | AXSheet | Modal sheets | | AXImage | Images | | AXLink | Hyperlinks | | AXGroup | Generic container |

Workflow Patterns

Pattern 1: Find and Click

# Always focus app first
macpilot app focus "Safari"
# Find the element to verify it exists
macpilot ui find "Downloads" --app "Safari" --role AXButton --json
# Click it
macpilot ui click "Downloads" --app "Safari"

Pattern 2: Read Form State

macpilot app focus "System Settings"
macpilot ui get-value "Computer Name" --app "System Settings" --json

Pattern 3: Fill a Form

macpilot app focus "MyApp"
macpilot ui set-value "Name" "John Doe" --app "MyApp"
macpilot ui set-value "Email" "[email protected]" --app "MyApp"
macpilot ui click "Submit" --app "MyApp"

Pattern 4: Explore Unknown UI

# Start with a broad scan
macpilot ui tree --app "SomeApp" --depth 2 --json
# Narrow down to specific elements
macpilot ui list --app "SomeApp" --role AXButton --json
# Inspect a specific element
macpilot ui attributes "Settings" --app "SomeApp" --json

Pattern 5: Click by Coordinates (Fallback)

# When label-based clicking fails, find coordinates first
macpilot ui find-text "Submit" --app "MyApp" --json
# Use returned x,y to click
macpilot click 450 320

Tips

• Always call macpilot app focus before any UI interaction
• Use --json output for reliable parsing of element properties
• When ui click fails, fall back to ui find-text + macpilot click x y
• Use ui tree --depth 2 first to understand app structure before deep scans
• AX permission must be granted to MacPilot.app in System Settings
• The --role filter significantly speeds up element searches in complex apps