automateyournetwork/desktop-gui-inspect

Desktop GUI Inspect Skill

Purpose

browser-gui-inspect (spec 048) fills gaps in web-based controller dashboards NetClaw's API-based skills can't reach. But some legacy network/security tooling has no web GUI or API at all — only a native desktop application. Browser automation cannot reach those; this skill can. It drives OpenClaw's own ClawHub computer-use skill — a real virtual X11 desktop (Xvfb + XFCE) on this NetClaw host, controlled via xdotool-based mouse/keyboard/screenshot scripts — to read, navigate, and confirm state in a desktop-only application.

This skill deliberately contains no per-application knowledge. Every invocation is directed by the operator's own target_application/intent at request time — nothing about any specific legacy tool's UI layout is hardcoded here (Constitution VI — Multi-Vendor Neutrality), the same design principle browser-gui-inspect follows. See specs/050-computer-use-desktop/ for the full spec, research, and design behind this skill.

Golden Rule — Read/Confirm/Search Only, Never a Config-Change Mechanism

The underlying desktop actions (click, type, drag, key combinations, scrolling) are for reading, navigating, and confirming state on the virtual desktop — never for submitting, applying, or committing a configuration change to network infrastructure. If a request would require clicking something like "Apply," "Commit," "Save," or "Deploy" in a legacy tool's configuration workflow, stop and tell the operator that this must go through the relevant API-based skill instead, which implements the proper observe→baseline→modify→verify workflow and ServiceNow CR gating (Constitution I, II, III, VIII). Using this skill to bypass that workflow would create an unaudited, unverified side door around NetClaw's safety model — the same reasoning browser-gui-inspect documents for its own interactive tools (see that skill's own Golden Rule, and specs/048-chrome-devtools-browser-inspection/research.md R8).

Underlying Skill

This skill orchestrates OpenClaw's ClawHub computer-use skill directly — it is not an MCP server. Its actions are individual bash scripts, invoked against a virtual display, documented in ~/.openclaw/skills/computer-use/SKILL.md:

| Action | Script | Example | |--------|--------|---------| | Screenshot | screenshot.sh | ./scripts/screenshot.sh | | Click (left/right/middle/double/triple) | click.sh | ./scripts/click.sh 512 384 left | | Type text | type_text.sh | ./scripts/type_text.sh "hello" | | Key / key combo | key.sh | ./scripts/key.sh "ctrl+s" | | Scroll | scroll.sh | ./scripts/scroll.sh down 5 | | Drag | drag.sh | ./scripts/drag.sh 100 100 400 400 | | Cropped screenshot | zoom.sh | ./scripts/zoom.sh 0 0 512 384 | | Wait (then screenshot) | wait.sh | ./scripts/wait.sh 2 | | Cursor position | cursor_position.sh | ./scripts/cursor_position.sh |

All actions run against DISPLAY=:99 (the virtual desktop's display, per the computer-use skill's own default). Most action scripts automatically chain a confirmation screenshot onto their own output (e.g. click.sh's last line execs screenshot.sh) — a single call to click.sh/type_text.sh/key.sh/etc. returns both a status line and a base64 PNG, so Step 4 of Workflow 1 below is often already satisfied by the action call itself. This skill never installs or modifies the underlying computer-use skill's own scripts — it only calls them, per the "no forked upstream code" pattern already used for chrome-devtools-mcp.

Known install quirk: openclaw skills install writes these scripts as non-executable (confirmed live, spec 050 research.md R2) — NetClaw's installer runs chmod +x on them automatically. If a manual/out-of-band skill install ever produces "Permission denied" errors, run chmod +x ~/.openclaw/skills/computer-use/scripts/*.sh once.

Configuration

No environment variables — the computer-use skill needs no credentials (research.md R6). The virtual desktop and its live-viewing service (VNC on port 5900, noVNC on port 6080) are provisioned once, at install time, via ./scripts/install.sh --components computer-use (or the equivalent at the time you're reading this) — see mcp-servers/ is not applicable here; there is no vendored server, only the installer catalog entry in scripts/lib/catalog.sh / scripts/lib/install-steps.sh.

---

Workflow 1: Operate a Legacy Desktop-Only Tool (P2)

Use this when an operator needs information from, or a read/confirm interaction with, a desktop-only application that has no browser or API path.

Input:

• intent (required) — plain-language description of what to read or confirm (e.g., "open the legacy NMS client and tell me the current alarm count")
• target_application (optional) — the application name/identifier, if already known

Step 1: Confirm the virtual desktop is available

Take a screenshot first, always — this is the computer-use skill's own documented workflow pattern (Screenshot → Analyze → Act → Screenshot → Repeat):

./scripts/screenshot.sh

If this fails (virtual desktop not running), stop and return:

status: "virtual_desktop_unavailable"

Step 2: Locate or open the target application

Analyze the screenshot for the target application. If it's not already open and needs to be launched, do so via the desktop (e.g., a terminal command through xfce4-terminal, or an existing desktop icon/menu). If the application appears to need first-time interactive setup (a license dialog, an initial login) that cannot be completed programmatically, stop and return:

status: "manual_setup_required"

Do not guess or fabricate a result — ask the operator to complete setup via Watch Mode (Workflow 2) instead.

Step 3: Perform the read/confirm/search actions needed to satisfy intent

Using click, type, scroll, key, etc. as needed to navigate to the information described in intent — e.g., opening a menu, switching a tab, searching a list. Reiterate: these actions are for navigating to the information, never for submitting a change (Golden Rule above).

Step 4: Capture the result

./scripts/screenshot.sh

Read the relevant value(s) from the screenshot content, using zoom.sh on a specific region if the full screenshot is too coarse to read reliably.

Step 5: Return the result

{
  "result": "<the extracted value/text/confirmation answering intent>",
  "screenshot_path": "<path to the captured screenshot>",
  "status": "ok" | "manual_setup_required" | "virtual_desktop_unavailable" | "conflict"
}

Worked Example

"Open the legacy NMS client and tell me the current alarm count."

1. screenshot.sh — confirm the virtual desktop is up.
2. Locate the NMS client icon/window; if not running, launch it via xfce4-terminal or the desktop's own launcher.
3. wait.sh 3 — give the legacy app time to render (many older Java/Swing apps are slow to start).
4. screenshot.sh again — read the alarm count directly from the rendered UI, or zoom.sh on the specific panel if needed.
5. Report the alarm count back, with the screenshot as supporting evidence.

---

Workflow 2: Watch Mode via Live Viewer (P3)

Use this when the operator wants to watch NetClaw operate the virtual desktop live, or wants to take over manually (e.g. to complete a manual_setup_required step from Workflow 1) — the desktop analogue of browser-gui-inspect's Chrome DevTools Watch Mode.

Unlike the browser skill, there is no headless/headed choice to make here — Xvfb is inherently a virtual display with no physical monitor, so NetClaw always "operates headless" in that sense. Watching is purely a matter of whether a viewer is connected to the same virtual desktop while actions run; it never changes how the actions themselves execute.

Local viewing (NetClaw and the operator are on the same machine)

Point a VNC client, or a browser via noVNC, at the loopback-bound live-viewing service:

VNC:   localhost:5900
noVNC: http://localhost:6080

Both are confirmed loopback-only by the installer (scripts/lib/install-steps.sh's component_install_computer_use() verifies this with ss -tlnp after every install — see FR-004, research.md R5). If either is reachable from a non-loopback address, treat that as a real, unresolved security exposure — stop and fix it (re-run the installer's computer-use step, or manually reapply the -localhost / --listen 127.0.0.1:6080 patches documented in research.md R5) before continuing, rather than proceeding anyway.

Remote viewing (operator is on a different machine than NetClaw)

The live-viewing service being loopback-only is deliberate, not a bug to route around — never expose port 5900 or 6080 directly. Instead, tunnel:

ssh -L 6080:localhost:6080 <netclaw-host>

Then open http://localhost:6080 in a local browser. This is the same secure-remote-access pattern documented for chrome-devtools-mcp's Watch Mode in spec 048.

During a Watch Mode session

Continue running Workflow 1's actions normally — screenshot, click, type, etc. There is no special "watch flag" to pass to the action scripts; a connected viewer simply sees the same virtual display those scripts are already driving in real time. If the operator wants to take over and complete a step manually (e.g. an interactive login manual_setup_required stopped on), they can click and type directly into the viewer — the virtual desktop has no concept of whose input is "real," so operator and NetClaw actions on the same display are equivalent.

Error Handling

| Status | Meaning | What to tell the operator | |--------|---------|----------------------------| | ok | Requested information/interaction succeeded | Return the result and the screenshot | | manual_setup_required | The target application needs interactive first-time setup | Ask the operator to complete it via Watch Mode (Workflow 2), then retry | | virtual_desktop_unavailable | The computer-use component isn't installed, or its virtual desktop failed to start | Point at ./scripts/install.sh --components computer-use (or the equivalent at the time) | | conflict | Another task is already using the single shared virtual desktop | Retry once the other task completes — only one virtual display exists (data-model.md) |