accomplish-ai/desktop-control

Desktop Control

Native desktop automation using MCP tools. Use these tools for interacting with native desktop applications.

CRITICAL: Every Action Requires User Approval

ALL desktop actions require per-action user approval. The user will see a permission dialog describing the exact action before it executes. If denied, the action returns an error.

CRITICAL: Sensitive App Blocklist

The following apps are automatically blocked and cannot be automated:

• Password managers: 1Password, Bitwarden, LastPass, KeePass, Dashlane
• System security: Keychain Access, Credential Manager, Windows Security
• Admin tools: Registry Editor

Attempting to target a blocklisted window returns a 403 error with details.

IMPORTANT: Use needs_planning: true

Desktop automation is inherently destructive. ALL tasks involving desktop.* tools MUST use needs_planning: true in the start_task call. Plan your steps before executing them.

Tools

desktop.click(x, y) — Click at pixel coordinates

• x, y: Required pixel coordinates
• Example: desktop.click(640, 300) → clicks at center of screen

desktop.doubleClick(x, y) — Double-click at coordinates

• x, y: Required pixel coordinates

desktop.rightClick(x, y) — Right-click at coordinates

• x, y: Required pixel coordinates

desktop.moveMouse(x, y) — Move cursor without clicking

• x, y: Required pixel coordinates

desktop.scroll(direction, amount?) — Scroll the view

• direction: up, down, left, right
• amount: Number of scroll units (default: 3)

desktop.type(text) — Type text at the current cursor focus

• text: The text string to type
• IMPORTANT: Make sure the target input is focused first (use desktop.click on it)

desktop.hotkey(keys[]) — Press a keyboard shortcut

• keys: Array of key names, e.g. ["LeftControl", "C"] for Ctrl+C
• Keys are pressed simultaneously and released
• Key names follow nut.js Key enum (e.g. LeftControl, LeftAlt, LeftShift, Return, Space, Tab, Escape, Delete, Backspace, A-Z, F1-F12)

desktop.pressKey(keys[]) — Press and hold key(s)

• keys: Array of key names to press down
• Use with desktop.releaseKey for complex interactions

desktop.releaseKey(keys[]) — Release held key(s)

• keys: Array of key names to release

desktop.screenshot() — Take a screenshot of the entire desktop

• Returns: base64-encoded image data with width/height

desktop.listWindows() — List all open application windows

• Returns: Array of { id, title, appName, bounds } objects
• Use this to discover available windows before targeting them

desktop.findWindow(title) — Find windows matching a title pattern

• title: Regex pattern to match against window titles (case-insensitive)
• Returns: Array of matching window objects

desktop.focusWindow(title) — Bring a window to the foreground

• title: Window title (or partial match) to focus
• The window must exist and not be on the blocklist

desktop.resizeWindow(title, width, height) — Resize a window

• title: Window title to target
• width, height: New dimensions in pixels

desktop.repositionWindow(title, x, y) — Move a window on screen

• title: Window title to target
• x, y: New position coordinates

Workflow

ALWAYS follow this pattern:

1. desktop.screenshot() — See what's currently on screen
2. desktop.listWindows() — Discover available windows
3. Plan your actions (what to click, where to type)
4. Execute actions ONE AT A TIME, getting approval for each
5. desktop.screenshot() — Verify the result after each step

Example: Open Notepad and Type

1. desktop.focusWindow("Notepad")       → Bring Notepad to front
2. desktop.click(400, 300)              → Click in the text area
3. desktop.type("Hello World!")         → Type the text
4. desktop.screenshot()                 → Verify the text appeared

Example: Take a Screenshot

1. desktop.screenshot()                 → Capture the current screen

Example: Send a Slack Message

1. desktop.listWindows()                → Find Slack window
2. desktop.focusWindow("Slack")         → Bring Slack to front
3. desktop.screenshot()                 → See current state
4. desktop.click(x, y)                  → Click message input
5. desktop.type("Hello team!")          → Type the message
6. desktop.hotkey(["Return"])           → Press Enter to send
7. desktop.screenshot()                 → Verify message sent

Error Recovery

| Error | What it means | What to do | | -------------------------------------------------- | ---------------------------------------- | ---------------------------------------------------------------------------- | | "Action denied by user" | User clicked Deny in the approval dialog | Respect the denial, explain what you were trying to do | | "Blocked: [app] is on the sensitive app blocklist" | Target window is a protected app | Do NOT try to work around this — inform the user | | "Failed to load @nut-tree/nut-js" | Native module not available | Ensure Accessibility permission is granted (macOS) or run as admin (Windows) | | "x and y coordinates are required" | Missing required parameters | Always provide coordinates for mouse actions |

Platform Considerations

• macOS: Requires Accessibility permission in System Preferences → Privacy & Security → Accessibility
• Windows: Some window management actions may require Administrator privileges
• Linux: Window management uses wmctrl (must be installed)
• Display scaling: Coordinates are in physical pixels — use desktop.screenshot() to determine actual positions
• Multi-monitor: Coordinates span across all displays