suman73/desktop-agent-builder

Desktop Agent Builder

A comprehensive skill for building production-ready desktop AI agent applications that combine a Python backend with a React frontend, packaged as a standalone executable.

Architecture Overview

The pattern is a local client-server architecture bundled into a single distributable package:

project/
├── backend/                 # Python FastAPI server + AI agent
│   ├── main.py              # FastAPI app, API routes, SSE streaming
│   ├── run.py               # Entry point (dev + frozen exe)
│   ├── build_exe.py         # Build script (uv + PyInstaller)
│   ├── requirements.txt     # Python dependencies
│   └── app/
│       ├── agent_factory.py # Agent creation, tools, system prompt
│       ├── sessions.py      # Session management, credential validation
│       ├── models.py        # Pydantic request/response models
│       ├── config.py        # App configuration dataclass
│       ├── profiles.py      # AWS profile discovery
│       └── ...              # Additional modules as needed
├── frontend/                # React + TypeScript + Vite SPA
│   ├── src/
│   │   ├── App.tsx          # Root component, routing, state
│   │   ├── api.ts           # API client + SSE streaming helper
│   │   ├── types.ts         # Shared TypeScript types
│   │   └── components/      # UI components
│   └── vite.config.ts
└── infra/                   # Optional: CDK/CloudFormation for cloud resources

How It Works

1. User launches the .exe — it starts a local FastAPI server on localhost:8000
2. Server serves the React SPA as static files from the bundled frontend_dist/
3. Browser opens automatically to http://localhost:8000
4. User interacts with the chat UI, which calls backend API endpoints
5. Backend runs the AI agent, streams responses via Server-Sent Events (SSE)
6. Only prerequisite on the user's machine: AWS CLI (or whatever external tool the agent needs)

Key Design Decisions

Read references/architecture.md for detailed rationale on each decision. The highlights:

• Local server, not Electron: Avoids Chromium bloat. The user's default browser is the UI. FastAPI serves both the API and the SPA static files.
• SSE over WebSockets: Simpler, works through proxies, natural fit for streaming LLM responses. The agent runs synchronously in a thread pool; events are pushed to a queue that the SSE generator reads from.
• Session-based, not persistent: Sessions live in memory. No database needed. The exe is stateless between runs.
• Lazy agent creation: Credentials are validated on connect, but the agent (which is expensive to initialize) is only created on the first chat message. This keeps the connect flow fast.
• uv for builds: uv resolves and installs Python packages 10-50x faster than pip. The build script creates a clean venv with uv venv, installs deps with uv pip install, then runs PyInstaller from that venv.
• onedir not onefile: PyInstaller --onedir produces a folder with the exe + dependencies. Startup is instant. --onefile extracts to a temp dir on every launch — slow and triggers antivirus.

Building a New Desktop Agent App

Step 1: Scaffold the Backend

Create the FastAPI server with these core components:

1. run.py — Entry point that works both in dev (uvicorn with reload) and frozen (PyInstaller) mode
2. main.py — FastAPI app with CORS, API routes, SSE streaming, and static file serving
3. app/agent_factory.py — Agent creation with configurable tools and system prompt
4. app/sessions.py — Session lifecycle (create, get, destroy)
5. app/models.py — Pydantic models for all API request/response schemas

See references/backend-patterns.md for code patterns for each component.

Step 2: Scaffold the Frontend

Create the React SPA with Vite:

1. api.ts — API client with SSE streaming helper that parses data: lines
2. App.tsx — Root component managing session state and view routing
3. components/ChatWindow.tsx — Chat UI with message bubbles, tool call display, streaming indicator
4. components/ProfileSelector.tsx — Login/connect page (pre-auth)
5. components/Sidebar.tsx — Navigation sidebar (post-auth)

See references/frontend-patterns.md for the SSE streaming pattern and component structure.

Step 3: Wire SSE Streaming

The critical integration point. The pattern:

Frontend (EventSource) ←SSE→ Backend (StreamingResponse) ←Queue→ Agent Thread

1. Agent runs in a ThreadPoolExecutor thread
2. A ToolStreamingHook pushes events (tool_call, tool_result, content) to a Queue
3. The SSE endpoint reads from the queue with timeout, yields data: {json}\n\n lines
4. Frontend reads the SSE stream, updates UI in real-time
5. Final data: [DONE]\n\n signals stream end

Step 4: Build the Executable

The build script (build_exe.py) does:

1. uv venv .build_venv — Create clean venv
2. uv pip install --python .build_venv/Scripts/python.exe <packages> — Install deps fast
3. npm run build — Build frontend
4. Copy frontend/dist → backend/frontend_dist
5. Run PyInstaller with --onedir, --add-data for frontend + app modules, --hidden-import and --collect-submodules for dynamic imports, --exclude-module for unused heavy packages

See references/pyinstaller-guide.md for the full list of hidden imports and exclusions needed for common packages (FastAPI, uvicorn, boto3, strands, etc.).

Step 5: Test the Executable

1. Run the exe from the dist/ folder
2. Verify browser opens to localhost:8000
3. Verify the SPA loads (static files served correctly)
4. Verify API endpoints work (create session, send message, stream response)
5. Verify the agent can use its tools (AWS calls, etc.)

Common issues and fixes are in references/troubleshooting.md.

Adding Features

Custom Tools for the Agent

Define tools using the @tool decorator pattern:

from strands import tool

@tool
def my_custom_tool(param: str) -> str:
    """Description of what this tool does."""
    # Implementation
    return result

Add to the agent's tool list in agent_factory.py.

System Check / Prerequisites Page

Add a GET /api/system-check endpoint that verifies the user's machine has required software (AWS CLI, etc.). Display results on the login page so first-time users know what to install.

Report Generation

For data-heavy features, generate HTML reports server-side and serve them via a download endpoint with Content-Disposition: attachment header. Avoid blob URLs — they're blocked by some corporate browser policies.

Configuration from Cloud (SSM Parameter Store)

Fetch config from AWS SSM Parameter Store in a single batch get_parameters call. This lets you update config (model IDs, prompts, feature flags) without rebuilding the exe.

What NOT to Do

• Don't use --onefile for PyInstaller — it's slow to start and triggers antivirus
• Don't open the browser with webbrowser.open() for SSO auth — corporate environments block it. Show the URL in the UI instead.
• Don't use WebSockets for LLM streaming — SSE is simpler and sufficient
• Don't bundle a database — use in-memory sessions for desktop apps
• Don't hardcode credentials or config — use profiles and cloud config
• Don't use npm run dev in the exe — build the frontend and serve static files