agent-desktop
Computer useApache-2.0
agent-desktop is a command line tool from Vercel Labs that lets an agent use desktop applications by reading their accessibility tree instead of looking at screenshots. It prints JSON with a reference for each element that can be clicked or typed into, then the agent sends commands such as click against those references. It contains no model; the loop lives in the developer's agent. It runs on macOS, Windows and Linux and can also expose each command as an MCP tool over stdio.
Each command prints one JSON envelope, references belong to the snapshot that produced them, and ref actions run headless by default.

Pinout
- MCP server: supported, verified
- HTTP API: not checked yet
- command line: supported, verified
- Python SDK: not checked yet
- TypeScript SDK: not checked yet
- OpenAPI description: not checked yet
| Interface | Status | Source we read | Read on |
|---|---|---|---|
| MCP server | Supported, verified | GitHub README, vercel-labs/agent-desktop, Interfaces | |
| HTTP API | Not checked yet | — | — |
| Command line | Supported, verified | GitHub README, vercel-labs/agent-desktop, Install | |
| Python SDK | Not checked yet | — | — |
| TypeScript SDK | Not checked yet | — | — |
| OpenAPI description | Not checked yet | — | — |
Facts
| Approach | Reads the accessibility tree and returns JSON with a ref for each actionable element, without screenshots or pixel matching. GitHub README, vercel-labs/agent-desktop, read . |
|---|---|
| Model | The README states the loop lives in the agent and agent-desktop has no model inside. GitHub README, vercel-labs/agent-desktop, read . |
| Install | npm install -g agent-desktop; Homebrew is also listed for macOS and Linux. GitHub README, vercel-labs/agent-desktop, Install, read . |
| Operating systems | macOS 13 or later, Windows 10 1809 or later, Ubuntu 24.04 or later. GitHub README, vercel-labs/agent-desktop, OS support, read . |
| MCP | agent-desktop mcp exposes each command as an MCP tool over stdio. GitHub README, vercel-labs/agent-desktop, Interfaces, read . |
| C library | Each release ships libagent_desktop_ffi for in-process use. GitHub README, vercel-labs/agent-desktop, Interfaces, read . |
| Privacy note | The README warns snapshots, clipboard reads and screenshots can contain private content. GitHub README, vercel-labs/agent-desktop, read . |
| License | Apache-2.0. Repository LICENSE file, read . |
| Last commit | . GitHub API, latest commit on the default branch of vercel-labs/agent-desktop, read . |
Limits
- Per the README, the key-down, key-up, mouse-down and mouse-up commands stay disabled until a daemon can release keys or buttons left pressed.
- References are valid only for the snapshot that produced them, so the UI must be re-read after changes.
- The README states there is no official Scoop package.
Strengths
- Both a command line and an MCP interface are documented.
- Ref actions run headless by default.
- Three operating systems are documented.
Alternatives to agent-desktop
All computer use tools for agents
MCP server that lets AI agents operate Windows applications.

Simular's open-source framework for computer-use agents, plus a hosted API.

Python SDK that drives a Playwright browser from natural-language steps.

Open-source drivers and sandboxes that let AI agents use computers.
Last check: . Method: public repository and docs read automatically, page and day written down. Next check due within 90 days. How we check
Is this your tool? Write to hello@agent-tools.directory from an address on its domain to correct a pin or a fact.