agent-desktop

Computer useApache-2.0

agent-desktop is a command line tool from Vercel Labs that lets an agent use desktop applications by reading their accessibility tree instead of looking at screenshots. It prints JSON with a reference for each element that can be clicked or typed into, then the agent sends commands such as click against those references. It contains no model; the loop lives in the developer's agent. It runs on macOS, Windows and Linux and can also expose each command as an MCP tool over stdio.

Each command prints one JSON envelope, references belong to the snapshot that produced them, and ref actions run headless by default.

Pinout

  • MCP server: supported, verified
  • HTTP API: not checked yet
  • command line: supported, verified
  • Python SDK: not checked yet
  • TypeScript SDK: not checked yet
  • OpenAPI description: not checked yet
Where each pin comes from
InterfaceStatusSource we readRead on
MCP serverSupported, verifiedGitHub README, vercel-labs/agent-desktop, Interfaces
HTTP APINot checked yet——
Command lineSupported, verifiedGitHub README, vercel-labs/agent-desktop, Install
Python SDKNot checked yet——
TypeScript SDKNot checked yet——
OpenAPI descriptionNot checked yet——

Facts

Facts about agent-desktop
ApproachReads the accessibility tree and returns JSON with a ref for each actionable element, without screenshots or pixel matching. GitHub README, vercel-labs/agent-desktop, read .
ModelThe README states the loop lives in the agent and agent-desktop has no model inside. GitHub README, vercel-labs/agent-desktop, read .
Installnpm install -g agent-desktop; Homebrew is also listed for macOS and Linux. GitHub README, vercel-labs/agent-desktop, Install, read .
Operating systemsmacOS 13 or later, Windows 10 1809 or later, Ubuntu 24.04 or later. GitHub README, vercel-labs/agent-desktop, OS support, read .
MCPagent-desktop mcp exposes each command as an MCP tool over stdio. GitHub README, vercel-labs/agent-desktop, Interfaces, read .
C libraryEach release ships libagent_desktop_ffi for in-process use. GitHub README, vercel-labs/agent-desktop, Interfaces, read .
Privacy noteThe README warns snapshots, clipboard reads and screenshots can contain private content. GitHub README, vercel-labs/agent-desktop, read .
LicenseApache-2.0. Repository LICENSE file, read .
Last commit. GitHub API, latest commit on the default branch of vercel-labs/agent-desktop, read .

Limits

  • Per the README, the key-down, key-up, mouse-down and mouse-up commands stay disabled until a daemon can release keys or buttons left pressed.
  • References are valid only for the snapshot that produced them, so the UI must be re-read after changes.
  • The README states there is no official Scoop package.

Strengths

  • Both a command line and an MCP interface are documented.
  • Ref actions run headless by default.
  • Three operating systems are documented.

Alternatives to agent-desktop

All computer use tools for agents
  1. MCP server that lets AI agents operate Windows applications.

    Computer use
  2. Simular's open-source framework for computer-use agents, plus a hosted API.

    Computer use
  3. Python SDK that drives a Playwright browser from natural-language steps.

    Computer use
  4. Open-source drivers and sandboxes that let AI agents use computers.

    Computer use

Last check: . Method: public repository and docs read automatically, page and day written down. Next check due within 90 days. How we check

Is this your tool? Write to hello@agent-tools.directory from an address on its domain to correct a pin or a fact.