OmniParser

Computer use

OmniParser is a Microsoft research tool that parses user interface screenshots into structured, easy-to-read elements. The repository describes it as a screen parsing tool towards pure vision based GUI agents. The repository includes a Gradio demo, a Python notebook with examples, and OmniTool, which controls a Windows 11 virtual machine with a vision model of the developer's choice.

The license is shown inconsistently on the repository page, so the license field is left empty.

Pinout

  • MCP server: not checked yet
  • HTTP API: not checked yet
  • command line: not checked yet
  • Python SDK: supported, verified
  • TypeScript SDK: not checked yet
  • OpenAPI description: not checked yet
Where each pin comes from
InterfaceStatusSource we readRead on
MCP serverNot checked yet——
HTTP APINot checked yet——
Command lineNot checked yet——
Python SDKSupported, verifiedOmniParser repository README, demo.ipynb examples
TypeScript SDKNot checked yet——
OpenAPI descriptionNot checked yet——

Facts

Facts about OmniParser
DescriptionA simple screen parsing tool towards pure vision based GUI agent. OmniParser repository README, read .
License shownThe repository sidebar lists CC-BY-4.0 while the README badge links to MIT; the page does not settle it. OmniParser repository, sidebar and README, read .
Installconda environment with Python 3.12, pip install -r requirements.txt, model weights fetched from Hugging Face. OmniParser repository README, install, read .
Demopython gradio_demo.py starts a Gradio demo. OmniParser repository README, usage, read .
Python usageExamples are provided in demo.ipynb. OmniParser repository README, usage, read .
OmniToolControls a Windows 11 virtual machine with OmniParser and a vision model of the user's choice. OmniParser repository README, OmniTool, read .
OmniTool model providers namedOpenAI, DeepSeek, Qwen and Anthropic Computer Use. OmniParser repository README, OmniTool, read .
Last commit. GitHub API, latest commit on the default branch of microsoft/OmniParser, read .
FrameworksOpenAI (OmniParser repository README, OmniTool, read 2026-11-16); DeepSeek (OmniParser repository README, OmniTool, read 2026-11-16); Qwen (OmniParser repository README, OmniTool, read 2026-11-16); Anthropic Computer Use (OmniParser repository README, OmniTool, read 2026-11-16).

Limits

  • One weight is fetched from an open Hugging Face pull request until it is merged.
  • The detector licenses differ between releases, as the README notes.

Strengths

  • Model weights are hosted on Hugging Face.
  • A Gradio demo and a notebook of examples are included.
  • OmniTool pairs the parser with several named model providers.

Alternatives to OmniParser

All computer use tools for agents
  1. CLI that lets an agent operate desktop apps through accessibility trees.

    Computer use
  2. MCP server that lets AI agents operate Windows applications.

    Computer use
  3. Simular's open-source framework for computer-use agents, plus a hosted API.

    Computer use
  4. Python SDK that drives a Playwright browser from natural-language steps.

    Computer use

Last check: . Method: public repository and docs read automatically, page and day written down. Next check due within 90 days. How we check

Is this your tool? Write to hello@agent-tools.directory from an address on its domain to correct a pin or a fact.