agent-desktop is a free, open-source tool from Vercel Labs that lets AI agents see and use desktop apps on macOS, Windows and Linux without taking screenshots. Instead of looking at pixels, it reads each app’s accessibility tree and gives the agent a clean list of buttons and fields it can click or type into. With about 19,700 GitHub stars, it was one of the most viral AI projects of early November 2026.
What problem does agent-desktop solve?
Most computer-use agents work like a person squinting at a screen. They take a screenshot, ask an AI model where the button is, then click at those coordinates. That is slow, expensive and fragile: a moved window or a dark mode switch can break it. agent-desktop skips the picture entirely.
How does it work, in plain English?
Every operating system keeps a hidden map of what is on screen so that screen readers can describe it to blind users. That map is called the accessibility tree, and it lists every window, button, menu and text box with its name. agent-desktop reads that map and returns it as structured data, giving each element a short ID called a ref.
- The agent asks for a snapshot of an app, for example Finder or Slack
- It picks the ref for the element it wants, such as a Save button
- It tells agent-desktop to click or type on that ref
- It takes a new snapshot to check the result, and repeats
The AI “brain” stays in your own agent. agent-desktop has no model inside, it is just the hands and eyes. It uses Apple’s AX API on macOS, UI Automation on Windows and AT-SPI on Linux, per its GitHub README.
What can it do?
It ships around 56 working commands: clicking, typing, scrolling, dragging, opening and closing apps, moving windows, reading notifications and using the clipboard. By default it works “headless,” acting through the accessibility system without moving your mouse or stealing focus. It installs with npm or Homebrew and also runs as an MCP server, so assistants like Claude Code or Cursor can call it as a set of tools.
How is it different from screenshot-based agents?
Speed and reliability. Text is far cheaper for a model to read than an image, and a ref points to an exact element, so there is no guessing at pixel positions. The trade-off is coverage: apps with poor accessibility support, some games, and custom-drawn interfaces expose little or nothing in the tree, so a screenshot approach can still see things agent-desktop cannot. Our computer-use agent comparison shows how the main approaches stack up.
Is it safe to let an agent control my desktop?
Treat it with care. The project itself warns that snapshots, clipboard reads and screenshots can contain private content, so keep them out of public logs. An agent that can click anything can also click the wrong thing, or be tricked by text hidden in an email or web page. Read our guide to computer-use security risks, grant accessibility permission only to tools you trust, and keep a person in the loop for anything that sends, buys or deletes.
Who should try it?
Developers building agents that must work in desktop software with no API, such as older business apps, design tools or chat clients. For everyday users it is still a building block, not a finished assistant: you need an agent to drive it.




