Pixel-based computer use fails in ways operators can already name. The coordinate is a few pixels off. The model assumes the click landed. Every look costs another image. And in the setup Anthropic documents, the desktop being driven is a sandboxed Linux machine in a container, not the Windows estate on the floor.
There is an older map of that screen. Operating systems already expose roles, names, and actions so screen readers can operate apps without seeing pixels. RPA platforms have treated that map as an API for years. UiPath and Power Automate Desktop were driving native controls through the accessibility layer long before anyone called the pattern computer use.
Vercel Labs’ agent-desktop is a narrower bet, and it is worth separating from the pitch that usually rides along with it. It is an open-source command-line tool a coding agent can call. It reads that same accessibility tree and returns element references, so the agent can act without a vision model. The tree was already there. The addition is an agent-owned observe, act, and verify loop, with references that expire when the interface changes, and with snapshots trimmed so the model does not swallow the whole app.
The ceiling belongs with the pitch. If an app publishes no tree, the tool cannot see it. It does not fall back to pixels. Screenshot computer use, and the RPA jobs you already run on those screens, are not retired by a cleaner snapshot.
Name the control, or do not click
The project’s own walkthrough uses GNOME Calculator, which is small enough that the rules show. A skeleton snapshot, three levels deep, comes back with two references. Drilling into one region raises the count to 33. A click that offers only a bare element id is refused. A reference left over from an earlier snapshot comes back undelivered, and the error says a retry is safe.
The reference is qualified. It carries the snapshot id, so an id from a previous look cannot be replayed by accident. The docs use a form like @s8f3k2p9:e8 as the example. If the interface has shifted, or two targets match, resolution refuses to guess. Two similar buttons do not become a coin flip. The agent takes a fresh snapshot and names the element, or it does not click.
Before it acts, the tool waits up to 5, 000 milliseconds for a target that is visible, enabled, and stable. By default the action goes through the accessibility API. The pointer stays put, and focus is not stolen, so a person can keep working on the same machine. Headed mode allows real pointer input. Cursors drawn for several agents are decoration. A machine-wide lock queues the mouse.
A tool the agent calls
agent-desktop embeds no model and has no window of its own. A coding agent that can run a shell, or a plain script, calls it, reads one JSON snapshot, and chooses the next move. The loop stays with the caller. If you are buying AI agents rather than another closed assistant, that split is the part that matters. The tool is model-agnostic on purpose. It returns structure, and whatever model you already run does the reasoning.
It will not improvise on your behalf. Unclear targets are rejected. They are not approximated. That is useful discipline, and a reminder that a careless caller does not get a safer desktop just because the CLI is strict. The agent still has to look again, and a person still has to decide when looking again is no longer required.
A cost hypothesis, not a scoreboard
A full accessibility tree can read like a novel. Skeleton mode stops at three levels. Containers that were cut report how many children they still hold and hand back a drill reference. The next read opens that one region, the way a careful person opens a folder instead of printing the building directory.
The token figures attached to this idea are self-reported. They are not an independent benchmark. On dense Electron apps, Slack, VS Code, and Notion, skeleton traversal is said to use 78 to 96 percent fewer tokens than dumping the full tree. A separate look at Apple Maps is reported at 3, 811 tokens per look, falling to 273. A simulated 52-step task is reported at 198, 000 tokens, falling to 31, 000.
Those comparisons measure how much structure the model has to read. They do not measure whether the task finished correctly, and it is not clear they used the same method. The Electron figure is against a full-tree dump, not against screenshot computer use. Treat the set as a cost hypothesis, not a scoreboard.
Screenshots still set the terms elsewhere
According to Anthropic’s computer-use documentation, the reference design still looks at the screen. Claude works against a sandboxed Linux desktop: a virtual display in a container, with preinstalled apps, not the live Mac or Windows machine an employee is using. The model does not connect to that desktop itself. It names an action. Your application runs it and returns a result, often another screenshot.
The same docs name a failure that should humble any tree-based design. The model will sometimes proceed as if a click worked without checking. Anthropic’s remedy is another screenshot and a short evaluation after the step. Dropdowns and scrollbars are awkward for mouse movement. Keyboard shortcuts often behave better. Handing over login credentials raises prompt-injection risk, because the model is reading an interface other software can influence.
agent-desktop is aimed at the inverse observation: structure from the machine the person actually uses, not an image from a virtual Linux desktop. The check does not leave with the pixels. A command that succeeds means the action was handed to the accessibility API. That does not mean the interface changed. The invoice is not saved, the record is not updated, and the dialog is not closed because the call returned clean.
Vercel already uses accessibility snapshots in a separate browser tool, agent-browser. That tool still offers screenshots, and it recommends skipping unchanged images so you do not spend tokens on them. Its examples accept bare element refs. Do not assume the desktop tool shares those rules. As described, it rejects a bare ref, and it does not match pixels.
Electron sits in the messy middle. For web contents inside those apps, the desktop project describes opening a DevTools port on launch so a browser agent can take the page. A cold Chromium or Electron app can take 10 to 25 seconds before that web tree appears. If you do not budget for the wait, the agent may decide the app is empty.
Invisible apps, and a permission that is not a formality
Custom-drawn canvases, some older line-of-business tools, and interfaces that never implemented assistive technology publish little or no useful tree. They are invisible here. Teams that already run desktop automation have often sorted the estate into screens with a tree and screens that are just a picture. Start from that list. Do not discover it on a live quote.
The permission matches the reach. On macOS the process needs Accessibility access, and that grant is broad: the same class of access a screen reader needs, now held by an agent that can type. On Windows, apps at the terminal’s integrity level need no extra grant, but higher-integrity apps refuse input. On Linux you need a graphical session with the accessibility service available.
Snapshots and clipboard reads can contain customer records, credentials, and half-written mail. Project notes put screenshots in the same warning, even though the control path does not match pixels. Whatever the integration captures, keep it out of shared logs. There is no described audit log, sandbox, or admin control plane. For AI automation inside a real company, that gap sets the pilot boundary: one known machine, one known app, private logs. Not a fleet.
One workflow, then a week of counting
Chat can draft, summarize, and route. It cannot press Save in the desktop app a revenue team still lives in. Pixel computer use can press Save, and it pays for an image on every look. In Anthropic’s reference design it is also pointed at a virtual desktop rather than the estate you support.
An agent on top of the accessibility tree is a new caller. It is not a new category of software. One plausible workflow. This is not a case study. In an AI for sales setup, the agent reads a quoting application the operating system already describes, fills a discount field by reference, and looks again to confirm the total. The step that fails is predictable. A custom-drawn price grid often has no useful tree. That step stays with a person, with the RPA bot that already owns it, or with a headed pointer if you accept the misses.
The test that matches the claim is narrow. Pick one app you know exposes a usable tree. Run the agent on a machine whose Accessibility grant you understand. Log privately. Count finished tasks, stale references, and screens where the tree was missing. Token savings are a side number until those counts exist.
Questions worth asking before a pilot
Does this replace a vision model?
Only where the app publishes a usable accessibility tree. The design returns structure so the calling agent can act without screenshots. Interfaces with no tree are invisible, because this tool does not match pixels, and vision-based computer use still covers those screens.
Are the token savings proven?
No. The cuts are self-reported, mostly against full-tree dumps, and they are not tied to an independent measure of task success. Read them as a cost hypothesis.
Is agent-desktop itself an AI agent?
No. It embeds no model. Your coding agent, or a script, owns the loop. The CLI returns a snapshot and accepts a qualified reference. It does not decide the task is finished.
How is this different from the RPA tool we already run?
RPA already drives native apps through the same accessibility APIs, usually along a path someone recorded or designed. The difference here is a model choosing the next action from a fresh snapshot, with references that expire when the UI changes. That can handle more variation, and it can wander. It does not replace a stable bot on a screen that never changes, and it cannot see apps your RPA team has already marked as picture-only.
What permission are we actually granting?
On macOS, Accessibility access, which is broad. On Windows, same-integrity apps are reachable and higher-integrity apps refuse input. On Linux, a graphical session with the accessibility service running. There is no described enterprise audit or sandbox, and snapshots can contain private desktop data.
Does a successful action mean the job is done?
No. Delivery is not the same as a changed interface. The agent has to look again and check, the same discipline Anthropic recommends after clicks, only with a tree instead of another image.
How does this differ from Anthropic computer use?
Anthropic’s documented path is screenshot and coordinate based, inside a sandboxed Linux desktop, with your application executing the actions. agent-desktop is a model-free CLI aimed at the user’s native apps through OS accessibility APIs. They have not been benchmarked against each other.
What should we measure in week one?
Task completion on one known-good app, how often references go stale, how often the tree is missing, and whether private desktop content is leaking into logs. Token count is a secondary number.