In partnership with
Most AI automation runs on APIs, which works great until the app you want to automate doesn't have one (looking at you, every legacy tool your company still runs on).
Vercel Labs open-sourced agent-desktop, which lets AI agents use desktop apps through your operating system's accessibility APIs. It can look at windows, find buttons and text fields, click, type, and read what changed on screen.
And honestly, this one surprised me.
Claude's computer use and OpenAI's computer-using agents both work off screenshots, while agent-desktop reads the app the way a screen reader does, which is way more reliable than guessing pixels.
So today, we're making an AI open TextEdit on a Mac, write a document and save it, all by clicking around the way you would.
But first, a quick pause for two tools that don’t make you guess either.
Presented by TLDR
Most AI newsletters get skimmed, then archived. TLDR AI is the one worth opening. It's a free daily email curated by engineers from OpenAI, Adobe, and Google DeepMind. They read everything, then send the model releases, research, and tools that actually matter. It’s a 5-minute read, delivered every weekday morning.
Presented by TigerData
Analytics on Live Data Without Leaving Postgres
When analytics on Postgres slows down, most teams add a second database. Then they manage pipelines, sync lag, and drift forever. TimescaleDB extends Postgres instead. Analytics run on live data, in the database you already have. No pipeline. No migration. No new query language.
How does agent-desktop work?
Screenshot-based agents use vision to figure out where things are and then click on coordinates, which is a bit like playing darts with one eye closed.
agent-desktop reads the accessibility layer instead, which every operating system has (macOS calls it the Accessibility API, Windows has UI Automation, and Linux uses AT-SPI), and it describes every window, button, text field and menu in a structured way.
So a snapshot of a Save button looks something like this:
{
"role": "button",
"name": "Save",
"ref_id": "@s8f3k2p9:e8",
"available_actions": ["Click"]
}The agent clicks the button using the ref_id. Here's how the pieces fit together:
Step 1: Install agent-desktop
We'll use macOS and TextEdit for the demo.
Install the CLI:
npm install -g agent-desktopThen check your setup:
agent-desktop status
agent-desktop permissions --guideGrant Accessibility permission first, since macOS blocks desktop control without it.
Now launch TextEdit and take a snapshot of its interface:
agent-desktop launch TextEdit
agent-desktop snapshot --app TextEdit -iYou'll get back JSON with every interactive element and its reference. If you're curious how much your Mac exposes, try the same thing on Finder:
agent-desktop snapshot --app Finder -iStep 2: Connect the desktop to an AI agent
agent-desktop doesn't come with an LLM, so we need an agent that can run its commands. We're using Codex CLI for this one.
Install the official agent-desktop skill and pick Codex when prompted:
npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktopThe skill teaches the agent how to take snapshots, interact with elements, and recover when a command fails. You can read those instructions yourself too:
agent-desktop skills get desktop
agent-desktop skills get platformStart a new Codex session and give it this task:
Prompt:
Use agent-desktop to open TextEdit and create a plain-text file named launch-checklist.txt.
Add these three lines:
Review the landing page
Test the signup form
Publish the announcement
Use desktop interactions only. Do not write the file through shell commands. Inspect the UI after each important action and verify that the file was saved.
Stop and ask if you need to access another application or perform an unrelated action.
We never told it where to click or how to get through the Save dialog, so the agent has to figure all of that out from the interface.
Step 3: What the agent is doing under the hood
The agent starts with the same snapshot command from Step 1, reads the JSON, and picks the control it needs. Then it clicks:
agent-desktop click '@s8f3k2p9:e8'And types into an editable element:
agent-desktop type '<current-ref>' "Review the landing page"
It keeps going like this for the rest of the checklist, through TextEdit's menus and the Save dialog, taking a fresh snapshot after every action.
That fresh snapshot is important because references go stale the moment a dialog opens or the screen changes, and agent-desktop returns a STALE_REF error so the agent knows to look again.
Step 4: Verify the result
Did the agent save the file because it said "Task completed"??? Nope, not until you check 😅 For our demo, that means three things:
TextEdit shows the expected checklist.
The document is saved as launch-checklist.txt.
The saved file contains all three lines.
The agent should verify this through the UI, and you can open the file yourself afterwards to double-check.
Quick warning: run this on a separate desktop environment with no sensitive files or logged-in accounts, keep the agent limited to TextEdit, and make it ask before doing anything destructive.
My take
TextEdit is a simple demo, but the approach scales. It has its limits though.
Some apps (especially ones with custom interfaces) don't expose much accessibility info, and clicking through a UI will always be slower and flakier than a direct API call.
APIs will always win when they exist. But most of the software people use at work never got one, and that's where agent-desktop gets interesting.
If your team still copy-pastes data out of some ancient desktop app every week, this is the first thing I'd try.
Until next time,
Vaibhav 🤝
