GUIRILLA
A scalable framework for automated desktop UI exploration
Desktop agents are starving for training data - especially on macOS. So we built a gorilla: a crawler that installs real apps, explores them through the accessibility tree, and turns every screen into grounded tasks. No hand labeling.
Meet the crawler
Feed it an application bundle. It installs the app, walks every reachable state, and leaves behind a structured graph of the entire interface - then cleans up after itself.
- Application bundle + input parameters
- Installer
- GUIrilla Crawler
Pop-up handlerGraph orderingElement orderingEmpty elements handlerInvisible elements handlerUnrolling menu items handler
- Output Graph
- Un-installer
- Task Postprocessing Agent (GPT-4)
- Graph cleanup
- Task list
The problem
Desktop automation remains constrained by the scarcity of high-quality, publicly available desktop interaction data:
Manual annotation doesn't scale
Human-curated benchmarks are high quality but narrow or closed-source.
Single-window datasets misrepresent real usage
Real desktops are cluttered, multi-window, modal.
Automated collection demands OS-specific engineering
macOS lacks easy virtualisation; off-the-shelf pipelines don't apply.
Three GPT-4 agents ride along
Input Agent
Generates realistic input for input fields.
Order & Login Agent
Orders elements so irreversible or dangerous actions are queued last - and pings a human when a login is required.
Task Postprocessing Agent
Generates tasks based on the action and its UI context.
Are agents necessary? On three test apps - Stocks, Maps, Weather - enabling the handlers and agents increased task discovery 5× in Stocks and 3× in Maps, with fewer duplicates and faster parse times. The handlers target platform-agnostic pain points (inconsistent labels, hidden components, dynamic content), so the logic carries over to Windows, Linux, Android and other platforms.
GUIrilla-Task
Each task is a screenshot + the accessibility state + one grounded action (click or type), typed by task category and target element.
Tasks per element type - the most prevalent category is buttons.
MacApp Trees - 561 GB of structured interaction graphs
Every explored interface is organized into a hierarchical MacApp Tree derived from accessibility states and user actions. We release the trees - and macapptree ↗, the open-source library that turns the raw Accessibility API into clean hierarchical JSON.
{
"name": "empty-name-AXImage",
"role": "AXImage",
"description": "UpgradeToProHomeBanner",
"role_description": "image",
"value": null,
"absolute_position": "241.00;582.00",
"size": "212;252",
"enabled": true,
"bbox": [19, 517, 231, 769],
"visible_bbox": [19, 517, 231, 769],
"children": []
}
…and the tasks the tree yields:
Click on text input field "Intention"Press on button "Deep Work Session"Tap on button "SKIP BREATHE"4.2K synthetic images vs ~20M real ones
Models trained purely on GUIrilla data - GUIrilla-See - against the field on UI grounding benchmarks:
| Model | Platform | Data | # Images | ScreenSpot-v2 | ScreenSpot-Pro | Pro (macOS) |
|---|---|---|---|---|---|---|
| Florence-2 (0.7B) | Multi-OS | Real + Synthetic | - | 1.80% | 0.12% | 0.16% |
| Qwen 2.5 VL (3B) | Multi-OS | Real + Synthetic | - | 62.34% | 15.93% | 18.37% |
| Qwen 2.5 VL (7B) | Multi-OS | Real + Synthetic | - | 87.50% | 27.13% | 34.27% |
| CogAgent (18B) | Multi-OS | Real + Synthetic | 40M | 52.80% | 7.70% | 4.60% |
| UGround (7B) | Web + Android | Synthetic | 1.3M | 76.30% | 16.50% | 12.30% |
| ShowUI (2B) | Multi-OS | Real + Synthetic | 256K | 77.30% | 7.70% | 10.80% |
| OS-Atlas (7B) | Multi-OS | Synthetic | 2.2M | 83.30% | 18.90% | 20.00% |
| UI-TARS (2B) | Multi-OS | Real + Synthetic | ~20M (est.) | 84.70% | 27.70% | 15.40% |
| UI-TARS (72B) | Multi-OS | Real + Synthetic | ~20M (est.) | 90.30% | 38.10% | 40.00% |
| UI-TARS (7B) | Multi-OS | Real + Synthetic | ~20M (est.) | 91.60% | 35.70% | 27.70% |
| GUIrilla-See (0.7B) | macOS | Synthetic | 4.2K | 53.55% | 7.34% | 7.95% |
| GUIrilla-See (3B) | macOS | Synthetic | 4.2K | 89.54% | 29.35% | 32.62% |
| GUIrilla-See (7B) | macOS | Synthetic | 4.2K | 94.73% | 35.36% | 41.39% |
How do you word a grounding task?
Accessibility metadata often reduces to generic labels ("button", "text") that don't convey intent. Training on GPT-rewritten tasks that focus on the functional role of a UI element - rather than its visual description or raw value - gave 53.55% accuracy vs 40.35%. A 13-point gap from the same data, described differently.
Is accessibility metadata reliable?
The dataset covers almost every app on the App Store, brew, and MacUpdate - popular and very niche. Only about 9% ship reliable accessibility metadata (it's developer-dependent), but the ones that do provide very useful context. Five accessibility-expert annotators verified a test subset - GUIrilla-Gold, 1,283 tasks - and reached 90.26% task alignment.
27,171 tasks.
Zero hand labeling.
The paper covers the crawler, the agents, the dataset, and the full evaluation. The framework and macapptree are open source.