MACARENA
Benchmarking computer use agents on an online macOS environment
Computer-use agents are improving fast - almost entirely on Linux and Windows. MacArena puts them in the ring on real macOS: 421 human-verified tasks across 50 applications, running on Apple Silicon VMs that reset to a clean state for every episode.
Step into the arena
The loop is simple: the agent sees a screenshot (optionally the accessibility tree), produces one executable action, and the VM answers with the next observation. When the agent calls it done, an evaluator inspects the final state - files, app state, shell output - and scores the round.
421 verified tasks · 50 apps · 3 sources
Task mix across the full benchmark - the pink tail is the new MacArena set. Full reproducibility: all code and virtual machines publicly released.
The problem
macOS is underserved
The only existing macOS benchmark, macOSWorld, covers a narrow slice: mostly first-party apps, with simpler and less ambiguous tasks than cross-platform benchmarks.
Outdated hardware
macOSWorld runs on x86 virtual machines - incompatible with the entire modern Apple Silicon lineup. Cloud Apple Silicon is technically possible but too costly for large-scale benchmarking or RL.
No third-party coverage
Real macOS usage is dominated by third-party software, which existing benchmarks barely evaluate.
Strong scores on existing benchmarks can reflect familiarity with task distributions, not genuine cross-platform GUI skill. On macOS-native tasks, a leading model trails by over 26 percentage points.
The environment
POMDP formulation
At each step the agent gets an observation (screenshot, optionally the accessibility tree via macapptree) and produces an executable action.
Native virtualization
VMs run under UTM on Apple's Virtualization framework - near-native speed on M-series chips.
Copy-on-use snapshots
Each episode gets a fresh copy of the VM image, guaranteeing a clean, reproducible starting state.
Execution-based reward
r ∈ [0, 1]: an evaluator inspects final files, app state, and shell output. Not LLM-judged.
Anatomy of a task
Instruction
A natural-language goal, e.g. "Open Preview, export the open PDF as PNG, then set it as wallpaper."
Pre command / config
VM initialization - download files, launch apps, open documents.
Evaluator
A deterministic script that verifies completion from the final state.
Two formats are supported - OSWorld config-based and macOSWorld shell-based - and each of the 49 new MacArena tasks ships with its own hand-crafted evaluation script.
Results - nobody wins the arena yet
Success rates of four baseline agents, 15 steps max per task, two runs each. Best result in each row in pink:
| Category | UI-TARS-1.5 7B | Qwen3-VL 2B | Qwen3-VL 4B | OpenAI CUA |
|---|---|---|---|---|
| OSWorld subset | ||||
| Overall | 21.27% | 9.95% | 16.36% | 16.74% |
| Chrome | 28.12% | 15.62% | 37.50% | 37.50% |
| GIMP | 41.67% | 8.33% | 12.50% | 4.17% |
| LibreOffice Calc | 6.67% | 2.22% | 4.44% | 4.44% |
| LibreOffice Writer | 17.39% | 13.04% | 17.39% | 21.74% |
| Multi-App | 0.00% | 1.72% | 3.45% | 1.72% |
| OS | 50.00% | 16.67% | 36.36% | 33.33% |
| Thunderbird | 25.00% | 25.00% | 25.00% | 37.50% |
| VS Code | 68.42% | 31.58% | 36.84% | 47.37% |
| macOSWorld subset | ||||
| Overall | 24.50% | 15.89% | 39.74% | 52.32% |
| Advanced | 0.00% | 14.29% | 14.29% | 42.86% |
| File Management | 25.93% | 7.41% | 37.04% | 40.74% |
| Media | 40.00% | 20.00% | 30.00% | 30.00% |
| Multi-App | 5.00% | 10.00% | 15.00% | 15.00% |
| Productivity | 26.67% | 20.00% | 63.33% | 70.00% |
| Sys & Interface | 32.00% | 32.00% | 44.00% | 60.00% |
| System Apps | 28.12% | 9.38% | 40.62% | 71.88% |
| MacArena subset | ||||
| Overall | 10.20% | 4.08% | 12.24% | 36.73% |
| Productivity | 7.14% | 7.14% | 14.29% | 28.57% |
| Sys & Interface | 25.00% | 0.00% | 12.50% | 37.50% |
| File Management | 0.00% | 0.00% | 0.00% | 0.00% |
| Advanced | 11.11% | 11.11% | 11.11% | 55.56% |
| System Apps | 8.33% | 0.00% | 16.67% | 50.00% |
| Overall | 21.14% | 11.40% | 24.23% | 31.83% |
OpenAI CUA leads overall at 31.83% - and still no agent cracks 37% on the macOS-native subset. The arena is wide open.
What the fights revealed
Does macOS actually pose a harder challenge than Linux for CUA?
Yes. On the identical OSWorld task set, every model with a public reference score drops when moved to macOS: OpenAI CUA −9.26, Qwen3-VL 4B −9.84, Qwen3-VL 2B −7.05, UI-TARS-1.5 7B −3.23. Different app appearance, keyboard shortcuts, window management and system behavior all penalize models trained mainly on Linux and Windows.
Do model rankings hold across task sources?
No - they invert. UI-TARS-1.5 7B beats OpenAI CUA on the OSWorld subset (21.27% vs 16.74%), but on the new macOS-native subset the order flips hard: OpenAI CUA 36.73% vs UI-TARS-1.5 7B 10.2% - a swing of over 26.5 points in the opposite direction. Pattern-matching previously seen task structures is not the same as generalizing to a new platform.
How do we keep the benchmark trustworthy?
100% human-verified. Every one of the 421 tasks was reviewed to be executable, unambiguous and correctly specified - and evaluation is execution-based, not LLM-judged. MacArena is the only macOS online benchmark combining third-party app coverage with full manual verification.
421 tasks.
Every one human-verified.
The paper covers the environment, the task suite, and the full evaluation. All code and virtual machines are publicly released.