# MacPaw Research — Full Content
> MacPaw Research — advancing Mac-native intelligence. On-device AI, LLM efficiency, AI memory, and human–computer interaction. Papers, tech notes, and highlights from a research department at MacPaw.
This document is a single-fetch dump of every article currently published on
https://research.macpaw.com. Companion to [/llms.txt](https://research.macpaw.com/llms.txt) (curated map) and
[/feed.xml](https://research.macpaw.com/feed.xml) (RSS subscription).
Articles are sorted newest first. Each article is preceded by a metadata
block (authors, publication date, conference, topics, canonical URL).
Image references use absolute URLs.
---
# Memory Belongs to the User: Portable Memory, an Open Standard Proposal for Cross-Vendor AI Memory
**Authors**: Sergii Kryvoblotskyi, Nataliia Stulova, Vladyslav Hamolia
**Published**: 2026-07-06
**Topics**: artificial-intelligence
**Type**: Paper
**URL**: https://research.macpaw.com/publications/portable-memory
AI assistants became systems that *remember*: preferences, projects, relationships, and history now persist across sessions and increasingly shape what these systems do for us. That memory is the most personal dataset a person delegates to software — and today it is locked inside each vendor's proprietary storage, in incompatible shapes, with no way to move it and no way to prove it has ever been deleted. We believe this is the data portability gap of the AI era, and that it should be closed the way the web closed it for documents, mail, and calendars: with an open format. This paper proposes **Portable Memory**, an open, vendor-neutral standard for carrying AI memory across applications, devices, and vendors — lossless, local, and with **verifiable deletion** as its guarantee of integrity. We present the vision and principles; an overview of the format (a self-contained `.mem` bundle with a canonical, byte-reproducible serialization, portable deletion tombstones carrying proof-of-reach, and optional Ed25519 signing); and two open reference implementations (in Python and Swift) with proven byte-for-byte interoperability through a shared conformance fixture and a golden-vector suite, ingest adapters for mem0, ChatGPT, and Claude memory, a production integration, and a four-level conformance program gated on deletion propagation. We state plainly what this proposal is not yet — independently implemented, neutrally governed, widely adopted — and close with an explicit invitation: implement it, adopt it, challenge it, and help govern it.
Reference implementations are open source (MIT): [Python](https://github.com/MacPaw/portable-memory) and [Swift](https://github.com/MacPaw/portable-memory-swift). Community discussion is welcome at tech-research@macpaw.com or by opening an issue on either repository.
```bibtex
@misc{kryvoblotskyi-2026-portable-memory,
author = {Sergii Kryvoblotskyi and Nataliia Stulova and Vladyslav Hamolia},
title = {Memory Belongs to the User: Portable Memory, an Open Standard Proposal for Cross-Vendor AI Memory},
month = {July},
year = {2026},
note = {Standards proposal — preprint for community review},
url = {https://research.macpaw.com/publications/portable-memory}
}
```
---
# MacArena: Benchmarking Computer Use Agents on an Online macOS Environment
**Authors**: Victor Muryn, Maksym Shamrai, Sofiia Mazepa, Yehor Khodysko
**Conference**: 2nd AIWILD Workshop @ ICML 2026
**Published**: 2026-06-08
**Topics**: artificial-intelligence
**Type**: Paper
**URL**: https://research.macpaw.com/publications/macarena
Computer-use agents (CUAs) operate graphical user interfaces (GUIs) through vision and control primitives, and their capabilities have advanced rapidly, driven in part by standardized online evaluation benchmarks such as OSWorld, which serve both as evaluation tools and as training environments for reinforcement learning. However, macOS remains underserved in this landscape: the only existing benchmark, macOSWorld, covers a narrow slice of first-party applications with simpler tasks, and runs on x86 virtual machines incompatible with Apple Silicon. We introduce MacArena, a benchmark of 421 manually verified tasks spanning 50 applications that combines a curated port of OSWorld tasks, content sourced from macOSWorld, and 49 new macOS-native tasks, all running on Apple's native Virtualization framework on Apple Silicon. We argue that macOS presents distinct GUI challenges beyond what Linux-based benchmarks capture, and our evaluation supports this claim: strong model performance on existing benchmarks can reflect familiarity with task distributions rather than genuine cross-platform GUI competence. Notably, model rankings invert between ported and macOS-native tasks, with a leading model trailing by over 26% on the MacArena subset, suggesting that macOS poses a genuinely harder environment for current GUI agents.
```bibtex
@misc{muryn-etal-2026-aiwild-macarena,
author = {Victor Muryn and Maksym Shamrai and Sofiia Mazepa and Yehor Khodysko},
title = {MacArena: Benchmarking Computer Use Agents on an Online macOS Environment},
month = {June},
year = {2026},
eprint = {2606.06560},
eprinttype = {arxiv},
eprintclass = {cs.LG},
url = {https://arxiv.org/abs/2606.06560},
urldate = {2026-06-08},
note = {\emph{Accepted to AIWILD @ ICML 2026}},
}
```
---
# Help, Don’t Control: Measuring User Emotional Background in AI Assistant Interaction
**Authors**: Anastasiia Satarenko
**Conference**: Conversational AI: Agency and Identities 2026
**Published**: 2026-06-04
**Topics**: human-computer-interaction
**Type**: Paper
**URL**: https://research.macpaw.com/publications/help-dont-control-measuring-user-emotional-background-in-ai-assistant-interaction
As AI assistants shift from answering questions to executing actions, small interaction choices can influence users’ moment-to-moment experience. We report an exploratory lab study (N=20) measuring users’ Emotional Background (EB) during three local device-maintenance tasks in a prompt-based assistant and a menu-based baseline interface. We derived an EEG-based EB index and triangulated it with eye tracking and post-task questionnaires. EB trajectories were task-dependent: the prompt-based assistant produced higher EB for straightforward app deletion, while the baseline yielded higher EB for malware checking and troubleshooting. EB decreases appeared to cluster around three interaction moments: formulating an initial request, encountering negatively framed system-health terminology, and processing rapid or ambiguous automation of consequential actions. Survey responses indicated greater willingness to delegate non-personal tasks than personal communications. We discuss these patterns as preliminary evidence and outline design considerations for pacing, framing, transparency, and prompt scaffolding.
```bibtex
@misc{satarenko-2026-cui-measuring-emotional-background-assistant-interaction,
author = {Anastasiia Satarenko},
title = {Help, Don’t Control: Measuring User Emotional Background in AI Assistant Interaction},
month = {June},
year = {2026},
note = {\emph{Accepted to CUI 2026.} \url{https://research.macpaw.com/publications/help-dont-control-measuring-user-emotional-background-in-ai-assistant-interaction}},
}
```
---
# Composable AI Pipelines: Write Intent, Not Infrastructure
**Authors**: Maksym Kotliar
**Published**: 2026-05-18
**Topics**: artificial-intelligence
**Type**: Tech Note
**URL**: https://research.macpaw.com/publications/composable-ai-pipelines
## Introduction
AI product features routinely embed scheduling, model selection, and retry logic directly into business code. This couples _what a workflow does_ to _how it runs_, creating fragility as runtime capabilities evolve.
Composable Pipelines separates those concerns by design. Developers express intent in a SwiftUI-inspired Swift DSL — property wrappers, result builders, composable value types — making the authoring model immediately familiar to any Swift developer. Our proprietary runtime handles dependency analysis, parallel batching, model ranking, and re-execution mechanics. The developer-facing surface stays stable while the execution layer improves.
This note describes that boundary: the DSL contract, the compilation and execution model, and the open questions that remain. The authoring layer is open source — [MacPaw/ComposablePipelines](https://github.com/MacPaw/ComposablePipelines) on GitHub.
## Motivation
Most AI feature code today is a mix of business logic and infrastructure: loops that retry, branching that routes between models, glue that parallelizes independent calls. That works at small scale, but it accumulates fast. Every new feature re-solves the same scheduling problems, and every runtime improvement requires touching product code.
Our design target is a clean separation:
- developers write _intent_ in a DSL,
- the compiler and executor decide _scheduling_ and _model routing_,
- local-first execution is a first-class mode, not an afterthought.
This note demonstrates the current architecture direction. Breaking changes should be expected as we iterate.
## System Model
The pipeline lifecycle follows four stages:
```mermaid
flowchart LR
DSL["DSL"] --> AST["AST"]
AST --> Compiler["Compiler"]
Compiler --> Executor["Executor"]
```
### DSL
Swift-native declarative authoring via `@PipelineBuilder`, `@State`, and composable primitives: `Model`, `Guardrail`, `While`, `Group`, `ClientTask`. The construction is intentionally SwiftUI-inspired — property wrappers, result builders, and composable value types — so the authoring model feels immediately familiar to any Swift developer who has built a view hierarchy. Developers express intent; the DSL makes that intent inspectable by the compiler.
### AST
Pipelines lower to `PipelineGraph`—a codable graph representation that serves as the stable transport boundary between authoring and compilation.
### Compiler
`PipelineCompiler` transforms the AST into a `PipelineExecutionGraph` by performing dependency analysis, slot hazard ordering (RAW / WAR / WAW), gate and barrier edge construction, and topological grouping for safe parallel batches.
### Executor
`ExecutionEngine` evaluates compiled graphs with deterministic walk semantics, state update commits, parallel branch batching, and graph replacement when committed state triggers re-execution.
### Re-execution and Epoch Resolution
When a model step completes and writes to state, the pipeline may need to continue along a different path. The naive solution—re-run everything from the top—is correct but wasteful. The obvious alternative—assign stable identities to tasks and skip the ones already done—sounds reasonable until you try to implement it. Task identity in a dynamic graph turns out to be fragile: two executions of the same pipeline body can produce structurally similar but semantically different graphs, and there is no reliable anchor for "this task in pass two is the same task as in pass one."
The underlying tension comes from how pipelines lower to graphs. Developers read the DSL as conventional code: steps flow top to bottom, state accumulates, branches resolve. But the executor never sees the full structure at once. Because the graph re-lowers only when committed state changes, the executor receives one DAG slice per pass — a partial view of a pipeline that may branch or extend itself based on values it has not yet produced.
This asymmetry pointed us toward a different invariant: instead of tracking task identity, **guarantee that the prefix of each re-lowered graph is identical to the prefix of the previous one**. If that holds, the executor can skip the prefix entirely without knowing anything about individual tasks — positional order is enough.
Enforcing that guarantee is where epoch-based time travel comes in. Each committed state write advances a monotonic **epoch** counter. When the DSL body re-evaluates for a new pass, `@State` reads are answered using the slot history as it stood at the epoch recorded by the previous pass — not the current live values. This means the re-lowered graph replays the same decisions the first pass made for every step that was already executed, producing a structurally identical prefix. Only the suffix, where new state values change the outcome, diverges.
At the boundary between passes, the engine produces an `ExecutionCursor` carrying the latest epoch and a **walk offset** — the count of surface tasks allocated so far. The client trims the re-lowered graph by that offset and hands only the suffix back to the executor. The executor skips the prefix, slot values from prior passes remain available in place, and execution continues from exactly where the graph changed.
The result is incremental re-execution with a provable prefix guarantee: a pipeline can iterate, branch differently, and extend itself in response to model output, while the engine avoids redundant work without needing to track or compare individual tasks across passes.
## Developer Contract
The contract separates what product teams own from what the runtime owns.
**Developers control:** pipeline business logic, control flow and state transitions, task intent via model hints, and tool usage inside client tasks.
**Our runtime controls:** scheduling and batching strategy, parallel safety and hazard ordering, model candidate ranking and selection, and execution bookkeeping.
This separation is deliberate. DSL stays stable and readable; the runtime keeps improving without forcing product teams to rewrite logic.
**Product boundary:** `PipelineDSL` syntax and semantics are the developer-facing contract. The compiler and executor are proprietary.
## Examples
The snippets below are intentionally compact and illustrate contract shape, not full production implementations.
### Intent-First Composition
The simplest form: a two-step pipeline where the reply step depends on a prior summarization.
```swift
import PipelineDSL
struct SupportReply: Pipeline {
typealias Output = String
let userMessage: String
@State var summary: String = ""
var body: some Pipeline {
$summary.set {
Model(
instructions: "Summarize user intent",
hints: [.fastResponse, .instruct],
input: userMessage
)
}
Model(
instructions: "Write a short helpful reply",
hints: [.reasoning, .streaming],
input: summary
)
}
}
```
Hints describe intent, not a hardcoded model ID. The runtime selects a candidate at execution time.
### Parallel Prep + Guarded Exit
Independent branches can batch in parallel when optimization is enabled. A guardrail check allows early return before the expensive synthesis step.
```swift
import PipelineDSL
struct AnalyzeMessage: Pipeline {
typealias Output = String
let message: String
@State var intent: String = ""
@State var sentiment: String = ""
@State var risk: Bool = false
var body: some Pipeline {
$intent.set {
Model(
instructions: "Classify intent",
hints: [.fastResponse, .instruct],
input: message
)
}
$sentiment.set {
Model(
instructions: "Classify sentiment",
hints: [.fastResponse, .instruct],
input: message
)
}
$risk.set {
Model(
instructions: "Reply true if the message contains PII, false otherwise",
hints: [.fastResponse, .instruct],
input: message
)
}
if risk {
Self.return("Request blocked by guardrails.")
} else {
Model(
instructions: "Synthesize final response",
hints: [.reasoning, .streaming],
input: """
message=\(message)
intent=\(intent)
sentiment=\(sentiment)
"""
)
}
}
}
```
The diagram below illustrates how this pipeline actually executes at runtime — the compiler detects that the three state-writes have no data dependencies between them and groups them into a parallel batch.
```mermaid
flowchart TD
Start([Pipeline Start]) --> Compiler["Compiler: detects independent branches"]
Compiler --> I & S & R
subgraph Parallel ["Parallel Batch"]
I["Model: Classify intent → $intent"]
S["Model: Classify sentiment → $sentiment"]
R["Model: PII check → $risk"]
end
I & S & R --> Gate["Barrier — all branches complete"]
Gate --> Cond{risk?}
Cond -- yes --> Blocked([Return: request blocked])
Cond -- no --> Synth["Model: Synthesize final response"]
Synth --> Output([Output])
```
### Agent Loop Shape
Iterative tool-call loops remain declarative. The `While` block drives refinement without exposing scheduling machinery to product code.
```swift
import PipelineDSL
struct AgentTurnLite: Pipeline {
typealias Output = String
@State var conversation: String
@State var reply: String = ""
var body: some Pipeline {
While { reply.isEmpty } body: {
let modelOutput = Model(
instructions: "Return either final reply or tool plan",
hints: [.reasoning, .streaming],
input: conversation
)
// If tool plan → run ClientTask → append tool result to conversation
// Else set reply
}
reply
}
}
```
### Pipeline Composition
Pipelines compose. A small reusable sub-pipeline can be embedded into any larger product flow without duplicating logic.
```swift
import PipelineDSL
struct TranslateAndSummarize: Pipeline {
typealias Output = String
let text: String
@State var translated: String = ""
var body: some Pipeline {
$translated.set {
Model(
instructions: "Translate to English",
hints: [.fastResponse, .instruct],
input: text
)
}
Model(
instructions: "Summarize in two sentences",
hints: [.reasoning],
input: translated
)
}
}
struct ResearchBrief: Pipeline {
typealias Output = String
let articles: [String]
@State var summary: String = ""
var body: some Pipeline {
ForEach(in: { articles }) { article in
$summary.set { TranslateAndSummarize(text: article) }
}
Model(
instructions: "Synthesize a research brief from the collected summaries",
hints: [.reasoning, .streaming],
input: summary
)
}
}
```
## Model Hints and Selection
Model hints are soft runtime signals, not hardcoded model references. The current vocabulary includes `fastResponse`, `instruct`, `reasoning`, and `streaming`.
Developers express workload intent. The runtime ranks available model candidates and selects one at execution time. Pipeline logic is therefore not coupled to a single vendor or model ID by default—this is a deliberate core design decision, not an incidental property.
## Local-First Thesis
We believe local execution will become the default for many production AI workflows.
The reasoning is practical: the privacy boundary stays near user data; the security posture is easier to reason about; the cost profile is more predictable; and product teams retain more freedom from external provider economics.
## Platform Architecture
The same pipeline architecture runs on both macOS and iOS. The developer-facing API is identical across platforms — what changes is only how the runtime is deployed.
### macOS: Out-of-Process Agent
On macOS, we ship a system service — the **Elix Agent** — that runs as a launch agent and owns execution. Client apps link against `ElixClient`, which lowers the DSL to a `PipelineGraph` and sends it to the agent over XPC. The agent compiles the graph, runs it through `ExecutionEngine`, and calls back to the client whenever pipeline state changes require re-emitting the body or executing a `ClientTask` closure.
```
Client app
│ ElixClient.execute(pipeline:)
│ lower DSL → PipelineGraph
▼
ElixAgentRuntime (XPC service)
│ compile PipelineGraph → PipelineExecutionGraph
│ ExecutionEngine walks graph
│ → calls back for ClientTask and state re-emission
▼
Result returned to client
```
This split has real benefits beyond privacy. The agent can serve multiple client processes, enforce load-balancing per process key, and validate client code signatures and entitlements before accepting any work. The agent binary is distributed and updated independently from client apps — `ElixAgentBootstrap` handles resolving, downloading, signature-verifying, and installing the correct agent version before a client connects.
The XPC contract is intentionally small: `RunPipeline` from client to agent, and `HandlePipelineStateUpdated` / `ProvideUpdateGraph` / `ExecuteClientTask` from agent back to client. No broader execution surface is exposed.
### iOS: Embedded Runtime
On iOS, cross-process services are not a viable pattern. Rather than a different architecture, the same runtime is embedded directly inside the client process. `ElixClient` switches to a local implementation that runs `ExecutionEngine` in-process, with the same compilation path and the same re-execution semantics.
The pipeline code is identical. No conditional compilation, no platform-specific branching in product code. The difference is entirely in how `ElixClient` is backed at the deployment layer.
## Cloud Compatibility
Cloud execution remains important for burst capacity and large shared infrastructure. Batching economics at cloud scale can be significant, and our pipeline model targets both environments with the same developer contract: write DSL intent once, let the runtime choose execution strategy.
Clients plug in remote models by implementing the `ModelProvidingHeap` protocol and advertising a `[RemoteModelDescriptor]` catalog. Each descriptor declares a model ID and the `ModelSelectionHints` it advertises — the same hint vocabulary used in the DSL (`fastResponse`, `reasoning`, `streaming`, etc.). When the executor encounters a `Model` step, it ranks descriptors by hint-intersection score and invokes the winning entry. Clients separately provide a closure that maps descriptor IDs to `ModelConfiguration` values holding the endpoint URL and credentials — keeping connection details out of the authoring layer entirely.
```swift
let catalog: [RemoteModelDescriptor] = [
RemoteModelDescriptor(id: "fast-cloud", advertisedHints: [.fastResponse, .instruct]),
RemoteModelDescriptor(id: "reasoning-cloud", advertisedHints: [.reasoning, .streaming]),
]
let config = ElixClientConfiguration(
modelCatalog: catalog,
modelConfigurationForDescriptor: { id in
switch id {
case "fast-cloud": return ModelConfiguration(modelID: id, baseURL: "...", apiKey: "...")
case "reasoning-cloud": return ModelConfiguration(modelID: id, baseURL: "...", apiKey: "...")
default: return nil
}
}
)
```
For teams already running OpenAI-compatible inference (Ollama, vLLM, Groq, and similar), the built-in `.openAICompatibleHTTP` endpoint type covers the transport layer without requiring a custom `ModelProvidingHeap` implementation. Both approaches coexist in the same catalog, so a pipeline can route fast classification steps to a local model and reasoning steps to a remote one — driven entirely by hints, with no routing logic in pipeline code.
## Client Authority and Permissions
Every action that touches the outside world — file access, network calls, keychain reads, user prompts, API calls — runs inside a `ClientTask` closure on the client side. The executor never crosses that boundary directly. It schedules a `ClientTask` step, hands control back to the client process, waits for the result, and continues.
This means the client is the sole gatekeeper between the execution engine and the environment. Permissions, entitlements, secrets, and user-facing authorization dialogs all live in client code, not in pipeline graph definitions. A pipeline that calls a calendar API, reads a file, or requests a user confirmation does so through a `ClientTask` that the client registers and controls. The DSL describes _that_ a task should happen; the client decides _whether_ and _how_.
```swift
ClientTask(
input: query,
action: { encodedQuery in
// Running in client process: full access to keychain, entitlements, UI
let token = try Keychain.read("calendar-api-token")
let result = try await CalendarAPI(token: token).search(encodedQuery)
return try JSONEncoder().encode(result)
}
)
```
The practical consequence is a clean capability model: the executor is stateless with respect to user secrets and system access. It cannot exfiltrate credentials, trigger permission dialogs, or access protected resources on its own. All of that capability is owned and brokered by the client, which can enforce its own audit, rate-limiting, or user-consent policies around each task registration.
On macOS, this boundary is also a process boundary — `ClientTask` callbacks cross XPC back to the originating client, so the agent never holds secrets in its process space at all. On iOS, where execution is in-process, the same logical boundary holds: task closures are registered by client code and remain under client control.
## Open Source
We have open-sourced the [`ComposablePipelines` package](https://github.com/MacPaw/ComposablePipelines) — the authoring layer developers write against — while keeping the compiler and executor proprietary. The code is available on GitHub at [MacPaw/ComposablePipelines](https://github.com/MacPaw/ComposablePipelines).
**What is public:** `PipelineDSL` syntax, the `Pipeline` protocol, all composable primitives (`Model`, `Guardrail`, `While`, `Group`, `ClientTask`, `ForEach`), `@State` and `@PipelineBuilder`, and the `PipelineGraph` AST representation. This gives developers a stable, inspectable surface they can build product code against, share pipelines, and contribute to tooling.
**What stays closed:** `PipelineCompiler`, `ExecutionEngine`, model ranking and selection logic, scheduling heuristics, and the re-execution mechanics. These are where our runtime differentiation lives.
The rationale is straightforward: an open authoring surface makes pipelines portable and testable in isolation, and lowers the barrier for early adoption. A closed execution layer lets us continue optimizing scheduling, batching, and model selection without exposing proprietary logic or locking the DSL contract to implementation details.
This is the same pattern as many successful platform designs — open interface, differentiated engine. Developers never need to know which execution path their pipeline takes; they just write intent.
---
# GUIrilla: A Scalable Framework for Automated Desktop UI Exploration
**Authors**: Sofiya Garkot, Maksym Shamrai, Ivan Synytsia, Mariya Hirna
**Conference**: 3rd DATA-FM Workshop @ ICLR 2026
**Published**: 2026-05-12
**Topics**: artificial-intelligence
**Type**: Paper
**URL**: https://research.macpaw.com/publications/guirillascalableframeworkautomated
The performance and generalization of foundation models for interactive systems critically depend on the availability of large-scale, realistic training data. While recent advances in large language models (LLMs) have improved GUI understanding, progress in desktop automation remains constrained by the scarcity of high-quality, publicly available desktop interaction data, particularly for macOS. We introduce GUIRILLA, a scalable data crawling framework for automated exploration of desktop GUIs. GUIRILLA is not an autonomous agent; instead, it systematically collects realistic interaction traces and accessibility metadata intended to support the training, evaluation, and stabilization of downstream foundation models and GUI agents. The framework targets macOS, a largely underrepresented platform in existing resources, and organizes explored interfaces into hierarchical MacApp Trees derived from accessibility states and user actions. As part of this work, we release these MacApp Trees as a reusable structural representation of macOS applications, enabling downstream analysis, retrieval, testing, and future agent training. We additionally release macapptree, an open-source library for reproducible accessibility-driven GUI data collection, along with the full framework implementation to support open research in desktop autonomy.
```bibtex
@misc{garkot2026guirillascalableframeworkautomated,
title={GUIrilla: A Scalable Framework for Automated Desktop UI Exploration},
author={Sofiya Garkot and Maksym Shamrai and Ivan Synytsia and Mariya Hirna},
year={2026},
eprint={2510.16051},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2510.16051},
}
```
---
# Fast-dLLM on MLX: Training-Free Acceleration for Diffusion Language Models on Apple Silicon
**Authors**: Kyrylo Yemets
**Published**: 2026-04-20
**Topics**: artificial-intelligence
**Type**: Tech Note
**URL**: https://research.macpaw.com/publications/fast-dllm-mlx
## Introduction
We present **Fast-dLLM-mlx**, a project that brings the core Fast-dLLM ideas to **Dream-style diffusion language models** running on Apple Silicon.
The project adapts the training-free acceleration strategy introduced in *Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding* to the MLX ecosystem, with a practical focus on local inference, reproducibility, and benchmarking on macOS.
Rather than introducing a new model family, this project ports an inference strategy: it aims to make diffusion LLM decoding faster on Apple hardware by combining **KV-cache reuse** and **confidence-based parallel token finalization** in MLX.
## Background
Diffusion language models differ from standard autoregressive LLMs in how they generate text. Instead of predicting the next token one step at a time, they iteratively refine a partially masked sequence across multiple denoising steps. This gives them a different decoding profile, but it also makes inference more expensive if implemented naively.
In this note, **Dream-style diffusion language models** refers to diffusion LMs built around the Dream family of iterative text generation methods, introduced in *Dream 7B: Diffusion Large Language Models* ([https://arxiv.org/abs/2508.15487](https://arxiv.org/abs/2508.15487)), where a transformer repeatedly revises masked token positions instead of committing tokens strictly left-to-right. In practice, that means they keep the diffusion-style refinement loop, but often reuse much of the standard LLM transformer stack, making them a useful bridge between familiar autoregressive architectures and non-autoregressive generation.
The Fast-dLLM paper:
> Wu, C., Zhang, H., Xue, S., Liu, Z., Diao, S., Zhu, L., Luo, P., Han, S., & Xie, E. (2025). *Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding*.
> [https://arxiv.org/abs/2505.22618](https://arxiv.org/abs/2505.22618)
proposes a training-free way to speed up diffusion LLM inference by reusing attention state and decoding multiple positions in parallel instead of fully recomputing the sequence at every refinement step.
The original reference implementation is available at:
[https://github.com/NVlabs/Fast-dLLM](https://github.com/NVlabs/Fast-dLLM)
## Motivation
Most experimental work around diffusion LLMs is still centered on Python stacks tuned primarily for CUDA environments. That leaves a gap for developers and researchers who want to:
- Run diffusion LLM inference locally on Apple Silicon
- Benchmark new decoding strategies in MLX
- Compare diffusion decoding against `mlx_lm` autoregressive baselines
- Avoid introducing training or architecture changes just to test inference improvements
Fast-dLLM-mlx addresses this gap as a project focused on **inference-time acceleration only**. It is designed for experimentation with decoding behavior rather than for changing the underlying model weights.
This project brings these ideas to **MLX**, Apple's machine learning framework for Apple Silicon, and applies them to **Dream-style** diffusion language model inference.
## Fast-dLLM for Dream Models in MLX
This project implements Dream architecture inference in MLX and adds a first MLX adaptation of the Fast-dLLM approach for that setting.
The current implementation includes:
- **Dream architecture inference in MLX**
- **Dual-cache support** for prompt-prefill and cache-aware token updates
- **Parallel token generation** with probability thresholding
- **Confidence-threshold token finalization** to lock in easy tokens early
- Benchmark scripts for comparing Fast-dLLM-mlx, Dream-MLX, and `mlx_lm` baselines
In practice, the Fast-dLLM path attempts to finalize multiple masked positions in parallel when their predicted confidence is high enough. This reduces redundant computation relative to repeatedly reprocessing the refinement state from scratch.
Example benchmark invocation:
```bash
uv run python -m benchmarks.fast_dllm_mlx_benchmark \
--model mlx-community/DiffuCoder-7B-cpGRPO-8bit \
--trust-remote-code \
--max-new-tokens 128 \
--steps 20 \
--block-length 32 \
--threshold 0.9 \
--warmup
```
## Design Principles
Fast-dLLM-mlx follows several practical design principles:
1. **Training-free acceleration:** improve inference without retraining the model
2. **Apple-native execution:** target MLX and Apple Silicon directly
3. **Architectural restraint:** adapt the decoding strategy without overcomplicating the model stack
4. **Benchmarkability:** make comparisons with baseline MLX paths straightforward
5. **Research utility:** keep the project clear enough for further experimentation
## Structure of project
The implementation consists of three main pieces:
1. MLX inference of Dream-style architecture.
2. Dual KV-cache.
3. Parallel decoding with confidence thresholding.
First, it provides an MLX implementation of the **Dream-style transformer stack**, including attention, RoPE handling, RMSNorm, and the language modeling head.
Second, it extends the standard cache behavior with a **dual KV-cache** design. This cache supports both:
- appending prompt and decoded states in the normal way
- replacing cached spans in place during iterative refinement
This matters because Fast-dLLM-style decoding reuses previously computed attention state while iteratively updating masked positions, rather than recomputing every token position from scratch.
Third, the generator adds **confidence-based parallel decoding**. During refinement, the implementation:
- predicts candidate tokens for masked positions
- computes per-position confidence
- finalizes tokens whose confidence exceeds a threshold
- guarantees progress by forcing at least one update when masked positions remain
This mechanism preserves the spirit of the original Fast-dLLM approach while fitting the execution model of MLX.
The repository currently depends on:
- **MLX / MLX-LM** for tensor execution and model loading
- **Transformers** for tokenizer support
- Apple Silicon hardware as the primary target environment
The benchmark examples in the repository focus on:
- `mlx-community/DiffuCoder-7B-cpGRPO-8bit` for diffusion-style generation
- `mlx-community/Qwen2.5-Coder-7B-Instruct-8bit` as an autoregressive `mlx_lm` baseline
## Performance
The repository includes dedicated benchmark entrypoints for three comparison paths:
- `fast_dllm_mlx_benchmark.py`
- `dream_mlx_benchmark.py`
- `qwen_mlx_lm_benchmark.py`
These scripts evaluate prompts from the local `prompts/` set and record per-prompt runtime summaries in CSV and JSON form. The current benchmark was run on a **limited number of samples**, not on a full dataset.
The benchmark slice currently used in the repository compares three prompt categories, `coding`, `general`, and `math`, across:
- autoregressive Qwen baselines in `mlx_lm`
- Dream MLX diffusion decoding
- Fast-dLLM MLX decoding on top of the same Dream-style model family
For the MLX diffusion path, the experiments use **DiffuCoder**, and we include **Qwen2.5-Coder** as an autoregressive baseline because the Dream architecture used here is based on Qwen2.5, making the comparison more meaningful.
Average throughput by category is shown below:
| Category | Qwen MLX 4-bit | Qwen MLX 8-bit | Dream MLX 4-bit | Dream MLX 6-bit | Dream MLX 8-bit | Fast-DLLM MLX 4-bit | Fast-DLLM MLX 6-bit | Fast-DLLM MLX 8-bit |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| coding | 50.664 | 30.334 | 13.567 | 13.521 | 12.973 | 38.227 | 37.492 | 38.015 |
| general | 50.648 | 30.164 | 13.554 | 13.477 | 13.016 | 36.983 | 39.137 | 35.729 |
| math | 50.103 | 30.096 | 11.748 | 11.489 | 10.989 | 34.381 | 34.887 | 37.022 |
| average | 50.472 | 30.198 | 12.956 | 12.829 | 12.326 | 36.530 | 37.172 | 36.922 |
The main result is straightforward: **Fast-dLLM on MLX is much faster than plain Dream MLX decoding across all measured categories and quantization variants**.
Using the average row as a summary:
- **Fast-DLLM MLX 4-bit** reaches `36.53` versus `12.96` for **Dream MLX 4-bit**
- **Fast-DLLM MLX 6-bit** reaches `37.17` versus `12.83` for **Dream MLX 6-bit**
- **Fast-DLLM MLX 8-bit** reaches `36.92` versus `12.33` for **Dream MLX 8-bit**
This places the Fast-dLLM MLX variants at roughly **2.8x to 3.0x faster** than the corresponding Dream MLX baselines in this benchmark slice.
The autoregressive `mlx_lm` Qwen baselines still set the upper bound in this comparison, especially in 4-bit form. However, the important outcome is that Fast-dLLM substantially narrows the gap between diffusion decoding and autoregressive generation while preserving the diffusion-style inference setup.
These numbers should still be treated as **directional rather than exhaustive**. The benchmark was run on a limited number of samples rather than a full dataset, and the measurements are best interpreted as evidence that the Fast-dLLM decoding strategy transfers effectively to MLX and Apple Silicon.
## Applications and Impact
Fast-dLLM-mlx is useful as both a project and a practical starting point for on-device experimentation with diffusion LLM inference on Apple hardware.
It enables:
- Faster local experimentation with diffusion decoding strategies
- Side-by-side comparison between diffusion and autoregressive MLX inference
- Exploration of cache-aware and parallel diffusion decoding techniques on Apple Silicon
- A clearer path toward privacy-preserving, local-first diffusion LLM tooling on macOS
More broadly, this project shows that Fast-dLLM-style inference optimizations are not limited to CUDA-first environments. They can also be adapted to Apple's local inference stack in a way that remains faithful to the original idea while being practical for MLX-based workflows.
## Code
- [Fast-dLLM-mlx on GitHub](https://github.com/MacPaw/Fast-dLLM-mlx)
---
# Do Our Design Principles Ever Really Change, or Do We Just Change the Methods of Achieving Them?
**Authors**: Anastasiia Satarenko
**Conference**: The International Conference on Computer Graphics, Interaction, Visualization Theory and Applications (GRIVAPP 2026)
**Published**: 2026-04-17
**Topics**: human-computer-interaction
**Award**: Best Poster Award
**Type**: Paper
**URL**: https://research.macpaw.com/publications/design-principles-change
Human–computer interaction has long been guided by principles such as system feedback, directness, and usability, most clearly articulated in Direct Manipulation Interfaces (DMIs). With the emergence of adaptive, AI-driven systems, new interaction paradigms raise the question of whether these foundations still hold or must be redefined. In this paper, we introduce Infinite Interfaces (IIs), a novel paradigm that integrates natural language processing, contextual awareness, and multimodal outputs into a single, adaptive entry point. To examine whether traditional principles extend to this new context, we conducted a moderated usability study with 10 participants, testing an II prototype on tasks ranging from simple actions to complex, multi-step problem-solving. The findings reveal that while IIs introduce new requirements—such as semantic alignment, personalization, progressive disclosure of functionality, and visually distinguishable multimodal outputs—they remain grounded in the same core principles as DMIs. Rather than replacing established foundations, IIs reinterpret and extend them. This work contributes to HCI theory by demonstrating HCI Researchers and UI/UX Designers that design principles remain stable across technological shifts, with innovation emerging in the methods used to achieve them.
```bibtex
@conference{grivapp26,
author={Anastasiia Satarenko},
title={Do Our Design Principles Ever Really Change, or Do We Just Change the Methods of Achieving Them?},
booktitle={Proceedings of the 21st International Conference on Computer Graphics, Interaction and Visualization Theory and Applications - GRIVAPP},
year={2026},
pages={230-237},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0014252800004728},
isbn={978-989-758-803-7},
}
```
---
# GLiNER2 for Swift: Unified Schema-Based Information Extraction on Apple Silicon
**Authors**: Hlib Koshurnykov
**Published**: 2026-02-19
**Topics**: artificial-intelligence
**Type**: Tech Note
**URL**: https://research.macpaw.com/publications/gliner2-swift
## Introduction
We present **GLiNER2Swift**, a native Swift/MLX implementation of *GLiNER2*, a unified schema-based information extraction framework. GLiNER was originally introduced by Zaratiana et al. as a *generalist and lightweight Named Entity Recognition (NER)* model. GLiNER2 subsequently extended this paradigm toward a broader schema-driven information extraction framework.
Our work brings this unified, schema-based extraction approach to the Apple ecosystem with first-class Apple Silicon support. Designed as a CPU-first, production-ready library for macOS, GLiNER2Swift enables developers and researchers to deploy advanced information extraction pipelines directly in Swift applications without relying on Python runtimes or GPU acceleration.
## Background
The GLiNER framework was introduced as a generalist, lightweight alternative to traditional NER task systems, allowing entity extraction without task-specific retraining. The original GLiNER paper:
> Zaratiana, U., Tomeh, N., Holat, P., & Charnois, T. (2023). *GLiNER: Generalist and Lightweight Model for Named Entity Recognition*. arXiv preprint arXiv:2311.08526.
> [https://arxiv.org/abs/2311.08526](https://arxiv.org/abs/2311.08526)
proposes a model that performs NER using label descriptions rather than fixed classification heads. As stated in the paper:
> "GLiNER is a generalist model for Named Entity Recognition that can generalize to arbitrary entity types defined at inference time without task-specific fine-tuning."
> — Zaratiana et al., 2023
GLiNER2 extends this idea into a broader **schema-based information extraction** framework, supporting not only NER but also classification and structured extraction under a unified architecture. The reference Python implementation is available at:
[https://github.com/fastino-ai/gliner2](https://github.com/fastino-ai/gliner2)
## Motivation
Despite rapid progress in NLP research, most modern frameworks remain Python-centric and GPU-dependent. This creates friction for macOS-native development, especially in contexts where:
- On-device inference is required for privacy or latency
- Deployment targets Apple Silicon hardware
- Swift is the primary programming language
- Tight integration with native macOS applications is needed
GLiNER2Swift addresses this gap by providing a numerically faithful port of GLiNER2 implemented fully in Swift and optimized for macOS 14+ on Apple Silicon (M1/M2/M3).
## Unified Schema-Based Extraction in Swift
GLiNER2Swift preserves the schema-driven paradigm introduced by GLiNER. Instead of training separate models for NER, classification, and structured extraction, developers define extraction tasks dynamically via label schemas.
The library currently supports:
- **Named Entity Recognition (NER)**
- **Text Classification**
- **Structured Data Extraction**
Relation extraction support is currently in progress.
Example usage:
```swift
let model = try await GLiNER2.fromPretrained("macpaw-research/gliner2-base-v1")
let entities = try model.extractEntities(
from: "Tim Cook is CEO of Apple in Cupertino.",
labels: ["person", "company", "location"]
)
```
This design enables rapid adaptation to new domains without retraining, aligning with the original GLiNER philosophy of inference-time label flexibility.
## Design Principles
GLiNER2Swift follows several core principles:
1. **Native-first:** pure Swift implementation without Python bridges
2. **On-device by default:** privacy-preserving local inference
3. **Reproducibility:** architectural fidelity to the reference implementation
4. **Developer ergonomics:** Swift Package Manager integration
5. **Extensibility:** foundation for future fine-tuning and adapter support
## Architecture and Implementation
GLiNER2Swift is a direct architectural port of the Python GLiNER2 implementation and aims to achieve numerical parity with it. The model architecture includes the following modules:
- **Encoder:** DeBERTa v3 with disentangled attention
- **Span Marker:** MLP-based span scoring
- **Count LSTM:** entity count prediction
- **Downscaled Transformer:** schema embedding
The implementation leverages:
- **MLX** ([https://github.com/ml-explore/mlx](https://github.com/ml-explore/mlx)) for tensor computation
- macOS-native [Swift concurrency mechanisms](https://developer.apple.com/documentation/swift/concurrency) for asynchronous loading and inference
The system is CPU-first and does not require GPU acceleration.
Currently supported models:
- `fastino/gliner2-base-v1` (205M parameters)
- `macpaw-research/gliner2-base-v1_mlx` (FP-16 instead of FP-32 to reduce model size)
Planned features include training loops, relation extraction, and additional model variants.
## Performance
We ran the benchmarks for our model on Apple M3 Pro, macOS 14+, CPU-only inference, using the `macpaw-research/gliner2-base-v1_mlx` model.
| Task | Mean | Min | Max |
|---|---|---|---|
| Entity Extraction | 334.8 ms | 324.9 ms | 339.6 ms |
| Classification | 59.6 ms | 58.9 ms | 60.1 ms |
| Structured Extraction | 288.3 ms | 284.5 ms | 291.0 ms |
| Combined (all 3 tasks) | 351.9 ms | 344.0 ms | 358.3 ms |
## Applications and Impact
GLiNER2Swift enables:
- Intelligent document parsing
- Flexible, on-device zero-shot NER
- Structured extraction from various document types
- Real-time zero-shot classification pipelines
By bringing GLiNER2 to Swift, we bridge modern NLP research and native Apple platform development. This work demonstrates that transformer-based schema-driven extraction can run efficiently on-device, without GPU dependency or server infrastructure.
GLiNER2Swift represents a step toward privacy-preserving, local-first AI tooling for macOS — bringing unified information extraction directly into Swift applications while staying faithful to the original GLiNER research vision.
## Code
- [GLiNER2Swift on GitHub](https://github.com/MacPaw/Gliner2Swift)
---
# Demonstrating Headphone-Sensed, Accessibility-Informed Head Pointing with Snapping for Inclusive Interaction
**Authors**: Mykhailo Tarasenko, Iryna Pastukhova, Oleksandr Frankiv, Anastasiia Satarenko, Nataliia Stulova, Sergii Kryvoblotskyi
**Published**: 2026-01-21
**Topics**: human-computer-interaction
**Type**: Paper
**URL**: https://research.macpaw.com/publications/head-pointing-with-snapping
The mouse pointer is central to direct-manipulation graphical user interfaces, and modern desktop operating systems provide accessibility features that enable pointer control via eye or head movement. Existing solutions, such as Eye Control on Windows and Head Pointer on macOS, rely on continuous video capture, which makes them sensitive to lighting conditions and user position and raises privacy concerns. In this work, we demonstrate a head-pointing approach based on head movements captured by the gyroscope and accelerometer sensors of commercial headphones. To improve pointing precision, we implement pointer snapping that leverages accessibility information from application user interfaces. This approach is independent of camera placement and lighting conditions, offers privacy advantages, and requires no specialized hardware beyond commonly used headphones, supporting more inclusive and accessible interaction. Our demo highlights how reusing existing accessibility infrastructure can support more inclusive pointing interactions and contribute to creating more accessible interactive systems.
https://www.youtube.com/watch?v=PHnWZzRSgjE
> This is an independent research and has not been authorized, sponsored, or otherwise approved by Apple Inc. The video contains depictions of Apple AirPods Max, a product of Apple Inc. Apple and AirPods Max are trademarks of Apple Inc. All trademarks, сopyrighted works, and designs appearing in this video are the property of their respective owners.
```bibtex
@unpublished{tarasenko-etal-2026-preprint-head-pointing-with-snapping,
author = {Mykhailo Tarasenko and Iryna Pastukhova and Oleksandr Frankiv and Anastasiia Satarenko and Nataliia Stulova and Sergii Kryvoblotskyi},
title = {Demonstrating Headphone-Sensed, Accessibility-Informed Head Pointing with Snapping for Inclusive Interaction},
note = {\emph{Preprint.} \url{https://research.macpaw.com/publications/head-pointing-with-snapping}},
month = {Jan},
year = {2026}
}
```
---
# Deep Language Geometry: Constructing a Metric Space from LLM Weights
**Authors**: Maksym Shamrai, Vladyslav Hamolia
**Conference**: RANLP 2025 Recent Advances in Natural Language Processing
**Published**: 2025-09-08
**Topics**: artificial-intelligence
**Type**: Paper
**URL**: https://research.macpaw.com/publications/digital_language_geometry
We introduce a novel framework that utilizes the internal weight activations of modern Large Language Models (LLMs) to construct a metric space of languages. Unlike traditional approaches based on hand-crafted linguistic features, our method automatically derives high dimensional vector representations by computing weight importance scores via an adapted pruning algorithm. Our approach captures intrinsic language characteristics that reflect linguistic phenomena. We validate our approach across diverse datasets and multilingual LLMs, covering 106 languages. The results align well with established linguistic families while also revealing unexpected inter-language connections that may indicate historical contact or language evolution.
```bibtex
@misc{shamrai-hamolia-2025-ranlp-deep-language-geometry,
title={Deep Language Geometry: Constructing a Metric Space from {LLM} Weights},
author={Maksym Shamrai and Vladyslav Hamolia},
month={September},
year={2025},
eprint={2508.11676},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2508.11676},
note={\emph{Accepted to RANLP'25} \url{https://arxiv.org/abs/2508.11676}},
}
```
---
# Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation
**Authors**: Viktor Muryn, Marta Sumyk, Mariya Hirna, Sofiya Garkot, Maksym Shamrai
**Published**: 2025-07-22
**Topics**: artificial-intelligence
**Type**: Paper
**URL**: https://research.macpaw.com/publications/screen2axvisionbasedapproachautomatic
Desktop accessibility metadata provides textual representations of applications that support both AI agents and users relying on screen readers. Our investigation shows that 33\% of macOS applications provide complete accessibility support. Yet, prior work on structured screen representation has not attempted to reconstruct the full hierarchical structure of desktop interfaces.
We present Screen2AX, the first framework that automatically generates real-time, tree-structured accessibility metadata from a screenshot. Screen2AX combines vision-language and object detection models to detect, describe, and organize UI elements hierarchically, mirroring macOS's system-level accessibility structure.
We compiled and publicly released three datasets covering 302 macOS apps, annotated with accessibility trees. Screen2AX reconstructs accessibility hierarchies with an F1 score of 79\% and substantially improves GUI grounding in complex desktop environments. We show that Screen2AX improves grounding accuracy by 2.2× over native accessibility metadata and outperforms state-of-the-art OmniParser V2 on the ScreenSpot benchmark from textual representations.
```bibtex
@misc{muryn2025screen2axvisionbasedapproachautomatic,
title={Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation},
author={Viktor Muryn and Marta Sumyk and Mariya Hirna and Sofiya Garkot and Maksym Shamrai},
year={2025},
eprint={2507.16704},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2507.16704},
}
```
---
# Measure Twice, Cut Once: the Best* Way to Calculate macOS File Hierarchy Size
**Authors**: Artem Levchenko, Oleksandr Frankiv, Yevhenii Peteliev
**Published**: 2025-05-19
**Topics**: software-engineering
**Type**: Paper
**URL**: https://research.macpaw.com/publications/disco
Efficient scanning and sizing of files and directories in the Apple File System (APFS) is critical for effective storage management and system monitoring.
This study investigates algorithms for measuring the size of APFS file hierarchies with a focus on speed, scalability, and resource efficiency. The proposed solution integrates three APFS access methods—NSFileManager, URLResourceKey, and the UNIX utility du—with four traversal strategies and three multithreading approaches (serial processing, Grand Central Dispatch, and Swift concurrency).
Experimental results on synthetic hierarchies show that URLResourceKey achieves scan times an order of magnitude shorter than NSFileManager and consistently outperforms du. Furthermore, multithreaded processing with Swift concurrency or GCD reduces execution time by 2–3× compared to serial methods. The use of traversal algorithms further improves scanning performance, enhances user experience, and enables tailored optimization for specific tasks.
```bibtex
@unpublished{levchenko-etal-2025-report-macos-dirsize,
author = {Artem Levchenko and Oleksandr Frankiv and Yevhenii Peteliev},
title = {Measure Twice, Cut Once: the Best* Way to Calculate macOS File Hierarchy Size},
note = {\emph{Preprint.} \url{https://research.macpaw.com/publications/disco}},
month = {May},
year = {2025}
}
```
---
# Mousetraps: Reliably distinguishing free apps and trial versions of paid macOS apps on public app marketplaces
**Authors**: Zakhar Litvinchuk, Oleksandr Frankiv, Yevhenii Peteliev, Sergii Kryvoblotskyi, Nataliia Stulova
**Published**: 2025-05-19
**Topics**: software-analysis
**Type**: Paper
**URL**: https://research.macpaw.com/publications/mousetraps
Transparency and clear labeling of macOS applications in public marketplaces are essential for fostering user trust and enabling informed decision-making.
This research focuses on distinguishing truly free applications from those containing in-app purchases and classifying them into free, freemium, and paid categories. The analysis combines technical and language-based approaches to identify key indicators of each class, followed by classification using a supervised machine learning algorithm.
Among the evaluated models, Random Forest and Histogram-Based Gradient Boosting demonstrated the highest performance, each achieving 90% precision, 90% recall, 89% F1-score, and 90% accuracy.
These findings support the development of automated app classification systems and hold practical value for enhancing transparency within the software distribution ecosystem.
```bibtex
@unpublished{litvinchuk-etal-2025-report-macos-paid-apps-analysis,
author = {Zakhar Litvinchuk and Oleksandr Frankiv and Yevhenii Peteliev and Sergii Kryvoblotskyi and Nataliia Stulova},
title = {Mousetraps: Reliably distinguishing free apps and trial versions of paid macOS apps on public app marketplaces},
note = {\emph{Preprint.} \url{https://research.macpaw.com/publications/mousetraps}},
month = {May},
year = {2025}
}
```
---
# SwiftEval: Developing a Language-Specific Benchmark for LLM-generated Code Evaluation
**Authors**: Ivan Petrukha, Yana Kurliak, Nataliia Stulova
**Conference**: FORGE 2025 The 2nd ACM International Conference on AI Foundation Models and Software Engineering
**Published**: 2025-03-04
**Topics**: artificial-intelligence
**Type**: Paper
**URL**: https://research.macpaw.com/publications/swift_eval
In recent years, large language models (LLMs) have showcased significant advancements in code generation. However, most evaluation benchmarks are primarily oriented towards Python, making it difficult to evaluate other programming languages, such as Swift, with high quality. By examining widely established multilingual benchmarks like HumanEval-XL and MultiPL-E, we identified critical issues specific to their Swift components, making them insufficient or even irrelevant for assessing LLM coding capabilities on Swift. Unlike these existing approaches, which prioritize rapid scaling and generalization by automatically translating Python-centric benchmarks with LLMs, we adopt a quality-over-quantity methodology. We present SwiftEval, the first Swift-oriented benchmark consisting of 28 carefully hand-crafted problems, and evaluate 44 popular Code LLMs on it. Our results show significant LLM scores drop for problems requiring language-specific features, most noticeable in the models of smaller sizes.
```bibtex
@misc{petrukha-etal-2025-forge-swifteval,
author = {Ivan Petrukha and Yana Kurliak and Nataliia Stulova},
title = {{SwiftEval}: Developing a Language-Specific Benchmark for {LLM}-generated Code Evaluation},
month = {March},
year = {2025},
eprint = {2505.24324},
eprinttype = {arxiv},
eprintclass = {cs.LG},
url = {https://arxiv.org/abs/2505.24324},
note = {\emph{Accepted to FORGE'25.} \url{https://arxiv.org/abs/2505.24324}},
}
```
---
# Towards Generating App Feature Descriptions Automatically with LLMs: the Setapp Case Study
**Authors**: Yevhenii Peteliev, Ivan Synytsia, Nataliia Stulova
**Conference**: FORGE 2025 The 2nd ACM International Conference on AI Foundation Models and Software Engineering
**Published**: 2025-03-04
**Topics**: artificial-intelligence, software-analysis, software-distribution
**Type**: Paper
**URL**: https://research.macpaw.com/publications/theseus
For any application to understand what it allows its users to do, we must rely on app functionality descriptions provided by software developers on app pages in marketplaces and in the release notes, the developer view or claimed features. User reviews and public discussions on thematic forums can serve as another source of information about app's features, and sometimes new features are inspired by such user view. However, little research has been done on app artifact analysis to distill actual high-level features, with researchers focusing on bytecode analysis to understand low-level app behaviors, such as API calls, without necessarily mapping those to features. Herein, we explore the possibilities of LLMs to reconstruct the app features and functionality descriptions from the (middle-level) app artifact information to bridge the perspective and knowledge gaps. We extract diverse unstructured text strings from 235 macOS app artifacts obtained from the Setapp app store and prompt the GPT-4o LLM for a list of possible feature descriptions, which we later compare with the human-written app's feature list in the app store. We observe minor differences in lexical structure in terms of part-of-speech counts, and the semantic similarity (cosine) score varies between 0.47–0.76 with GloVe embeddings and between 0.57–0.77 with BERT ones, meaning that even naive prompting can produce similar enough app feature descriptions w.r.t. the human-produced oracle. Our results show the potential of the LLM use for automatic/assisted app feature description generation in marketplaces and for contrasting the claimed and actual app behavior for detecting any discrepancies.
```bibtex
@misc{peteliev-etal-2025-forge-setapp-study,
author = {Yevhenii Peteliev and Ivan Synytsia and Nataliia Stulova},
title = {Towards Generating App Feature Descriptions Automatically with {LLM}s: the {Setapp} Case Study},
month = {May},
year = {2025},
note = {\emph{Accepted to FORGE'25} \url{https://research.macpaw.com/publications/theseus}},
}
```
---
# Case Study: Using EEG to Assess Cognitive Load in Infinite Interfaces
**Authors**: Anastasiia Satarenko
**Conference**: HCI 2025 27TH International Conference on Human-Computer Interaction
**Published**: 2025-01-14
**Topics**: human-computer-interaction
**Type**: Paper
**URL**: https://research.macpaw.com/publications/using-eeg-to-assess-cognitive-load
In the realm of Human-Computer Interaction, cognitive load plays a crucial role in determining user satisfaction, product acquisition, and user retention. We are introducing a new type of interface - Infinite Interfaces, that dynamically adapts to user inputs and, therefore, offers personalized and efficient user experiences.
However, the novelty of such interfaces may lead to increased cognitive effort, as most users are accustomed to traditional predetermined interfaces. This study seeks to evaluate the cognitive load associated with Infinite Interfaces using two approaches: subjective user ratings, which are simple and accessible, and EEG technology, providing an objective measure.
The research aims to compare cognitive load in Infinite Interfaces versus traditional interfaces, using both methods to identify any similarities or differences in the results. Results indicate that while novel interfaces like Infinite Interfaces initially induce higher cognitive load, they offer long-term potential for improving user efficiency in complex tasks.
```bibtex
@misc{satarenko-2025-hcii-cognitive-load-infinite-interfaces,
author = {Anastasiia Satarenko},
title = {Case Study: Using {EEG} to Assess Cognitive Load in Infinite Interfaces},
month = {January},
year = {2025},
note = {\emph{Accepted to HCII'25.} \url{https://research.macpaw.com/publications/using-eeg-to-assess-cognitive-load}},
}
```
---
# Datasets: Inside Mac App Store Applications Metadata
**Authors**: Iryna Pastukhova
**Published**: 2024-09-23
**Topics**: software-analysis
**Type**: Tech Note
**URL**: https://research.macpaw.com/publications/macappsstore-metadata-dataset
## Overview and Data Collection
This data aims to cover our internal company research needs and start collecting and sharing the macOS app dataset since we have yet to find a suitable existing one.
Full application metadata for over 87,000 samples was sourced by the public iTunes search API for the US, Germany, and Ukraine between December 2023 and January 2024.
For the convenience of further analysis, we present the data as three separate datasets.
## Metadata Dataset
The data is provided in its original form without any additional cleaning or transformations. It is organized into 43 columns ranging from such essential details as app names, descriptions, and genres, along with release and version information, user ratings, and asset links. All the data is grouped by the country API and is presented in three corresponding CSV files.
## Release Notes Dataset
This dataset is a combined and refined subset of the metadata dataset. The main idea behind its creation is to separate the release notes text of the macOS apps for further analysis.
Key fields related to release notes were selected, and entries were additionally classified by language using the [langdetect](https://pypi.org/project/langdetect/) library. The relevant fields include the app name, release date, current version release date, language, and the release notes themselves. The dataset primarily focuses on the release notes texts, so was additionally deduplicated by this attribute, resulting in over 24,000 apps.
The dominant languages are English (75%) and German (13%); the others have fewer than 600 entries (3%) each.
The data is organized in a single CSV file.
## Descriptions Dataset
By analogy with the Release Notes dataset, we also formed a separate one from the metadata dataset with app descriptions. The corresponding fields were selected, and each record was additionally assigned with the description language detected by the [langdetect](https://pypi.org/project/langdetect/) library so that the dataset is organized into five columns, including app name, bundle ID, ID, language, and description text. In the final step, the descriptions were deduplicated, resulting in almost 39,000 unique records.
The prevalent language is English (78%), followed by German (16%); the rest are less than 2% each.
The data is organized in a single CSV file.
## Links
- https://huggingface.co/datasets/MacPaw/mac-app-store-apps-metadata
- https://huggingface.co/datasets/MacPaw/mac-app-store-apps-release-notes
- https://huggingface.co/datasets/MacPaw/mac-app-store-apps-descriptions
- https://pypi.org/project/langdetect/
---
# R+R: State of the Application Sandboxing on macOS: A Differentiated Replication
**Authors**: Iryna Pastukhova, Nataliia Stulova, Ivan Synytsia
**Published**: 2024-05-29
**Topics**: software-analysis
**Type**: Paper
**URL**: https://research.macpaw.com/publications/sandbox
Application sandboxing is a common practice in mobile and desktop operating systems, employed to prevent unauthorized resource access and protect users from malicious apps. In the context of macOS, the second most widespread desktop operating system, little has been studied on the topic of application security outside of malware research. Blochberger et al. performed an exploratory study of two application marketplaces for macOS in 2019, comparing sandboxing adoption in apps from the official app marketplace and a third-party one.
While their study provides very useful insights both for researchers and practitioners, by design, it excludes open-source apps, a limitation we aim to address in our current work.
We conduct a differentiated replication study, extending the application dataset to include open-source apps. Given the time difference between the original study and ours, we also check if the original results hold in the newer macOS version.
We confirm the original observations on sandbox adoption practices, noticing a significant improvement in sandboxing mechanism adoption in the closed-source apps from a third-party store. We also demonstrate the similarity of the results between sandbox adoption in closed- and open-source applications.
Our work confirms the applicability of the original sandboxing analyses in different contexts, such as dataset or test OS environment changes. We also demonstrate that open-source applications could be included in datasets used for the Apple ecosystem analysis.
```bibtex
@unpublished{pastukhova-etal-2024-preprint-macos-sandbox,
author = {Iryna Pastukhova and Nataliia Stulova and Ivan Synytsia},
title = {State of the Application Sandboxing on {macOS}: A Differentiated Replication},
note = {\emph{Preprint.} \url{https://research.macpaw.com/publications/sandbox}},
month = {May},
year = {2024}
}
```
---
# Evaluating a multi-method approach for user emotional dynamics assessment in Software UX
**Authors**: Daria Polishchuk, Kateryna Filipchenko, Alina Khomenko
**Published**: 2024-03-27
**Topics**: human-computer-interaction
**Type**: Paper
**URL**: https://research.macpaw.com/publications/emotional-assessments-ux
User experience research plays a crucial role in software product development, focusing on user perceptions of the product and the emotions it invokes.
However, many methods for measuring emotions still remain subjective and can lack sufficient accuracy and objectivity.
We aim to address the subjectivity concern by proposing a multi-method user research approach, which could be applied in the context of interactions with software products and would be scalable and repeatable in remote user testing conditions.
We combine self-reporting, behavioral observation analysis, direct user speech, and AI-powered facial expression analysis.
We evaluate our method in two case studies with 15 participants, analyzing the emotional responses of users interacting with a Utility App and an App Marketplace, utilizing the Customer Journey Map (CJM) Framework for deeper insights into emotional dynamics shifts.
The analysis results indicate that, although AI analysis of emotions has limitations, the overall methodology partially correlates with observer analysis.
Both methodologies are more effective in reporting emotional downs, while self-reported data tends to show emotional shifts more boldly.
```bibtex
@unpublished{polishchuk-etal-2024-preprint-emotional-assessments-ux,
author = {Daria Polishchuk and Kateryna Filipchenko and Alina Khomenko},
title = {Evaluating a multi-method approach for user emotional dynamics assessment in Software {UX}},
note = {\emph{Preprint.} \url{https://research.macpaw.com/articles/emotional-assessments-ux/}},
month = {March},
year = {2024}
}
```
---
# Position Paper: Think Globally, React Locally — Bringing Real-time Reference-based Website Phishing Detection on macOS
**Authors**: Ivan Petrukha, Nataliia Stulova, Sergii Kryvoblotskyi
**Conference**: STAST'24, 14th International Workshop on Socio-Technical Aspects in Security, Affiliated with the 9th IEEE European Symposium on Security and Privacy
**Published**: 2024-03-26
**Topics**: security, artificial-intelligence
**Type**: Paper
**URL**: https://research.macpaw.com/publications/website-phishing-detection-macos
The recent surge in phishing attacks keeps undermining the effectiveness of the traditional anti-phishing blacklist approaches. On-device anti-phishing solutions are gaining popularity as they offer faster phishing detection locally.
We aim to eliminate the delay in recognizing and recording phishing campaigns in databases via on-device solutions that identify phishing sites immediately when encountered by the user, rather than waiting for a web crawler’s scan to finish. Additionally, utilizing operating system-specific resources and frameworks, we aim to minimize the impact on system performance and depend on local processing to protect user privacy.
We propose a phishing detection solution that uses a combination of computer vision and on-device machine learning models to analyze websites in real-time. Our reference-based approach analyzes the visual content of web pages, identifying phishing attempts through layout analysis, credential input areas detection, and brand impersonation criteria combination.
Our case study shows it’s feasible to perform background processing on-device continuously, for the case of the web browser requiring the resource use of 16% of a single CPU core and less than 84MB of RAM on Apple M1, while maintaining the accuracy of logo detection at 46.6% (comparable with baselines), and of Credential Requiring Page detection at 98.1% (improving the baseline by 3.1%), within the test dataset.
Our results demonstrate the potential of on-device, real-time phishing detection systems to enhance cybersecurity defensive technologies and extend the scope of phishing detection to more similar regions of interest, e.g., email clients and messenger windows.
```bibtex
@misc{petrukha-etal-2024-stast-extended-macos-phishing,
author = {Ivan Petrukha and Nataliia Stulova and Sergii Kryvoblotskyi},
title = {Position Paper: Think Globally, React Locally — Bringing Real-time Reference-based Website Phishing Detection on {macOS}},
month = {May},
year = {2024},
eprint = {2405.18236},
eprinttype = {arxiv},
eprintclass = {cs.LG},
url = {https://arxiv.org/abs/2405.18236},
urldate = {2025-05-30},
note = {\emph{Accepted to STAST'24.} \url{https://arxiv.org/abs/2405.18236}},
}
```
---
# Parsing macOS Application UI: Techniques and Tools for Effective Analysis and Automation
**Authors**: Ivan Synytsia
**Published**: 2024-02-15
**Topics**: software-analysis
**Type**: Tech Note
**URL**: https://research.macpaw.com/publications/how-to-parse-macos-app-ui
## Introduction
Parsing macOS application UI using code tools is a technique that allows developers to extract information from the graphical user interface (GUI) of different apps, such as the layout and interface elements (buttons, labels, text fields, images). This operation can be helpful for various purposes:
1. Testing:
- Workflow automation to find regressions
- Input/output data validation
2. Debugging:
- Validate real-time UI layout issues and check off-screen rendering and component frames
- Localization errors
3. Automation:
- Validate UI consistency and scale across multiple platforms
- Batch Processing of multiple elements
4. Analysis
- Data extraction for external tools
- UI Analysis for UX experts and researchers
In the research, we will explore how to parse macOS application UI using code tools and compare the advantages and disadvantages of different methods. We will also show examples of using various tools for parsing macOS application UI and extracting useful information. By the end of this article, you will better understand how to perform parsing and have an example of applying this technique to generic scenarios.
## Possible ways to perform parsing
If there is access to the source code, it is pretty trivial to work with the window structure of an app, and there are techniques to parse `Xib` or `SwiftUI` files. If we want to explore a distributed app with a GUI, more sophisticated approaches must be used.
### Appium framework
There is a well-known open-source test automation tool, Appium, which facilitates UI automation of all modern platforms (Web, macOS, Windows, iOS, and Android). It also supports code written in different languages (JavaScript, Java, Python, etc.) The tool provides a cross-platform API for interacting with platform-specific drivers and performing tests. For macOS and iOS testing, Appium uses the XCUITest framework from Apple, which has various possibilities to interact with apps and can fetch UI layout and perform actions if available on the components.
Appium installation can be done by using the command in the Terminal app. The Node package manager (`NPM`) is the fastest way to do this is to perform the following installation steps:
1. Install Brew
```bash
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
```
2. Install NodeJS
```bash
brew install node
```
3. Install Appium
```bash
npm i -g appium
```
#### Configure and use Appium
The Appium tool API provides the driver for macOS to allow launching an application using bundle ID, listening to a specific IP and port, and handling timeouts. Before launching any tests, driver configuration should be done, and the CLI or GUI Appium version must be launched for handling requests. To get an entire application structure, the following code is used:
```python
@pytest.fixture()
def driver():
options = Mac2Options()
options.bundle_id = application_bundle_id
drv = webdriver.Remote("http://127.0.0.1:4723", options=options)
yield drv
drv.quit()
def test_edit_text(driver):
source = driver.page_source
print(source)
```
As a result, `print()` will provide a list of parsed UI components in the XML format:
```xml
```
The TextEdit app has a window with an "Untitled" title, a content scroll view, close, full screen, and minimize buttons, a menu bar, and touch bar items. To perform the analysis, we can recursively check all child elements of the window and try to fetch essential data like type, title, position, size, etc. These components allow us to build the window structure in code. Also, Appium can provide a screenshot for the parsed components in base64 or binary formats. A sample script that can run Appium tests is included in the attached archive.
### XCText framework
Appium exploits a framework provided by Apple that can create and run unit, performance, and UI tests from Xcode or the terminal. The easiest way to run tests is to install Xcode from the Mac App Store or [Apple Developer Portal](https://developer.apple.com).
#### Installation and usage
No additional installation steps are required to install the framework. It can be used by creating a test target in the Xcode project and adding test cases for each task. Below is an example of getting a complete window structure in a text format. A trivial iteration can do further element processing through the child elements.
```swift
import XCTest
final class ApplicationParserUITests: XCTestCase {
func testAppStructure() throws {
let app = XCUIApplication(bundleIdentifier: "com.apple.TextEdit")
app.launch()
let windows = app.windows
for i in 0.. **Important:**
> Using the Accessibility API system requires granting permissions for the application before launching any code or scripts:
>
> System Settings -> Privacy & Security -> Accessibility -> "An App"
The Accessibility API can be used from the Xcode or the Python package `ApplicationServices`, which has a bridge to system calls and provides the same functionality as Xcode. The API is simple and straightforward.
A general algorithm for extracting UI element structure is displayed below:
```python
import ApplicationServices
import AppKit
func parse_window():
# get the shared workspace
workspace = AppKit.NSWorkspace.sharedWorkspace()
# search for launched app
app_bundle_id = "com.apple.TextEdit"
running_app = search_for_running_app(workspace, app_bundle_id)
# search for all the windows
windows = windows_for_application(application)
# get window children
for window in windows:
err, children = ApplicationServices.AXUIElementCopyAttributeValue(
window, kAXChildrenAttribute, None
)
# serialize UI structure in JSON
json_string = store_elements_in_json(children)
# write JSON to file
write_json_file(json_string)
```
The complete example of application UI parsing and extracting to a file is attached to the `prototype.zip` archive.
## Tricky case
On macOS, applications are built using AppKit, SwiftUI, Electron, Atom, Flutter frameworks, etc. These various implementations can affect the structure of user-interface elements, and sometimes, the Accessibility API does not provide a correct hierarchy while fetching property value by using the attribute `kAXChildrenAttribute.` This behavior can be reproduced using all the provided tools, Appium, XCTest, and Accessibility subsystem. Let us check the ClearVPN app UI structure:
In the image, an additional layer is added by `NSHostingView`, which is not reflected in the structure fetched for the window. Other children's elements are placed on the hidden layer; none of the mentioned tools can track this layer out of the box. If fetching children from the root elements is not helpful, we can find any child element and then go back to the root. The hit-test technique was used to resolve the issue. Accessibility API has the option to obtain an element that is located under a provided point. After obtaining the element, we can return to the root element and find a window container.
One of the top elements can be retrieved by using the next Accessibility API calls:
```python
component = ApplicationServices.AXUIElementCreateSystemWide()
err, value = ApplicationServices.AXUIElementCopyElementAtPosition(component, 0, 50, None)
```
Here, we fetch an accessibility object that provides access to system attributes and get a single top element by coordinates. After retrieving an element, we must find a root window or `AXHostingView` container. This container contains all his children required for parsing the structure.
To complete a parsed structure, we can combine regular and hit-test parsing into one flow and then export all the data to a suitable format like JSON or XML.
## Conclusion
Application window parsing is expected to be a regular task, and the practice shows that the task is not very trivial because of the various technologies used for implementation. There are multiple frameworks for building apps where UI might be constructed using native or web technologies. For performed tasks, there are also multiple tools for analyzing UI structure, and Apple provides the most powerful ones, bundled with Xcode and its operating system. An output of the solution can be used to understand available actions for apps from UI representation and interaction with its components. These results may be consumed by third-party applications, neural networks, or automation, provide simplified flows for users, and perform complex automation flows.
## Links
- [Appium](http://appium.io)
- [Brew](https://brew.sh)
- [NodeJS](https://nodejs.org/en)
- [Apple Developer Portal](https://developer.apple.com)
- [Accessibility](https://support.apple.com/guide/mac-help/mobility-mchl654a7cc7/mac)
- [Accessibility Inspector](https://developer.apple.com/documentation/accessibility/accessibility-inspector)
- [Xcode](https://developer.apple.com/xcode/)
- [Application Services API](https://developer.apple.com/documentation/applicationservices)
---
# AI Opportunities in Operating Systems
**Authors**: Iryna Pastukhova
**Published**: 2024-02-05
**Topics**: software-engineering, artificial-intelligence
**Type**: Highlight
**URL**: https://research.macpaw.com/publications/ai-in-os
## Understanding AIOS
Today, AI is everywhere, and you won't surprise anyone with one more AI assistant that not only answers questions but also takes some actions for you. Moreover, assistants are supposed to offer task completions and suggestions even before you ask them. On the one hand, assistants are just programs; on the other, agents, that is, besides doing some tasks, they can behave adaptively and react to environmental changes, providing the best result under the given conditions.
And what about the operating system (OS)? Being the platform for running software, it is also software itself and, consequently, could be an AI-boosted agent.
According to modern understanding, AI is composed of the following six disciplines:
- [Natural language processing](https://en.wikipedia.org/wiki/Natural_language_processing) (NLP) communicating in human language
- [Knowledge representation](https://en.wikipedia.org/wiki/Knowledge_representation_and_reasoning) storing what it knows
- [Automated reasoning](https://en.wikipedia.org/wiki/Automated_reasoning) focusing on developing algorithms and systems capable of making logical inferences and drawing conclusions automatically
- [Machine learning](https://en.wikipedia.org/wiki/Machine_learning) (ML), acquiring knowledge from data and adapting to new circumstances
- [Computer vision](https://en.wikipedia.org/wiki/Computer_vision) and [Speech recognition](https://en.wikipedia.org/wiki/Speech_recognition) perceiving the world
- [Robotics](https://en.wikipedia.org/wiki/Robotics) performing physical tasks in the real world
Thus, the AI operating system should be a combination of the above features and still do the needed OS tasks, like managing hardware and software resources and providing services for them.
Searching for information about AI operating systems reveals two possible approaches to treating this term: an OS with at least one AI agent or a well-optimized platform for running AI algorithms efficiently. The topic of our interest is an OS, the main features of which are managed by AI. In other words, we're interested in AI running the OS rather than the OS running the AI.
This investigation was inspired by the relatively recent paper ['AI based OS'](https://www.ijtrd.com/papers/IJTRD22752.pdf), in which the idea of an AI operating system (AIOS) is raised with the following main features:
- Monitoring and optimizing its own performance
- Smart process and memory management
- Predicting results
- Using the right resources
- Making sure that resources are available by predicting before a process asks for them
- Suggesting alternative options to a problem
- Search internally and externally for information to help themselves
- Deriving a solution to a problem by itself
- Self-healing and getting immune to viruses after an attack
- Communicating with other OS
## Current State of AIOS
One of the latest results on the topic proposes the idea of a large language model (LLM) as an OS. Relying on the most recent advancements in AI, especially the capabilities of LLMs, the authors introduce the concept of the [LLMOS](https://arxiv.org/pdf/2312.03815.pdf) - the operating system with the LLM 'soul' and argue its relevance by establishing analogies with the traditional OS.
In particular, sharing the LLM backbone between multiple users and processes requires scheduling much like CPU management; handling the limitations of the model's context window corresponds to memory management, and accessing the external data storage - to storage management. Agents could play the role of applications, libraries, and SDKs, natural language prompts correspond to system calls, and some pre-defined prompts provide an analogy for the standard CLI commands in traditional OS. GUI will convert different user interactions into prompts understandable by LLM.
LLM can also learn from the feedback of its actions (reinforcement learning) or the examples (supervised learning) and, thus, permanently improve, similar to how the kernel is updated based on bug reports and performance issues.
A somewhat more concrete theorization on the topic is [F4OS](https://arxiv.org/pdf/2312.07813.pdf). The authors insist on using a foundation OS model (F4OS) capable of managing all of the OS responsibilities instead of using existing partial task-specific AI solutions: the strong inter-dependence of OS components implies the non-optimality of fine-tuning them separately. In particular, the approach relies on the fact that every modern OS collects the logs from its components, hardware, applications, etc., thus automatically providing not only the training data but potentially capturing the relationships between OS processes. As an example, the case of scheduling and cache replacement tasks is considered. Along with the OS traces for each of them, there are naturally gathered traces of their relationship since, for example, the process completion times depend on resources and, particularly, the cache.
The foundation model approach is also actively popularized and investigated in the Cloud Systems field. Such systems can be compared to OS in complexity and have many similar tasks; consequently, the ideas can be potentially reused and shared.
Although the ideas of the universal (foundation) model, controlling and managing OS responsibilities sound great, they are mostly just theoretical envisions rather than concrete implementations. The applications of AI to single tasks of OS, such as network managing, memory managing, CPU scheduling, etc., seem ‘more alive’ and have, if not production-ready implementations, at least working prototypes.
Many such results are related to tuning some OS parameters. For example, storage components in the OS work with heuristic algorithms and allow tuning some of their parameters via system calls. There is [the prototype](https://dl.acm.org/doi/pdf/10.1145/3465332.3470875) of the ML framework for the memory management system’s performance improvement. In particular, it solves the problem of adaptively setting the heuristics values for `readahead`. `Readahead` is a technique that reduces the latency associated with accessing data from storage by bringing the required information into the system's memory before an application or process actually requests it. Obviously, its effectiveness is highly dependent on the nature of the workload and access patterns, making it and similar tasks perfect candidates for applying AI-based solutions.
Also, there are investigations into solving the potential problems of the LLM-based OS. The very obvious one is context window limitation (the constraint on the number of preceding words or tokens that the model considers when predicting the next word in a sequence), which must play the 'Memory' role in such an OS. The proposed [solution](https://arxiv.org/pdf/2310.08560.pdf) in the form of virtual context management is inspired by the hierarchical memory structure in traditional OS. The LLM is complemented with 'LLM OS', providing it with the illusion of unbounded context: by analogy with a traditional OS, there is the main context corresponding to main memory, RAM, and external, corresponding to disk storage, and function calls allowing to move data between them. The LLM processor takes the main context and outputs a result parsed to a yield or function call.
## Conclusions
For now, the AIOS is definitely a promising direction for making theoretical and scientific investigations. However, the concrete realization remains an open question and seems to be too far from being answered: the recent publications and conference talks state only envisions of such systems. Most of them are based on the LLMs capabilities and assume natural language as a key communication channel in user-machine interaction.
Obviously, there are a lot of potential risks of using such systems. One of the main shortcomings of LLMs is their 'black-boxiness' - it is hard to predict and control their intrinsic state and behaviour. In some sense, giving the model full control over the OS could be analogous to putting a self-driving car on the road without a driver. Thus, it seems like some additional external control still may be needed.
All the concepts suggest that AIOS should be highly customizable for a concrete user by permanent training itself, which is indeed a must feature. But will it be able to re-train itself quickly enough once the user behaviour changes drastically? Additionally, training huge and powerful models usually requires a lot of specific resources; it is not clear if the usual machine could manage it.
Although most of the ideas rely on the concept of OS as a general artificial intelligence and seem to be unachievable yet, integrating AI solutions for some specific tasks is definitely worth considering. As for now, the agents are already successful in API manipulation; accessing the internet resources; writing and executing code, testing, and finding system issues like race conditions; simulating behaviour, etc. The main challenges are adaptivity to environmental changes and enough data. The first can be ensured with Machine learning, especially Reinforcement learning methods. The need for training data is also covered by using OS traces simulated by running automated common usage workflows.
## Links
- [Artificial Intelligence: A Modern Approach, 4th US ed.](https://aima.cs.berkeley.edu/)
- [AI based OS](https://www.ijtrd.com/papers/IJTRD22752.pdf)
- [AI is about to completely change how you use computers](https://www.gatesnotes.com/AI-agents)
- [A Machine Learning Framework to Improve Storage System Performance](https://dl.acm.org/doi/pdf/10.1145/3465332.3470875)
- [LLM as OS, Agents as Apps: Envisioning AIOS, Agents and the AIOS-Agent Ecosystem](https://arxiv.org/pdf/2312.03815.pdf)
- [MemGPT: Towards LLMs as Operating Systems](https://arxiv.org/abs/2310.08560)
- [On a Foundation Model for Operating Systems](https://arxiv.org/pdf/2312.07813.pdf)
- [On the Promise and Challenges of Foundation Models for Learning-based Cloud Systems Management](https://mlforsystems.org/assets/papers/neurips2023/paper2.pdf)
- [Operating system](https://en.wikipedia.org/wiki/Operating_system)
---
# Collecting a Dataset of macOS Apps: Pains, Gains, Lessons Learned. Part 1
**Authors**: Sergii Kryvoblotskyi, Nataliia Stulova
**Published**: 2024-01-31
**Topics**: software-analysis
**Type**: Tech Note
**URL**: https://research.macpaw.com/publications/macos-app-dataset
## Introduction
The desktop OS share of the macOS operating system keeps steadily increasing, approaching 1/5, according to [statcounter.com](https://gs.statcounter.com/os-market-share/desktop-tablet/worldwide/#monthly-200901-202401) data:
At the same time, the academic research on this part of the Apple ecosystem remains scarce, as noted by researchers in the macOS forensics [12] and [13] malware analysis areas.
More specifically, prior work involving app dataset collection is limited to the security domain, specifically malware detection and analysis techniques based on machine learning and vulnerability analyses.
This contrasts sharply with the situation in security research for iOS [10] or Android [1], where large-scale app datasets are available and reused.
To overcome this issue, we collect an open and diverse dataset of macOS application bundles from self-hosted distribution platforms (i.e., outside MAS and excluding system binaries and default pre-installed apps).
## Fantastic apps and where to find them
In malware research, the datasets include a mix of malware and benignware binaries, and the sources of benign apps vary greatly:
- in [5], authors collect 1000 macOS app binaries from OS X Mavericks and open-source software (OSS).
- in [11], the authors collect 460 app binaries from the [Mac App Store](https://www.apple.com/app-store/) (MAS).
- in [6], authors work with 461 app binaries from MAS and the Homebrew package manager.
- in [3], authors also use 853 binaries from the `/usr/bin/` system directory, MAS, and the [Softonic](https://en.softonic.com/mac).
The body of work on vulnerability research considers larger app datasets, and in contrast with malware studies, authors collect not just the binaries but the [whole application bundles](https://developer.apple.com/library/archive/documentation/CoreFoundation/Conceptual/CFBundles/BundleTypes/BundleTypes.html#//apple_ref/doc/uid/10000123i-CH101-SW13):
- in [4], authors collect 1612 apps from MAS, and
- in [2], authors collect 13038 apps from MAS and the [MacUpdate](https://www.macupdate.com/) platform.
To the best of our knowledge, only the dataset of [11] is used in subsequent work [7, 8, 9].
Factors that affect dataset scope (that we and previous studies had faced):
- regional accessibility, as some apps are only present in specific country markets
- free vs paid app versions — some paid apps are not accessible without purchase
## Collecting the data
In our work, we consider self-hosted distribution platforms explored in the previous work or listed at [alternativeto.net](https://alternativeto.net/software/mac-app-store/?platform=mac), and exclude Steam (game-specific), Setapp (only has paid apps and has <300 apps), MacApps (has <300 apps), Homebrew/MacPorts/Nix (only have CLI binaries), and AppAgg (only has links to MAS).
Furthermore, we exclude Softonic, as it is similar to MacUpdate but has fewer apps.
We select GitHub over SourceForge to get macOS bundles, as the repositories of the former provide compiled applications as ready-to-download assets.
We finally chose a [curated OSS app list on GitHub](https://github.com/serhii-londar/open-source-mac-os-apps/tree/master), [Homebrew Cask](https://formulae.brew.sh/cask/), and [MacUpdate](https://www.macupdate.com/) as app source platforms.
We implement a Python script for automated app retrieval via API access (GitHub, Homebrew Cask) or website crawling (MacUpdate), and we use Google Cloud Bucket to store the full dataset.
We collect 12095 free/freemium/trial app archives (ZIP or DMG), totaling over 800 GB of storage.
In the case of GitHub, we also fetch TAR archives.
### App categories
In some cases (Table 1, row “Bundle N/A”), we fail to extract and analyze the app bundle due to the broken or missing APP files or the DMG archives being protected or requiring explicit license approval from the user.
|| GitHub | Homebrew Cask | MacUpdate |
|-|-|-|-|
| Size | 393 | 4270 | 7432 |
| Bundle N/A | 12 (2.44%) | 717 (19.1%) | 2616 (42.49%) |
| Top-3 Categories | dev-tools (25%), utilities (21.75%), productivity (14.43%) | utilities (14.73%), dev-tools (14.7%), productivity (10.12%) | utilities (11.99%), productivity (6.82%), dev-tools (4.97%) |
We observe that apps in the dev-tools, utilities, and productivity categories constitute 1/5 to 1/2 of the dataset, a significant proportion not previously reported in the literature.
### Bundle sizes
For bundles extracted successfully, we report their size in MB:
| Dataset | 0–5 | 5–50 | 50–100 | 100–500 | 500+ | N/A |
| ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |
| **GitHub** | 11.99% | 38.82% | 5.89% | 38.62% | 2.24% | 2.44% |
| **Brew Cask** | 9.08% | 25.15% | 8.82% | 31.67% | 6.18% | 19.1% |
| **MacUpdate** | 11.4% | 24.87% | 6.63% | 12.61% | 2.0% | 42.49% |
At least 1/3 of the bundles have <50 MB in size, suggesting a fair share of native low-resource apps with little framework use.
### Update frequency
For bundles extracted successfully, we report their size and last update date from the `Info.plist` file.
| Dataset | 30 days | 6 months | 1 year | 3+ years | N/A |
| ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |
| **GitHub** | 59.55% | 10.98% | 5.89% | 21.14% | 2.44% |
| **Brew Cask** | 41.08% | 14.94% | 6.39% | 18.49% | 19.1% |
| **MacUpdate** | 25.79% | 5.7% | 3.35% | 22.67% | 42.49% |
While at least 1/3 of the apps are maintained up-to-date across all datasets, almost 1/5 of the apps in two of our datasets might represent abandonware.
## Peek into the data
### GitHub
```json
{
"short_description": "App for macOS with a minimalistic UI which lets you quickly throttle down the CPU usage of any running process. ",
"categories": [
"system"
],
"repo_url": "https://github.com/AppPolice/AppPolice",
"title": "AppPolice",
"icon_url": "",
"screenshots": [
"https://cloud.githubusercontent.com/assets/1557716/12860558/11908a78-cc66-11e5-9998-b4bec11dbfeb.png",
"https://cloud.githubusercontent.com/assets/1557716/12860551/ffff72d8-cc65-11e5-9304-4f1341657b5a.png",
"https://cloud.githubusercontent.com/assets/1557716/12860559/1193fe42-cc66-11e5-9d4f-8b8af842ea72.png",
"https://cloud.githubusercontent.com/assets/1557716/12860549/fdffd054-cc65-11e5-8405-cc224ea4ab3b.png",
"https://cloud.githubusercontent.com/assets/1557716/12860557/118f5fcc-cc66-11e5-8822-dc85cbe7bbb9.png"
],
"official_site": "",
"languages": [
"objective_c"
]
}
```
### Homebrew Cask
```json
{
"token": "1password",
"full_token": "1password",
"old_tokens": [],
"tap": "homebrew/cask",
"name": [
"1Password"
],
"desc": "Password manager that keeps all passwords secure behind one password",
"homepage": "https://1password.com/",
"url": "https://downloads.1password.com/mac/1Password-8.10.20-x86_64.zip",
"url_specs": {},
"appcast": null,
"version": "8.10.20",
"installed": null,
"installed_time": null,
"outdated": false,
"sha256": "4fe7f5f50fe9cec0ba961f0e17fc3fa7344b05f01d964d60c10cfc7ee9e4c7c4",
"artifacts": [
{
"app": [
"1Password.app"
]
},
{
"zap": [
{
"trash": [
"~/Library/Application Scripts/2BUA8C4S2C.com.1password*",
"~/Library/Application Scripts/2BUA8C4S2C.com.agilebits",
"~/Library/Application Scripts/com.1password.1password-launcher",
"~/Library/Application Scripts/com.1password.browser-support",
"~/Library/Application Support/1Password",
"~/Library/Application Support/Arc/User Data/NativeMessagingHosts/com.1password.1password.json",
"~/Library/Application Support/com.apple.sharedfilelist/com.apple.LSSharedFileList.ApplicationRecentDocuments/com.1password.1password.sfl*",
"~/Library/Application Support/CrashReporter/1Password*",
"~/Library/Application Support/Google/Chrome Beta/NativeMessagingHosts/com.1password.1password.json",
"~/Library/Application Support/Google/Chrome Canary/NativeMessagingHosts/com.1password.1password.json",
"~/Library/Application Support/Google/Chrome Dev/NativeMessagingHosts/com.1password.1password.json",
"~/Library/Application Support/Google/Chrome/NativeMessagingHosts/com.1password.1password.json",
"~/Library/Application Support/Microsoft Edge Beta/NativeMessagingHosts/com.1password.1password.json",
"~/Library/Application Support/Microsoft Edge Canary/NativeMessagingHosts/com.1password.1password.json",
"~/Library/Application Support/Microsoft Edge Dev/NativeMessagingHosts/com.1password.1password.json",
"~/Library/Application Support/Microsoft Edge/NativeMessagingHosts/com.1password.1password.json",
"~/Library/Application Support/Mozilla/NativeMessagingHosts/com.1password.1password.json",
"~/Library/Application Support/Vivaldi/NativeMessagingHosts/com.1password.1password.json",
"~/Library/Containers/2BUA8C4S2C.com.1password.browser-helper",
"~/Library/Containers/com.1password.1password*",
"~/Library/Containers/com.1password.browser-support",
"~/Library/Group Containers/2BUA8C4S2C.com.1password",
"~/Library/Group Containers/2BUA8C4S2C.com.agilebits",
"~/Library/Logs/1Password",
"~/Library/Preferences/com.1password.1password.plist",
"~/Library/Preferences/group.com.1password.plist",
"~/Library/Saved Application State/com.1password.1password.savedState"
]
}
]
}
],
"caveats": null,
"depends_on": {
"macos": {
">=": [
"10.15"
]
}
},
"conflicts_with": {
"cask": [
"homebrew/cask-versions/1password-beta",
"homebrew/cask-versions/1password-nightly"
]
},
"container": null,
"auto_updates": true,
"tap_git_head": "c256ad3460b28b2b41e46045fb7871406c46b0c5",
"languages": [],
"ruby_source_path": "Casks/1/1password.rb",
"ruby_source_checksum": {
"sha256": "f761848b5482f70c4df09158ac756c31f13bce989aa0f490956d738878521a3a"
},
"variations": {
"arm64_sonoma": {
"url": "https://downloads.1password.com/mac/1Password-8.10.20-aarch64.zip",
"sha256": "9266a5e707ea08a02583b73ec551ccdfdaabb9cafe29ac6be2934163029fb398"
},
"arm64_ventura": {
"url": "https://downloads.1password.com/mac/1Password-8.10.20-aarch64.zip",
"sha256": "9266a5e707ea08a02583b73ec551ccdfdaabb9cafe29ac6be2934163029fb398"
},
"arm64_monterey": {
"url": "https://downloads.1password.com/mac/1Password-8.10.20-aarch64.zip",
"sha256": "9266a5e707ea08a02583b73ec551ccdfdaabb9cafe29ac6be2934163029fb398"
},
"arm64_big_sur": {
"url": "https://downloads.1password.com/mac/1Password-8.10.20-aarch64.zip",
"sha256": "9266a5e707ea08a02583b73ec551ccdfdaabb9cafe29ac6be2934163029fb398"
}
}
}
```
### MacUpdate
```json
{
"id": 27559,
"group_id": 27559,
"title": "123 Flash Chat",
"title_slug": "123-flash-chat",
"seo_title": null,
"description": "
Note: 123 Flash Chat is no longer under development, and it is no longer available for download.
\r\n
123 Flash Chat can add a chat room to website in minutes. It is a live chat solution for networking with multiple skins & chat clients,such as iPhone Chat,Avatar Chat. Other features: New Post Notifier; enhanced HTML Chat supports iPad.
",
"short_description": "Flash chat client.",
"version": "10.0",
"date": {
"date": "2013-11-05T13:40:13+00:00",
"timestamp": 1383658813
},
"requirements": {
"minimum_os": "Mac OS X 10.1.5",
"supports_macos": true,
"supports_ios": false,
"is_64bit": true,
"is_32bit": true,
"architectures": [
"Intel 64",
"Intel 32",
"PPC 64",
"PPC 32"
],
"other_list": [],
"other": null
},
"file_size": 127590,
"release_notes": "Version 10.0: \r\n \r\n Note: Please be aware that all the changes are on the HTML client. \r\n
\r\n\t
Add hand-raising function
\r\n
\r\n\t
Regular users can apply to queue for broadcasting video once it is enabled
\r\n\t
Admins can approve/disapprove users' request to broadcast video with hand-raising mode
\r\n
\r\n\t
Push to talk mode is enabled in Audio/Video settings
\r\n\t
Media Player is enabled
\r\n
\r\n\t
Mp3/mp4/ogg/webm,etc. are supported
\r\n\t
Youtube is supported
\r\n\t
Pause/Play function is realized
\r\n\t
Volume control is realized
\r\n\t
Add full screen toggle to the media player, and each video-window can be minimized or maximized.
\r\n\t
Repeating/previous/next is realized for the playlist
\r\n\t
Auto resizing is realized, media player will be minimized when height is dwindled to a set value
\r\n
\r\n\t
YouTube message is supported, media player window will be displayed in the chat area which can be popped up or folded
\r\n\t
New Web version design
\r\n
\r\n\t
UI is changed
\r\n\t
Current Skins are polished
\r\n\t
A new light red skin is added
\r\n
\r\n\t
Access configuration is supported for moderated-chat mode. Admin's access to enable/disable moderated-chat mode is supported
\r\n\t
Access configuration is supported for silent mute function. Admin's access to enable/disable silent mute function is supported
\r\n\t
IP Range Ban function
\r\n
\r\n\t
Admin can ban a specific IP range
\r\n\t
User will still stay in the current room even if his IP range is banned
\r\n\t
User cannot come back once he has exited the room by himself after his IP range was banned
\r\n
\r\n\t
Clear screen for all users function. Admin can clear the screen for all users
\r\n\t
User list position can be changed in the admin panel, right or left
\r\n
",
"bundle_identifiers": [],
"price": {
"value": 19900,
"currency": "USD"
},
"download_count": 5046,
"download_url": {
"type": "download_file",
"url": "http://www.123flashchat.com/dl/v100/123flashchat_s.tar.gz"
},
"purchase_url": "",
"mudesktop_url": "mudesktop://download/27559/123flashchat-s-tar-gz/10.0/123flashchat_s.tar.gz/1383658813/",
"video": [],
"logo": {
"id": 131927,
"source": "https://dl2.macupdate.com/images/icons256/27559.png",
"s_webp": null,
"s_png": null,
"m_webp": null,
"m_png": null,
"l_webp": null,
"l_png": null,
"created": {
"date": "2019-09-12 15:28:14.000000",
"timezone_type": 1,
"timezone": "+00:00"
},
"updated": {
"date": "2019-09-12 15:28:14.000000",
"timezone_type": 1,
"timezone": "+00:00"
}
},
"screenshots": [
{
"id": 1174505,
"source": "https://screenshots.macupdate.com/JPG/27559/27559_1569584753_scr.jpg",
"s_webp": null,
"s_png": null,
"m_webp": "https://static.macupdate.com/screenshots/221310/m/123-flash-chat-screenshot.webp?v=1571060627",
"m_png": "https://static.macupdate.com/screenshots/221310/m/123-flash-chat-screenshot.png?v=1571060628",
"l_webp": null,
"l_png": null,
"created": {
"date": "2019-10-14 13:43:48.000000",
"timezone_type": 1,
"timezone": "+00:00"
},
"updated": {
"date": "2019-10-14 13:43:48.000000",
"timezone_type": 1,
"timezone": "+00:00"
}
}
],
"rating": 0.5,
"review_count": 3,
"rate_count": 1,
"license": "Demo",
"vendor": null,
"computed_rank": 748,
"category": {
"id": 10,
"parent_id": null,
"parent_name": null,
"slug": "internet-utilities",
"name": "Internet Utilities",
"description": "Browse our curated collection of messengers, email clients, file managers, and other apps to simplify your online experience.",
"url": "https://www.macupdate.com/explore/categories/internet-utilities",
"children": null
},
"subcategory": {
"id": 160,
"parent_id": 10,
"parent_name": "Internet Utilities",
"slug": "internet-utilities/messengers",
"name": "Messengers",
"description": "Send instant messages or make video calls with fast and secure chat apps.",
"url": "https://www.macupdate.com/explore/categories/internet-utilities/messengers",
"children": null
},
"unsupported": true,
"redirect": null,
"faq": null,
"learn_more": null,
"is_following": null,
"is_mud_install_available": false,
"versions": null,
"last_scan": null,
"monetization": null,
"member_rating": null,
"member_review": null,
"review_report": {
"rating": [],
"label_positive": [],
"label_negative": []
},
"developer": {
"name": "Topcmm Computing Inc.",
"url": "http://www.123flashchat.com/",
"support": "",
"email": "info@123flashchat.com"
},
"nofollow": [],
"is_hidden": 0
}
```
## Conclusions
Our method of collecting macOS apps does not suffer from any regional restrictions imposed by the app stores and results in 12k items.
There are several ways to further improve it, starting with a broader selection of OSS macOS apps to reduce possible biases of the currently used curated collection from GitHub.
We would also like to complement our dataset with MAS data and verify if we would get similar results to those of [2] when studying the dataset overlap.
## Links
- [1] Allix, Kevin et al. "AndroZoo: collecting millions of Android apps for the research community." *Proceedings of the 13th International Conference on Mining Software Repositories*, 2016, pp. 468-471. ACM, https://doi.org/10.1145/2901739.2903508
- [2] Blochberger, Maximilian et al. "State of the Sandbox: Investigating macOS Application Security." *Proceedings of the 18th ACM Workshop on Privacy in the Electronic Society*, 2019, pp. 1150-161. ACM, https://doi.org/10.1145/3338498.3358654
- [3] Bumanglag, Kimo. "An Application of Machine Learning to Analysis of Packed Mac Malware." Ph. D. Dissertation, Dakota State University, 2022, https://scholar.dsu.edu/cgi/viewcontent.cgi?article=1382&context=theses
- [4] Xing, Luyi et al. "Cracking App Isolation on Apple: Unauthorized Cross-App Resource Access on MAC OS." *Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security*, 2015, pp. 31-43. ACM, https://doi.org/10.1145/2810103.2813609.
- [5] Walkup, Elizabeth. "Mac Malware Detection via Static File Structure Analysis." *University of Stanford*, 2014, https://cs229.stanford.edu/proj2014/Elizabeth%20Walkup,%20MacMalware.pdf.
- [6] Burgardt, Caio Augusto Pereira. "Malware detection in macOS using supervised learning." M. Sc. Thesis, Universidade Federal de Pernambuco, 2022, https://repositorio.ufpe.br/handle/123456789/46235
- [7] Chen, Alex Chenxingyu and Wulff, Kenneth. "Machine learning for OSX malware detection." *Handbook of Big Data Analytics and Forensics*, pp. 209-222, 2022. Springer
- [8] Gharghasheh, Samira Eisaloo and Hadayeghparast, Shahrzad. "Mac OS X Malware Detection with Supervised Machine Learning Algorithms." *Handbook of Big Data Analytics and Forensics*, pp. 193-208, 2022. Springer
- [9] Sahoo, Dilip and Dhawan, Yash. "Evaluation of supervised and unsupervised machine learning classifiers for Mac OS malware detection." *Handbook of Big Data Analytics and Forensics*, pp. 159-175, 2022. Springer
- [10] Orikogbo, Damilola et al. "CRiOS: Toward Large-Scale iOS Application Analysis." *Proceedings of the 6th Workshop on Security and Privacy in Smartphones and Mobile Devices*, 2016, pp. 33-42. ACM, http://dl.acm.org/citation.cfm?id=2994473
- [11] Pajouh, Hamed Haddad et al. "Intelligent OS X malware threat detection with code inspection." *Journal of Computer Virology and Hacking Techniques*, vol. 14, no. 3, 2018, pp. 213-223. https://doi.org/10.1007/s11416-017-0307-5
- [12] Manna, Modhuparna, et al. "Modern macOS userland runtime analysis." *Forensic Science International: Digital Investigation 38 (2021): 301221*. https://www.sciencedirect.com/science/article/abs/pii/S2666281721001293
- [13] Pham, D.P., et al. "Mac-A-Mal: macOS malware analysis framework resistant to anti-evasion techniques." *J Comput Virol Hack Tech* 15, 249–257 (2019). https://doi.org/10.1007/s11416-019-00335-w
---
# Application Streaming: Use macOS Apps Remotely
**Authors**: Anton Barkov
**Published**: 2023-10-13
**Topics**: software-distribution
**Type**: Tech Note
**URL**: https://research.macpaw.com/publications/app-streaming
## Introduction
In an age where the streaming revolution continues to reshape our digital landscape, it's not just about binge-watching our favorite TV shows anymore. For instance, as of late, the concept of streaming has been applied to gaming: you no longer need to have a high-end PC or even install a game to play it. Instead, you can use a service like [Nvidia GeForce Now](https://www.nvidia.com/en-ua/geforce-now/) to launch a game on a remote server and interact with it through the web. The remote machine performs all of the necessary processing to render the frames of the game that you're playing and sends the result to you. Meanwhile, all your computer does is draw the frames received from the server on the screen and send your input events back to the server so that you can interact with the game.
Sounds pretty cool. So, what if we could apply the same approach to usual macOS apps? Could we launch an app remotely and display it on our screen like it's actually running on our Mac? We decided to test just that.
Below is the story of our first attempt at implementing an application streaming prototype. Spoilers: there's still much to do and even some stuff to redo, but we have decided to document our current achievements in this field anyway. Let's dive in!
## Technical Task
First, let's define what we'll try to implement in technical detail. We'll be using two MacBooks; one will serve as the streaming Server, and the other will be the Client. The workflow between them should be the following:
1. The Client decides which application to use and sends a specific web request containing the desired app's bundle ID to the Server.
1. The Server launches the specified application if it is present on the machine, sets up any functionality needed for its streaming, and responds to the Client on whether the preparations were successful.
1. The Client establishes a WebSocket connection with the Server, and a two-way communication commences:
* The Server sends the streamed application's UI to the Client, frame by frame. The Client draws these frames on the user's display.
* The Client sends the user's input events to the Server, which it repeats to simulate interaction with the streamed application.
**Important notes:**
* The streamed application must appear on the Client's side without any other UI from the remote display. We don't want to recreate the experience of a TeamViewer session where you see the entire remote desktop — we'll be streaming only the desired app. The idea is to make the UX as close to running a native app as possible.
* The streamed application is a 3-rd party app we cannot modify or control. In other words, there's no way to establish an XPC connection for data exchange.
The idea looks straightforward on paper, but we've never tried implementing anything similar before. We'll tackle this problem by dividing it into separate development steps.
## Development Steps
### Basic application development
We'll start off by creating two macOS applications, we'll call them **AppStreamingClient** and **AppStreamingServer**. Their names speak for themselves — each one is to be launched on its designated MacBook and perform the role of a server or client. We'll do some basic UI setup as well, and the apps will end up looking like this:
The Client app requires the user to specify the IP address of the Server computer and the bundle identifier of an application they would like to use remotely. The Server's UI is primarily informational and doesn't require any interaction for it to work — it'll simply output info on established connections to Clients (plural, there may be more than one), e.g., how much traffic has been sent to each Client.
We can go ahead and implement the first two steps of our tech task right away, the ones where the Client asks the Server to launch an app and gets a response. We'll be sending a GET request of the following type:
```swift
let url = URL(string: "http://\(host):\(port)/init/\(bundleID)")
```
Sending the request is easy; use a `URLSession.dataTask(request: ...)`, and you're good to go. Listening to incoming requests is a bit less trivial, though. To save time on the implementation, we'll use a tiny Swift Package called [Swifter](https://github.com/httpswift/swifter). It will allow us to quickly implement an HTTP server and listen to specific requests with only a few lines of setup, like so:
```swift
private lazy var server: HttpServer = {
let server = HttpServer()
server.get["/init/:bundleID"] = { [weak self] request in
// Get the bundleID from the request's path
let bundleID = request.params.first?.value
// Launch the specified application.
// `applicationLauncher` is a wrapper over the asynchronous `NSWorkspace.openApplication(...)` call,
// which performs the job synchronously instead.
applicationLauncher.launchApplication(with: bundleID)
// Return the response after the application is launched.
// The response should contain a port number to use to establish a WebSocket connection
// for the upcoming streaming data transmissions. In this example, for the sake of brevity,
// we've specified the port 5901 statically, though in true implementation we should look for
// a currently unused port and provide it instead.
return .ok(.json(["data": ["port": 5901]]))
}
return server
}()
do {
// Start the server on port 5900.
try server.start(5900)
} catch {
preconditionFailure("HTTP server init failed: \(error.localizedDescription)")
}
```
Once the Client receives the response, it should establish a WebSocket connection to the same computer using the provided port number.
### Establishing the streaming communication
Now, we must figure out how to make two computers continuously communicate with each other and transmit specific types of data. They should be able to exchange at least two messages: one containing a frame of the streamed application and another providing a user's input event. It is definitely possible to implement this from scratch, but it would take a lot of time and effort to do so in high quality.
The communication protocol we're after looks quite familiar, and you've probably dealt with it yourselves if you have ever used any remote desktop software before (e.g., TeamViewer mentioned above). In fact, this technology is about two decades old, and its name is **RFB protocol** or **VNC systems**.
> RFB ("remote framebuffer") is a simple protocol for remote access to graphical user interfaces. Because it works at the framebuffer level, it is applicable to all windowing systems and applications, including X11, Windows, and Macintosh. RFB is the protocol used in VNC (Virtual Network Computing).
> The remote endpoint where the user sits (i.e., the display plus keyboard and/or pointer) is called the RFB client or viewer. The endpoint where changes to the framebuffer originate (i.e,. the windowing system and applications) is known as the RFB server.
> [The RFB Protocol Specification](https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#introduction)
So, in other words, RFB is a standardized and time-tested way of controlling a remote desktop through the web, which fits our needs just right. Just instead of interacting with the entire remote desktop, we'll settle on a single on-screen app, but we'll get to that shortly. VNC is another term frequently used to describe basically the same thing: a VNC system is a combination of multiple computers that conform to the RFB protocol and can communicate with each other.
Despite having the blueprints for implementing an RFB-conforming piece of software, it would still prove quite a challenging task — just take a glance at the [protocol specification](https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst) to get the basic idea of how much there is to account for. Luckily for us, the developer community behind VNC is quite large and active, and it didn't take long to find an open-source solution for our needs. Meet [libvnc](https://github.com/LibVNC/libvncserver) — a cross-platform set of libraries written in C that trivialize implementing a VNC server or client.
We can test their capabilities without any prior coding:
1. Clone the [repository](https://github.com/LibVNC/libvncserver) and [build the sources](https://github.com/LibVNC/libvncserver#how-to-build) (make sure you have CMake installed).
1. The newly-created build folder contains the distributable `libvncserver` and `libvncclient` libraries, which we'll use in our apps. But for now, find the `examples/server/mac` binary in the same folder and launch it — a Terminal window will appear, and if there are no errors, you've successfully launched a VNC server on your Mac.
1. Go to [novnc.com](https://novnc.com/noVNC/vnc.html) — this is a distribution of an open-source VNC web client, which allows you to connect to any VNC server out there, no matter the platform it's running on (pretty awesome, right?). Go ahead and connect it to `127.0.0.1:5900`.
If everything is done right, you should see your desktop appear in your browser window like so:
The libraries will cover the most challenging part of the job: they'll perform all of the necessary handshakes to setup a WebSocket connection between our two Macs, and handle all of the complexities of data transmission between them. As the libraries' clients, we only have to provide them with framebuffers/events to send to the other side.
For details on libs usage, please refer to [libvnc documentation](https://libvnc.github.io/doc/html/libvncserver_doc.html).
### Server-side screen capture
With the communication capabilities successfully set up, we can move on to implementing specific on-device logic. We'll start with capturing our streamed application's UI, which is pretty straightforward, thanks to Apple.
At WWDC 2022, Apple announced its new framework — [ScreenCaptureKit](https://developer.apple.com/documentation/screencapturekit/). It allows for effortless screen capture (duh 🙂) and provides many ways to customize what exactly you want to include in or leave out of the final result. Obviously, due to the framework being relatively new, it raises our minimum macOS requirement all the way to macOS 13 (Ventura). We consider this acceptable, as the Server machine will surely be set up on the side of the streaming service provider; it can be configured in whichever way is needed. We won't have similar restrictions on the Client's side.
The streaming process is handled through a `SCStream` object, which relies on two other configuration objects: `SCContentFilter` and `SCStreamConfiguration`. Out of the box, with the most basic setup, you can grab the entirety of your display like so:
Let's say we want to stream Setapp to our Client, the window in the center of the screenshot. The current picture won't work for us — we don't want the Client to see our whole screen with other unrelated windows open, we want to stream just the specific app. Luckily, `SCContentFilter` lets us define which apps we want to include in our stream. If we specify Setapp there, the output will change in the following way:
The picture looks better, but we still have the desktop background image that our Client doesn't need to see, and this is where `SCStreamConfiguration` comes to the rescue. If we set its `backgroundColor` property to `.clear`, the background fades away completely, and all that's left is Setapp UI and the menubar:
The picture now fits our needs precisely, and this is the exact kind of framebuffer we'll be sending to the Client.
### Server-side framebuffer handling
Handling framebuffers received from `SCStream` and sending them to the Client has a few pitfalls. We learned about them the hard way and decided they're worth a special mention.
What exactly is a framebuffer? Simply put, it's a region of memory (RAM, to be precise) that stores pixel data for continuous on-screen rendering. How much memory does a framebuffer occupy? In bytes, exactly as much as the number of pixels in a given image multiplied by the number of color components in a pixel. For example, a 100x100 pixel image in RGBA format will occupy 100 * 100 * 4 = 40000 bytes, or roughly 39 kilobytes. Pretty simple so far.
Notice how we wrote "*continuous* on-screen rendering" back there? That word is elaborate. Let's say you're watching a video, dozens of frames changing on-screen every second. And yet, only one framebuffer is used to draw them all, as the image data is constantly being overwritten and read from the same memory region. This purposeful optimization skips the process of allocating/deallocating graphics in the memory — things just work faster if we allocate the needed memory size once and keep using that same region for frequent visual updates.
Cool, you might say, so what's the issue here? A possible data race condition is the issue. If we don't synchronize our read-write operations properly, we might end up with screen tearing and similar visual artifacts on the Client's side. Simply put, it's like reading a book page that is being constantly rewritten before your eyes — things just won't make sense. Roughly the same will happen to our Client: the framebuffer gets rewritten at the exact moment it is being sent to the Client, which results in broken pixel data or image misalignment.
There's a known solution to this problem, a technique called **framebuffer swapping**. We allocate two framebuffers and make one readable to the Client, while the other is writeable for our `SCStream` output. As soon as we get an update from the stream, we swap those framebuffers, and the process repeats. Here's the simplified code:
```swift
typealias FrameBuffer = UnsafeMutablePointer
final class VNCServer: NSObject {
// Framebuffers.
// One of them is readable to the client while the other is being updated with new pixel data,
// then they get swapped and the process repeats.
// This protects from screen tearing & similar artefacts on the client's side.
private var frameBufferOne: FrameBuffer?
private var frameBufferTwo: FrameBuffer?
private var backBuffer: FrameBuffer? // points to the currently writeable buffer
// Pointers to objects provided by libvncserver.
// One represents our server and the other — a possibly connected client.
private var libServer: rfbScreenInfoPtr?
private var libClient: rfbClientPtr?
override init() {
super.init()
mallocFrameBuffers(displaySize: NSScreen.main!.frame.size)
swapFrameBuffers()
// Do all of the other necessary server setup here
}
func mallocFrameBuffers(displaySize: CGSize) {
frameBufferOne?.deallocate()
frameBufferTwo?.deallocate()
let bytes = Int(displaySize.width) * Int(displaySize.height) * 4
frameBufferOne = FrameBuffer.allocate(capacity: bytes)
frameBufferTwo = FrameBuffer.allocate(capacity: bytes)
}
func swapFrameBuffers() {
if backBuffer == frameBufferOne {
backBuffer = frameBufferTwo
libServer?.pointee.frameBuffer = frameBufferOne
} else {
backBuffer = frameBufferOne
libServer?.pointee.frameBuffer = frameBufferTwo
}
}
}
extension VNCServer: SCStreamOutput {
// An SCStream delegate method used to handle stream output.
func stream(_ stream: SCStream, didOutputSampleBuffer sampleBuffer: CMSampleBuffer, of type: SCStreamOutputType) {
guard let libServer, let screen = NSScreen.main, sampleBuffer.isValid else { return }
guard let attachmentsArray = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, createIfNecessary: false) as? [[SCStreamFrameInfo: Any]],
let attachments = attachmentsArray.first,
let statusRawValue = attachments[SCStreamFrameInfo.status] as? Int,
let status = SCFrameStatus(rawValue: statusRawValue),
status == .complete else { return }
// Get the pixel buffer that contains the image data.
guard let pixelBuffer = sampleBuffer.imageBuffer else { return }
// Get the backing IOSurface.
guard let surface = CVPixelBufferGetIOSurface(pixelBuffer)?.takeUnretainedValue() else { return }
// Lock client from reading the buffer
if let libClient {
guard pthread_mutex_lock(&libClient.pointee.sendMutex) == 0 else {
preconditionFailure("pthread_mutex_lock failed")
}
}
// Lock the surface and copy it to our backBuffer
IOSurfaceLock(surface, .readOnly, nil)
memcpy(
backBuffer,
IOSurfaceGetBaseAddress(surface),
Int(screen.frame.width) * Int(screen.frame.height) * 4
)
IOSurfaceUnlock(surface, .readOnly, nil)
// Swap the buffers!
swapFrameBuffers()
// Reenable client reads
if let libClient {
guard pthread_mutex_unlock(&libClient.pointee.sendMutex) == 0 else {
preconditionFailure("pthread_mutex_unlock failed")
}
}
}
}
```
### Client-side frame drawing
The image data we receive on the Client's side is, of course, also provided in the form of a framebuffer. So, it might be a little tricky to figure out how to transform that raw pixel data into something we can output into the UI.
In a nutshell, the approach we took is the following:
1. The AppStreamingClient app presents a transparent borderless window stretched across the user's screen. It's pretty basic stuff; we won't go into detail on this one.
1. We implement a custom `VNCFramebufferView` as the window's content view. The view has a backing `CALayer`, and we'll use its `contents` to display the framebuffer data after we complete the necessary conversions.
1. `VNCFramebufferView` has two public methods:
* `setFrameBuffer(_ frameBuffer: FrameBuffer)` — we'll use it once we establish a connection to the Server and provide the view with a pointer to the memory region holding the pixel data. We'll be dealing with a single framebuffer this time; there's no need for swapping like on the Server.
* `handleFrameBufferUpdated(rect: NSRect)` — we'll call this whenever we receive a framebuffer update from the Server. The `rect` parameter represents the image coordinates that must be redrawn.
1. The view will redraw its contents with the help of a `CVDisplayLink` to achieve smooth and synchronized picture updates.
1. To get a usable image representation of the framebuffer data, we must first transform it back into an `IOSurface`. The surface can then be easily fed to a `CIImage` initializer for further output on-screen.
Here's the `VNCFramebufferView` implementation:
```swift
struct FrameBuffer {
let pointer: UnsafeMutableRawPointer
let width: Int
let height: Int
let pixelFormat: OSType
let bytesPerPixel: Int
}
final class VNCFramebufferView: NSView {
// MARK: Init
override init(frame frameRect: NSRect) {
super.init(frame: frameRect)
wantsLayer = true
layer?.drawsAsynchronously = true
layer?.masksToBounds = true
layer?.allowsEdgeAntialiasing = false
}
required init?(coder: NSCoder) {
fatalError("init(coder:) has not been implemented")
}
deinit {
removeDisplayLink()
}
// MARK: Public
func setFrameBuffer(_ frameBuffer: FrameBuffer) {
self.frameBuffer = frameBuffer
self.surface = IOSurface(properties: [
.width: frameBuffer.width,
.height: frameBuffer.height,
.pixelFormat: frameBuffer.pixelFormat,
.bytesPerElement: frameBuffer.bytesPerPixel,
.bytesPerRow: frameBuffer.width * frameBuffer.bytesPerPixel,
.allocSize: frameBuffer.width * frameBuffer.height * frameBuffer.bytesPerPixel
])
}
func handleFrameBufferUpdated(rect: NSRect) {
updateSurface(rect: rect)
}
// MARK: Private
private let ciContext = CIContext()
private var frameBuffer: FrameBuffer?
private var surface: IOSurface?
private var displayLink: DisplayLink?
}
// MARK: - Overrides
extension VNCFramebufferView {
override func viewDidMoveToWindow() {
super.viewDidMoveToWindow()
addDisplayLink()
}
}
// MARK: - Display link
extension VNCFramebufferView: DisplayLinkDelegate {
private func addDisplayLink() {
removeDisplayLink()
guard let window = window,
let screen = window.screen ?? NSScreen.main else {
return
}
guard let displayLink = DisplayLink(screen: screen) else { return }
displayLink.delegate = self
displayLink.isEnabled = true
self.displayLink = displayLink
}
private func removeDisplayLink() {
guard let oldDisplayLink = self.displayLink else { return }
oldDisplayLink.delegate = nil
oldDisplayLink.isEnabled = false
self.displayLink = nil
}
func displayLinkDidUpdate(_ displayLink: DisplayLink) {
DispatchQueue.main.async { [self] in
guard let surface else { return }
surfaceLockReadOnly(surface)
let ciImage = CIImage(ioSurface: surface)
surfaceUnlockReadOnly(surface)
layer?.contents = ciContext.createCGImage(ciImage, from: ciImage.extent, format: .BGRA8, colorSpace: CGColorSpaceCreateDeviceRGB(), deferred: true)
}
}
}
// MARK: - Private
private extension VNCFramebufferView {
func surfaceLockReadOnly(_ surface: IOSurface) {
IOSurfaceLock(surface, [.readOnly], nil)
}
func surfaceUnlockReadOnly(_ surface: IOSurface) {
IOSurfaceUnlock(surface, [.readOnly], nil)
}
func surfaceLockReadWrite(_ surface: IOSurface) {
IOSurfaceLock(surface, [], nil)
}
func surfaceUnlockReadWrite(_ surface: IOSurface) {
IOSurfaceUnlock(surface, [], nil)
}
func updateSurface(rect: NSRect) {
guard let surface, let frameBuffer else { return }
surfaceLockReadWrite(surface)
defer { surfaceUnlockReadWrite(surface) }
let bytesPerPixel = surface.bytesPerElement
let bytesPerRow = surface.bytesPerRow
let x = Int(rect.minX)
let y = Int(rect.minY)
let width = Int(rect.width)
let height = Int(rect.height)
for row in y.. Some of them are described previously in our previous research article [Single Input Interfaces](https://research.macpaw.com/articles/single-input-interfaces-ui-ux/).
-->
What combines them all is a similar approach to user-computer interaction. The user has a single input field, where they describe what they are researching for, or a task, and the system creates the output based on it, showing the user only the relevant information to them. The output can also be in different formats — such as web results, text suggestions, calculations, opening 3rd party apps (Siri, Alfred), or even transcribed mathematical equations (Wolfram).
### Dynamic vs Infinite interfaces
Dynamic Interface is also a design practice based on a user intent. Basically, the interface changed according to the user's needs and adapts to the conditions. Choices made by users drive what controls are displayed so that only relevant fields are presented. This approach helps the user achieve their goal faster, reduces visual clutter, and shows what's only relevant to the user. In the Dynamic interface, the user is still presented with an interface
Infinite Interface is an approach that can grow further from the dynamic design practice. Having AI on our hands, we can now not just adapt the interface to the user's need, but **completely get rid** of the interface itself and base the interface on the user's intent only. Having that approach, we believe we can personalise the experience for the user, don't overload them with unneeded or unused features, and make the task completion faster, more precise, user-friendly, and simplify the products and interfaces that require domain knowledge. By reducing the distance between the user and the interface, the interface creates a more direct feel, minimising the user's effort to achieve their goals.
## User Interaction and Engagement
### User interaction in infinite interfaces
One of the first mentions of a similar approach was in the 1985 work "Human-computer interaction" by Lawrence Erlbaum Associates, Inc., Edwin L. Hutchins, James D. Hollan, and Donald A. Norman. In this work, they describe the first interfaces that could be interacted with directly but not by commands.
The first examples they described as follows:
> For example, Bill Budge's Pinball Construction Set (Budge, 1983) permits a user to construct an infinite variety of electronic pinball games by directly manipulating graphical objects that represent the components of the game surface.
Naturally, years have passed, and technology has evolved immaculately, but the approach to human-computer interaction in general hasn't changed as much as we think it has. Now, we still have to consider the same things within user-computer interactions — the user's approach to a task, the cognitive load, the language of the interface, the speed of the output, and the overall directness of the manipulation.
**Shneidermnan** (Ben Shneiderman, Behaviour and Information Technology, 1982, vol.1 no.3, 237-256) has suggested that direct manipulation systems have the following virtues:
- Novices can learn basic functionality quickly, usually through a demonstration by a more experienced user.
- Experts can work extremely rapidly to carry out a wide range of tasks, even defining new functions and features.
- Knowledgeable intermittent users can retain operational concepts.
- Error messages are rarely needed.
- Users can see immediately if their actions are furthering their goals, and if not, they can simply change the direction of their activity.
**And to this day, those criteria fit perfectly with the approach of the Infinite Interface practice that we will be covering in this work.**
#### Two aspects of directness: distance and engagement
The distance in this case depicts the distance between the user's intent and the output of the system. The engagement represents the feeling of manipulating the interface and object of interest directly.
In the "Human-computer interaction" work, the authors introduce terms as "Gulf of execution" and "Gulf of evaluation", depicted in the following picture.
So, in the Infinite Interfaces approach, our goal is to shorten or completely erase those gulfs in order to create a smooth human-computer interaction.
The Gulf of execution can be bridged by matching the output of the system to the user's thoughts and goals, and the Gulf of Evaluation — by making the output represent the result in the most clear way, that user can easily perceive.
The bigger are the Gulfs of Execution and Evaluation, the more cognitive load we put on the user in the human-computer interaction, and the goal is to take this load from the user and make it work on the end of the product.
#### Two forms of distance: semantic and articulatory
> Historically, most interfaces have been built on the conversation metaphor.
Previously, in order to achieve their goal, the user had to use the language of the interface, learn what the interface meant by specific terms, and adapt to using them.
With the AI technology we have now, this conversation can become not a metaphor, but a reality, and this conversation between the user and the system shouldn't feel difficult, technical, and frustrating. This conversation can now be human-centred, using natural language, that will not only help the user, but also make them feel understood, and also in charge of their interaction.
**With the Infinite Interface approach, our goal is to flip this approach and make the system understand the user, not vice versa.**
In order to shorten the gulf between the user and the system, the output vocabulary of the system should match the input of the user's request. If the output doesn't match the user's language, the cognitive load increases and therefore put the burden of shortening the gulf of distance onto the user.
> The more that the user must provide, the greater the distance to be bridged.
One way to achieve articulatory directness at the input side is to provide an interface that allows the user to use more direct sources of input — such as voice commands, or using keyboard to select the options, so that the user's actions and output are perceived more naturally.
#### The Gulf of Evaluation
When users observe the output value, they must evaluate it in terms of how it fits their request. If the action or the output only shows the current value, the user must make a mental effort to evaluate and compare the output with the current state and understand whether it changed or not.
In the "Human-computer interaction" work, it's represented by a simple case of filling a water tank. If the interface doesn't show the changes, the burden of evaluating them goes to the user, increasing their mental and cognitive load and making the Gulf of evaluation their responsibility.
By clearly indicating the changes in the interface, we can bridge the Evaluation Gap on the side of the system.
Another example of shortening this gulf is to provide consistency with semantics and actions in the interface, or provide a higher-level language that will include the requested input.
That approach will surely require more effort from the system, but can achieve the shortening of the gap between human-computer interaction.
#### The learning curve
In the Infinite Interface approach, as well as with all other interfaces, we also should consider the user's learning curve. As with all interfaces, the more familiar the user becomes with it, the easier and smoother the interaction will become. Meaning that with time, users will adapt to the system more and more and will understand what they, as a user, are expected to provide to the system.
The full schema of the Execution and Evaluation Gulfs is represented below:
### Cognitive load aspect within the infinite interface design framework
At the root of the Infinite Interface approach, we have the assumption that by taking out the unneeded and unused features from the interface, let the user directly manipulate the interface by their input, and only propose the results fitting the user's needs — we will lessen the cognitive load the user experiences.
To address the cognitive load within the interface, firstly, we need to understand the Cognitive Load Theory (CLT).
Here is a quote from Fred Paas, Juhani E. Tuovinen, Huib Tabbers & Pascal W. M. Van Gerven (2003) "Cognitive Load Measurement as a Means to Advance Cognitive Load Theory", Educational Psychologist, 38:1, 63-71
> CLT is concerned with the design of instructional methods that efficiently use people's limited cognitive processing capacity to apply acquired knowledge and skills to new situations (i.e., transfer). CLT is based on a cognitive architecture that consists of a limited working memory with partly independent processing units for visual and auditory information, which interacts with an unlimited long-term memory. These structures and functions of human cognitive architecture have been used to design a variety of novel, efficient instructional methods.
According to CLT, multiple elements of information can be chunked as single elements in cognitive schemas, which can be automated to a large extent. Therefore, the main goal of a CLT-based interface would be to create and automate schemas. However, before information can be stored in schematic form in long-term memory, it must be extracted and manipulated in working memory.
These instructional methods have proven successful for children, young adults (for an overview, see Sweller et al., 1998), and older adults (e.g., Paas, Camp, & Rikers, 2001; Van Gerven, Paas, van Merriënboer, & Schmidt, 2002a) in a wide variety of domains, such as physics, mathematics, statistics, computer programming, electrical engineering, paper folding, understanding empirical reports, and learning to use computer programs.
**Compared with conventional instructional tasks, CLT-based tasks have been found to be more efficient because they require less training time and less mental effort to attain the same or better learning and transfer performance.**
According to Nielsen Norman Group, when talking about the user interface, cognitive load can be caused by 3 main factors:
- too many choices
- too much thought required
- lack of clarity
> In the field of user experience, we use the following definition: the cognitive load imposed by a user interface is the amount of mental resources that is required to operate the system. Informally, you can think of mental resources as "brain power" — more formally, we're talking about slots in working memory.
Intrinsic cognitive load is the effort of absorbing that new information and of keeping track of their own goals.
In the basis of Infinite Interfaces lays the foundation of eliminating the cognitive load for the user as much as possible, which means reducing visual clutter by taking out the features the user doesn't need to use, giving them full control to describe their intent, propose fitting and relevant options to the user's request, and offload the tasks from the user, providing them the actionable results as well.
Short-term memory limitations dictate a whole range of other design guidelines:
- quick response time, so that the user doesn't have to keep in mind what task they were performing
- providing clarity within the interface, to show what task is being performed
> At the extreme, only about 5% of the population has enough brainpower to perform complex tasks with computers, such as making high-level inferences using specialized background knowledge. Most likely, you're in this elite group. And, worse yet, so are many other members of your Internet team. (And the rest are definitely in the top 25%, which is also much better than average users.)
> Even though it's a bad course title, it's a good overall mnemonic to design for cavemen and their literal-minded and limited-capacity brains. After all, your paying customers are only one step out of the cave.
**There are also basic UX Laws that correspond with the Cognitive Load Theory:**
- [Hick's Law](https://lawsofux.com/hicks-law/)
The time it takes to make a decision increases with the number and complexity of choices
- [Miller's Law](https://lawsofux.com/millers-law/)
The average person can only keep 7 (plus or minus 2) items in their working memory.
- [Jakob's Law](https://lawsofux.com/jakobs-law/)
Users spend most of their time on other sites. This means that users prefer your site to work the same way as all the other sites they already know.
In Design Principles for Infinite Interfaces, we will go over how we should approach designing the interface with Cognitive Load Theory and UX Laws in mind.
## Testing the interfaces
### The task
For this test, we conducted moderated user testing with 2 different interfaces that have the same functionality: 1 Infinite interface and 1 Dynamic user interface. Both products are system utilities related to computer maintenance.
**User's number**: 10
**Type:** Moderated
**Task:** users were given 3 tasks. The first 2 were straightforward - to find or request a needed feature within the app. The third task was formulated as a problem, that the user faces, so the user needed to understand the root of the problem and complete several steps to resolve it.
Although it is quite difficult to measure cognitive load exactly, we used one of the simplest yet effective rating-scale techniques for this test.
> Rating scale techniques are based on the assumption that people are able to introspect on their cognitive processes and to report the amount of mental effort expended. Although self-ratings may appear questionable, it has been demonstrated that people are quite capable of giving a numerical indication of their perceived mental burden (Gopher & Braune, 1984).
**Metric used**: users' rating of the simplicity of completing the task
### Results
We noticed that with complex task that require more than 1 action, users rate the simplicity of Infinite Interfaces by **22%** better (even if having issues with wording the request), as they only needed to formulate the request in the app, and the actionable items were presented as an outcome already, so the user didn't need to dive into understanding the problem itself, but rather just inform about it and have the proposed solution.
This result was calculated based on the average score users gave to rate the simplicity of completing the given task in both interfaces.
However, even though the task completion rate was much better in the infinite interface, most of the time users spent on the task was **formulating the request**, which is connected directly to the Semantic Gulf explained earlier. In the **Design Principles for Infinite Interfaces** we go into detail describing how to avoid this issue and make the user-computer interaction as smooth and fast as possible.
### Important notes
CLT research has identified various task characteristics, including task format, task complexity, use of multimedia, time pressure, and pacing.
It is also important to note that learners can compensate for increased mental load, such as higher task complexity, by putting in more mental effort to maintain performance. Hence, the cognitive costs associated with a particular performance level cannot be solely inferred from performance-based measures. Instead, combining measures of mental effort and performance can provide valuable insights into cognitive load.
An interesting observation is that even though researchers are continuously trying to find or develop physiological and secondary task measures of cognitive load, subjective workload measurement techniques using rating scales remain popular, because they are easy to use; do not interfere with primary task performance; are inexpensive; can detect small variations in workload (i.e., sensitivity); are reliable; and provide decent convergent, construct, and discriminate validity (Gimino, 2002; Paas et al., 1994). However, we must stress that the internal consistency of these measures requires further studies.
Determining the effectiveness of instructional conditions requires considering the relationship between cognitive load and performance. Paas and van Merriënboer (1993) introduced a computational method to analyze mental effort and performance to assess instructional efficiency. This approach has been successfully applied in numerous studies. Learners' behavior in specific instructional conditions is deemed more efficient if their performance surpasses expected levels based on their mental effort. Similarly, learners' mental effort is considered lower than expected if their performance exceeds anticipated levels. Learners can compensate for increased mental load by investing more mental effort. However, performance-based measures do not consistently reflect the cognitive costs associated with a particular level of performance.
## Design Principles for Infinite Interfaces
- **Provide clear instructions in the beginning**
The way customers perceive your product is greatly influenced by the information they gather from other products and apps, forming a significant part of their mental models. (Jakob's Law)
This type of interface is rather new and unusual for the average user, so in the beginning of the interaction it is critical to minimise the stress and cognitive load as to how to interact with it.
Enhance users' mental models by aligning them more accurately with your system. This can be achieved through improved explanations and clearer labels, leading to a more transparent UI. These aspects can be provided during onboarding, as well as through tips and hints at the beginning of the user interaction with the product.
- **It's all about the semantics**
Taking that Infinite Interface approach is based solely on the user input; this part is the most critical to approach in the right way.
- In the initial phase, help the user understand what types of input they can do. Give the user an example or a prompt to start with
- Output should semantically match the input. If the user doesn't have enough domain knowledge, the different output can confuse them about whether it is their desired outcome or not.
- Take into account human errors, typos, and mistakes. The system should recognise the request even with typos in it, as we don't want the user to spend the majority of their time in the product correcting their mistakes.
- The system should understand the general idea of the input and not solely base it on the words in the input. For example, is the user types "my computer is hot", the system should also recognise requests such as "overheated", "warm", "slowing down", "noisy" etc., and give the same relevant output, resolving the core of the issue.
- **Personalise the user's experience**
Automate what can be automated. Interfaces have the advantage of having access to contextual data, allowing them to anticipate users' needs rather than relying on the end user to initiate actions.
If the user goes through certain scenarios on a regular basis - propose them to them first, without having to request it every time.
You can also base the output on the previous user's actions or propose next steps for the user in a certain scenario.
- **Different types of output should be displayed differently**
Let's say your types of output could be a text suggestion, a link, a picture, or another app proposed to use. When users see different types of output, they should understand they are different from one another, and after some time with the learnability curve, they should immediately distinguish which output they need without reading all the options, but just having a glance.
It can be done via color difference, icons, sizes, etc.
- **Take learnability into account**
When you put this design to test, take into account that this type of interfaces is not yet very familiar to the user. So, to understand the real user flow, test the design in the beginning, when user is just getting familiar with it, and also some time later. As the user get's more familiar with system's capabilities, you might see changes in the user's requests and general approach to the tasks, and improve the product based on the new data.
- **Don't clutter the space**
The idea of Infinite Interfaces is not to propose as many options as we can to the user, but to propose only relevant ones at the specific time and place. So, if the number of outputs is more than 5-7, the system should filter the most relevant to the user itself, not putting this task on the user. Again, minimising the cognitive load is one of the bases of this approach, so in order to do that, we make the system do the thinking instead of the user.
- **Propose options**
If the user's input is not clear enough to propose one certain solution, the system should provide a few relevant options to choose from, so that the user doesn't have to rethink their request and do it again. This causes frustration, more cognitive load, and an overall poor product satisfaction rate.
#### User experience considerations
**User is put in control of the action**
While we are trying to minimise the cognitive load with the Infinite Interface approach, we still have to remember that the user is in charge of your product's actions. No matter how intuitive a product may be, there will always be moments when the user needs to take control, make a decision, or formulate their intent. By keeping this in mind, we can ensure that our designs are both user-friendly and empowering.
**Formulating the intention**
If the user is not introduced to the product and its features and approach properly, there is a chance they might struggle to formulate the intent at first. Having that issue will completely disregard the product's output, no matter how good the system is in the background, if the user cannot reach the results they need. It is critical to adapt to the user's language, not make the user guess how they should form their inputs, and make the input for the user as clear and user-friendly as possible from the very beginning.
To avoid that, we can propose to the user a variety of relevant options, which can minimise the chance of an invalid result.
## Potential Future Directions
We envision an interface where a solitary input can bring diverse outputs: text, apps, actions, ushering an era where evolving interfaces reshape engagement paradigms and redefine human-computer interactions.
We see multiple areas where such interfaces can be used:
- **Personal Assistants and AI Companions**
This interface can function as an advanced personal assistant. Users can express their needs using natural language, and the system can offer text suggestions, launch relevant apps, display images or videos, and even perform intricate tasks like disk space cleanup, all tailored to the user's context and requirements.
- **AR/VR**
The interface in AR and VR environments effortlessly merges the digital and physical realms. Users can voice their desires and receive visual or auditory feedback that overlays their surroundings, providing just the needed information without cluttering the space.
- **Internet of Things (IoT) Devices**
By utilising this interface, IoT devices can enhance usability. Users can interact with their smart home systems simply by expressing their needs, while the system takes care of controlling lights, adjusting thermostats, playing music, and displaying relevant information on smart displays. Users can also use natural language to ask the system to create a certain atmosphere or a setup.
- **Healthcare and Accessibility**
This interface is transformative for individuals with disabilities. Voice-activated control over a wide range of functionalities promotes inclusivity and accommodation.
- **Education and Learning**
Such an interface can also be used as a learning material. Users can ask questions related to their studies, receive visual aids, interactive learning materials, and games, and even practice speaking with an AI assistant within the system.
## Conclusion
Infinite Interfaces is a design practice that is logically evolving from the Dynamic Interfaces. Having the AI technology we have now, we can now personalise user experience and reduce the time for the user to complete the task, and use cognitive effort to do so. Using the described Design Principles for Infinite Interfaces, products can create interfaces that are easy to use and human-friendly. We see a big potential in using those interfaces in various fields, such as Smart assistants, AR/VR using apps, IoT, Healthcare, Accessibility, and the Educational field.
## References
1. Cognitive, physical, sensory, and functional affordances in interaction design. Rex Hartson. Pages 315-338
3. SYSTEMS DESIGN User Interface Design Output Design Input Design Menu Design. Paul Moya
5. [Minimize Cognitive Load to Maximize Usability. Kathryn Whitenton on December 22, 2013](https://www.nngroup.com/articles/minimize-cognitive-load/)
6. [Short-Term Memory and Web Usability. Jakob Nielsen on December 6, 2009](https://www.nngroup.com/articles/short-term-memory-and-web-usability/)
7. [Website Response Times. Jakob Nielsen on June 20, 2010](https://www.nngroup.com/articles/website-response-times/)
8. [Improving the Design with Dynamic Interface. Test-Driven Development for Embedded C — Pragmatic Programmers (106 / 173) ](https://medium.com/pragmatic-programmers/improving-the-design-with-dynamic-interface-242b7d973814)
9. Direct Manipulation Interfaces. Edwin L. Hutchins, James D. Hollan &Donald A. Norman. Pages 311-338
10. System Dynamics Review - Winter 1996 - Vicente - Improving dynamic decision making in complex systems through ecological
11. Cognitive Load Measurement as a Means to Advance Cognitive Load Theory. Fred Paas, Juhani E. Tuovinen, Huib Tabbers & Pascal W. M. Van Gerven
12. BEN SHNEIDERMAN (1982) The future of interactive systems and the emergence of direct manipulation, Behaviour & Information Technology, 1:3, 237-256
---
# Improving macOS Accessibility Features with W1 Chip sensors
**Authors**: Sergii Kryvoblotskyi
**Published**: 2022-12-05
**Topics**: human-computer-interaction, software-engineering
**Type**: Tech Note
**URL**: https://research.macpaw.com/publications/headphones-accessibility
## Introduction
In today's world, accessibility has become essential in technology design.
Apple's macOS devices and software offer a range of features and settings that assist in making devices more accessible for people with disabilities or special needs.
In this article, we'll cover some ways to improve accessibility using Apple AirPods or any other device with a W1 chip onboard.
We will implement a prototype of controlling macOS by head movements.
## macOS accessibility
With more than 1 billion active devices in use worldwide, the impact of Apple and its products on our lives cannot be overstated.
Yet for approximately 1.3 billion people, or 16% of the world's population according to the [World Health Organization](https://www.who.int/news-room/fact-sheets/detail/disability-and-health), access to smart technology is a challenge due to various physical or cognitive impairments.
To address this issue, Apple has committed to providing innovative accessibility features that enable users with disabilities — both visible and invisible — to take full advantage of their beloved iPhone, iPad, Mac, Apple Watch, and other products.
From built-in apps like VoiceOver and Zoom that provide customizable options for vision and hearing impairments to advanced assistive technologies such as Switch Control that allow users with mobility limitations greater freedom of movement than ever before, these features are transforming how we interact with technology daily.
All Apple products come with a built-in suite of accessibility features that people can use to adjust to their vision, hearing, physical motor needs, and more.
These features allow users to customize their experience, and many can be turned on or off as needed.
On a Mac, many ways exist to access and adjust these settings.
For example, users can click on the Apple logo in the top left corner of the screen and select System Preferences from the drop-down menu.
From there, they can click Accessibility to view a list of available options.
Alternatively, users can also use the Keyboard Viewer to see a list of all available keyboard shortcuts.
Each method provides users with a way to tailor their macOS experience to their individual needs.
## Apple W1 chip
Apple's introduction of the new W1 chip has been widely heralded as an innovative leap forward for Bluetooth audio products, allowing manufacturers to design headphones and earbuds that offer enhanced convenience and performance.
The W1 chip was introduced alongside Apple wireless AirPods and is already being integrated into more and more third-party headsets — giving users even greater choice in sound quality, portability, and battery life from their preferred headphone brand.
### Spatial audio
The Apple W1 chip has revolutionized how we experience audio.
With its [Spatial Audio](https://support.apple.com/en-mk/guide/airpods/dev00eb7e0a3/web) feature, the sound we hear will feel as if it’s coming from all around in an incredibly immersive listening experience.
From wireless earbuds to headphones, this technology is changing what we expect from our music-listening devices and allowing us to indulge in a new world of sound that blurs the boundaries between headphone sensation and surround-sound feelings.
The Spatial Audio feature is powered by an integrated motion sensor, which we will use as hardware for the prototype.
### Enhancement idea
Our idea is to use the W1 Chip Accelerometer to control macOS with head movements.
This method is pretty close to Pointer Control that macOS has on board, but it has some key advantages:
1. No camera is required. Pointer Control requires good light and high precision, which are not necessarily available all the time.
2. "Shortcuts over pointer": we will implement "shortcuts" mapping, so there is no need for precise pointing.
3. The user doesn't need to sit exactly in front of the computer.
## Recognizing head movements
We will use [Apple AirPods Pro](https://www.apple.com/airpods-pro/) for the prototype.
Motion sensor data is available in the SDK in the [CoreMotion](https://developer.apple.com/documentation/coremotion) framework and the `CMHeadphoneMotionManager` class.
### API
However, there is a key problem here: this API, for some unknown reason, is not available on the macOS platform, so we will need to get the data from another device.
```swift
COREMOTION_EXPORT API_AVAILABLE(ios(14.0)) API_UNAVAILABLE(macos)
@interface CMHeadphoneMotionManager : NSObject
```
Fortunately, this API works well on iOS, so we can use mobile devices as a proxy to get sensor data and transfer it to the computer.
So the high-level architecture will look like this:
```mermaid
sequenceDiagram
W1 Chip Sensor->>Proxy: Raw Sensor Data
Proxy->>Client: Mapped Head Movements
```
### Proxy
As a proxy, we will use an iOS mobile device.
The implementation on mobile devices consists of 3 key steps:
1. Handle the headphones connection.
2. Read the sensor data.
3. Transfer the data to the computer.
#### Handle headphones connection
Headphones connection state is available via the CoreMotion framework and the `CMHeadphoneMotionManagerDelegate` class:
```swift
@available(iOS 14.0, *)
public protocol CMHeadphoneMotionManagerDelegate : NSObjectProtocol {
/*
* headphoneMotionManagerDidConnect:
*
* Discussion:
* Invoked when a headphone is connected.
* Execution of the delegate callback occurs on the operation queue used to deliver the device motion updates.
* The main thread is used if the queue was not specified.
*/
optional func headphoneMotionManagerDidConnect(_ manager: CMHeadphoneMotionManager)
/*
* headphoneMotionManagerDidDisconnect:
*
* Discussion:
* Invoked when a headphone is disconnected.
* Execution of the delegate callback occurs on the operation queue used to deliver the device motion updates.
* The main thread is used if the queue was not specified.
*/
optional func headphoneMotionManagerDidDisconnect(_ manager: CMHeadphoneMotionManager)
}
```
#### Read sensor data
Reading the sensor data is also available via the CoreMotion framework and the `CMHeadphoneMotionManager` class:
```swift
/*
* startDeviceMotionUpdatesToQueue:withHandler:
*
* Discussion:
* Starts device motion updates, providing data to the given handler through the given queue.
*
*/
open func startDeviceMotionUpdates(to queue: OperationQueue, withHandler handler: @escaping CMHeadphoneMotionManager.DeviceMotionHandler)
```
Once the headphones are connected and motion updates are started, the `CMDeviceMotion` class is delivered to the callback with every head motion update.
Head's attitude is available via the `CMAttitude` class as its [pitch, yaw, and roll](https://simple.wikipedia.org/wiki/Pitch,_yaw,_and_roll):
```objc
/*
* roll
*
* Discussion:
* Returns the roll of the device in radians.
*
*/
@property(readonly, nonatomic) double roll;
/*
* pitch
*
* Discussion:
* Returns the pitch of the device in radians.
*
*/
@property(readonly, nonatomic) double pitch;
/*
* yaw
*
* Discussion:
* Returns the yaw of the device in radians.
*
*/
@property(readonly, nonatomic) double yaw;
```
> CoreMotion outputs a direction cosine matrix — a rotation from the last "old" orientation to the new orientation of the device.
For this prototype, we will use 6 types of movements:
```swift
enum HeadGesture: String, Codable {
case up, down, right, left, forward, back
}
```
| Gesture Visualization | Gesture Direction | Gesture Visualization | Gesture Direction |
| --------------------- | ----------------- | --------------------- | ----------------- |
|  | Up |  | Left |
|  | Down |  | Forward |
|  | Right |  | Back |
So once roll, pitch, and yaw are delivered, a simple mapping can be used:
```swift
private var headThreshold: Double = 0.4
private var headThresholdCorrection: Double = 0.5
extension ViewController {
func handleMotion(with motion: CMDeviceMotion) {
let pitch = motion.attitude.pitch
let yaw = motion.attitude.yaw
let roll = motion.attitude.roll
let pitchHandled = pitch > (headThreshold * headThresholdCorrection) || pitch < -headThreshold
let yawHandled = yaw > headThreshold || yaw < -headThreshold
let rollHandled = roll > headThreshold || roll < -headThreshold
if pitchHandled {
if pitch > (headThreshold * headThresholdCorrection) {
handleGesture(gesture: .up)
} else if pitch < -headThreshold {
handleGesture(gesture: .down)
}
} else if yawHandled {
if yaw > headThreshold {
handleGesture(gesture: .left)
} else if yaw < -headThreshold {
handleGesture(gesture: .right)
}
} else if rollHandled {
if roll > headThreshold {
handleGesture(gesture: .forward)
} else if roll < -headThreshold {
handleGesture(gesture: .back)
}
}
}
}
```
#### Transfer movements to the computer
Once movements are handled and mapped, they have to be delivered to the computer within a local network environment.
We used [Bonjour](https://developer.apple.com/bonjour/) broadcasting to get this done.
```swift
import DTBonjour
final class Broadcaster: NSObject {
private let server: DTBonjourServer
override init() {
self.server = DTBonjourServer(bonjourType: "_mt_broadcasting_protocol._tcp.")
super.init()
self.server.delegate = self
}
func start() {
self.server.start()
}
func broadcast(gesture: HeadGesture) {
let encoder = JSONEncoder()
do {
let data = try encoder.encode(gesture)
self.server.broadcast(data)
} catch {
print(error)
}
}
}
```
The video demonstrates how an iOS application connects to the headphones, gets the motion data, and handles the head movements:
./motion-control.mp4
### Client
Once the Proxy side is ready, we can switch to the client side.
The main tasks here are:
1. Observe head movements from the proxy.
2. Send corresponding events to the system events chain to control the computer.
3. Provide a user interface to get the user acquainted with available movements.
#### Observe head movements
Since we have used [Bonjour](https://developer.apple.com/bonjour/) service to broadcast mapped movements to the local network, the same technology should be used to handle them.
Browse available services:
```swift
import Foundation
final class ServersBrowser: NSObject, ObservableObject {
var onServersChanged: (([Server]) -> Void)?
@Published private(set) var servers: [Server]
private let mockupServiceType = "_mt_broadcasting_protocol._tcp.";
private let workingQueue = DispatchQueue(label: "ServersBrowser.workingQueue")
private let receiverQueue = DispatchQueue.main
private let bonjourBrowser: NetServiceBrowser
init(bonjourBrowser: NetServiceBrowser = .init(), servers: [Server] = []) {
self.bonjourBrowser = bonjourBrowser
self.servers = servers
super.init()
self.bonjourBrowser.delegate = self
}
func start() {
bonjourBrowser.searchForServices(ofType: mockupServiceType, inDomain: "")
}
func stop() {
bonjourBrowser.stop()
}
}
```
Connect to the service:
```swift
import AppKit
class BonjourConnection: NSObject {
var onGestureReceived: ((HeadGesture) -> Void)?
let internalQueue = DispatchQueue(label: "BonjourConnection.internal")
let connection: DTBonjourDataConnection
init(service: NetService) {
self.connection = DTBonjourDataConnection(service: service)
super.init()
self.connection.delegate = self
}
func open() {
connection.open()
}
func close() {
connection.close()
}
}
extension BonjourConnection: DTBonjourDataConnectionDelegate {
func connection(_ connection: DTBonjourDataConnection!, didReceive object: Any!) {
self.internalQueue.async {
guard let data = object as? Data else { return }
let decoder = JSONDecoder()
do {
let gesture = try decoder.decode(HeadGesture.self, from: data)
DispatchQueue.main.async {
self.onGestureReceived?(gesture)
}
} catch {
print(error)
}
}
}
}
```
#### Update system events stream
Once head movements are handled, the system events stream should be updated to control the system.
We will simplify this task by simulating keyboard events, so the system will handle them accordingly.
```swift
private extension Keyboard {
func simulateKeyboardKey(_ key: UInt16) {
let rightArrowKeyCode: UInt16 = key
let keyDownEvent = CGEvent(keyboardEventSource: nil, virtualKey: rightArrowKeyCode, keyDown: true)
keyDownEvent?.post(tap: CGEventTapLocation.cghidEventTap)
let keyUpEvent = CGEvent(keyboardEventSource: nil, virtualKey: rightArrowKeyCode, keyDown: false)
keyUpEvent?.post(tap: CGEventTapLocation.cghidEventTap)
}
}
```
This code uses system Accessibility features to simulate keyboard events. The system will prompt the user with consent to accept those events.
The client application will be added to the Privacy->Accessibility section of the System Preferences:
#### Preferences Window
To get the users acquainted with the movements available, we have implemented a preferences window.
It demonstrates supported head movements and the keystrokes that bind to them.
Now, a macOS system can be controlled by head movements using Apple AirPods.
## Live Demo
https://www.youtube.com/watch?v=2bQwUyRaa78
## Conclusion
Regardless of the system limitations and unavailability of some CoreMotion features on macOS, W1-chip-powered headphones could be a great addition to the built-in accessibility features.
We can use various head movements to map system events and simplify navigation by implementing key binding or other types of events.
## Source Code
* [Proxy. iOS App](./HeadNavigationProxy.zip)
* [Client. macOS App](./HeadGesturesClient.zip)
## Links
* [Apple's Accessibility](https://www.apple.com/accessibility/)
* [Spatial Audio](https://support.apple.com/en-mk/guide/airpods/dev00eb7e0a3/web)
* [CoreMotion](https://developer.apple.com/documentation/coremotion)
* [Bonjour](https://developer.apple.com/bonjour/)
* [Pitch, Yaw, Roll Wiki](https://simple.wikipedia.org/wiki/Pitch,_yaw,_and_roll)
> This is an independent publication, and it has not been authorized, sponsored, or otherwise approved by Apple Inc.
---
# Passwordless Authentication: Possible Implementation of User Authentication During Registration and Login Using Passkeys
**Authors**: Taras Neporozhniy
**Published**: 2022-10-24
**Topics**: security
**Type**: Tech Note
**URL**: https://research.macpaw.com/publications/passwordless-authentication
Passwords are the original sin of authentication on the web.
Strong and unique passwords are hard to remember and still phishable, while memorable passwords are easy to guess, and people often use one such password across several different accounts.
The introduction of the second authentication factor does fix most of the security issues above, but makes the authorization process more complicated.
Authorization through other providers is one of the attempts to implement authentication without a password.
However, it has multiple disadvantages on its own: the need to support many providers, the issue of correctly merging the accounts of a user with different emails (for example, the user *cat* can use `cat.github@example.com` for GitHub and `cat.fb@example.com` for Facebook), last but not least comes the cluttered UI.
[Passkeys](https://developer.apple.com/passkeys/) is an authentication method based on biometric recognition.
By using Touch ID and Face ID, available on over a billion devices, for authentication with passkeys, a user gets a better experience than in a password-based flow, plus security problems like weak and reused credentials, leaks, and phishing are no longer possible.
The web authentication [specification](https://w3c.github.io/webauthn/), which defines the requirements for acceptable credentials — like passkeys — for strongly authenticating users, is still in draft, so there may be changes in the future.
## Authentication without passkeys
Passkey credentials are always attached to an existing user, so some user registration flow must already be in place before using passkeys for authentication.
This registration flow can also be useful when a user does not have any biometric scanners on their device (so passkeys cannot be used in principle) and can be based on a no-password authentication method.
This way, we can eliminate the need for passwords from the start, for example, by implementing an email-only authentication with a "Magic Link" method, which we showcase below.
**Registration flow:**
- User fills in their email address and submits the form.
- Server checks that the user with this email doesn't exist.
- The registration button is shown to the user.
- Server sends a verification letter to the email address on the registration button click.
- User opens the email and clicks the verification link.
The server creates a new account and redirects the new user to a private page, for example, a user profile.
./magic-link.mp4
**Login flow:**
- User fills in the email address and submits the form.
- Server checks that there is a user with this email.
- The login button is shown to the user.
- Server sends a sign-in letter to the email address on the login button click.
- User opens the email and clicks the sign-in link.
- Server redirects the existing user to a private page, for example, a user profile.
There [are a lot of articles](https://auth0.com/docs/authenticate/passwordless/authentication-methods/email-magic-link) on the web about the technical details of Magic Link authentication method implementation.
This method can be used for authentication by itself, but it has some disadvantages:
- The authentication email might get into spam, making it challenging for the user to understand what is wrong.
- Each time during login, the user needs to interact with an email client. It's even worse if the user needs to log in on somebody else's device.
Using passkeys as an additional authentication method would eliminate these disadvantages altogether.
## Using passkeys
Before we continue, let's clarify several passwordless authentication terms that are often confused:
- **FIDO** - **F**ast **Id**entity **O**nline - the consortium that develops secure, open, and phishing-proof passwordless authentication standards
- **FIDO2** - a passwordless authentication protocol
- **WebAuthn** - a specification for browser API written by W3C and FIDO with the participation of other companies
- **passkeys** - implementation of WebAuthn, which involves credentials sharing between devices (for example, with iCloud)
### Registering passkeys
Passkeys need to be registered before they are used in an authentication flow: the authenticator creates a pair of keys (private and public), the public key should be validated and saved on the server side and the private key is stored on the authenticator side.
There are two common ways to suggest to the users to provide biometric data:
- The first one is right after the user is registered (after checking whether the client supports WebAuthn at all):
- The second one, which can be used for existing users, is typically the "Add passkeys" button somewhere in the user profile interface:
./register-attach.mov
For the second case, the client and the server should correctly follow the WebAuthn specification to avoid multiple registrations for the same device (`excludeCredentials` parameter in registration options).
Passkey registration is as simple as scanning a fingerprint or a face.
Below are examples of registration flow for an iOS application:
./example-ios.mp4
Starting with macOS Ventura, all passkey credentials are shared through iCloud.
It is important to note that different authenticators on the same device can have their own private keys.
For now, Chrome and Safari [don't share credentials](https://developers.google.com/identity/passkeys/supported-environments) among themselves.
It is important to consider during development that the relationship between the user and their credentials is one-to-many, not one-to-one, as with a password.
### Authentication flows
Two common login scenarios with WebAuthn are Modal UI and Conditional UI.
#### Modal UI flow
1. User fills in the username (or their email, depending on what is used by the server as a unique identifier).
2. Server checks if the user has previously registered some public keys and returns to the client `allowCredentials` options key with all the list `ids` of the user's public key identifiers.
3. Client calls WebAuthn `getCredentials` method with this `ids` list.
4. If the authenticator (for example, a browser) has one of these credentials registered, it will ask the user for a biometric scan.
5. In response to a successful biometric scan, the authenticator returns the signature and all other data needed for the server to be verified by a previously registered public key.
./login-modal.mov
#### Conditional UI flow
[Conditional UI](https://github.com/w3c/webauthn/wiki/Explainer:-WebAuthn-Conditional-UI) flow is based on autofill and is supported on the new macOS Ventura.
A browser can check if this flow is supported with `.isConditionalMediationAvailable()` method.
With Conditional UI flow, the client does not need to send a username or email to the server.
The browser will auto-suggest all accounts for which there were previously registered credentials for this website.
The server can filter this list with `allowCredentials` option if the client sends a username or email, but it is not required.
To retrieve data for autocomplete, the `getCredentials` method, in this case, should be called as soon as possible on the page.
The previous Modal UI flow example can be easily transformed into a Conditional UI flow (technical details will come a little later):
./login-conditional.mov
The browser retrieves all available passkeys for the current website from iCloud — Suggests them to autofill — Scans Touch ID — All done.
If the user hasn't registered passkeys or the client (browser) doesn't support passkeys at all, the application can go the backup way with Magic Link authentication — no passwords are used in any case here.
#### Login by phone
Passkeys brought one more feature to WebAuthn - authentication by QR code.
The feature comes in handy when the user needs to log in on someone else's computer without sharing any credentials (making unnecessary "Do not remember me" checkbox).
Logging in with a QR code works by default between devices (for now, only the ones with the latest versions of iOS and macOS can do this) without a single extra line of code.
./login-phone.mp4
Communication between devices is handled securely by Bluetooth, and sharing or stealing the QR code will not lead to authentication.
### Sharing credentials
Users can also share passkeys via AirDrop.
Here is an example of sharing passkeys through AirDrop from Apple Keynote:
./sharing.mp4
Most likely, applications that are open almost all the time and keep a limit of active sessions will not care about this.
Also, sharing credentials will most likely not bother native apps that link their license to the device id.
But for browser applications that perform some action for a short time and close immediately (for example, BrowserStack), sharing one user account between multiple people is not a very good idea for license reasons.
### Removing passkeys
Before macOS Ventura, removing passkeys from Safari was not possible.
No,w passkeys can be managed from the Passwords settings in Safari:
Keychain Access doesn't seem to know anything about passkeys yet.
## Technical details
Let's define more terms before proceeding with the technical details:
- **Base64url encoding** - base64 encoding using the URL- and filename-safe character set defined in Section 5 of [RFC4648](https://www.w3.org/TR/webauthn/#biblio-rfc4648), with all trailing `=` characters omitted and without the inclusion of any line breaks, whitespace, or other additional characters.
- **_Challenge_** - a random 32 bytes. In JavaScript, it can be created with a call to `crypto.getRandomValues(new Uint8Array(32)` method. The _Challenge_ should not be a static value. It should be re-generated each time before the relying party sends it to the client. In the Modal UI flow, the _Challenge_ can be associated with the user by a one-to-one relation. But the relying party doesn't know anything about the user yet during the Conditional UI flow authentication step. That's why storing challenges separately from users with a nullable `user_id` field is better.
- **Relying Party** (`rp`) - below, we assume it is a web server.
- **Relying Party Identifier** (`rp.id`) - it is a caller's origin's [effective domain](https://html.spec.whatwg.org/multipage/browsers.html#concept-origin-effective-domain).
- **User Name** - while this field is called *name*, it seems that there are no restrictions on using an email instead of a name. There must be an advantage for the Modal UI flow if this string is unique for each user, because this string is used to select all the public key IDs from the storage to set the `allowCredentials` authentication parameter.
- **User Handle** - specified by the _Relying Party_, as the value of the `user.id` key. A user handle is an opaque byte sequence with a maximum size of 64 bytes and is not meant to be displayed to the user.
- **Assertion** - a signed statement returned by the authenticator carrying data sent by the _Relying Party_ and the client.
### Registration
```mermaid
sequenceDiagram
Client->>Server: generate credential options for the user
Server->>Client: options (challenge, user.id, rp.id, ...)
Client->>Authenticator: navigator.credentials.create(options)
Authenticator->>Client: credential (id, clientDataJSON, publicKey, authenticatorData, ...)
Client->>Server: validate and save credentials for the user
```
First, [options should be generated](https://www.w3.org/TR/webauthn-2/#dictionary-makecredentialoptions) on the server side in response to the client's request. The browser API expects some of these options parameters, like `challenge` or `user.id`, to have the `BufferSource` format. The options can only be passed to the client if they are compatible with JSON format, so they are encoded to a `base64url` string:
```js
const challenge = crypto.getRandomValues(new Uint8Array(32));
const options = {
challenge: Base64URL.fromBuffer(challenge),
rp: {
id: 'example.com', // effective domain
name: 'Example Inc',
},
user: {
id: 'some_user_id', // e.g. crypto.randomUUID()
name: 'username_or_email',
displayName: 'username_or_email',
},
pubKeyCredParams: [
{ type: 'public-key', alg: -7 }, // ES256
{ type: 'public-key', alg: -257 }, // RS256
],
authenticatorSelection: {
userVerification: 'required', // or preferred
authenticatorAttachment: 'platform',
},
attestation: 'none',
timeout: 60000,
};
// attach to options created user's credential ids to avoid duplicates:
if (user.credentials) {
options.excludeCredentials = user.credentials.map((id) => ({
type: 'public-key',
id,
}));
}
```
On the client side, some values should be decoded back to binary format and passed as options to `navigator.credentials.create` method:
```js
const options = {
challenge: Base64URL.toBuffer(json.challenge),
rp: json.rp,
user: {
id: UTF8.toBuffer(json.user.id),
name: json.user.name,
displayName: json.user.displayName,
},
pubKeyCredParams: json.pubKeyCredParams,
authenticatorSelection: json.authenticatorSelection,
attestation: json.attestation,
timeout: json.timeout,
};
if (json.excludeCredentials) {
options.excludeCredentials = json.excludeCredentials.map((credential) => ({
...credential,
id: Base64URL.toBuffer(credential.id),
}));
}
const credential = await navigator.credentials.create({ publicKey: pkOptions });
```
Convert the created `credential` to JSON format and send it to the server:
```js
const attestationResponse = /** @type {AuthenticatorAssertionResponse} */ (credential.response);
const json = {
credentialId: credential.id,
credentialType: credential.type,
clientDataJSON: Base64URL.fromBuffer(attestationResponse.clientDataJSON),
authenticatorData: Base64URL.fromBuffer(attestationResponse.authenticatorData),
signature: Base64URL.fromBuffer(attestationResponse.signature),
};
if (attestationResponse.userHandle) {
json.userHandle = Base64URL.fromBuffer(attestationResponse.userHandle);
}
return json;
```
On the server side, `credential` should be [validated](https://www.w3.org/TR/webauthn-2/#sctn-registering-a-new-credential) and saved to storage:
```js
if (!attestationJSON.credentialId) {
throw Error('Attestation must contain credential id');
}
if (attestationJSON.credentialType !== 'public-key') {
throw Error('Credential type must be public-key');
}
// parse client data
const clientDataBuffer = Base64URL.toBuffer(attestationJSON.clientDataJSON);
const clientDataJSON = JSON.parse(UTF8.fromBuffer(clientDataBuffer));
// validate type
if (clientDataJSON.type !== 'webauthn.create') {
throw Error('Response type must be webauthn.create');
}
// validate challenge
if (clientDataJSON.challenge !== expectedChallenge) {
throw Error('Response challenge does not match the expected');
}
// validate origin
if (!expectedOrigin(clientDataJSON.origin)) {
throw Error('Response origin does not match the expected');
}
// algorithm should be one of those passed to publicKey creation options
if (![-7, -256].includes(attestationJSON.algorithm)) {
throw Error('Algorithm code must be one of: -7, -257');
}
// parse authenticatorData
const authDataBuffer = Base64URL.toBuffer(attestationJSON.authenticatorData);
const authData = parseAuthenticatorData(authDataBuffer);
// validate relying part id
const authDataRpIdHex = SHA256.toHex(authData.rpId);
const expectedRpIdHash = await SHA256.fromString(expectedRpId);
const expectedRpIdHex = SHA256.toHex(expectedRpIdHash);
if (expectedRpIdHex !== authDataRpIdHex) {
throw Error('Relying party id does not match the expected');
}
// expect user presence flag
if (!authData.flags.up) {
throw Error('User presence flag expected to be true');
}
// expect user verified flag
if (!authData.flags.uv) {
throw Error('User verified flag expected to be true');
}
// object to save in storage along with user.id
return {
id: attestationJSON.credentialId,
publicKey: attestationJSON.publicKey,
algorithm: attestationJSON.algorithm,
counter: authData.counter,
};
```
The authenticator (for example, browser) increments the` counter` for each successful `getCredential` operation by some positive value.
A _Relying Party_ stores the signature counter of the most recent `getCredential` operation.
If either counter value is non-zero, and the new `counter` value is less than or equal to the stored value, a cloned authenticator may exist, or the authenticator may be malfunctioning.
### Authentication
Modal and Conditional UI flows are slightly different.
In the Modal UI flow, the _Relying Party_ must know which user is trying to log in; in the Conditional UI flow, the _Relying Party_ gets the user ID (`userHandle`) from the authenticator response.
#### Modal UI
```mermaid
sequenceDiagram
Client->>Server: generate challenge options for the user
Server->>Client: options (challenge, allowCredentials, rp.id, ...)
Client->>Authenticator: navigator.credentials.get(options)
Authenticator->>Client: credential (id, clientDataJSON, authenticatorData, signature, ...)
Client->>Server: validate signature, set new counter value
```
#### Conditional UI
```mermaid
sequenceDiagram
Client->>Server: generate random challenge
Server->>Client: options (challenge, rp.id, ...)
Client->>Authenticator: navigator.credentials.get(options)
Authenticator->>Client: credential (userHandle, id, clientDataJSON, authenticatorData, signature, ...)
Client->>Server: validate signature for the userHandle, set new counter value
```
Relying party [creates and sends the options in JSON format](https://www.w3.org/TR/webauthn-2/#dictionary-assertion-options) to the client:
```js
const challenge = crypto.getRandomValues(new Uint8Array(32));
const options = {
challenge: Base64URL.fromBuffer(challenge),
rpId: settings.rpId,
userVerification: 'required',
timeout: 60000,
};
// required for modal ui, optional for conditional ui
if (user.allowCredentialIds) {
options.allowCredentials = settings.allowCredentialIds.map((id) => ({
type: 'public-key',
id,
}));
}
return options;
```
The client decodes some option values to `ArrayBuffer` format and passes them to `navigator.credentials.get`:
```js
const options = {
challenge: Base64URL.toBuffer(json.challenge),
rpId: json.rpId,
userVerification: json.userVerification,
timeout: json.timeout,
};
if (json.allowCredentials) {
options.allowCredentials = json.allowCredentials.map((credential) => ({
...credential,
id: Base64URL.toBuffer(credential.id),
}));
}
return options;
```
For the Conditional UI flow, `webauthn` should be added to `autocomplete` input attribute:
```js
```
and `mediation` option should be set to `conditional` value:
```js
const credential = await navigator.credentials.get({ publicKey: pkOptions, mediation: "conditional" });
```
The method `navigator.credentials.get` should be called as early on the page as possible to fetch all passkeys for the current user and suggest them for autocomplete input.
The client converts the output credential from the `navigator.credentials.get` method to JSON and passes it to the _Relying Party_:
```js
const attestationResponse = /** @type {AuthenticatorAssertionResponse} */ (credential.response);
const json = {
credentialId: credential.id,
credentialType: credential.type,
clientDataJSON: Base64URL.fromBuffer(attestationResponse.clientDataJSON),
authenticatorData: Base64URL.fromBuffer(attestationResponse.authenticatorData),
signature: Base64URL.fromBuffer(attestationResponse.signature),
};
if (attestationResponse.userHandle) {
json.userHandle = Base64URL.fromBuffer(attestationResponse.userHandle);
}
return json;
```
The _Relying Party_ [validates the assertion](https://www.w3.org/TR/webauthn-2/#sctn-verifying-assertion) and updates the counter value (if the authenticator supports it, the value should not be equal to 0):
```js
const { expectedChallenge, expectedRpId, expectedOrigin } = options;
if (!assertionJSON.credentialId) {
throw Error('Attestation must contain credential id');
}
if (assertionJSON.credentialType !== 'public-key') {
throw Error('Credential type must be public-key');
}
// parse client data
const clientDataBuffer = Base64URL.toBuffer(assertionJSON.clientDataJSON);
const clientDataJSON = JSON.parse(UTF8.fromBuffer(clientDataBuffer));
// validate type
if (clientDataJSON.type !== 'webauthn.get') {
throw Error('Response type must be webauthn.get');
}
// validate challenge
if (clientDataJSON.challenge !== expectedChallenge) {
throw Error('Response challenge does not match the expected');
}
// validate origin
if (!expectedOrigin(clientDataJSON.origin)) {
throw Error('Response origin does not match the expected');
}
// parse authenticatorData
const authDataBuffer = Base64URL.toBuffer(assertionJSON.authenticatorData);
const authData = parseAuthenticatorData(authDataBuffer);
// validate relying part id
const authDataRpIdHex = SHA256.toHex(authData.rpId);
const expectedRpIdHash = await SHA256.fromString(expectedRpId);
const expectedRpIdHex = SHA256.toHex(expectedRpIdHash);
if (expectedRpIdHex !== authDataRpIdHex) {
throw Error('Relying party id does not match the expected');
}
// expect user presence flag
if (!authData.flags.up) {
throw Error('User presence flag expected to be true');
}
// expect user verified flag
if (!authData.flags.uv) {
throw Error('User verified flag expected to be true');
}
// validate counter
if ((credential.counter > 0 || authData.counter > 0)) {
if (authData.counter <= credential.counter) {
throw Error('Invalid counter');
}
}
// validate signature
const clientDataHash = await SHA256.fromBuffer(clientDataBuffer);
const concatenatedBuffer = concatenateBuffers(authDataBuffer, clientDataHash);
const publicKey = Base64URL.toBuffer(credential.publicKey);
const algorithm = SHA256.identify(Number(credential.algorithm));
let signature = Base64URL.toBuffer(assertionJSON.signature);
if (algorithm.name === 'ECDSA') {
signature = SHA256.der2raw(signature);
}
const cryptoKey = await crypto.subtle.importKey('spki', publicKey, algorithm, false, ['verify']);
const verified = await crypto.subtle.verify(algorithm, cryptoKey, signature, concatenatedBuffer);
return {
credentialId: credential.id,
verified,
counter: authData.counter,
};
```
A prototype for this project is available [from this link](./prototype.zip).
It implements the full WebAuthn Client / Server working example with [Deno](https://deno.land/) and SQLite.
## Conclusion
Passkeys are a reliable and convenient solution to get rid of passwords.
They can work as the second step of two-factor authentication or independently with a password as a fallback authentication method.
Sharing credentials via iCloud provides the same advantages as a keychain with a password currently has.
With passkeys, users can safely log in on someone else's device without the risk of password spying.
Unlike many previous attempts to make authorization easier, passkeys look very promising.
## Code
- [Research prototype (Deno, sqlite)](./prototype.zip)
## Links
- [W3C specification](https://www.w3.org/TR/webauthn/)
- [Apple WWDC 2022 keynote](https://developer.apple.com/videos/play/wwdc2022/10092/)
- [Resources about passkeys](https://www.passkeys.io/)
- [About passkeys by the W3C WebAuthn Community and the FIDO Alliance (WIP)](https://passkeys.dev/)
- Javascript helpers for WebAuthn: [SimpleWebAuthn](https://simplewebauthn.dev/), [fido2-lib](https://github.com/webauthn-open-source/fido2-lib)
- [Matthew Miller's blog about passkeys with reviews of new features on different platforms](https://blog.millerti.me/)
- [WebAuthn test app](https://webauthntest.identitystandards.io/)
- [W3C GitHub discussions about WebAuthn](https://github.com/w3c/webauthn)
- [Ackermann Yuriy's blog about FIDO (FIDO Alliance member)](https://medium.com/webauthnworks/introduction-to-webauthn-api-5fd1fb46c285#16e8)
- [FIDO Alliance \"Passkeys in Action\" Demo](https://www.youtube.com/watch?v=SWocv4BhCNg)
> This is an independent publication, and it has not been authorized, sponsored, or otherwise approved by Apple Inc.
---
# Diskoverse. Visualizing macOS Filesystem Using Augmented Reality
**Authors**: Sergii Kryvoblotskyi
**Published**: 2022-04-21
**Topics**: human-computer-interaction
**Type**: Highlight
**URL**: https://research.macpaw.com/publications/diskoverse
## Introduction
At MacPaw, we place a lot of emphasis on user interfaces and innovation.
We believe that the way the product communicates and presents itself to the user is one of the fundamental components of its success and applicability in the real world.
The faster and easier the service solves the user's problem, and the clearer the service looks, the more likely the service will become a habit.
## Interfaces
_Human–computer interaction (HCI)_ research in the design and the use of computer technology focuses on the interfaces between people (users) and computers.
HCI researchers observe how humans interact with computers and design technologies that allow humans to interact with computers in novel ways.
Humans interact with computers in many ways, and the interface between the two is crucial to facilitating this interaction.
HCI is also sometimes termed human-machine interaction (HMI), man-machine interaction (MMI), or computer-human interaction (CHI).
Desktop applications, internet browsers, handheld computers, and computer kiosks use today's prevalent graphical user interfaces (GUIs).
Voice user interfaces (VUI) are used for speech recognition and synthesizing systems, and the emerging multi-modal user interfaces allow humans to engage with embodied character agents in a way that cannot be achieved with other interface paradigms.
The growth in the human-computer interaction field has led to an increase in the quality of interaction and resulted in many new areas of research beyond it.
Instead of designing regular interfaces, the different research directions explore the concepts of multimodality over unimodality, intelligent adaptive interfaces over command- and action-based ones, and active interfaces over passive ones.
The interaction between users and computers has also changed with the development of technologies and interfaces. If initially keyboards were used for input, and tapes and punch cards were used for output, then with technological progress, we got a large variety of input and output devices.
Let's look at a brief comparison of how HCI evolved through time.
### Artificial reality, virtual reality
AR/VR devices have become another step in the evolution of I/O devices and user interaction with a computer.
**Augmented reality (AR)** is an interactive experience of a real-world environment where the objects in the real world are enhanced by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory, and olfactory.
**Virtual reality (VR)** is a simulated experience similar to or completely different from the real world.
Applications of virtual reality include entertainment (video games), education (medical or military training), and business (virtual meetings).
Other types of VR-style technology include augmented reality and mixed reality, sometimes referred to as **extended reality (XR)**.
Augmented reality (AR) and virtual reality (VR) technology originated in 1838 when Charles Wheatstone invented the stereoscope. This technology uses an image for each eye to create a 3D image for the viewer.
Since then, this technology has evolved rapidly but has remained niche.
However, as graphics and computing technologies have evolved in the past few years, AR and VR have experienced a renaissance.
Consumer headsets like the Oculus Rift and the HTC Vive are helping gamers and designers reimagine their interfaces. In the same way, businesses are using these technologies to train employees and market new products.
Currently, they are widely used in medicine, science, entertainment, games, etc.
### Apple glasses
Based on patent filings, Apple has been exploring virtual reality and augmented reality technologies for more than 10 years, but with virtual and augmented reality exploding in popularity with the launch of ARKit, Apple's dabbling is growing more serious and could lead to an actual dedicated AR/VR product in the not too distant future.
Apple is rumored to have a secret research unit with hundreds of employees working on AR and VR and exploring ways to use the emerging technologies in future Apple products.
VR/AR hiring has ramped up over the last several years, and Apple has acquired multiple AR/VR companies as it furthers its work in the AR/VR space.
Former Apple hardware engineering chief Dan Riccio in January 2021 transitioned to a new role, where he is overseeing Apple's work on an AR/VR headset.
The project has faced development challenges, and Apple execs believe that Riccio's focus may help.
Though he oversees the overall project, Mike Rockwell continues to lead the day-to-day efforts.
Apple is also rumored to be working on at least two AR projects, including an augmented reality headset set to be released right around 2022, followed by a sleeker pair of augmented reality glasses coming later.
Many rumors have focused solely on the glasses, leading to some confusion about Apple's plans, but it appears the headset will be the first product launched.
### realityOS
2022 brings more evidence that Apple is working on some AR/VR technologies.
There were different leaks that independent researchers have detected.
Apple has accidentally [force-pushed](https://twitter.com/stroughtonsmith/status/1491266972226490368?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1491266972226490368%7Ctwgr%5E%7Ctwcon%5Es1_&ref_url=https%3A%2F%2F9to5mac.com%2F2022%2F02%2F09%2Frealityos-apple-headset%2F) `TARGET_FEATURE_REALITYOS` macros to the `oss-distribution` repository and removed it eventually.
The same thing [happened](https://en.wikipedia.org/wiki/BridgeOS) before with `bridgeOS`.
Later this year, the `com.apple.platform.realityos` key was [spotted](https://twitter.com/renssies/status/1483038609208975360?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1483038609208975360%7Ctwgr%5E%7Ctwcon%5Es1_&ref_url=https%3A%2F%2F9to5mac.com%2F2022%2F02%2F09%2Frealityos-apple-headset%2F) in the App Store application logs.
Also, _realityOS_ trademark has been [registered](https://9to5mac.com/2022/05/29/realityos-trademark-related-to-apple-ar-vr-headset/) by Realityo System LLC, which points to "Corporation Trust Center" — a real company that provides trademark services and of which Apple is a client.
### WWDC22
Although _realityOS_ was not presented at WWDC22, some user interface parts of the updated operating systems have changed: macOS and iOS 16 got [Stage Manager](https://www.apple.com/cz/newsroom/2022/06/ipados-16-takes-the-versatility-of-ipad-even-further/).
Stage Manager is an entirely new multitasking experience that automatically organizes apps and windows, making it quick and easy to switch between tasks.
For the first time on iPad, users can create overlapping windows of different sizes in a single view, drag and drop windows from the side, or open apps from the Dock to create groups of apps for faster, more flexible multitasking.
This feature brought back 3D transitions for their groups that could lead to the future of _realityOS_ and AR/VR interfaces.
## Use case: filesystem visualization
Given the growing interest in AR/VR technologies and the potential release of the headset from Apple, our goal is to be ready for this event and validate to which extent, with the current set of tools, it is possible to create AR interaction interfaces and how comfortable they are for the user.
Since we have extensive experience working with the file system, we decided to take the visualization of computer disk space using AR technologies as a test task.
Our goal is to create a PoC application that will make it possible to visualize a MacBook file system using AR.
This task consists of 4 key parts.
### There is no Apple AR headset yet
At this point, Apple still hasn't unveiled a prototype of Apple Glasses, and there's no guarantee it will ever happen.
Existing AR headsets do not have native integration with the macOS SDK, and the implementation will be different for each platform, which is inconvenient.
Nevertheless, Apple puts a lot of effort into developing AR/VR technologies on its iOS devices.
It provides a set of APIs for implementing AR, in particular, these:
* [ARKit](https://developer.apple.com/augmented-reality/arkit/)
* [RealityKit](https://developer.apple.com/documentation/realitykit)
* [Scenekit](https://developer.apple.com/documentation/scenekit?language=objc)
* [SpriteKit](https://developer.apple.com/documentation/spritekit/)
* [Vision](https://developer.apple.com/documentation/vision)
Configuring horizontal planes tracking with ARKit:
```swift
func setUpSceneView() {
let configuration = ARWorldTrackingConfiguration()
configuration.planeDetection = .horizontal
if ARWorldTrackingConfiguration.supportsFrameSemantics(.personSegmentationWithDepth) {
configuration.frameSemantics.insert(.personSegmentationWithDepth)
}
sceneView.session.delegate = self
sceneView.session.run(configuration)
sceneView.delegate = self
}
```
Placing a folder node on a plain:
```swift
class FolderNode: FileNode {
override init(file: File) {
let geometry = SCNBox(width: 0.1, height: 0.1, length: 0.1, chamferRadius: 0.005)
let material = SCNMaterial()
material.locksAmbientWithDiffuse = true
material.isDoubleSided = false
material.ambient.contents = UIColor.red
geometry.materials = [material]
super.init(file: file)
}
}
func addFileNode(with file: File, to planeNode: SCNNode, at position: SCNVector3? = nil) {
let node = nodesBuilder.makeFileNode(with: file)
node.position = position ?? sceneView.convenientCenterPointToPlaceAnObject(for: planeNode)
sceneView.scene.rootNode.addChildNode(node)
}
```
As a result, there is a possibility to place SceneKit objects on a plane:
Based on this, these technologies will likely be used in a future prototype.
Thus, Apple gives us a hint that using the current technology on the market, we can already simulate Apple's AR headset using any iOS device that supports the technologies from the above.
### Handling user input in AR
User input in AR/VR is entirely different from that we have on touch- and keyboard-based devices.
To navigate within the disk structure, we will use recognized hand gestures by implementing hand pose detection by the Vision framework:
- Finger-point gesture: access a file or a folder
- Swipe down gesture: dismiss the details
```swift
final class HandTracking {
private let videoProcessingQueue = DispatchQueue(label: "HandsTrackingQueue", qos: .userInitiated)
private let handPoseRequest = VNDetectHumanHandPoseRequest()
func processPixelBuffer(pixelBuffer: CVPixelBuffer, completion: @escaping (CGPoint) -> Void) {
self.videoProcessingQueue.async {
let orientation = CGImagePropertyOrientation(UIDevice.current.orientation)
let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, options: [:])
do {
// Perform VNDetectHumanHandPoseRequest
try handler.perform([self.handPoseRequest])
// Continue only when a hand was detected in the frame.
// Since we set the maximumHandCount property of the request to 1, there will be at most one observation.
guard let observation = self.handPoseRequest.results?.first else {
return
}
// Get points for thumb and index finger.
let thumbPoints = try observation.recognizedPoints(.thumb)
let indexFingerPoints = try observation.recognizedPoints(.indexFinger)
// Look for tip points.
guard let thumbTipPoint = thumbPoints[.thumbTip], let indexTipPoint = indexFingerPoints[.indexTip] else {
return
}
// Ignore low confidence points.
guard thumbTipPoint.confidence > 0.3 && indexTipPoint.confidence > 0.3 else {
return
}
DispatchQueue.main.async {
completion(CGPoint(x: indexTipPoint.location.x, y: indexTipPoint.location.y))
}
} catch {
print(error)
}
}
}
}
```
Our implementation in action:
https://www.youtube.com/watch?v=Oe33e6MMELE
### Connecting a Mac to a device that doesn't exist
The next step is to link the AR device to a Mac.
Since we have no idea about the future protocol, we currently use the technology available.
For device discovery purposes, Apple has a technology called [Bonjour](https://developer.apple.com/bonjour/), which works on the "broadcast" principle.
A Mac, in our case, will act as a broadcaster and publish a service with a deterministic name to the local network.
Knowing the name of the service, our AR headset will be able to conduct a lookup and, if a service is found, will know the name of the server in the local network to implement communication using a standardized API — this will be enough for the connection to happen.
We implement a classic client-server architecture where the computer acts as a server and the AR headset as a client.
A quote from the official Bonjour documentation:
>Bonjour, also known as zero-configuration networking, enables automatic discovery of devices and services on a local network using industry-standard IP protocols. Bonjour makes it easy to discover, publish, and resolve network services with a sophisticated, easy-to-use programming interface that is accessible from Cocoa, Ruby, Python, and other languages.
Publish a service (server-side):
```swift
let netService = NetService(domain: domain, type: type, name: name)
netService.delegate = self
netService.publish()
```
Lookup for a service (client side):
```swift
final class DiskoverseFinder: NSObject {
var onFindDiskoverse: ((Diskoverse) -> Void)?
var onDiskoverseDisappeared: ((String) -> Void)?
private let browser = NetServiceBrowser()
func findDiskoverse() {
self.browser.delegate = self
self.browser.searchForServices(ofType: "_mo_broadcasting_protocol._tcp.", inDomain: "")
}
}
extension DiskoverseFinder: NetServiceBrowserDelegate {
func netServiceBrowser(_ browser: NetServiceBrowser, didFind service: NetService, moreComing: Bool) {
onFindDiskoverse?(Diskoverse(name: service.name))
}
func netServiceBrowser(_ browser: NetServiceBrowser, didRemove service: NetService, moreComing: Bool) {
onDiskoverseDisappeared?(service.name)
}
}
```
### Streaming data and service API from a Mac to an AR device and vice versa
Once we address the discovery problem, we need a standardized API that allows the AR device and the computer to transfer data to each other.
An important concept here is that the whole service should be "app-context" and not "machine-context."
In other words, the AR device would have access only to what the server provides, not the computer.
As a server part that can provide an interface in the form of an API, we use an HTTP server that will work on the local network.
We can use the REST API for communication once the service is started.
```swift
//Lists files
server["/list"] = {...}
//Returns file path for details
server["/file/:path"] = {...}
//Go backwards in file hierarchy
server["/backward"] = {...}
//Returns file's thumbnail path
server["/thumbnail/:path"] = {...}
//Change hierarchy pointer
server["/change/:path"] = {...}
```
macOS (server part) could be launched with no UI, like a service.
#### Components diagram
1. macOS (server) starts an HTTP service for REST API availability
2. macOS (server) starts Bonjour service broadcasting for discoverability
3. iOS (client) locates the service and executes the REST API to get the file system information
4. iOS (client) handles the user's input and notifies the server via REST API that the default folder has been changed and new content is required
## Conclusion
The study has shown that with a set of existing technologies available on Apple platforms, it is possible to simulate AR wearable headsets to implement prototypes.
iOS devices with the support of ARKit and Vision could be used as simulators.
**Demo**
https://www.youtube.com/watch?v=DR9O4oCgrkQ
## Links
* [YouTube](https://www.youtube.com/watch?v=V2PPjObxxXs)
* [ARKit](https://developer.apple.com/augmented-reality/arkit/)
* [RealityKit](https://developer.apple.com/documentation/realitykit)
* [Scenekit](https://developer.apple.com/documentation/scenekit?language=objc)
* [SpriteKit](https://developer.apple.com/documentation/spritekit/)
* [Vision](https://developer.apple.com/documentation/vision)
> This is an independent publication, and it has not been authorized, sponsored, or otherwise approved by Apple Inc.
---
# Static Analysis of macOS Application Bundles for Territorial Affiliation
**Authors**: Sergii Kryvoblotskyi
**Published**: 2022-04-21
**Topics**: software-analysis, security
**Type**: Tech Note
**URL**: https://research.macpaw.com/publications/static-analysis-country
## Introduction
Due to the full-scale invasion of the russian federation on February 24, 2022, on the territory of Ukraine, it became necessary to strengthen not only the country's physical defenses but also its digital protection.
In July 2016, russia enforced Federal bills No. 374-FZ and 375-FZ, which require telecom providers to store the content of voice calls, data, images, and text messages on russian servers for 6 months, and their metadata (e.g., time, location, message sender, and recipients) for 3 years.
Online services such as messengers, emails, and social networks that use encrypted data are required to permit the Federal Security Service (FSB) to access and read their encrypted communications.
It means that all internet and telecom companies that have some presence in russia are obliged to disclose these communications and metadata and "all other information necessary" to authorities on request without a court order.
With the above considerations, we need a way to discover potentially unwanted software with connections to the aggressor for further decision-making regarding its removal.
We will use static analysis in this research to examine macOS application bundles.
## Inspecting the bundle contents
Software development and quality assurance teams use static analysis in various software engineering tasks.
Static analysis is a computer program debugging method that examines the code and supplementary resources without executing the program.
The process provides an understanding of the code structure and can help ensure that the code adheres to industry standards.
Dedicated automated tools will scan all code in a project to check for vulnerabilities while validating the code.
We are primarily interested in discovering territorial affiliation information, so we experiment with several heuristics for code inspection that allow us to extract relevant pieces of data.
### Bundle identifiers
In the Apple ecosystem, a bundle ID or bundle identifier uniquely identifies an application, and no two applications can have the same bundle identifier.
To avoid identifier conflicts, Apple encourages developers to use reverse domain name notation to choose an application's bundle identifier.
The bundle identifier could contain specific country detection information, such as country ISO identifiers as parts of the reverse domain name: `ru.keepcoder.Telegram`
### Using a static database as a deny list
One basic and most common way to affiliate an application with its origin country is to have a static deny list, where the bundle ID can identify individual applications.
Example of a deny list database entry:
```json
{
"webLink" : "https:\/\/www.themoscowtimes.com\/2021\/03\/24\/telegram-raises-1bln-with-russian-direct-investment-fund-buying-bonds-a73346",
"reason" : "Probable ties to the Russian Federation",
"bundleIdSubstring" : "telegram",
"humanReadableDescription" : "Caution: Telegram doesn't verify or fact-check information shared by users or media in groups; bad actors can use this to spread disinformation. Probable ties to the Russian Federation, according to media publications."
}
```
In the example above, we use the "telegram" substring to detect the Telegram application family.
### Code signing information
As described in the Apple [Code Signing Guide](https://developer.apple.com/library/archive/documentation/Security/Conceptual/CodeSigningGuide/Introduction/Introduction.html):
> Code signing is a macOS security technology that you use to certify that you created an app. Once an app is signed, the system can detect any change to the app — whether the change is introduced accidentally or by malicious code.
> You participate in code signing as a developer when you obtain a signing identity and apply your signature to the apps you ship. A certificate authority (often Apple) vouches for your signing identity.
Bundle's code signature contains various security information along with the certificates that were used with the signature. Code signing information is [available](https://developer.apple.com/documentation/security) by `Security.framework`:
>A `SecCertificateRef` object for a certificate that is stored in a keychain can be safely cast to a `SecKeychainItemRef` for manipulation as a keychain item. On the other hand, if the `SecCertificateRef` is not stored in a keychain, casting the object to a `SecKeychainItemRef` and passing it to Keychain Services functions returns errors.
```swift
var codeRef: SecStaticCode?
guard let appURL = CFURLCreateWithFileSystemPath(kCFAllocatorDefault, (path as NSString), .cfurlposixPathStyle, true) else {
return
}
SecStaticCodeCreateWithPath(appURL, .init(rawValue: 0), &codeRef)
guard let codeRef = codeRef else {
return
}
var signingDictionary: CFDictionary?
SecCodeCopySigningInformation(codeRef, SecCSFlags(rawValue: kSecCSSigningInformation), &signingDictionary)
```
However, certificate information is not available publicly, only through a private API:
```objc
extern CFArrayRef SecCertificateCopyCountry(SecCertificateRef certificate);
@implementation CertificateOriginProvider
- (NSArray *)certificateOriginsFromCertificates:(SecCertificateRef)certificate {
if (&SecCertificateCopyCountry == NULL) {
return @[];
}
return (__bridge NSArray *)(SecCertificateCopyCountry(certificate));
}
@end
```
### Information property list file
Bundles, which represent executables of different kinds, contain an information property list file. This collection of key-value pairs specifies how the system should interpret the associated bundle. Some key-value pairs characterize the bundle itself, while others configure the app, the framework, or another entity the bundle represents. Some keys are required, while others are specific to particular features of the executable.
The point of interest here is two keys: `SUFeedURL` and `NSHumanReadableCopyright`.
`SUFeedURL` is the key that comes from a 3rd party open-source [Sparkle](https://sparkle-project.org) framework. It represents the URL where the appcast feed is hosted. The approach described in [AppStore metadata](#appstore-metadata) section can be applied here.
`NSHumanReadableCopyright` contains the copyright text shown in the "About" window. It often represents the developer's website or company name.
### Bundle localization
Localization is a common practice of making the application accessible across the world. macOS applications can be localized by including string representations of translations that are applied according to the device location.
`AppKit` has native APIs to retrieve the bundle's localizations list.
```swift
let bundle = Bundle(url: url)
let localizations = bundle.localizations
```
### Mach-O files
[Mach-O](https://developer.apple.com/library/archive/documentation/Performance/Conceptual/CodeFootprint/Articles/MachOOverview.html) is the native executable format of binaries in OS X and is the preferred format for shipping code. It determines the order in which the code and data in a binary file are read into memory. The ordering of code and data has implications for memory usage and paging activity, and thus directly affects the performance of a program.
A Mach-O binary is organized into segments. Each segment contains one or more sections. Code or data of different types go into each section. Segments always start on a page boundary, but sections are not necessarily page-aligned. The size of a segment is measured by the number of bytes in all the sections it contains and rounded up to the next virtual memory page boundary. Thus, a segment is always a multiple of 4096 bytes, or 4 kilobytes, with 4096 bytes being the minimum size.
The `__cstring` section contains literal string constants (quoted strings in source code), so they are definitely subjects for investigation for stored URLs in the binary.
In macOS, there are built-in tools to parse binary files and search for URLs therein.
```shell
$ strings /Applications/Setapp/CleanMyMac\ X.app/Contents/MacOS/CleanMyMac | grep "http:"
http://www.homesweeklies.com/homepage
http://www.weknow.ac
http://www.google.
http://www.apple.
http://www.news.com
```
Using the [MachO-Kit](https://github.com/DeVaukz/MachO-Kit) open-source 3rd party library:
```swift
func parseBinaryURLs(applicationURL: URL) -> [String] {
guard let bundle = Bundle(url: applicationURL), let executableURL = bundle.executableURL else {
return []
}
do {
let memoryMap = try MKMemoryMap(contentsOfFile: executableURL)
let offset: UInt32 = {
guard let fatBinary = try? MKFatBinary(memoryMap: memoryMap),
let slice64 = fatBinary.architectures.first(where: { $0.cputype == CPU_TYPE_X86_64 }) else {
return 0
}
return slice64.offset
}()
let macho = try MKMachOImage(name: "MKMemoryMap", flags: .init(rawValue: 0), atAddress: mk_vm_address_t(offset), inMapping: memoryMap)
guard let strings = macho.sections.first(where: { $0.value.name == "__cstring" }), let stringSection = strings.value as? MKCStringSection else {
return []
}
let siteURLs = stringSection
.strings
.compactMap { $0.string?.lowercased() }
.filter(stringsFilter)
return siteURLs
} catch {
return []
}
}
func stringsFilter(_ string: String) -> Bool {
let searchedStrings = [".ru", "vk.com", "yandex.com"]
return (string.hasPrefix("http") || string.contains("yandex")) && searchedStrings.contains(where: { string.lowercased().contains($0) })
}
```
## Augmenting analysis with external information
### AppStore metadata
For every application distributed through the App Store, we can obtain its metadata by using [App Store Search API](https://developer.apple.com/library/archive/documentation/AudioVideo/Conceptual/iTuneSearchAPI/index.html):
> The Search API allows you to place search fields in your website to search for content within the iTunes Store, App Store, iBooks Store, and Mac App Store. You can search for a variety of content, including apps, iBooks, movies, podcasts, music, music videos, audiobooks, and TV shows. You can also call an ID-based lookup request to create mappings between your content library and the digital catalog.
A Search API request example:
```bash
curl "https://itunes.apple.com/lookup?entity=software&bundleId=com.macpaw.CleanMyMac-mas" \
-H 'Content-Type: multipart/form-data; charset=utf-8; boundary=__X_PAW_BOUNDARY__'
```
Search API response:
```json
"kind": "mac-software",
"minimumOsVersion": "10.10",
"trackCensoredName": "CleanMyMac X",
"languageCodesISO2A": [
"NL",
"UK"
],
"fileSizeBytes": "88072777",
"sellerUrl": "https://macpaw.com",
"formattedPrice": "Free",
"contentAdvisoryRating": "4+",
"averageUserRatingForCurrentVersion": 0,
"userRatingCountForCurrentVersion": 0,
"averageUserRating": 0,
"trackContentRating": "4+",
"bundleId": "com.macpaw.CleanMyMac-mas",
"releaseDate": "2020-04-28T07:00:00Z",
"trackName": "CleanMyMac X",
"primaryGenreName": "Utilities",
"isVppDeviceBasedLicensingEnabled": true,
"sellerName": "MacPaw Inc.",
"currentVersionReleaseDate": "2022-05-24T09:22:29Z",
"releaseNotes": "Improved:\nEnhanced our malware detection system to make sure your Mac is protected at all times.\n\nFixed:\nMinor issues and known bugs.",
"primaryGenreId": 6002,
"currency": "USD",
"description": "Delete megatons of junk, malware, and make your Mac faster & more organized.\n\nCleanMyMac X packs 30+ tools to help you solve the most common Mac issues. You can use it to manage storage, apps, and monitor the health of your computer. There are even personalized cleanup tips based on how you use your Mac.\n\n\nKEY FEATURES\n\nFree up space\n\nDelete gigabytes of system junk, broken data, and caches.\nFind large and old files scattered across all folders.\nVisualize your storage and find your largest space-wasters.\n\nProtect your Mac\n\nScan your Mac for the latest viruses and adware.\nDelete malware agents like keyloggers, spyware, etc.\nClear out browsing history and tracking cookies.\n\nUninstall apps\n\nFind and delete unwanted apps completely.\nReset broken apps to their default state.\nRemove extensions and background plugins.\n\nMonitor Mac's health\n\nSee real-time data about battery and processor load.\nMonitor network speed and available memory.\nGet personalized Mac cleanup tips.\n\n\nAWARD-WINNING DESIGN\n\nWinner of iF Design Award 2020\nProduct Hunt "App of the month”\nMacStories "Must-Have Mac App” 2019\n\nCleanMyMac X turns the not so exciting task of cleaning your computer into a stylish and interactive ride. It places simplicity at the core of its design. With smart and self-learning algorithms under the hood, the app stays incredibly easy to use. \n\n\nWHAT MAC EXPERTS SAY \n\n"If you've found yourself struggling with a nearly full Mac, check out CleanMyMac X. The app has been an excellent way to recover space with minimal effort for many years and I expect it will continue to be so for many more.”\n\nMacStories\n\n"CleanMyMac X makes it easy to maintain a healthy Mac. Its built-in tools make it easy to rid your machine of unwanted apps and files, protect it against malware, and more.”\n\nCult of Mac\n\n"Users will appreciate CleanMyMac X's streamlined, attractive interface, which includes clear icons and gentle animations to make the scrubbing process pleasant.”\n\nVentureBeat\n\n"From insane speed improvements to malware removal, a new menu design, and more, this release is packed with new features that you are going to want to check out.”\n\niMore\n\n\nSUBSCRIPTION AND PRICING\n \nSome features are only partially available for non-paying users and require an in-app purchase. For example, the non-paying users can clean 500 MB of junk across all modules and up to 1 GB of junk in the Space Lens module.\n \nSee the pricing details in the Information section under In-App Purchases.\n \nHave questions? We are always here to help. Please message our support using the contact below.\nhttps://macpaw.com/support/contact\n\nTerms of Service https://macpaw.com/cleanmymac-x-terms-of-service-mas\nPrivacy Policy https://macpaw.com/cleanmymac-x-privacy-policy \n \nMac is a trademark of Apple Inc.\niTunes is a trademark of Apple Inc.",
"artistId": 403752295,
"artistName": "MacPaw Inc.",
"price": 0.00,
"version": "4.10.6",
"wrapperType": "software",
"userRatingCount": 0
```
Some of the fields are relevant to our investigation.
Fields `sellerName` and `artistName` can be checked for well-known country-affiliated developers or developer websites written there.
The field `sellerURL` can be checked for a match with an affiliated country domain name. For example, if the country of interest is russia, we can check if the "ru" subdomain appears in the full domain name.
Another important part of the `sellerURL` investigation is GeoIP information.
Each domain corresponds to a particular IP address which points to some physical server. To get the country of the server location, we can use Geo IP services. A built-in service in macOS is `whois`.
A query can be used to obtain an IP address from the domain name:
```bash
$ dig macpaw.com
;; ANSWER SECTION:
macpaw.com. 163 IN A 104.18.31.100
$ whois 104.18.30.100
% IANA WHOIS server
% for more information on IANA, visit http://www.iana.org
% This query returned 1 object
OrgName: Cloudflare, Inc.
OrgId: CLOUD14
Address: 101 Townsend Street
City: San Francisco
StateProv: CA
PostalCode: 94107
Country: US
RegDate: 2010-07-09
Updated: 2021-07-01
Ref: https://rdap.arin.net/registry/entity/CLOUD14
```
Same functionality is [available](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/getnameinfo.3.html) from `getnameinfo` POSIX function and CoreFoundation `CFHostStartInfoResolution`:
```swift
class DNSResolver {
func resolve(domain: String) -> String? {
let host = CFHostCreateWithName(nil, domain as CFString).takeRetainedValue()
CFHostStartInfoResolution(host, .addresses, nil)
var success: DarwinBoolean = false
if let addresses = CFHostGetAddressing(host, &success)?.takeUnretainedValue() as NSArray?,
let theAddress = addresses.firstObject as? NSData {
var hostname = [CChar](repeating: 0, count: Int(NI_MAXHOST))
if getnameinfo(theAddress.bytes.assumingMemoryBound(to: sockaddr.self), socklen_t(theAddress.length),
&hostname, socklen_t(hostname.count), nil, 0, NI_NUMERICHOST) == 0 {
let numAddress = String(cString: hostname)
return numAddress
}
}
return nil
}
}
```
Another important key in the Search API response is `languageCodesISO2A`. It represents the languages of the AppStore page localizations.
```json
"languageCodesISO2A": [
"NL",
"UK"
],
```
### App title and description
Another part of bundle analysis is to detect the application's App Store title and description language with Natural Language Processor. On macOS and iOS, this is possible with [Natural Language](https://developer.apple.com/documentation/naturallanguage) framework:
>The Natural Language framework provides a variety of natural language processing (NLP) functionality with support for many different languages and scripts. Use this framework to segment natural language text into paragraphs, sentences, or words, and tag information about those segments, such as part of speech, lexical class, lemma, script, and language.
Language definition is available in `NLLanguageRecognizer`:
```swift
let languageRecognizer = NLLanguageRecognizer()
languageRecognizer.processString(description)
if languageRecognizer.dominantLanguage == .russian {
//Description is written in Russian
}
languageRecognizer.reset()
languageRecognizer.processString(title)
if languageRecognizer.dominantLanguage == .russian {
//Title is written in Russian
}
```
## Conclusion
Static analysis of macOS application bundles is a straightforward and powerful approach for getting the country affiliation of the software.
This is a well-extensible mechanism that could be improved and reconsidered from time to time, and could be a basis for security-related software like SpyBuster.
## Links
* [CleanMyMac Suspicous Applications](https://macpaw.com/news/cleanmymac-suspicious-apps-detection)
* [SpyBuster Case](https://macpaw.com/news/data-protector-spy-buster)
* [SpyBuster](https://spybuster.app)
* [MachO Binaries](https://developer.apple.com/library/archive/documentation/Performance/Conceptual/CodeFootprint/Articles/MachOOverview.html)
* [MachO-Kit](https://github.com/DeVaukz/MachO-Kit)