# LLM Inference on the Apple Neural Engine

**Authors**: Andrii Yaroshevych  
**Published**: 2026-10-10  
**Topics**: artificial-intelligence  
**Type**: Tech Note  
**URL**: https://research.macpaw.com/publications/llm-inference-apple-neural-engine

## Introduction

Established local LLM engines on Apple Silicon, including [MLX](https://github.com/ml-explore/mlx) and [llama.cpp](https://github.com/ggml-org/llama.cpp), primarily execute on the GPU, leaving the Apple Neural Engine (ANE) unused in these inference paths. Sustained GPU inference consumes battery power, generates heat and fan activity, and competes with the user interface and other models. We therefore evaluate local inference by end-to-end latency, time to first token (TTFT), energy per request, temperature, fan activity and contention with other work. These measures need not favor the same execution strategy.

The ANE executes fp16 tensor programs at lower power than the GPU and shares unified memory with the CPU and GPU through a common fabric and system level cache (Figure 1). Its programming model has limited its use for LLM inference: programs have static shapes, whereas autoregressive decoding grows a key-value (KV) cache with each token, and its effective memory bandwidth is lower than the GPU's. Before macOS 27, applications accessed the ANE through Core ML, with limited control over operator placement and model state. [Core AI](https://developer.apple.com/documentation/coreai), introduced with macOS 27, supports stateful programs that share weights across cache sizes and retain the KV cache, enabling the execution strategy studied here.

![Figure 1. Schematic of an Apple Silicon chip package. The CPU, the GPU and the Neural Engine reach the unified memory through a shared fabric, the system level cache and the memory controllers.](https://research.macpaw.com/publications/llm-inference-apple-neural-engine/fig1-soc.svg)

[LFM2.5-2.6B](https://huggingface.co/LiquidAI/LFM2.5-2.6B) is a compact chat model suited to on-device assistants, and local engines serve it through MLX on the GPU. This study asks whether the ANE can serve the same model at lower energy while preserving competitive latency and numerical fidelity. We describe the ANE implementation and compare ANE-only, heterogeneous ANE–GPU and GPU inference on M2 Pro, M3 Pro and M4 Pro systems.

Tree speculative decoding exploits the ANE's nearly constant verification cost across token positions at a fixed context size, amortizing weight reads over candidate continuations. On an assistant workload, ANE-only inference used 24 to 34 percent of the energy of MLX 8-bit and 29 to 39 percent of that of MLX 8-bit with speculative decoding, with numerical fidelity close to MLX 8-bit. Relative to MLX 8-bit without speculation, request latency fell by 26 to 43 percent. For long prompts with 256-token answers, split execution gave the lowest measured latency from 8K tokens on the M4 Pro and from 16K on the M2 Pro, across the tested decoding precisions. On the M3 Pro laptop, ANE-only inference increased projected battery life from 7.1 to 13.0 hours at one request per 30 s; after 20 minutes of sustained inference, the median temperature across sensors was about 20 °C lower than with the GPU configurations.

The results motivate choosing the execution backend per request, according to the workload, machine and system state.

## Methods

### Experimental setup

The target model is [LFM2.5-2.6B](https://huggingface.co/LiquidAI/LFM2.5-2.6B) (Liquid AI), a 2.69-billion-parameter decoder with 30 layers: 22 gated short-convolution layers with a kernel size of three, and eight grouped-query attention layers. The hidden dimension is 2,048, the feed-forward dimension is 10,752, and the vocabulary contains 128,000 tokens. Each convolution layer keeps the two preceding frames as state, and each attention layer a KV cache. The drafter used for speculative decoding is [LFM2.5-2.6B-DSpark](https://huggingface.co/LiquidAI/LFM2.5-2.6B-DSpark), a 328-million-parameter model with five attention layers that, conditioned on hidden states of five target layers, drafts a block of nine tokens in one parallel pass; a low-rank Markov head conditions neighboring draft positions.

Measurements were made on the three systems in Table 1: two Mac minis, operated headless, and a MacBook Pro, measured on AC power, on battery and in Low Power Mode (see [Laptop operation](#laptop-operation)).

| Quantity | Mac mini, M2 Pro | MacBook Pro, M3 Pro | Mac mini, M4 Pro |
| --- | --- | --- | --- |
| CPU cores (performance + efficiency) | 10 (6 + 4) | 12 (6 + 6) | 12 (8 + 4) |
| GPU cores | 16 | 18 | 16 |
| Neural Engine cores | 16 | 16 | 16 |
| Unified memory (GB) | 16 | 18 | 24 |
| Specified memory bandwidth (GB/s) | 200 | 150 | 273 |
| Operating system | macOS 27.0 (26A428) | macOS 27.0 (26A5425a) | macOS 27.0 (26A428) |

*Table 1. Measured systems. Figures are taken from Apple's technical specifications.*

All configurations ran in our Swift inference library built on [MLX Swift](https://github.com/ml-explore/mlx-swift), to which we added a Core AI module for the ANE supporting ANE-only and heterogeneous inference, batched decoding and a prompt-routing encoder. The GPU baselines use the library's existing MLX path with Liquid AI's MLX checkpoints [LFM2.5-2.6B-MLX-4bit](https://huggingface.co/LiquidAI/LFM2.5-2.6B-MLX-4bit) and [LFM2.5-2.6B-MLX-8bit](https://huggingface.co/LiquidAI/LFM2.5-2.6B-MLX-8bit), including its DSpark speculative decoding, which the study extended with a 4-bit drafter, an output head computed only for verified draft tokens, and a configurable draft length. GPU measurements on the M2 Pro, whose GPU lacks native bf16 arithmetic, used fp16 activations, except for split prefill with GPU 8-bit decoding at long context, which was measured with bf16 activations (Table 8 and Figure 8). The MLX checkpoints use group-wise affine quantization with groups of 64 weights, whereas the ANE model uses per-channel INT8 (see [Model implementation](#model-implementation)), so equal nominal precision does not imply equal arithmetic.

The workload consists of 100 prompts from a chat dataset for a macOS assistant with tool calling. The prompts cover casual conversation, question answering, tool use and the extraction of tool parameters; 46 of the reference answers are tool calls and 54 are replies without tools. The workload includes substantial conversational text, for which acceptance was lower than for tool calls (see [Tree speculative decoding](#tree-speculative-decoding)). The prompts are rendered in the LFM2.5 chat format; including the beginning-of-sequence token, they contain 777 to 2,588 tokens (median 1,669) and share a 618-token system prefix, which was processed in full for every request. Generation is greedy, ends at the model's end token and is capped at 1,024 tokens. Long-context behavior was measured on summarization requests built from three disjoint regions of the [WikiText-2](https://huggingface.co/datasets/Salesforce/wikitext) test set, with prompts of 1K to 32K tokens followed by 256 generated tokens, the end token being ignored. Each length therefore has three source documents, and lengths derived from the same region are not independent.

Numerical fidelity was measured by teacher-forced top-1 agreement with the bf16 model on 30 workload prompts (9,442 positions): 98.4 percent for ANE INT8, 99.0 percent for MLX 8-bit and 90.6 percent for MLX 4-bit. We group ANE INT8 with MLX 8-bit for comparisons and refer to this grouping as the 8-bit quality class; it denotes similar measured fidelity, not equivalent task accuracy. Configurations with GPU decoding are grouped by their decoding model; this comparison convention does not establish the fidelity of the complete heterogeneous path. Agreement was used because WikiText-2 perplexity did not expose the failure of per-channel INT4: perplexity was 50.6 against 51.1 for bf16, despite only 57 percent top-1 agreement.

We report end-to-end latency, TTFT, prefill and decoding throughput, and energy per request and per token. A configuration lies on the latency–energy Pareto front if no alternative has both no greater latency and no greater energy, with a strict improvement in at least one measure. Rankings are interpreted within the quality classes defined above.

Energy is the time integral of the total DC power drawn by the system, as reported by the System Management Controller (SMC) under the key PSTR (System Total Power), sampled at 10 Hz and shifted back by the sensor's measured lag of about 1.9 s. This reading covers the system on chip (SoC) with its fabric, the unified memory, storage, the fans and the idle draw of the platform; on the laptop it also includes the display, which was held at 50 percent brightness in all measurements. On the Mac minis it agreed with the independent SMC readings of DC input power within 0.4 W; no external power meter was used. On battery, the energy drawn from the battery was recorded separately from the battery discharge-power sensor (PPBR), whose lag of 0.95 s was calibrated; per run, it agreed with the system reading within 1 to 2.5 percent. The per-unit power rails reported by powermetrics are model estimates and serve only to check placement and to decompose system power. Energies are not idle-subtracted.

Throughput and energy per token are pooled over all requests. Each configuration was run once over the workload, starting on an idle machine after a 60 s cool-down period and with 3 s of idle time between requests. Small differences are reported with 95 percent paired bootstrap intervals over the 100 prompts. On the M4 Pro, configurations measured in several sessions agreed within 1 percent in latency and 2.7 percent in energy.

### Execution on the ANE

Single-token autoregressive decoding is primarily limited by memory bandwidth: each forward pass reads the model weights to produce one additional token. In LFM2.5-2.6B, the feed-forward networks (FFNs) contain about three quarters of the weights, so dense matrix products account for much of this cost. The lower effective weight-read bandwidth of the ANE relative to the GPU (Table 2) therefore limits non-speculative decoding. On the M4 Pro, plain ANE decoding reached 32.2 tok/s, compared with 81.5 tok/s for MLX 8-bit without speculation (Tables 3 and 6).

The microbenchmarks show two distinct regimes (Table 2). With fp16 weights and one token, the ANE has lower effective weight-read bandwidth than the GPU. Its bandwidth approximately doubles from M2 Pro and M3 Pro to M4 Pro, making the relative performance chip-dependent. At 256 tokens, the ANE with INT8 weights achieves 2.0 to 2.7 times the GPU's fp16 compute throughput. Efficient ANE execution therefore requires sufficient weight reuse to exploit its arithmetic throughput.

| Quantity | M2 Pro | M3 Pro | M4 Pro |
| --- | --- | --- | --- |
| ANE weight-read bandwidth, 1 token (GB/s) | 60.4 | 60.8 | 122.4 |
| GPU weight-read bandwidth, 1 token (GB/s) | 138.8 | 110.9 | 175.6 |
| ANE compute, INT8 weights, 256 tokens (TFLOP/s) | 9.0 | 9.4 | 10.3 |
| GPU compute, fp16, 256 tokens (TFLOP/s) | 3.4 | 4.1 | 5.3 |

*Table 2. Microbenchmarks on the four-block FFN stack, with fp16 weights for the bandwidth rows.*

Efficient decoding on the ANE consequently requires amortizing weight reads over multiple tokens. Figure 2 shows the latency of one call through four FFN blocks of the model's dimensions with INT8 weights. On the ANE the latency is nearly constant up to 64 tokens, increasing on the M4 Pro from 2.34 ms for one token to 2.42 ms for 64, because execution is dominated by reading the weights, which occurs once per call; beyond approximately 64 to 128 tokens the call becomes compute-bound. On the GPU, running the same Core AI program, the latency increases from the first tokens onward, from 3.05 ms to 7.30 ms over the same range. Between one and 64 tokens, the latency per token therefore decreases about 60-fold on the ANE and about 27-fold on the GPU. Plain decoding operates the ANE at the point with the least weight reuse, whereas prefill, draft verification and batched decoding distribute each weight read over multiple token positions.

![Figure 2. Latency of one call through four FFN blocks of the model's dimensions with INT8 weights, as a function of the number of tokens in the call. The GPU runs the same Core AI program.](https://research.macpaw.com/publications/llm-inference-apple-neural-engine/fig2-call-latency.svg)

Prefill reuses weights across known prompt tokens. During decoding, speculative verification scores candidate continuations together. A chain proposes one path, so the first mismatch ends acceptance; a tree includes alternatives, increasing acceptance at the cost of scoring more positions. Within the ANE's nearly constant-latency regime, these additional positions are inexpensive. The implementation therefore uses a fixed 64-token window for tree verification.

#### Model implementation

The ANE executes precompiled programs with static shapes and computes in fp16 arithmetic; programs specified in bf16 or fp32 compile but silently fall back to GPU execution. Following Apple's guidance on [deploying transformers on the ANE](https://machinelearning.apple.com/research/neural-engine-transformers), the model was re-implemented in the channel-first layout that the ANE executes natively. Every call processes a fixed window of 64 tokens, and the KV cache is allocated in one of six size buckets, so that attention covers only the smallest bucket that holds the context. The output head is computed only for the positions that need a prediction: the last prompt token in prefill and the tree nodes in verification.

Transferring logits for all 64 positions and 128,000 vocabulary entries added 17 ms to a 30 ms call. Instead, the program returns the maximum and its index within each 2,000-token vocabulary subrange, and the host combines these into the exact argmax. Subrange indices remain exactly representable in fp16, whose exact integer range extends to 2,048.

Weights are stored as per-channel INT8 and activations are computed in fp16. Because the ANE flushes very small fp16 values to zero, some intermediate products in LFM2.5 vanish, while a few activations on real prompts exceed the fp16 maximum. Per-layer power-of-two scale factors, folded into the weights and calibrated on representative text that includes assistant prompts, reduce underflow and overflow; they raised the top-1 agreement of the fp16 model from 74 to 98 percent. Lower-precision formats were evaluated but not adopted (see [Alternative approaches](#alternative-approaches)).

#### Tree speculative decoding

A DSpark call yields ranked candidates at all nine draft depths. Tree verification adds alternatives best-first by cumulative draft log-probability until the 64-token window is full. Besides the tree, the window holds the root and the tokens accepted since the last 32-position boundary, because the program writes the KV cache in aligned blocks of 32 positions. The attention mask exposes to each node only its ancestors, and the convolution layers use parent and grandparent gather matrices, so that the convolution history follows the tree path rather than the storage order; one program therefore serves prefill as well as chain and tree verification. The longest path that agrees with the target's greedy choices is accepted, which makes decoding lossless relative to the same target: on the M4 Pro the outputs matched those of plain ANE decoding for all 100 requests.

The full-model verification measurements follow the scaling observed in the FFN benchmark. At a 1K-token cache on the M4 Pro, a chain verification call on the ANE took 29.8 ms and a tree verification call, with about 46 draft tokens, 30.6 ms. The additional positions share the same weight reads and add little to the call latency. Acceptance grows with the size of the tree and saturates near 48 draft tokens (Figure 3a). The window is not wider because the larger calls leave this efficient regime: at a 2K-token cache, a 96-token call took 59.4 ms against 29.7 ms for 64 tokens.

The measured GPU configuration has a different verification-cost profile. With MLX, the 8-bit model took 12.9 ms to verify one token, 17.6 ms to verify 8 and 39.9 ms to verify 10, so GPU speculative decoding in this study verifies a chain of at most 7 draft tokens. This cost profile in the measured MLX configuration motivates a shorter chain on the GPU and a wider tree on the ANE.

Decoding throughput is the mean number of generated tokens per cycle divided by the mean cycle time. Acceptance depends on the drafter and text; cycle time depends on the hardware and verification width. On the M4 Pro a cycle on the ANE takes 40.3 ms, of which 30.6 ms is verification, 7.2 ms drafting and 2.5 ms host work, whereas a cycle of GPU 8-bit decoding with DSpark takes 25.8 ms (Figure 3b). The ANE therefore matches the GPU's decoding rate only if it accepts about 1.6 times as many tokens per cycle, or 1.9 times on the M3 Pro, whose ANE is slower. On the workload the tree accepted 5.59 tokens per cycle against 3.74 for the GPU chain, a ratio of 1.5 (Figure 3c). Acceptance also depends on the text: on the ANE it ranged from 4.6 tokens per cycle for long prose to 6.3 for tool calls, and on the GPU from 3.1 to 4.3.

![Figure 3. Speculative decoding on the M4 Pro. (a) Tokens accepted per cycle as a function of the number of draft tokens verified, from an exact replay of the drafter against the target's greedy outputs on the workload. (b) Time per draft–verify cycle on the ANE and on the GPU with DSpark. (c) Distribution of tokens per cycle over the requests of the workload, with the pooled mean in parentheses.](https://research.macpaw.com/publications/llm-inference-apple-neural-engine/fig3-speculative.svg)

Tree verification increased mean generation from 3.9 to 5.6 tokens per cycle and improved decoding throughput on both machines (Table 3). On the M4 Pro, it reduced request latency by 69 percent and energy by 73 percent relative to plain ANE decoding. The [results](#typical-assistant-requests) compare the resulting engine with the GPU baselines and separate prefill from decoding.

| Decoding mode | Tokens per cycle | M2 Pro (s / J) | M4 Pro (s / J) | Decoding, M2 Pro / M4 Pro (tok/s) |
| --- | --- | --- | --- | --- |
| Plain decoding (one token per call) | 1.0 | 16.40 / 283 | 9.06 / 208 | 18.3 / 32.2 |
| Chain speculative decoding (9 draft tokens) | 3.9 | 6.28 / 107 | 3.54 / 74 | 56.4 / 98.1 |
| Tree speculative decoding | **5.6** | **4.90** / **84** | **2.77** / **57** | **78.8** / **136.9** |

*Table 3. Decoding modes of the ANE engine on the 100-request workload. The M2 Pro and M4 Pro columns give mean request latency (s) / energy (J). Tokens per cycle are generated tokens per draft–verify cycle and were nearly equal on the two chips. The best value in each column is in bold.*

Weight reuse addresses the cost of reading the model weights, but not the increasing cost of attention. The ANE programs compute attention over the entire allocated KV cache, so call latency also grows with context length. The nearly constant latency across token positions at fixed context therefore does not imply constant latency across context lengths. This limitation motivates configurations that combine the ANE with the GPU, particularly at long context (see [Attention at long context](#attention-at-long-context)).

### Heterogeneous execution

The execution characteristics above suggest assigning each phase or operation to the unit that performs it more efficiently. Combining the units is constrained, however, by their shared memory system. In a microbenchmark pairing the four-block FFN stack on the ANE with a matrix-vector benchmark on the GPU, concurrent weight reads raised the aggregate bandwidth by only 5, 11 and 5 percent over the GPU alone on the M2 Pro, M3 Pro and M4 Pro, respectively. Dividing the weight reads of one decoding step between the units yielded no gain. The combinations evaluated here therefore assign different work to the two units or overlap independent work, rather than treating their memory bandwidths as additive.

Table 4 lists the configurations together with the GPU baselines.

| Configuration | Prefill | Decoding | Principle |
| --- | --- | --- | --- |
| GPU (MLX 4-bit or 8-bit) | GPU | GPU | Baseline |
| GPU + DSpark | GPU | GPU, chain speculative decoding with 7 draft tokens and a 4-bit drafter | Baseline |
| ANE only | ANE | ANE, tree speculative decoding | Low power and many tokens per call |
| ANE prefill, GPU decoding | ANE | GPU (MLX) | Fast, low-energy prefill followed by GPU decoding |
| Co-prefill, GPU decoding | ANE for layers 1 to 20 and GPU for layers 21 to 30, pipelined over 128-token chunks | GPU (MLX) | Concurrent computation on both units during prefill |
| Split prefill, GPU decoding | Weights on the ANE, attention core on the GPU | GPU (MLX) | Attention moved off the ANE at long context |
| Split tree decoding | Weights on the ANE, attention core on the GPU | The same division, with tree speculative decoding | As above, while retaining ANE decoding |

*Table 4. Placement of prefill and decoding in each configuration.*

The simplest combination assigns prefill, which benefits from the ANE's higher compute throughput, to the ANE, and decoding to the GPU, which retains the decoding speed of MLX and the choice between its 4-bit and 8-bit models. The model's state is small, 16 KiB of KV cache per token in fp16 and two frames per convolution layer, so it can be transferred between units in milliseconds. After ANE prefill, the KV cache and the convolution states are handed off to MLX at a cost of about 1 percent of request latency.

Co-prefill adds overlap to this hand-off by dividing the layers between the units: the ANE executes layers 1 to 20 and the GPU layers 21 to 30, and the prompt passes through them in 128-token chunks, so that the GPU processes one chunk while the ANE processes the next. In a prompt-only benchmark on the M4 Pro, this division gave a lower TTFT than divisions after layers 16 and 24 (619 against 795 and 708 ms). Because prefill is compute-bound, the two units add their rates: the pipelined prefill reached 2,630 tok/s, close to the sum of the ANE's 1,584 and the GPU's 982, whereas the same division without overlap was slower than the ANE alone.

Split execution, the third combination, divides each layer by operation, with the weights on the ANE and the attention core on the GPU. It addresses the cost of attention at long context and is described below.

#### Attention at long context

An ANE call combines two costs that scale differently with context. Reading the projection and FFN weights takes about the same time at every length, whereas the cost of attention grows faster than the linear increase in KV-cache bytes: a 64-token target call on the M4 Pro took 27 ms at 1K tokens, 122 ms at 16K and 394 ms at 32K. Computing attention in eight chunks of the cache, merged exactly, reduced the 32K call to 159 ms and allowed the drafter to compile for the ANE at that length, but it added operations that made short contexts slower, and the call remained several times more expensive than at 1K.

Attention is confined to 8 of the model's 30 layers, and the attention core, softmax(QKᵀ)V, contains no learned weights. Split execution therefore keeps every weight read on the ANE and sends only the queries, keys and values to the GPU, which holds the KV cache and computes attention. It is used in two variants: split tree decoding applies this division to prefill and to tree speculative decoding, whereas split prefill applies it to the prompt only and hands the request off to GPU decoding.

Figure 4 shows the resulting decoding cycle on the M4 Pro: the ANE's weight work remained at 27 to 30 ms per cycle from 1K to 32K tokens, while GPU attention grew from 6 to 41 ms and host work, mainly the transfers between the units, from 7 to 20 ms. The full cycle grew from 47 to 116 ms, against 39 to 197 ms for ANE-only decoding. At short context, the transfers between the units and the separate dispatch of the segmented programs make split tree decoding slower than ANE-only decoding (119 against 138 tok/s at 1K); from 8K tokens the saving in attention outweighs them (39 against 22 tok/s at 32K). Moving entire attention blocks, including their weights, to the GPU was slower and consumed more energy at every length. At long context, split execution thus reduces the ANE's expensive attention computation while retaining its lower-power weight processing.

![Figure 4. Time per decoding cycle on the M4 Pro from the engine's timers, for ANE-only inference (left bar of each pair) and split tree decoding (right bar), with totals in ms. The ANE target time is divided into the weight part, measured in split execution, which holds all weights, and the remainder, attributed to attention; the remainder also includes differences in dispatch between one program and the segmented split programs. ANE-only inference uses 8-chunk attention from 16K tokens.](https://research.macpaw.com/publications/llm-inference-apple-neural-engine/fig4-split-cycle.svg)

Because the KV cache already resides on the GPU, the hand-off after split prefill transfers only the convolution states. Split prefill therefore combines this attention placement with GPU decoding. Its end-to-end benefit depends on both prompt length and answer length, because longer answers increase the decoding share of the request (see [Long-context inference](#long-context-inference)).

## Results

We compare typical assistant requests, long-context inference, laptop operation, and concurrent and batched workloads, and summarize the alternatives that were evaluated but not adopted.

### Typical assistant requests

Table 5 and Figure 5 compare latency and energy on the assistant workload. Table 6 separates prefill, which determines TTFT, from decoding on the M4 Pro, and Figure 6 repeats this separation on all three machines; Figure 7 shows the energy contributions of the two phases and the system-power breakdown.

| Configuration | Quality class | M2 Pro (s / J) | M3 Pro (s / J) | M4 Pro (s / J) |
| --- | --- | --- | --- | --- |
| ANE only | 8-bit | 4.90 / **84** | 4.46 / **72** | 2.77 / **57** |
| Split tree decoding | 8-bit | 5.70 / 96 |  | 3.35 / 66 |
| GPU 8-bit | 8-bit | 6.59 / 250 | 7.59 / 269 | 4.86 / 235 |
| GPU 8-bit + DSpark | 8-bit | 6.73 / 251 | 4.18 / 185 | 3.45 / 196 |
| ANE prefill, GPU 8-bit decoding | 8-bit | 6.11 / 210 | 7.66 / 229 | 4.26 / 169 |
| Co-prefill, GPU 8-bit decoding | 8-bit | 5.25 / 211 | 6.96 / 236 | 4.07 / 185 |
| GPU 4-bit | 4-bit | 4.83 / 185 | 5.31 / 202 | 3.65 / 184 |
| GPU 4-bit + DSpark | 4-bit | 7.08 / 246 | **4.06** / 182 | 3.88 / 206 |
| ANE prefill, GPU 4-bit decoding | 4-bit | 4.16 / 138 | 4.92 / 147 | 2.83 / 110 |
| Co-prefill, GPU 4-bit decoding | 4-bit | **3.54** / 147 | 4.36 / 153 | **2.69** / 128 |

*Table 5. Mean latency (s) / energy (J) per request on the 100-request workload, on AC power. GPU configurations on the M2 Pro use fp16 activations. Split tree decoding was not measured on the M3 Pro. The best value for each machine is in bold.*

![Figure 5. Latency and energy per request on the three machines, as in Table 5. Filled markers denote the 8-bit quality class and open markers the 4-bit quality class; the dashed line connects the plotted Pareto front. Quality classes must be distinguished when interpreting the front.](https://research.macpaw.com/publications/llm-inference-apple-neural-engine/fig5-pareto.svg)

| Configuration (M4 Pro) | TTFT (ms) | Prefill (tok/s) | Prefill (mJ/token) | Decoding (tok/s) | Decoding (mJ/token) |
| --- | --- | --- | --- | --- | --- |
| ANE only | 830 | 1,960 | **9.2** | 136.9 | **158** |
| Split tree decoding | 1,121 | 1,452 | 11.4 | 120.6 | 176 |
| ANE prefill, GPU 8-bit decoding | 831 | 1,959 | 10.8 | 79.8 | 553 |
| Co-prefill, GPU 8-bit decoding | 695 | 2,342 | 17.6 | 81.1 | 570 |
| GPU 8-bit + DSpark | 1,579 | 1,031 | 52.1 | **144.9** | 409 |
| GPU 8-bit | 1,553 | 1,048 | 49.4 | 81.5 | 575 |
| ANE prefill, GPU 4-bit decoding | 824 | 1,975 | 11.1 | 133.0 | 342 |
| Co-prefill, GPU 4-bit decoding | **667** | **2,438** | 17.1 | 134.8 | 365 |
| GPU 4-bit | 1,550 | 1,050 | 48.8 | 135.2 | 369 |
| GPU 4-bit + DSpark | 1,585 | 1,026 | 49.4 | 120.8 | 452 |

*Table 6. Prefill and decoding measured separately on the M4 Pro. The best value in each column is in bold.*

The separation is consistent across the three machines (Figure 6). Co-prefill with GPU 4-bit decoding had the highest prefill rate and the lowest TTFT on every machine, and ANE-only inference the lowest energy per token in both phases; GPU prefill required 2.8 to 5.7 times its energy per prompt token. Decoding rates follow the machines' memory systems: plain GPU 8-bit decoding reached 59, 45 and 82 tok/s on the M2 Pro, M3 Pro and M4 Pro. The effect of DSpark on the GPU also depends on the machine: on the M3 Pro it raised 8-bit decoding to 107 tok/s, whereas on the M2 Pro it left 8-bit decoding unchanged and reduced 4-bit decoding from 100 to 58 tok/s.

![Figure 6. Prefill and decoding measured separately on the three machines, from the runs of Table 5: (a) prefill and (b) decoding throughput, (c) prefill and (d) decoding energy per token. Split tree decoding was not measured on the M3 Pro.](https://research.macpaw.com/publications/llm-inference-apple-neural-engine/fig6-phases.svg)

ANE-only inference required the least energy on every machine, 3.0 to 4.1 times less than GPU 8-bit and 2.6 to 3.4 times less than GPU 8-bit with DSpark. Both phases contribute to this difference (Figure 7a): on the M4 Pro, ANE prefill ran 1.9 times faster than GPU prefill at about one fifth of its energy per token, and ANE decoding used 158 mJ per token against 409 to 575 mJ for the 8-bit GPU configurations. The hand-off configurations retain most of the saving in prefill, which required 18 to 29 J per request against 80 to 85 J on the GPU, but GPU decoding accounts for 78 to 90 percent of their energy. Figure 7b divides the system power during requests into the rails reported by powermetrics and the remainder. The ANE rail drew about 6 W with ANE-only inference, against 17 to 23 W on the GPU rail with GPU decoding, but most of the difference lies outside the accelerator rails: residual power after the modeled ANE, GPU and CPU rails was about 13 W with ANE-only inference and 26 to 33 W with GPU decoding. This residual includes the memory system and platform and is not a separate measurement of memory power. The whole-system ratio of about 2.4 is therefore smaller than the ratio of the rails.

![Figure 7. Energy and power per request on the M4 Pro: (a) energy divided into prefill and decoding, with the totals of Table 5, and (b) mean system power during requests, divided into the modeled ANE, GPU and CPU rails reported by powermetrics and the residual, which includes the memory system and platform.](https://research.macpaw.com/publications/llm-inference-apple-neural-engine/fig7-energy-power.svg)

At 8-bit quality, ANE-only inference was also the fastest configuration on the M2 Pro and M4 Pro; on the M4 Pro it completed a request in 2.77 s, against 3.45 s for GPU 8-bit with DSpark (latency ratio 0.80, 95 percent interval 0.76 to 0.85). On the M3 Pro, GPU 8-bit with DSpark was 6 percent faster (ratio 0.94, interval 0.89 to 0.99), since the ANE cycle on that chip lasts about 1.9 times as long as the GPU cycle (see [Tree speculative decoding](#tree-speculative-decoding)). Split tree decoding was slower than ANE-only inference at these prompt lengths, where the transfers between the units outweigh the saving in attention.

At 4-bit quality, co-prefill with GPU 4-bit decoding was the fastest configuration on the M2 Pro and M4 Pro and had the lowest TTFT on all three machines, at about twice the energy of ANE-only inference and 20 to 31 percent less than GPU 4-bit. Its lead of 3 percent over ANE-only inference on the M4 Pro is not resolved (ratio 0.97, interval 0.92 to 1.03), and on the M3 Pro GPU 4-bit with DSpark was faster. Configurations that hand off to GPU 8-bit decoding were slower and consumed more energy than ANE-only inference on every machine, because GPU 8-bit decoding dominates the request time.

Reusing the KV cache of the shared system prefix across requests is complementary to these configurations. In a separate evaluation with an earlier version of the ANE engine, it reduced the TTFT of ANE-only inference by about a third on both Mac minis, from 1,006 to 659 ms on the M4 Pro.

### Long-context inference

Long prompts, as in summarization or questions over documents, shift the cost of a request from decoding to prefill and from weight reads to attention (see [Attention at long context](#attention-at-long-context)). Table 7 reports latency and energy on the M4 Pro for prompts of 1K to 32K tokens followed by 256 decoded tokens, Figure 8 compares the M2 Pro and M4 Pro, and Table 8 separates the phases of a request at 32K tokens. Up to 8K tokens, the long-context ANE programs use cache buckets sized to the prompt plus the answer, which raised decoding at 4K tokens on the M4 Pro from 80 to 109 tok/s compared with power-of-two buckets. From 16K tokens they use chunked attention, which made the 32K-token target call 2.5 to 2.8 times cheaper.

| Prompt | GPU 4-bit | GPU 4-bit + DSpark | GPU 8-bit | GPU 8-bit + DSpark | ANE only | Split tree decoding | ANE prefill, GPU 4-bit decoding | ANE prefill, GPU 8-bit decoding | Split prefill, GPU 4-bit decoding | Split prefill, GPU 8-bit decoding |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1K | 2.81 / 132 | 3.32 / 167 | 4.06 / 185 | 2.86 / 147 | **2.38** / **48** | 2.93 / 56 | 2.45 / 96 | 3.71 / 145 | 2.57 / 95 | 3.85 / 152 |
| 2K | 3.72 / 181 | 4.55 / 242 | 4.98 / 239 | 3.83 / 222 | 2.96 / **61** | 3.63 / 71 | **2.94** / 115 | 4.23 / 163 | 3.20 / 110 | 4.48 / 164 |
| 4K | 5.60 / 293 | 6.44 / 358 | 6.86 / 361 | 5.84 / 341 | 4.50 / **91** | 5.38 / 109 | **4.13** / 137 | 5.41 / 190 | 4.52 / 141 | 5.79 / 193 |
| 8K | 9.52 / 527 | 10.53 / 596 | 10.86 / 602 | 10.04 / 594 | 8.48 / **164** | 8.24 / 178 | 7.58 / 202 | 8.82 / 258 | **7.39** / 211 | 8.66 / 268 |
| 16K | 18.19 / 1,016 | 20.00 / 1,133 | 19.75 / 1,081 | 19.37 / 1,105 | 21.88 / **343** | 15.99 / 394 | 18.71 / 362 | 19.95 / 424 | **13.83** / 396 | 15.13 / 453 |
| 32K | 37.22 / 2,173 | 40.39 / 2,348 | 38.61 / 2,295 | 39.81 / 2,378 | 67.99 / 991 | 34.45 / 984 | 58.91 / 939 | 60.02 / 1,021 | **29.77** / **923** | 31.14 / 981 |

*Table 7. Latency (s) / energy (J) per request on the M4 Pro for a prompt followed by 256 decoded tokens, averaged over three documents per length. The best latency and the best energy in each row are in bold.*

![Figure 8. Latency (top) and energy (bottom) per request on the M2 Pro and M4 Pro as a function of prompt length, for selected configurations. On the M2 Pro, split prefill with GPU 8-bit decoding uses bf16 activations and the other GPU configurations fp16. Open markers denote 4-bit decoding. Both axes are logarithmic.](https://research.macpaw.com/publications/llm-inference-apple-neural-engine/fig8-long-context.svg)

| Machine | Configuration | TTFT (s) | Decoding (tok/s) | Latency (s) | Energy (J) |
| --- | --- | --- | --- | --- | --- |
| M2 Pro | GPU 4-bit | 45.40 | **73.0** | 48.89 | 1,694 |
| M2 Pro | ANE only | 100.42 | 9.9 | 126.18 | 3,339 |
| M2 Pro | Split tree decoding | 43.59 | 15.6 | 59.89 | 1,172 |
| M2 Pro | Split prefill, GPU 4-bit decoding | 43.25 | 62.1 | **47.36** | **1,072** |
| M2 Pro | GPU 8-bit | 44.34 | 48.8 | 49.57 | 1,785 |
| M2 Pro | Split prefill, GPU 8-bit decoding \* | **43.00** | 47.8 | 48.34 | 1,130 |
| M4 Pro | GPU 4-bit | 34.82 | 106.1 | 37.22 | 2,173 |
| M4 Pro | ANE only | 56.33 | 21.9 | 67.99 | 991 |
| M4 Pro | Split tree decoding | 27.90 | 39.0 | 34.45 | 984 |
| M4 Pro | Split prefill, GPU 4-bit decoding | **27.38** | **106.5** | **29.77** | **923** |
| M4 Pro | GPU 8-bit | 34.86 | 68.0 | 38.61 | 2,295 |
| M4 Pro | Split prefill, GPU 8-bit decoding | 27.46 | 69.3 | 31.14 | 981 |

*Table 8. Phases of a request with a 32K-token prompt and 256 generated tokens, averaged over three documents. The decoding rate is pooled over the tokens after the first. On the M2 Pro, GPU configurations use fp16 activations, except the one marked \*, which was measured with bf16 activations. The best value for each machine is in bold.*

On the M4 Pro, ANE-only inference required the least energy up to 16K tokens and was the fastest 8-bit configuration up to 4K. Followed by GPU 4-bit decoding, split prefill was the fastest configuration from 8K tokens, 20 to 24 percent faster than GPU 4-bit with 58 to 61 percent less energy. At 8-bit quality, split tree decoding was the fastest configuration at 8K tokens, and split prefill with GPU 8-bit decoding from 16K.

These rankings hold for answers of 256 tokens; longer answers favor GPU decoding. The crossovers with the GPU occur at different lengths for different measures: the decoding rate of ANE-only inference fell below that of MLX 4-bit between 2K and 4K tokens, its request latency exceeded GPU 4-bit's between 8K and 16K on the M4 Pro, and its energy exceeded the GPU's only between 16K and 32K on the M2 Pro and not within 32K on the M4 Pro, because its system power fell from about 20 W at 1K to 15 W at 32K as it stalled on attention, while GPU 4-bit drew 47 to 58 W.

Measured numerical fidelity did not degrade with context: its top-1 agreement with MLX 8-bit remained at 95 to 96 percent in every position band up to 32K tokens, whereas MLX 4-bit agreed with MLX 8-bit at 77 to 79 percent.

On the M2 Pro, split prefill with GPU 4-bit decoding used 37 percent less energy than GPU 4-bit at 32K tokens at a latency within 3 percent, a difference small relative to the variation between documents, and ANE-only inference spent 100 of its 126 s in prefill (Table 8). On the M3 Pro (Table 9), whose GPU processes long prompts comparatively fast, GPU 4-bit remained the fastest configuration at 16K and 32K tokens, while split prefill with GPU 4-bit decoding took 13 percent longer at 32K and used 46 percent less energy.

| Configuration (M3 Pro) | 4K (s / J) | 16K (s / J) | 32K (s / J) |
| --- | --- | --- | --- |
| GPU 4-bit | 7.30 / 300 | **20.32** / 988 | **40.65** / 2,108 |
| GPU 8-bit | 9.73 / 376 | 22.88 / 1,160 | 43.33 / 2,251 |
| ANE only | 7.21 / **131** | 30.90 / 517 | 104.73 / 3,379 |
| ANE prefill, GPU 4-bit decoding | **7.06** / 183 | 26.15 / **494** | 84.02 / 2,748 |
| ANE prefill, GPU 8-bit decoding | 9.51 / 254 | 28.55 / 567 | 89.98 / 2,971 |
| Split prefill, GPU 4-bit decoding | 7.67 / 187 | 22.32 / 508 | 45.94 / **1,148** |
| Split prefill, GPU 8-bit decoding | 10.06 / 259 | 24.61 / 572 | 47.29 / 1,169 |

*Table 9. Latency (s) / energy (J) per request on the M3 Pro laptop on AC power for a prompt followed by 256 decoded tokens, averaged over three documents per length. GPU configurations use bf16 activations. At 32K tokens, ANE-only inference and ANE prefill raised system memory to 16.1 to 16.8 GB of 18 GB and ran under memory pressure. The best latency and the best energy in each column are in bold.*

### Laptop operation

On a laptop, energy per request determines battery life, and sustained power determines temperature and fan activity. The M3 Pro laptop was therefore measured in three ways. Requests were timed on AC power, on battery and in Low Power Mode (Table 10). Battery life was projected by dividing the full-charge capacity estimated by the battery controller, 86 to 88 Wh, by the average discharge power measured over 20 minutes of intermittent use, with the model kept loaded and one request every 30 s from 40 prompts sampled across output types and lengths; these are projections from finite windows, not full-discharge measurements. Under 20 minutes of back-to-back requests, starting with the fans stopped, temperatures, fan speed, GPU clock and thermal pressure were recorded (Figure 9).

| Configuration | AC power (s / J) | Battery (s / J) | Battery, Low Power Mode (s / J) | Average power, one request per 30 s (W) | Projected battery life (h) |
| --- | --- | --- | --- | --- | --- |
| ANE only | 4.46 / **72** | 4.49 / **77** | 4.85 / **58** | **6.7** | **13.0** |
| GPU 8-bit | 7.59 / 269 | 7.72 / 263 | 7.61 / 248 | 12.5 | 7.1 |
| GPU 8-bit + DSpark | 4.18 / 185 | **4.23** / 185 | **4.22** / 179 | 10.8 | 8.2 |
| ANE prefill, GPU 8-bit decoding | 7.66 / 229 | 7.70 / 215 | 7.66 / 200 |  |  |
| Co-prefill, GPU 8-bit decoding | 6.96 / 236 | 6.99 / 224 | 6.93 / 212 | 11.2 | 7.7 |
| GPU 4-bit | 5.31 / 202 | 5.34 / 193 | 5.31 / 188 | 10.0 | 8.6 |
| GPU 4-bit + DSpark | **4.06** / 182 |  |  |  |  |
| ANE prefill, GPU 4-bit decoding | 4.92 / 147 |  |  |  |  |
| Co-prefill, GPU 4-bit decoding | 4.36 / 153 |  |  |  |  |
| Idle, no model loaded |  |  |  | 4.1 | 21.1 |

*Table 10. M3 Pro laptop; empty cells were not measured. The best value in each column, excluding the idle row, is in bold.*

On battery in the default power mode, latency stayed within 2 percent of that on AC power for all configurations. Low Power Mode did not change GPU latency, whereas for ANE-only inference it increased latency by 9 percent and reduced energy by 19 percent relative to AC power. The projected battery life depends on the request rate: with fewer requests, idle power dominates and the difference narrows. Holding both an ANE and an MLX model, as co-prefill requires, raised the idle power with the model loaded to 6.5 W, against 4.4 to 4.7 W with one model.

![Figure 9. Continuous requests on the M3 Pro laptop on AC power, starting with the fans stopped: (a) system power as a 10 s moving average, (b) median temperature over the 201 sensors that reported valid readings throughout, as a 30 s moving average, and (c) speed of the faster fan.](https://research.macpaw.com/publications/llm-inference-apple-neural-engine/fig9-sustained.svg)

No sustained throughput decrease attributable to thermal throttling was observed: thermal pressure remained nominal, the GPU clock stayed near 1.38 GHz, and decoding in the final 3 minutes matched the first 3 within prompt-mix variation. ANE-only inference drew 17.6 W against 37 to 51 W for the GPU configurations and reduced the median temperature across sensors by 19 to 20 °C after 20 minutes (Figure 9 and Table 11). Fans remained active, so lower power did not imply fanless operation. These observations concern an actively cooled MacBook Pro over 20 minutes and do not establish steady-state behavior on fanless hardware or under additional application load.

| Configuration | Requests | System power (W) | Median sensor / hottest CPU sensor at 20 min (°C) | Fan speed, 20 min / peak (rpm) |
| --- | --- | --- | --- | --- |
| ANE only | 280 | **17.6** | **49** / **69** | **1,400** / 1,727 |
| GPU 8-bit | 154 | 37.5 | 69 / 98 | 1,545 / **1,663** |
| GPU 8-bit + DSpark | 283 | 50.6 | 68 / 96 | 2,205 / 2,551 |
| Co-prefill, GPU 8-bit decoding | 171 | 37.1 | 68 / 85 | 1,586 / 1,823 |
| GPU 4-bit | 229 | 40.3 | 68 / 86 | 1,678 / 1,726 |
| Co-prefill, GPU 4-bit decoding | 276 | 39.0 | 69 / 86 | 1,631 / 1,824 |

*Table 11. Continuous requests on the M3 Pro laptop on AC power over 20 minutes, starting with the fans stopped. The median is taken over the 201 temperature sensors that reported valid readings throughout all runs and averaged over the final minute; the hottest CPU sensor is a hot-spot reading. The lowest value in each of the last three columns is in bold.*

With its default unbounded buffer cache, MLX grew to 13.7 to 13.9 GB on this 18 GB machine and caused critical memory pressure with more than two million swap-outs; clearing the cache, as in all laptop measurements, kept the GPU configurations at 1.9 to 3.5 GB without slowing them.

### Concurrent and batched inference

An assistant runs more than a chat model: auxiliary models such as a prompt router and, in agentic use, several sequences at once. Two experiments test whether the ANE can take such work off the GPU: the router alone and alongside chat generation on the GPU, and batched decoding of independent sequences on the ANE.

| Prompt router | 128 tokens (ms / mJ) | 1,024 tokens (ms / mJ) |
| --- | --- | --- |
| ANE, INT8 | **5.5** / **126** | **59.8** / **968** |
| GPU, Core AI, fp16 | 18.9 / 948 | 125.5 / 7,220 |
| GPU, MLX, bf16 | 15.8 / 797 | 111.5 / 5,533 |

*Table 12. Time and energy per call of the prompt-routing encoder on the M4 Pro.*

The prompt router, a 350-million-parameter bidirectional encoder that scores an input against labeled routes, ran 2.9 times faster on the ANE than with MLX on the GPU at 128 tokens and required 6.3 times less energy (Table 12). On the M4 Pro, it agreed with the fp32 reference in 99 to 100 percent of cases on classification tasks and in 82 to 98 percent on tool ranking, against 98 to 100 percent for MLX. On the M2 Pro, the ranking agreement of the ANE router fell to 69 percent with 20 tools and 4 percent with 60 tools, against 100 percent for MLX. Agreement does not establish routing accuracy: with 20 tools, the fp32 reference itself ranked the tool that was actually called first in only 10 of 36 cases.

When the router was called twice per second with 1,024-token inputs during chat generation on the GPU, the GPU router reduced the generation rate by 20 percent and raised the 90th-percentile interval between tokens from 12.3 ms to 19 to 33 ms, whereas the ANE router left both nearly unchanged. Called back to back with 128-token inputs, the ANE router also reduced the generation rate by 20 percent, consistent with contention for the shared memory system.

Batched decoding interleaves several independent sequences in one ANE program, so that a single weight read serves all of them; an ordinary batch dimension gave no such reuse. Table 13 compares it with batched GPU decoding in llama.cpp. With 32 sequences, the ANE reached 398 tok/s in aggregate on the M4 Pro and 239 tok/s on the M2 Pro, at 38 and 54 mJ per token, the lowest reported energy per token among the measured decoding modes. The GPU energy estimates include model loading and prompt processing, so the energy ratios are approximate and should not be interpreted as comparisons with identical accounting boundaries.

Each sequence is decoded without speculation, at 35, 24 and 12 tok/s on the M4 Pro with 8, 16 and 32 sequences. Compared with serving sequences serially using tree speculative decoding at 137 tok/s, the estimated throughput break-even counts are about 4 concurrent sequences for the 8-sequence program, 6 for the 16-sequence program and 12 for the 32-sequence program. These break-even estimates assume that the measured per-sequence rates remain applicable at the scheduler's batch occupancy.

On the M4 Pro, ANE throughput exceeded llama.cpp Q8\_0 at all three measured batch sizes, while llama.cpp Q4\_0 reached the highest rate at 32 sequences (Table 13). Reported ANE energy per token was substantially lower, subject to the accounting qualification above. Q8\_0 is a different quantization scheme from MLX 8-bit, so this is a precision comparison rather than a separately validated fidelity grouping. Batched decoding therefore suits concurrent background agents or batch jobs, while a few interactive sequences are served faster one at a time.

| Engine | M2 Pro, 8 | M2 Pro, 16 | M2 Pro, 32 | M4 Pro, 8 | M4 Pro, 16 | M4 Pro, 32 |
| --- | --- | --- | --- | --- | --- | --- |
| ANE, INT8 | 146 / **111** | 222 / **69** | 239 / **54** | **278** / **78** | **378** / **52** | 398 / **38** |
| GPU, llama.cpp Q4\_0 | **173** / 266 | **266** / 139 | **407** / 100 | 133 / 316 | 346 / 145 | **540** / 94 |
| GPU, llama.cpp Q8\_0 | 151 / 321 | 205 / 192 | 278 / 141 | 128 / 353 | 275 / 196 | 390 / 143 |

*Table 13. Batched decoding with 8, 16 and 32 concurrent sequences: aggregate throughput (tok/s) / system energy per generated token (mJ). On the ANE the sequences are interleaved in one program with about 512 KV positions each. GPU energies include model loading and the 128-token prompts and are approximate. The highest throughput and the lowest energy in each column are in bold.*

### Alternative approaches

Table 14 lists the approaches that were evaluated but not adopted, together with the reason in each case.

| Approach | Outcome | Reason |
| --- | --- | --- |
| Per-channel INT4 weights | Calls 11 to 46 percent faster | Top-1 agreement fell to 57 percent, and group-wise INT4, which preserves quality, does not run on the ANE |
| INT8 activations (W8A8) | FFN calls 2.4 times faster | About ten times the error of weight-only INT8 on real activations; INT8 only at the input projections kept the error within about twice that of weight-only INT8 but gave a 1.25-fold speedup |
| Palettized (LUT) weights | Supported for single layers | The full-model program fell back to the GPU |
| Dividing the weight reads of a decoding step between the ANE and the GPU | At most 5 to 11 percent more bandwidth | The two units share one memory system |
| Attention blocks, including their weights, on the GPU | Slower and more energy-intensive than split execution at every length | Weight reads move from the ANE to the GPU |
| Verification on the GPU with drafting on the ANE | Far below the rate of the ANE engine | GPU verification cost rises steeply above 8 tokens |
| Verification on the ANE with drafting on the GPU | 3.02 s and 78 J per request on the M4 Pro, against 2.77 s and 57 J | Activating the GPU in every cycle added energy without shortening the cycle |

*Table 14. Approaches evaluated and not adopted.*

## Conclusions and discussion

The measurements identify weight reuse as the principal condition for efficient execution on the ANE: calls of a single token leave its arithmetic throughput unused, whereas prefill provides reuse directly, tree verification raises the tokens produced per weight read, and batched decoding applies the same principle to independent sequences.

Energy per token is system power divided by throughput. Plain ANE decoding's lower power did not compensate for its low throughput: on the M4 Pro it used 724 mJ per token against 369 mJ for GPU 4-bit and was about four times slower. Tree verification changes this tradeoff by increasing generation per target call. Its benefit depends on cycle time, which varies by chip, and acceptance, which varies by text. A drafter that increases acceptance without a compensating increase in cycle cost would reduce latency and energy.

Prefill and decoding favor different placements. ANE prefill at shorter contexts and split prefill at long context reduced request energy relative to the corresponding GPU-only configurations, while the latency advantage depended on the machine and context length. For typical requests, ANE decoding preserved the prefill saving; at long context, GPU decoding increasingly favored latency. The M3 Pro's comparatively fast GPU makes this tradeoff different from the two Mac minis.

In the measured M4 Pro configuration, speculation reduced both time and energy on the ANE. On GPU 8-bit it mainly reduced time: DSpark shortened requests by 29 percent but reduced energy by 17 percent. With GPU 4-bit, it increased both. Speculation therefore requires evaluation for each target precision and runtime, rather than being assumed beneficial.

Heterogeneous execution helped when it separated work by its limiting resource or overlapped compute. Split execution retained weight processing on the ANE and moved long-context attention to the GPU; co-prefill overlapped different layers. Concurrently dividing weight reads gave no useful decoding gain because the units share a memory system. At long context, execution time and memory pressure could also outweigh the power advantage of keeping the whole model on the ANE.

The ANE can also absorb auxiliary and background work. The router left GPU generation nearly unchanged at the measured moderate call rate, but back-to-back calls caused contention. Batched decoding improved aggregate throughput and reported energy per token at the cost of per-sequence speed. These results support moderate-rate auxiliary inference and concurrent background generation, subject to device-specific fidelity validation and the energy-accounting qualification in Table 13.

An execution policy must also distinguish first-token latency from complete-response latency: on the M4 Pro, co-prefill lowered TTFT without a resolved gain in total latency over ANE-only inference, at more than twice its energy. These outcomes do not imply a universal ranking of the two units. Table 15 summarizes the configurations with the lowest measured energy or latency, or highest throughput, by use case.

| Use case | Lowest measured energy | Lowest measured latency / highest throughput |
| --- | --- | --- |
| Typical requests (prompts up to about 3K tokens) | ANE only | 8-bit class: ANE only (M3 Pro: GPU 8-bit + DSpark). 4-bit class: co-prefill with GPU 4-bit decoding (M3 Pro: GPU 4-bit + DSpark), which also gives the lowest TTFT on all three machines |
| Prompts of about 4K tokens | ANE only | ANE prefill with GPU 4-bit decoding; at 8-bit quality on the M4 Pro, ANE only |
| Prompts of about 8K tokens | ANE only (M2 Pro, M4 Pro) | Split prefill with GPU 4-bit decoding (M2 Pro, M4 Pro); at 8-bit quality on the M4 Pro, split tree decoding |
| Prompts of 16K to 32K tokens | Split prefill with GPU decoding, except at 16K: ANE only on the M4 Pro, ANE prefill with GPU 4-bit decoding on the M3 Pro | Split prefill with GPU decoding (M2 Pro, M4 Pro); GPU 4-bit on the M3 Pro |
| Battery or sustained load | ANE only (M3 Pro, measured) | As for typical requests |
| Encoders alongside GPU generation | Encoder on the ANE (M4 Pro), at moderate call rates; ranking agreement was low on the M2 Pro | Encoder on the ANE (M4 Pro) |
| Many concurrent background sequences | Batched decoding on the ANE (M2 Pro, M4 Pro; approximate energy comparison) | Batched ANE decoding with 8 and 16 sequences on the M4 Pro; GPU 4-bit decoding for the other measured M2 Pro and M4 Pro batch sizes |

*Table 15. Configurations with the lowest measured energy or latency, or highest throughput, by use case. Entries apply to the measured systems and workloads; long-context entries assume answers of about 256 tokens and were measured at 4K, 16K and 32K on the M3 Pro. Entries summarize measured means rather than statistically resolved preferences in every comparison. The M4 Pro co-prefill lead for typical requests was not resolved, and M2 Pro 32K latency differences were small relative to document variation. Batched throughput comparisons include both quantization levels; energy comparisons have the accounting qualification in Table 13.*

### Practical implications

Table 15 motivates selecting the backend per request. A deployment policy should consider prompt length, expected answer length, required quality class and interactivity; chip, available memory and compilation state; power mode, temperature, fan activity and accelerator load; and user preferences such as power saving. These are policy inputs, rather than a routing policy validated by this study. Contention from other ANE users, including system features, was not measured.

Static window and cache sizes impose a separate capacity constraint. Requests beyond the largest compiled context require larger ANE programs or continuation on the GPU, either by handing off the model state, which costs little (see [Heterogeneous execution](#heterogeneous-execution)), or by reprocessing the context. Cache buckets should match the deployed distribution of context lengths; the gains from tight buckets were measured only at the tested lengths.

The ANE model is a separate artifact, produced by a conversion pipeline that applies INT8 quantization, calibrated scaling, tree-capable convolution and an in-program reduction of the output logits. Each model version requires conversion and calibration on representative prompts; calibration on generic text alone missed an fp16 overflow that occurred on assistant prompts.

Core AI compiles each program for the device on first load, which takes on the order of a minute per program, and caches the result, so that later loads take 12 to 16 s. Compilation is therefore best performed ahead of first use, and the cache grows with every compiled model. A program that fails to compile for the ANE, or that uses bf16 or fp32, runs on the GPU without an error, so deployments need to verify that the ANE is in use, for example from timing or ANE power. A single cache size that fails to compile moves the whole multi-size program to the GPU.

Memory budgets require system-level measurements. In hand-off configurations, macOS compressed the memory of the idle GPU model during long ANE prefill: after a 32K-token prefill, the first decoding step stalled for 2.6 s while 1.25 GB were decompressed, and keeping the GPU working set wired removed the stall. MLX's buffer cache likewise requires a bound (see [Laptop operation](#laptop-operation)).

ANE outputs were deterministic on a given chip but differed across generations at near-ties: the M3 Pro matched the M4 Pro on all 100 requests, and the M2 Pro on 81. Together with the M2 Pro router results, this is consistent with different treatment of small fp16 values, but does not establish that mechanism. Deployments should validate numerical fidelity and router agreement on each chip generation; cross-device tests should not require identical generated tokens.

### Limitations

The study concerns one model and one workload of 100 assistant requests with greedy decoding. Each configuration was measured once per machine, and the long-context results rest on three documents per length. The ANE model uses INT8 weights, and its quality, while close to that of MLX 8-bit (98.4 against 99.0 percent top-1 agreement), is not identical. Agreement measures numerical fidelity, not task accuracy, and no task benchmarks were run; the activation-scale calibration included ten assistant prompts, so agreement on the workload is not an independent estimate; the calibration also left fp16 overflow at 5 of 183,190 positions examined.

Speculative decoding speed depends on the generated text, and the reported rates are averages over the workload. Because the engines generate different texts, their output lengths also differ: on the M4 Pro the ANE produced 266.6 tokens per request against 284.6 for GPU 4-bit, so that per generated token ANE-only inference was 19 percent faster than GPU 4-bit at 33 percent of its energy, compared with 24 percent faster at 31 percent of the energy per request.

Energy was measured for the whole system, including idle power, with a sensor whose lag of about 1.9 s is comparable to the duration of a short request, so the lag limits the interpretation of individual short-request energies; we therefore interpret the reported values as workload averages. Memory use was not measured systematically. The three machines differ in form factor, memory and MLX cache policy, so differences between them describe the systems, not the chips alone.