Model on the ANE
Core AI
LFM2.5-2.6B, prefill and decoding entirely on the Neural Engine, with the 328M-parameter DSpark drafter.
LLM inference on the Apple Neural Engine
Energy-efficient local inference: ANE, GPU and heterogeneous execution
Established LLM engines execute on the GPU and leave the Neural Engine unused. Using Core AI, we ran inference entirely on the ANE and compared it with the GPU in terms of latency, energy, temperature and battery life.
2.6–3.4× less energy than MLX 8-bit with speculative decoding, on typical requests
26–43% lower latency than MLX 8-bit without speculation, on all three Macs
1.8× longer battery life than MLX 8-bit on an M3 Pro MacBook Pro, projected
19–20 °C lower temperature than the GPU configurations, median sensor after 20 min
Local inference leaves the Neural Engine unused
- Every Apple Silicon M-series chip includes a Neural Engine, a specialized, energy-efficient accelerator for neural networks.
- Its dedicated datapaths perform the dense matrix arithmetic of neural networks at lower power, and it works directly on the unified memory shared by all units.
- Yet local LLM engines such as MLX and llama.cpp run on the GPU, which draws considerable power under sustained load and is shared with the interface and other models.
- Applications previously reached the ANE through Core ML, with limited control over operator placement and model state. Core AI in macOS 27 provides this control.
- Using the ANE, alone or alongside the GPU, would make fuller use of the hardware available for local inference.
Objective Determine whether LLM inference on the ANE, alone or combined with the GPU, can reduce energy consumption relative to MLX on the GPU while maintaining latency and output fidelity.
Energy
lower than MLX on the GPU
Latency
comparable to MLX on the GPU
Fidelity
within the 8-bit quality class
Experimental setup
We implemented an ANE engine on Core AI and compared it with MLX on the GPU at 4-bit and 8-bit, with and without DSpark speculative decoding. Energy is the integral of whole-system power as reported by the System Management Controller.
Systems
3
Mac mini M2 Pro and M4 Pro, MacBook Pro M3 Pro on AC power, on battery and in Low Power Mode.
Workload
100
Assistant requests of 800 to 2,600 tokens, with and without tool calls; long context up to 32K tokens.
Quality
98.4%
Teacher-forced top-1 agreement with bf16, against 99.0% for MLX 8-bit, within the 8-bit quality class.
Efficient ANE decoding depends on weight reuse
Decoding is memory-bound, since each token requires a full read of the model weights. The ANE attains only 44 to 70 percent of the GPU's effective memory bandwidth, whereas its compute throughput is 2.0 to 2.7 times higher. Efficient decoding on the ANE therefore requires amortizing each weight read over several tokens.
Call latency and tokens per call
Four FFN blocks of the model, INT8 weights. The GPU runs the same Core AI program.
Latency per call, ms
Latency per token, ms (log)
Tokens per call
ANE 2.42 ms 0.038 ms per token, 62× lower than in single-token calls
GPU 7.30 ms 0.114 ms per token, 27× lower than in single-token calls
Chain and tree verification
One ANE call verifies a draft tree of up to 64 tokens, accepting the longest path that matches the target, plus the target's next token. Tokens and tree shape are illustrative.
Output runs ontheNeuralEngine +4 tokens this cycle
accepted rejected not reached target's token
Decoding throughput
M4 Pro, tok/s. With tree verification, ANE decoding throughput is about 5 percent lower than that of GPU 8-bit with DSpark.
Energy and latency on typical requests
Assistant requests with prompts of 800 to 2,600 tokens. ANE-only inference used the least energy on all three machines and had the lowest latency in the 8-bit quality class on the M2 Pro and M4 Pro; on the M3 Pro, GPU 8-bit with DSpark was 6 percent faster.
ANE onlyGPU 8-bitGPU 8-bit + DSparkGPU 4-bit Rest of system
Energy per request, J
Latency per request, s
Mean system power, W
M4 Pro: ANE-only inference used 3.4× less energy than GPU 8-bit + DSpark and 4.1× less than GPU 8-bit, with −20% and −43% latency, respectively. The system drew 21 W against 48 W with GPU 8-bit.
2.6–3.4×
less energy per request
than MLX 8-bit + DSpark; 3.0–4.1× less than MLX 8-bit
26–43%
lower latency
than MLX 8-bit without speculation, on all three machines
20%
lower latency
than MLX 8-bit + DSpark on the M4 Pro, 2.77 s against 3.45 s
The advantage arises in both phases. On the M4 Pro, ANE prefill was 1.9 times faster than GPU prefill and consumed about one fifth of the energy per token, while ANE decoding required 158 mJ per token, compared with 409 to 575 mJ for the 8-bit GPU configurations.
Battery life and thermals on a laptop
MacBook Pro with M3 Pro. Battery life is projected from 20 minutes of intermittent use; power, temperature and fan speed were recorded over 20 minutes of continuous requests, each run starting from an idle machine after a cooldown.
Projected battery life, hours
One request every 30 s. The idle projection without a model is 21.1 h.
1.5 to 1.8 times the GPU configurations. Projected from average discharge power, not measured to full discharge.
2.1–2.9×lower power17.6 W against 37–51 W
19–20 °Ccoolermedian sensor, 20 min
9–37%lower fan speedafter 20 min
ANE onlyGPU 8-bitGPU 8-bit + DSparkGPU 4-bit
Hover over the chart to read values. No thermal throttling was observed in any configuration.
Weights on the ANE, attention on the GPU
On the ANE, the cost of attention grows faster than the KV cache, so attention dominates each call at long context. MLX computes attention on the GPU in a single fused kernel that scales considerably better, whereas weight reads remain more energy-efficient on the ANE. Split execution therefore assigns only the weight-free attention core to the GPU and keeps all weight reads on the ANE.
Time per decoding cycle, ms
M4 Pro, tree decoding with ANE-only and split execution.
Weights, ANE Attention, ANE Attention, GPU Drafter, host and transfers
1K
ANE
Split
4K
ANE
Split
8K
ANE
Split
16K
ANE
Split
32K
ANE
Split
Split execution, per layer
Split prefill
Split tree decoding
split between the ANE and the GPU GPU
Against GPU 8-bit + DSpark at 16K tokens
M4 Pro, 8-bit quality class, 256 generated tokens.
ANE only
−69%lowest energy
+13%latency
Split tree decoding
−64%energy
−17%latency
Split prefill + GPU 8-bit
−59%energy
−22%lowest latency
GPU 8-bit + DSparkANE onlySplit tree decodingSplit prefill + GPU 8-bit
Latency per request, s (log)
Energy per request, J (log)
These rankings hold for 256-token answers. Longer answers favor split prefill, because at long context the GPU decodes faster than the ANE.
A prompt router on the ANE
Local assistants may also rely on auxiliary models, for example a prompt router that selects a route or tool before generation. We evaluated such a router on the ANE, both in isolation and alongside chat generation on the GPU.
Prompt router
A 350M-parameter bidirectional encoder, M4 Pro.
ANE, INT8GPU, MLX bf16GPU, Core AI fp16
Time per call, ms
128 tokens
1,024 tokens
Energy per call, mJ
128 tokens
1,024 tokens
2.9×
faster
than MLX on the GPU at 128 tokens
6.3×
less energy per call
than MLX on the GPU at 128 tokens
<1%
lower chat generation rate
with the router on the ANE, against 20% with it on the GPU
The execution backend should be selected by use case
Neither unit was preferable for every workload. The results support selecting the execution backend per request, according to context size, power source, GPU load and user preference, such as a low-power mode. The configurations below had the lowest measured energy or latency on the tested machines.
Typical requests
ANE only
Lowest energy on every Mac; fastest at 8-bit quality on the M2 Pro and M4 Pro.
Lowest first-token latency
Co-prefill
ANE and GPU share the prefill; 2.3× lower time to first token on the M4 Pro.
Long contexts
Split execution
On the M4 Pro, 20–24% faster than GPU 4-bit at 58–61% less energy.
Battery and sustained load
ANE only
Projected battery life 1.5–1.8× longer than with the GPU configurations.
Routing next to generation
Encoder on the ANE
Leaves the GPU generation rate nearly unchanged at moderate call rates.
Methods and full results
The tech note provides the implementation details, the experimental methodology, the full results on all three machines, and a discussion of their practical implications and limitations.