Inside the Apple Neural Engine

The Apple Neural Engine is no longer just a black box behind Core ML. Bryngelson’s guide presents a full-stack, reverse-engineered map of the engine’s numerics, program path, compiler behavior, memory system, compression support, performance envelope, firmware protocol, and security boundary. The result is an unusually detailed systems artifact: part architecture note, part performance paper, part ABI guide, and part warning label.

The arXiv record describes the paper as a 302-page reference covering A11 through A18 and M1 through M5 families, with direct measurements on M1 and M5 and claims labeled as measured, decompile-derived, or predicted.1 The PDF itself states that the direct route below Core ML is callable from ordinary user space but remains private, unsupported, and version-fragile; the supported production path remains Core ML.2

Why the ANE matters

The ANE is one of the most widely deployed neural accelerators, yet Apple exposes it to third-party software mostly as an implementation detail behind Core ML. Apple’s Core ML materials present a high-level deployment model in which the operating system can use CPU, GPU, and Neural Engine resources to execute models efficiently.3 Apple’s own machine-learning posts also describe Core ML as a hybrid planner that blends CPU, GPU, and ANE execution rather than exposing a public ANE instruction set.4

That abstraction is useful for shipping apps. It hides per-chip quirks, absorbs model-format details, and allows fallback when a graph segment does not run on the engine. It also leaves unanswered questions that matter to compiler authors, performance engineers, and researchers: which operations actually lower to the ANE, what precision is used internally, why some ANE-friendly models run fast while others fall behind the GPU, whether compression reduces bandwidth or merely file size, and how dispatch overhead affects small graphs.

Bryngelson’s paper addresses those questions by moving below the public abstraction. The guide says the ANE is reached through a private runtime/compiler path, then triangulates observations from direct execution, static decompilation, live instrumentation, and compile-and-run probes.2 Companion work on ANEForge presents a Python package that programs the ANE directly through private stack components, again emphasizing that production access remains Core ML and that direct access is a research path rather than a public Apple contract.8

The paper’s evidence model

The paper’s best methodological decision is the separation of evidence types. Many reverse-engineering claims fail because they treat a symbol name, a capability bit, or a compiler parse result as if it proved executable hardware support. This guide uses a stricter definition: a capability counts only after the relevant operation compiles and runs on a target device.

The paper records four main evidence streams: direct private-runtime access, static analysis of private runtime/compiler/driver/firmware binaries, live read-only instrumentation, and compile-and-run probing.2 Appendix E makes the taxonomy explicit: claims are marked as measured on a named generation, decompile-derived from a named binary, or predicted from per-chip tables and not yet confirmed on silicon.2

The measurement-host accounting is subtle. The abstract emphasizes M1 and M5 as direct measurement anchors.1 Appendix E later states that M1/H13 and M5/H17s were the two main measured points, with an A14-class Apple M2/H14 point used where a claim required a middle generation.2 That clarification matters because some compression and operation-gating claims use the A14/M2 point rather than only the two headline silicon anchors.

The architecture model: fixed-function fp16 matrix engine, not a small GPU

The guide’s conceptual model is that the ANE is a fixed-function matrix accelerator with a compiler-managed working set, a private program format, and a host-to-firmware command path. It is not presented as a general programmable GPU, a public SIMD ISA, or a CPU coprocessor. The compiler lowers a graph into an ANE program; the runtime loads and dispatches that program; firmware and driver components manage execution, address translation, and security checks.2

Three architecture ideas are central:

1. Static graph shape matters. The engine wants concrete shapes and compile-time-visible structure. The direct path does not expose the symbolic-shape route described by the compiler machinery; variable-length workloads generally require padding, bucketing, or separate compiled programs.2
2. Residency matters. The M1 working-set threshold is reported around 2 MB. Below that level, intermediate values can remain resident; above it, the compiler tiles or streams operands through DRAM.2
3. Dispatch has a fixed cost. The M1 per-evaluation dispatch floor is reported around 0.23 ms, so tiny operations can be dominated by host/runtime/firmware overhead rather than arithmetic throughput.2

The compiler pipeline described in the paper has four broad phases: graph fusion, hardware legalization, scheduling and task-descriptor partitioning, then memory and DMA optimization.2 This explains why apparently equivalent graphs can have different performance. A convolution followed by bias and activation may become one fused operation; an unsupported reshape or dynamic feature can split a graph or send work off-engine.

Numerics: fp16 products, wider accumulation, and real limits

The paper’s numerical characterization is one of its most useful contributions. The reported datapath uses fp16 inputs and fp16 weights, returns fp16 outputs, and accumulates internally in a wider register. The wide accumulator is supported by cancellation and reduction probes, while the fp16 input/product behavior explains why some transformer projections still lose accuracy even when reductions themselves are wider than fp16.2

The distinction is important. A wide accumulator can preserve many well-conditioned partial sums, which is valuable for convolution, matrix multiply, pooling, and normalization. It does not reverse rounding that has already occurred in fp16 products. Cancellation-heavy down-projections can therefore remain numerically fragile even if the final sum is accumulated in a wider internal representation.

The paper lists several exact constants, including fp16 maximum finite magnitude 65,504, a width-slice crop-DMA gain of 16, a width-slice finite ceiling of 4,094, and activation-table error figures.2 These details make the guide more than a performance note: it becomes a set of numerical guardrails for anyone attempting ANE-targeted graph construction.

Operation reachability: capability bits are not enough

The operation-surface section distinguishes operations that are actually reachable from operations that merely appear in hardware tables, frontend validators, or framework-level descriptions. According to the guide, core perception and encoder primitives are native from the M1 generation onward: two-dimensional convolution and variants, matrix multiply, fused attention, normalization, pooling, elementwise arithmetic, activation tables, and data-movement operations.2

The hard-negative list is just as useful. The paper reports no ANE path on current families for reduce_prod, scatter variants, mod, one_hot, non_zero, shape, recurrent cells, inverse and hyperbolic trigonometric functions, and several other operations. Those operations must be reformulated, compiled off-engine, or placed on CPU/GPU.2

Three-dimensional convolution illustrates the paper’s stricter evidence rule. It is described as visible in capability machinery but not reachable through backend lowering; the engine does not actually run it on the tested direct path.2 That finding is valuable because it prevents a common category error: treating parser acceptance or capability attestation as proof of executable hardware support.

The direct route below Core ML

The paper describes a direct dispatch path through private Espresso/e5rt interfaces: construct a graph, compile it into the ANE program format, load the resulting program, bind buffers, and dispatch the work. For production use, the route is undocumented, unsupported, private, and version-fragile.2

The direct route does not mean arbitrary instruction injection. The kernel driver reportedly verifies submitted programs before firmware execution using a program signature, vnode trustcache validation, and client code-signing identity checks. A hand-built or self-compiled program is rejected unless it comes through the system daemon’s compile-and-sign route.2

graph description
          ↓
        compiler / daemon accepted representation
          ↓
        signed ANE program package
          ↓
        private runtime load + buffer binding
          ↓
        firmware-dispatched execution on the engine

This boundary is one reason the work is credible but not directly productizable. It gives researchers a way to measure the engine more precisely than Core ML placement allows, while Apple’s public support and App Store path remain centered on Core ML.

The roofline: two ceilings, one working-set limit, one dispatch floor

The paper’s most actionable performance model is a roofline with four control quantities: compute throughput, DRAM bandwidth, on-chip working-set threshold, and fixed dispatch cost. The M1 constants are reported as an overhead-isolated matmul slope near 12 fp16 TFLOP/s, DRAM bandwidth near 85 GB/s, a ridge point near 141 FLOP/byte, a 2 MB on-chip working-set threshold, and a dispatch floor near 0.23 ms.2

The paper follows the standard roofline tradition: attainable performance is bounded by the lower of the compute roof and the product of arithmetic intensity and memory bandwidth. The original Roofline model was introduced as a way to relate floating-point throughput, operational intensity, and memory bandwidth in a simple visual performance envelope.7 Bryngelson adapts the model to a fixed-function neural accelerator by adding the ANE-specific working-set and dispatch-floor constraints.

The M5 section reports scaling rather than a new programming model: matmul-slope peak around 19.6 fp16 TFLOP/s, single-program matmul around 9.5 fp16 TFLOP/s, convolution peak around 14.3 fp16 TFLOP/s, a working-set threshold around 4.72 MB, and DRAM weight-stream bandwidth around 145 GB/s.2 The key message is that the engine scales, but the design rules remain recognizable.

Compression: storage savings are not the same as bandwidth savings

The compression chapter is important because it separates two ideas that are often conflated. A compressed model file is not automatically a faster model. Speed improves when the compressed representation streams through the hardware in compressed form; if the compiler expands it to dense fp16 before the DRAM movement, the bandwidth benefit disappears.

The M1 result is the practical surprise: int4 lookup-table weights and structured sparsity stream natively, while int8 and blockwise affine fold to dense fp16 and therefore do not provide a stream-bandwidth gain on that generation.2 This explains why model-format compression and runtime acceleration have to be measured separately.

Where the ANE wins, and where it loses

The guide’s performance conclusions are not boosterish. It does not claim that an NPU is automatically best for all machine-learning work. The reported model is more specific: the ANE excels when graph work is static, fused, compute-heavy, resident, and built from supported primitives. It loses when work is tiny, dynamic, bandwidth-bound, or dominated by dispatch overhead.

The paper reports that a single 3×3 convolution at 256 channels runs around 3.8× faster and 9× more energy-efficient than the same chip’s GPU, and that a 16-layer convolution stack reaches a 14.5× efficiency advantage on M1.2 For encoder serving, the reported batch threshold is roughly 23 for a single-sentence encoder and roughly 6 for a self-attention block; beyond those thresholds, GPU throughput can overtake.2

Apple’s public ANE guidance for transformers is directionally consistent with this: Apple’s transformer post emphasizes ANE-optimized implementations to reduce latency and memory use, while its vision-transformer post discusses architecture and layout choices such as working around tensor-rank limits and using NHWC layout for efficient window partition/reverse operations.45 The reverse-engineering guide adds the lower-level reason: the engine rewards fused, resident, statically shaped dataflow.

Energy efficiency and the instrumentation caveat

The reported energy picture is one of the paper’s most striking results. Across measured classes, the ANE is described as drawing a few watts where the GPU can draw tens of watts. The M1 convolution-stack comparison reports 2063 GFLOP/s/W for the engine versus 142 GFLOP/s/W for the GPU. The M5 comparison reports 2289 GFLOP/s/W for the engine versus 175 GFLOP/s/W for the GPU.2

The caveat is measurement basis. The methodology section says per-rail power figures come from powermetrics, a system power estimator, rather than an independently validated wall-meter reading.2 The ML.ENERGY Zeus Apple Silicon project similarly notes that IOReport energy values are believed to be model-based estimates derived from utilization, frequency, and voltage rather than direct power sensor readings, and that Apple’s powermetrics man page warns reported values may be inaccurate.10

That caveat does not make the results useless. A large, consistent power gap across similar measurement methods is still informative. It does mean that exact energy ratios should be quoted with the instrumentation limitation attached.

LLMs: prefill and encoders fit; autoregressive decode does not

The LLM case study is a corrective to a common assumption: “neural engine” does not mean “best processor for every neural network phase.” The paper argues that autoregressive decode is both bandwidth-bound and dispatch-bound. A single-token decode step streams model weights with low arithmetic intensity, while the layer stack issues many small dispatches that each pay the fixed per-evaluation floor.2

The reported numbers are explicit: at batch 16, GPU decode runs about 2.7× faster and about 4.6× more energy-efficient than the ANE path; int8-hybrid decode reaches about 0.99× the fp16 rate because the step is dispatch-bound rather than weight-bandwidth-bound.2 Apple’s own Llama 3.1 Core ML article also targets the GPU for the example M1 Max deployment, explaining that LLMs are often memory-bandwidth constrained and that the GPU offered the best compute/bandwidth combination for that case.6

The practical placement rule is therefore hybrid: put prefill, encoders, and vision front ends on the ANE where they fit the static fused model; put token-by-token decode on the GPU unless a future compiler/runtime path removes the dispatch and bandwidth bottlenecks.

Training and scientific computing: possible, but bounded

The training chapter is interesting because it demonstrates that “inference accelerator” is not the same as “inference-only arithmetic.” The paper reports no native backward operation, but expresses forward, backward, and optimizer updates as ordinary graph operations, keeping optimizer state resident across steps. A small convolutional network reportedly trains to 0.9080 final test accuracy on M1 and 0.9070 on M5 after the described run.2

This should be read as a capability demonstration, not as proof that ANE training is broadly competitive. Larger models face graph-size limits, compile latency, optimizer-state pressure, and fp16 conditioning constraints. Related contemporary work, such as Orion, also explores direct ANE execution and training/inference paths for LLM workloads. This remains an active reverse-engineering and systems research area rather than a stable public development surface.11

The scientific-computing chapter makes a similar point. Fixed-trip-count loops can be unrolled into static graphs: stencils, explicit integration, some series expansions, and small dense factorizations can fit. Data-dependent trip counts, adaptive output sizes, large factorizations, and ill-conditioned fp16 problems remain poor fits. The paper reports small dense factorization and eigensolver limits around n ≈ 20–32 for some routines, while fixed iterative solvers can run larger when the matrix is embedded as a constant weight.2

System internals and security

The back half of the guide reads like a systems reference: compiler internals, program/container format, hardware-abstraction-layer gates, compression internals, hidden layers, kernel driver ABI, address translation, firmware scheduling, command protocol, power behavior, security, and telemetry.

The security model described by the paper places most trust enforcement in the kernel driver, not in the firmware. The driver checks program signatures, trustcache status, and code-signing identity before the work reaches the engine; the firmware then performs structural checks such as bounds and overlap checks.2

The paper also reports a cross-process timing and occupancy side channel. On M1, contention between processes leaks co-tenant presence and coarse duty cycle at roughly 20–50 bit/s, while data isolation held in the reported concurrent-program checks: 9,000 of 9,000 checks were correct with no evidence of outputs reflecting another process’s inputs or weights.2 That is a measured confidentiality caveat, not a reported arbitrary data leak.

What the paper gets right

It uses multiple independent evidence streams

The strongest feature of the work is triangulation. Reverse-engineering claims are supported through combinations of compile/run behavior, measured latency and correctness, binary-derived constants, and live instrumentation. This is especially important for a private accelerator where no public ISA or driver contract exists.

It separates reachability from appearance

The guide repeatedly distinguishes “recognized by a frontend,” “present in a table,” “accepted by a validator,” and “actually lowered and run on the engine.” That discipline is the reason the operation tables are useful.

It explains failures, not just wins

The best performance sections are the ones that explain why the ANE loses. Tiny operations fall below the dispatch floor. Large GEMM and long-sequence attention can become GPU territory. Autoregressive decode is bandwidth- and dispatch-bound. These negative results make the positive results more credible.

It turns optimization into a cost model

The article’s practical rules fall out of the model: fuse small operations, keep intermediate working sets resident, increase arithmetic intensity, avoid unnecessary inter-engine transfers, and verify compression streaming rather than assuming it.

Main caveats

Private-interface fragility

The methodology says the path calls private Apple framework symbols tied to ANECompiler 9.509.0 and that macOS updates can break the build or dispatch path.2 This is not a small detail. It defines the boundary between research utility and production suitability.

Energy measurements need independent validation

The reported energy ratios are large, but they depend on modeled system telemetry rather than independent wall-meter validation.2 Future work would be stronger with a limited external power sanity check, even if per-rail attribution still requires system telemetry.

Measured, decompiled, and predicted facts can blur

The paper makes a serious effort to label provenance, especially in Appendix E. Even so, the density and breadth of the text make it easy for readers to quote a predicted or decompile-derived family-wide claim as if it were measured on every chip. A front-loaded “claim matrix” would make the guide easier to cite accurately.

Headline TFLOP/s values need careful wording

The M1 ~12 fp16 TFLOP/s number is an overhead-isolated slope, while other tables report lower single-program and convolution ceilings.2 A reader quoting only the largest number can overstate typical end-to-end throughput.

The work is too large for normal paper review

At 302 pages, the guide is closer to a reference manual than a conventional paper. The breadth is valuable, but auditability suffers. A shorter companion paper focused on methodology, numerics, roofline, operation reachability, and workload placement would make the central claims easier to review.

Practical implications

For production Core ML deployment

The public path remains Core ML. Apple’s developer-facing model is to convert and deploy models through Core ML, allowing the system to use CPU, GPU, and Neural Engine resources as appropriate.3 The reverse-engineering guide is still useful for production teams because it explains why Core ML placement may behave the way it does. ANE-friendly models tend to have static shapes, supported primitives, fused operator patterns, moderate working sets, and layout choices that avoid unnecessary movement.

For researchers and compiler authors

The direct route enables better measurement and experimentation, but every result needs compiler-version and OS-version provenance. A reproducible ANE experiment should record the chip, macOS version, ANECompiler version, graph shapes, operation list, precision, compression form, and whether energy figures come from modeled telemetry.

For model designers

The design rules are concise:

1. Use static shapes where possible.
2. Fuse small operations to pay the dispatch floor once.
3. Keep intermediate working sets below the on-chip threshold.
4. Prefer supported convolution, matmul, normalization, pooling, elementwise, and activation patterns.
5. Measure numerical error for cancellation-heavy fp16 contractions.
6. Verify whether compression streams on the target chip.
7. Place decode-heavy LLM work on the GPU unless measurements prove otherwise.

Verdict

Bryngelson’s guide is a high-value technical reference for Apple Neural Engine architecture and performance. Its strongest contributions are the ANE roofline model, the fp16/wide-accumulator numerical characterization, the distinction between advertised and reachable operations, the compression streaming table, and the workload-placement logic that separates convolution and encoder wins from autoregressive decode losses.

The correct way to cite the work is with qualifiers. M1 and M5 measurements should be treated as strong evidence for those targets under the tested compiler/runtime stack. M2/A14 middle-generation claims should be quoted only where the paper says they were measured. Cross-generation claims derived from tables should remain labeled as decompile-derived or predicted unless the paper reports silicon confirmation. Energy-efficiency claims should include the modeled-power caveat.

The larger implication is that NPUs are not magic. The ANE is powerful because it is specialized: fixed-function, static, low-power, and efficient for the right graph shapes. The same specialization creates the limits. The paper is valuable precisely because it maps both sides.