Inside Apple M4’s SME engine

Taras Zakharko’s M4 SME exploration suggests that Apple’s implementation of Arm’s Scalable Matrix Extension is not best understood as a wider SIMD unit attached to every CPU core. The measurements point instead to a tile-centric accelerator shared by the cores of each CPU cluster.

On the performance side, one thread can drive approximately 2 FP32 TFLOP/s, provided it uses four independent ZA accumulators. Additional threads do not raise that ceiling. On the efficiency side, the corresponding implementation reaches roughly 360 GFLOP/s and behaves like a materially different execution engine.

The distinction matters. Software that treats SME as “SVE, but wider” will leave most of the M4’s matrix throughput unused. Software that assumes every CPU core owns an independent SME pipeline may oversubscribe a shared resource. Software that expects FP16 or BF16 to automatically double throughput will also be disappointed.

The M4’s fastest SME path does not merely use wider vectors. It routes computation through ZA, a separate matrix-oriented architectural state whose performance characteristics differ sharply from ordinary streaming-vector instructions.

A compact view of the machine

The repository records a 512-bit streaming vector length on the tested M4. That gives 64-byte streaming Z registers and a 64-by-64-byte ZA array, or 4,096 bytes of matrix state. Interpreted as FP32, ZA contains four independent 16-by-16 tiles.

These numbers come from a nine-core M4 iPad Pro configuration with three performance cores and six efficiency cores. They should not be generalized silently to every M4 product.

At a 512-bit streaming vector length, each Z register contains sixteen FP32 values. An FP32 outer product combines two such vectors and updates a 16-by-16 tile:

Because multiply-add counts as two floating-point operations, one instruction represents:

ZA exposes four such FP32 tiles. That architectural detail becomes central to the measured throughput.

Why the repository is unusually useful

The project is small, but it asks the right questions. Rather than running one matrix multiplication and reporting a headline number, it generates families of microbenchmarks that vary the instruction, the number of independent destinations, loop unrolling, thread count, quality-of-service class, working-set size, alignment, and whether data moves through Z registers or ZA-oriented operations.

The repository includes the generators, emitted benchmark code, Swift execution harness, processed JSON results, and analytical reports. That makes it possible to compare the interpretation against the code and raw measurements instead of treating the prose as the only source of truth.

The generated compute tests use eight million loop iterations. Variants change the number of independent output tiles or vectors, which lets the benchmark separate dependency latency from peak issue throughput. The Swift harness performs warm-up runs followed by repeated measured runs, exercises multiple thread counts, and uses different dispatch quality-of-service classes to encourage placement on performance or efficiency cores.

The 2 TFLOP/s ceiling

The FP32 outer-product benchmark is the clearest experiment. With one thread and one dependent FP32 destination tile, the M4 reaches roughly 500 GFLOP/s. Giving the same thread two independent tiles roughly doubles the available throughput. With four independent tiles, the result approaches 2 TFLOP/s.

One independent tile    → about 0.5 TFLOP/s
Two independent tiles   → about 1.0 TFLOP/s
Three independent tiles → about 1.5 TFLOP/s
Four independent tiles  → about 2.0 TFLOP/s

The same aggregate ceiling can be approached with multiple threads whose combined instruction streams provide enough independent work. Once the performance-side SME resource reaches approximately 2 TFLOP/s, however, adding more threads does not multiply the result by the number of performance cores.

A per-core interpretation would predict three independent 2 TFLOP/s engines on the tested device, or approximately 6 TFLOP/s in aggregate. The measurements do not show that. A more accurate description is approximately 2 FP32 TFLOP/s per performance cluster, accessible from one CPU thread but shared among threads executing on that cluster.

This is not merely a synthetic-loop curiosity. A separate University of Jena investigation reported approximately 1.825 FP32 TFLOP/s for an optimized SGEMM path on an M4 performance core, about 91 percent of the microbenchmark ceiling. That indicates that much of the measured limit can survive the transition to a blocked matrix kernel with loads, stores, and loop control.

Four accumulators reveal the pipeline

The four-accumulator requirement provides a plausible view into the implementation. An FP32 outer product performs 512 floating-point operations. At 2 trillion operations per second, the engine is completing roughly 3.9 billion such instructions per second:

The result is consistent with an execution engine that can accept one outer product each cycle at an operating rate around 3.9 GHz. A dependency chain targeting the same accumulator cannot issue at that rate because the previous update has not completed. Four independent destinations hide an apparent four-cycle dependency latency and keep the pipeline full.

The architectural alignment is striking: ZA provides exactly four full FP32 accumulator tiles, and four independent tiles are exactly what the measured pipeline needs to reach peak throughput.

The efficiency cluster is not a scaled copy

The efficiency-side SME result is approximately 360 GFLOP/s for FP32 outer products, or about 18 percent of the performance-side ceiling. FP64 reaches roughly 90 GFLOP/s, while several widening FP16, BF16, and integer operations cluster around the same 360 GOP/s region.

More important than the absolute number is the scaling pattern. The repository finds little evidence that independent accumulators or additional E-core threads produce the same pipelined throughput increase seen on the performance side.

Several designs could explain this: a narrower matrix datapath, a lower operating frequency, a non-pipelined or less deeply pipelined outer-product unit, shared issue or data-movement bottlenecks, or some combination of these factors. The benchmark cannot choose definitively among them.

There is also a scheduling caveat. The harness does not bind threads to specific physical cores. It uses a high-priority dispatch queue for the performance-side tests and a utility-priority queue intended to steer work toward efficiency cores. The observed split is credible, but it remains scheduler-mediated rather than established through hard affinity.

Lower precision is not automatically faster

FP64 lands at one quarter of FP32 throughput, which is consistent with each vector holding half as many elements in each dimension. FP16 and BF16 are more surprising. Both accumulate into FP32, but neither doubles the FP32 peak. They remain near 2 TFLOP/s and show longer apparent dependency latency.

The repository proposes that these widening operations may internally use two FP32-like passes. That explanation is plausible, but remains a hypothesis. The benchmark establishes equal peak throughput and different latency; it does not directly expose the internal sequence.

I8 is the notable exception. Its four-way dot-product structure reaches approximately 4 TOPS on the performance side, making it the tested low-precision format with a clear arithmetic-throughput increase over FP32.

Streaming SVE and ZA follow different paths

One of the most revealing comparisons is between ordinary streaming-vector arithmetic and ZA-oriented arithmetic. A conventional FP32 FMLA operating on streaming Z registers reaches only about 31 GFLOP/s in the optimized variants. An SME2 FP32 vector operation using ZA-oriented multi-vector forms reaches roughly 250 GFLOP/s. Outer-product instructions that fully use the two-dimensional tile structure reach approximately 2 TFLOP/s.

Streaming Z-register FMLA     ≈   31 GFLOP/s
ZA-oriented vector FMLA       ≈  250 GFLOP/s
ZA outer-product accumulation ≈ 2000 GFLOP/s

These are not minor tuning differences. They are different performance classes. All three execute inside the broad SME programming environment, but they do not appear to use equivalent hardware resources.

This makes SME a deliberately leaky abstraction on M4. The architecture standardizes instructions, register state, and programming behavior. It does not require every legal instruction form to receive comparable throughput.

Memory results require restraint

The memory generator tests loads, stores, and copies over working sets from 4 KiB to 64 MiB, with different alignments and variants using adjacent or strided Z registers and ZA array-vector accesses. For cache-sized working sets, the raw data contains aggregate rates in the hundreds of gigabytes per second. Some multithreaded configurations rise higher as multiple cores access data already resident in the cluster cache hierarchy.

Those figures should not be presented as external DRAM bandwidth. A 3 KiB or 12 KiB benchmark is primarily exercising registers, load/store pipelines, and on-chip cache. Aggregate rates can also count the same cache-resident bytes repeatedly.

The measurements demonstrate a high-bandwidth local path between the CPU cluster and the SME state. They do not independently establish the bandwidth of the system memory interface, nor do they reveal the exact physical connection between the matrix machinery and a named cache level.

There is also evidence that direct ZA memory operations are not always the fastest way to populate a tile. The University of Jena SGEMM work describes loading data into Z registers and moving it into ZA as part of an optimized kernel. In practical terms, Z registers can serve as the load-and-transform layer while ZA remains the accumulation layer.

Scheduling becomes a cluster problem

A shared accelerator changes the unit of optimization. Conventional CPU code often treats every runnable core as an additional source of arithmetic throughput. That assumption fails when several cores feed the same SME engine.

On the tested M4, one performance-core thread can saturate the performance-side resource. Additional threads can help only when the original thread fails to expose enough independent work. Once the cluster ceiling is reached, more threads compete rather than scale. Efficiency-side work uses a separate, slower resource.

The repository also reports throughput reductions for some mixtures of floating-point widths and integer instruction families. It interprets these as possible switching penalties inside the SME implementation. This is suggestive, but still speculative. Targeted alternating-instruction tests or performance counters would be needed to establish the cause.

The software problem therefore has two layers. At the application level, multiple libraries or worker threads may unknowingly contend for one cluster-level engine. At the operating-system level, the benchmark can request different QoS classes, but it cannot express “place this thread on the performance cluster and grant it exclusive access to that cluster’s SME resource.”

A useful performance report should distinguish throughput per issuing thread, throughput per cluster, throughput across all clusters, performance under concurrent SME users, and performance under mixed data types. Publishing only “GFLOP/s per core” hides the most important part of the architecture.

What the project gets right, and where caution remains

The strongest methodological choice is the deliberate variation of instruction-level parallelism. Many microbenchmarks accidentally measure dependency latency and call the result peak throughput. By changing the number of independent destinations, this project exposes both.

Thread-count scaling is equally important. A one-thread test would have revealed 2 TFLOP/s, but not whether that throughput belonged to the thread, the core, or the cluster. The thread scaling is what makes the shared-resource interpretation possible.

The work also benefits from publishing the raw material and checking exposed SME feature bits before running instruction families. SME is modular; support for the base architecture does not imply support for every floating-point, integer, or widening operation.

The main caveats are straightforward. The results come from one device and one hardware configuration. QoS steering is not hard core affinity. The pipeline model is inferred from timing curves rather than direct performance-counter evidence. Peak loops are not complete applications. Real kernels must handle packing, cache misses, edge tiles, reductions, stores, streaming-mode transitions, and competing system activity.

There are also limits to the reports themselves. Some sections read as working notes, and exact opcode, datatype, and operation-count details are better checked against the generated benchmark catalog and raw result files.

What software should do with this

The practical guidance follows directly from the measurements.

• Treat SME throughput as a cluster resource. Do not multiply the single-thread 2 TFLOP/s result by the number of performance cores.
• Use enough independent ZA accumulators. For FP32 outer products on this M4, four tiles are the key configuration.
• Do not assume lower precision is faster. FP16 and BF16 can save bandwidth without increasing peak matrix arithmetic.
• Prefer ZA-oriented execution paths. Merely enabling streaming mode and porting an SVE loop may capture little of the available hardware.
• Stage data deliberately. Keep working sets close to the cluster, use Z registers for loading and transformation, and retain partial sums in ZA.
• Control concurrency. Two SME-heavy worker pools on one cluster may divide a shared engine rather than receive independent engines.

For general workloads, mature numerical libraries remain the safer default. Apple’s Accelerate framework provides optimized numerical routines, and the independent SGEMM measurements indicate that a carefully tuned implementation can approach the hardware ceiling. Hand-written SME is most justified for specialized shapes, fused operations, unusual data layouts, or kernels not covered by standard library interfaces.

The repository’s most valuable contribution is not a benchmark number. It is a change in mental model.

Apple M4 SME should not be pictured as a 512-bit SIMD extension replicated inside every CPU core. The measurements instead describe a shared, tile-oriented matrix machine reached through architectural instructions issued by ordinary CPU threads.

On the performance cluster, that machine accepts enough independent FP32 outer products to sustain approximately 2 TFLOP/s. Four ZA accumulators expose a pipeline consistent with roughly four cycles of dependency latency and one accepted outer product per cycle. One thread can fill it. More threads share it.

The efficiency cluster exposes the same architectural interface but not the same machine. Its peak is much lower and its scaling behavior is different. FP16 and BF16 do not provide the expected arithmetic uplift. I8 does. Plain streaming-vector FMA remains comparatively slow. ZA-oriented operations are the path to substantial throughput.

The exact internal block diagram remains unknown. The benchmarks do not prove a particular array layout or cache connection. They do, however, reveal the constraints that determine software performance: organize work around ZA, maintain several independent accumulators, schedule for a cluster-level resource, distinguish instruction forms, and validate low-precision assumptions on the actual hardware.