In-depth analysis · About 4 min read
Apple explains how RDMA and MLX split large-model inference across Macs
The October 15 engineering report separates memory capacity, prefill and decoding; its four-Mac result is not a universal speed or power claim.
- Source published
- Published here
- Source checked
Apple's October 15 technical report shows how MLX combines unified memory and Thunderbolt RDMA to run models across Mac Studio computers. Its headline example is Muse Spark Open at 36.73 generated tokens per second on four M5 Ultra machines with an 8,192-token context. This is a particular local-inference benchmark, not a new release of the example models, a cloud subscription or a guarantee for every agent workload.
Listen to the news
Read this article aloud or keep listening to the news.
Preparing the reader…
Device voice
Speech is handled by your browser or device. AISubHub does not send the text to an audio API. Voice availability and playback with a locked screen vary by device.
Resuming may repeat a short phrase.
At a glance
- Muse Spark Open has 816B total and 42B active parameters; NVFP4 reduces its model footprint to about 493GB, but intermediates still prevent it fitting on one 512GB machine.
- Pipeline parallelism splits layers by depth; tensor parallelism splits each layer by width and needs frequent low-latency communication.
- The reported up-to-4.6× TTFT improvement compares single-machine M5 and M3 for Qwen and Mistral, not Muse against an impossible single-Mac baseline.
Capacity and speed are different questions
Apple benchmarks with mlx_lm.benchmark, batch size one, generating 128 tokens after contexts of 8,192 or 65,536 tokens. It distinguishes time to first token, decode throughput and peak power. The evaluated formats are NVFP4 for Muse Spark Open, FP8 for Mistral Medium 3.5 and BF16 for Qwen 3.8. Total versus active parameters matters: a mixture-of-experts model can reduce computation without loading only its active weights. Muse's BF16 weights alone would exceed 1.6TB; even approximately 493GB after quantization leaves insufficient room for intermediate work on a single 512GB Mac. Distributing the model first solves capacity, which does not automatically imply faster generation.
What pipeline and tensor splitting change
A two-Mac pipeline stores earlier layers on one machine and later layers on another. Decoding remains largely sequential, so it is generally similar to single-machine speed for models that fit. Long-context prefill can improve because chunks overlap across the stages. Tensor parallelism instead divides each layer across machines, letting them compute simultaneously but requiring many small broadcasts and reductions. Apple's setup uses macOS 26.2 RDMA over Thunderbolt 5, the jaccl backend in MLX 0.30.1 or later, and two cables per peer link. The fast CPU–GPU synchronization flag used in the examples is experimental, not a universal default.
Read performance with the correct baseline
At the visible 8,192-token setting, Muse decoding increases from 15.74 tokens per second with two-machine pipelining to 24.99 with two-machine tensor parallelism and 36.73 with four. These are different configurations, not single-Mac comparisons. The up-to-4.6× TTFT and approximately 1.6× decode improvements concern single M5 versus M3 tests for Qwen and Mistral. Tensor parallelism is not always best: at 65,536 tokens, Muse's two-machine prefill trails its two-machine pipeline because expert-output normalization adds communication. Four machines improve the trade-off. In another specified four-machine, long-context experiment, doubling cables reduces TTFT from about 50.5 to 40.6 seconds. Context and topology cannot be dropped from those figures.
Local agents still require system-level evaluation
The report includes launch and local-server examples that can connect MLX to an agentic coding client. This demonstrates a configuration path, not that every tool workflow, concurrency level or response-quality requirement was evaluated. The roughly 200W M3 and 300W M5 peak figures concern the single-machine tests; they are not four-machine-cluster totals. Our analysis is that local deployment decisions should separately ask whether weights fit, how quickly the prompt is processed, how fast output arrives and what networking is required. More machines may unlock a model without providing a proportional speedup, especially when communications dominate. Hardware cost, full-system energy and concurrent service reliability remain separate questions.
Why it matters
The useful advance is a documented route from one machine's memory limit to distributed local inference, with explicit trade-offs between capacity and latency. It may help developers plan private experiments. AISubHub has not run these benchmarks, bought or configured the cluster, started the supplied server or measured production agent performance.
Limits and availability
We read Apple's complete engineering article and its visible default tables, not an independent replication. Results depend on model format, context, generated length, software and cabling. We do not transfer an example model's benchmark into a new model announcement, quality ranking, subscription entitlement or whole-cluster energy claim.
Sources and evidence
Source published: Oct 15, 2026 · Source checked: Oct 20, 2026
AI-assisted editorial articles checked against linked primary sources. Vendor claims are attributed; inclusion is not endorsement.