In-depth analysis · About 6 min read
PyTorch MPS: decoding, correctness and driver memory need separate measurements
Hugging Face's PyTorch team explains MPS changes for the 2.15 release candidate, with distinct evidence for grouped decoding, FP8 storage, padding and future 2.16 work.
- Published here
- Source published
- Source checked
An October 20 technical article by Hugging Face's PyTorch contributors explains how dedicated Metal kernels change work performed by the MPS backend on Apple silicon. It provides a PyTorch 2.15 release-candidate installation path and a separate 2.16 preview. Its useful contribution is a set of specific bottlenecks and measurements, rather than one speedup that applies to every Mac workload.
Listen to the news
Read this article aloud or keep listening to the news.
Preparing the reader…
Device voice
Speech is handled by your browser or device. AISubHub does not send the text to an audio API. Voice availability and playback with a locked screen vary by device.
Resuming may repeat a short phrase.
At a glance
- The documented environment requires Apple silicon, macOS 15 or later and Python 3.11 or later. The article presents 2.15 as a release candidate and separates 2.16 nightly or in-review work.
- A grouped-matmul decoding specialization produces about 1.3 times the throughput in the shown M3 Ultra experiments with torch.compile. A separate softmax chart measures single-operator microseconds, not complete response latency.
- For one 28-tile Wan2.2 VAE decode, the reported total driver-memory peak falls from 270.135 to 8.108 GiB while allocator-reserved peak stays at 8.070 GiB. These are different memory measures.
1. Start with the version and the part of execution being changed
The authors identify themselves as Hugging Face's PyTorch team, working upstream with PyTorch maintainers and contributors. Their explanation centers on replacing selected MPSGraph paths with dedicated Metal kernels. It is backend engineering: models can benefit without becoming new model releases, and a faster kernel does not establish better model answers. The stated setup requires an Apple-silicon Mac, macOS 15 or later and Python 3.11 or later.
The article explicitly invites readers to try a 2.15 release candidate in a separate environment. Later sections discuss 2.16 changes that may be in nightlies or still under review. AISubHub has not independently checked subsequent stable-release status. For a reproducible comparison, record the exact build, device, dtype, compiler setting and workload before moving between these sections. Otherwise a prospective optimization can easily be mistaken for a feature in the tested release.
2. Decode specialization changes a narrow, repeated operation
During prefill, a model processes a block of input; during decoding it repeatedly processes a small input for the next token. The article connects F.linear to existing matrix-vector kernels and eligible fused bias additions. Across 96 BF16 layer shapes, it reports a 1.23-times geometric-mean speedup. Small projections gain most, while large layers already close to the bandwidth limit remain near one times. This is a collection of layer measurements, not a full-model average.
For mixture-of-experts execution, native grouped matmul reads group offsets on the GPU instead of copying them to the CPU and looping over groups. The displayed decode-specialization chart uses an M3 Ultra with torch.compile: Qwen3.6-35B-A3B at context 512 rises from 38.7 to 51.7 tokens per second, and at context 2048 from 36.8 to 48.7. The article separately reports larger whole-version comparisons against 2.14. Those comparisons, this roughly 1.3-times kernel comparison and later 2.16 measurements have different baselines; they must not be combined or multiplied.
3. FP8 storage and softmax timing answer different questions
The FP8 addition is torch.float8_e4m3fn tensor support, with one byte per value compared with two for FP16 or BF16. F.scaled_mm supports tensorwise and rowwise scaling. That can reduce the storage occupied by eligible tensors, but it does not halve the memory of an entire application: other tensors, working buffers and runtime allocations remain. The example's quantization and output dtype also matter to numerical behavior. The article discusses native FP8 matrix multiplication as groundwork for future Metal shader work, so current storage support should not be equated with a universal hardware speed claim.
The softmax chart is another distinct experiment. For one FP32 logits row with vocabulary size 128,256, it shows 57.4 microseconds on MPSGraph and 7.1 on the Metal kernel; the prose rounds those to 57 and 7. An approximately eight-times faster softmax can shorten one step in token sampling, but other work still determines total generation time. AISubHub analysis: benchmark the complete path as well as the expensive operator, and preserve units so microseconds per call do not turn into an unsupported claim about seconds per response.
4. Padding exposes memory outside the tensor allocator
The Wan2.2 example is specifically Wan-AI/Wan2.2-TI2V-5B-Diffusers, with 8,764 constant-padding calls in a 28-tile VAE decode. Moving padding to Metal changes the reported total driver-memory peak from 270.135 to 8.108 GiB. Yet allocator-reserved peak is 8.070 GiB in both configurations, and current tensor-allocation peak changes only from 4.422 to 4.095 GiB. A tensor allocator metric alone would miss most of this particular problem. The result is not a claim that every video model now needs only eight GiB.
Correctness is a separate benefit. For a Qwen-Image-2.1 VAE encode-decode round trip, the authors report agreement with the CPU reference improving from 10.06 to 29.95 dB PSNR after the padding fix. This concerns the tested reconstruction workflow, not a general image-generation quality score. Other listed changes include complex-valued linear algebra, backward paths and attention broadcasting. A useful upgrade check therefore includes numerical outputs and peak driver memory alongside throughput, with the same model and operation shapes.
5. Keep future work and local validation visible
The 2.16 preview includes command-buffer submission changes, attention running entirely on Metal and further MPSGraph migrations. The authors warn that some work is in review. Their future attention examples also use an M5 Pro and particular head dimensions, unlike the M3 Ultra decoding chart. Neither those figures nor a reduction in one bundled library's size should be presented as a measured benefit of installing the 2.15 candidate on any Mac.
Public MPS streams and tensor stream-recording support give Python callers more scheduling control, but the example still synchronizes before reading its result. The article also lists remaining operation limits and ongoing work to remove compilation graph breaks. AISubHub's practical interpretation is to test a separate environment against the application's actual dtype, shapes, synchronization and output tolerances. We read the complete article and its three charts; we did not install PyTorch, run its snippets, inspect the linked pull requests or reproduce any timing, quality or memory result.
Why it matters
AISubHub analysis: backend upgrades should be evaluated with separate records for complete decoding throughput, operator timing, output correctness and driver-memory peaks. This makes it possible to identify which reported change matches a real application bottleneck without turning a kernel result into a universal model claim.
Limits and availability
All measurements are attributed to the Hugging Face PyTorch contributors and their stated configurations. The article describes a 2.15 release candidate and distinguishes 2.16 work; later stable availability was not independently checked. AISubHub did not install or execute the software, reproduce benchmarks or inspect linked implementations. No universal speed, memory-fit or numerical-quality guarantee follows.
Sources and evidence
Source published: · Source checked:
AI-assisted editorial articles checked against linked primary sources. Vendor claims are attributed; inclusion is not endorsement.