01 / 03M5 Max · 128 GB
NVIDIA / Nemotron
3.5 Lightning
30B-A3B
188–206tokens / second
Short answer with thinking · 4-bit
Server measurement before MTP drafts; the included MTP head is now enabled by default.
Read kernel recipeAn inference engine for Apple Silicon
Draft wide.
Commit exact.
One stream in. Candidate tokens spread across parallel lanes. The target model checks them before accepted tokens are committed. TensorFold serves the result from your Mac.
MLX + Metal on Apple SiliconOpenAI-compatible APIOpen source on GitHub
01 / SERIAL REFERENCE02 / PARALLEL DRAFTS03 / EXACT ACCEPTANCE04 / LOCAL API
01 / Measured on real Macs
The numbers.
With the conditions.
Decode throughput measured through TensorFold’s server. Different models, machines, and prompts are shown separately; these are observations, not a cross-model race.
3.5 Lightning
02 / 03M5 Max · 128 GB
Qwen / Dense + DFlash2
Qwen3.8
27B
120–124tokens / second
Short answer with thinking · 4-bit
27 tok/s without drafts in this workload. DFlash2 is a separate optional download.
Read kernel recipe03 / 03M3 Ultra · 256 GB
Qwen / Flash Next
Qwen3.8
Flash Next
88–92tokens / second
Short answer with thinking · 4-bit
79 tok/s without drafts in this workload. The 113 GB checkpoint includes its MTP head.
Read kernel recipeResults vary with output content, context length, MLX version, and thermals. Full tables and comparisons live in the repository’s speed section.
02 / The fold
One stream.
Many lanes.
One answer.
Most inference engines decode one token, then repeat. TensorFold proposes several next tokens, checks their positions together in a target-model pass, and accepts only the prefix that matches the target model’s own samples.
Its model-specific kernels keep each verify row’s arithmetic consistent with a single-row step. Drafting changes the time it takes to arrive at an answer, not the answer from the same cache state.
Read the full methodTHE DECODE ROUND001 → 004
01CONTEXT STREAM
At + 1candidate
Bt + 2candidate
Ct + 3candidate
Dt + 4candidate
✓EXACT COMMITTED PREFIX
Parallel candidate workSingle committed stream
03 / On your Mac
From zero
to local API.
Apple Silicon, Python 3.11+, and enough unified memory for your chosen model. TensorFold downloads the checkpoint from Hugging Face and serves an OpenAI-compatible endpoint.
01
Install the CLI
Install directly from the public repository.
02
Choose a checkpoint
The default Nemotron checkpoint lives in the Vontra Hugging Face namespace and includes the MTP head.
03
Serve, then connect
Point any OpenAI-compatible client at http://127.0.0.1:8080/v1.
# Install
pip install git+https://github.com/ashhart/TensorFold.git
# Download and serve Nemotron 3.5 Lightning
tensorfold serve Vontra/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit --context 65536
# OpenAI-compatible endpoint
http://127.0.0.1:8080/v1 Model downloads automatically on first serve32 GB+ Mac for this checkpoint
OTHER CHECKPOINTS
Qwen3.8-27B 16.1 GB + 3.8 GB drafterQwen3.8 Flash Next 113 GB · 192 GB+ Mac
Compare models ↗04 / The engineering
Built by
measuring the gap.
Three checkpoints. Three kernel paths. One rule: every accepted draft must equal what serial decoding would have produced from the same state.
Find the floor.
Serve one token at a time. Measure weight-read bandwidth, GPU time, kernel count, and where the host waits.
Cut the overhead.
Write model-specific Metal kernels for the real bottleneck: projections, routing, experts, attention, or state updates.
Use the lanes.
Draft ahead, verify several rows, and commit matching tokens. Check row invariance on the Mac and MLX version in use.
THE RECIPE BOOK IS OPENSee each model’s kernel notes
Made to run locally
Put your Mac
to work.
Choose a checkpoint, set the context and sampling you need, and serve it to the tools you already use.
Get TensorFold on GitHub