180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers
by Hugging Face (Tarek Ziadé)
How expert streaming from disk runs a 180B mixture-of-experts model on a 32 GB Mac with Transformers, with benchmarks.
Overview
This Hugging Face blog post by Tarek Ziadé, published November 18, 2026, documents how he got Qwen3.8-Flash-Next, a 180B-parameter mixture-of-experts model with 48 layers of 512 experts (10 active) and 121 GB of FP8 experts plus a 51 GB n-gram table, running on a 32 GB MacBook M5 through Hugging Face Transformers. It opens with Strata, an MoE inference engine that keeps hot experts in VRAM and leaves the rest in RAM or on disk, then compares how llama.cpp, vLLM and Transformers offload experts, with decode benchmarks for Qwen3.6-35B-A3B on a 24 GB A10G that range from 82-88 tok/s for Strata at Q8_0 down to 0.16 tok/s for Transformers with Accelerate. The main section prototypes on OLMoE-1B-7B and builds an experts_offload option for from_pretrained: a shared device slot pool with router IDs remapped to slots so grouped_mm runs unmodified, a bounded LRU expert cache filled by 8 disk I/O threads, router-based prefetching for the next two layers, and 3-bit offline expert quantization. A final optimization pass (early MPS commits, Metal kernels for Gated DeltaNet, Strata's GGUF weights, larger prefill chunks) lifts Transformers from 10.0 to 13.9 tok/s on short decode and from 68 to 228 tok/s on a 4K prefill, about 6% behind Strata on short decode. The code is transformers PR #50717, an open draft the author calls an AI-written proof of concept that is not meant to be merged as is.
At a Glance
- Topic
- Models
- Level
- Advanced
- Format
- Tutorial
- Cost
- Free
- Duration
- One long-form article; reproducing the setup includes a ~5-hour expert conversion step
- Provider
- Hugging Face (Tarek Ziadé)
- Hands-on
- Yes — code/exercises
- Certificate
- None
What You’ll Learn
- ✓Compare where llama.cpp, vLLM and Transformers place MoE experts and what data moves per token
- ✓Read the llama.cpp --n-cpu-moe and vLLM --cpu-offload-params experts flags for offloading experts on limited VRAM
- ✓Load a model with a per-module device_map that sends experts to disk and dense layers to MPS
- ✓Design a bounded LRU expert cache with a shared slot pool and parallel disk reads
- ✓Prefetch experts by running the next layers' routers early on the current hidden states
- ✓Quantize experts offline to 3 bits and dense layers to 8 bits, and weigh the reported +0.2% perplexity cost
- ✓Diagnose why per-layer GPU sync overhead on Apple MPS blocks the last few percent of decode speed
Highlights
- •Real benchmark tables: nine engine configurations on an A10G, plus before-and-after Transformers versus Strata numbers on an M5
- •The memory budget is spelled out: 49 GB of 3-bit experts on disk, 12 GiB cached on device, 25.4 GiB peak on a 6K prompt
- •An honest limitations section: about 14 tok/s is still too slow for the author's coding-agent workflow, and the browser has to close to avoid swapping
- •The experts_offload API is unmerged: transformers PR #50717 was an open draft with no maintainer review as of November 19, 2026, so read it as a design study rather than a supported feature
Who It’s For
Best For
- ✓Engineers running large open-weight MoE models on a laptop or a single consumer GPU
- ✓Inference engineers choosing between llama.cpp, vLLM and Transformers expert offloading
- ✓Contributors interested in Transformers internals and Apple MPS performance
Prerequisites
- •Hands-on experience loading models with Hugging Face Transformers and PyTorch
- •Understanding of mixture-of-experts routing and basic weight quantization
- •To reproduce the main run: an Apple silicon Mac with 32 GB and Transformers installed from the PR branch
FAQ
What is 180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers?
A Hugging Face engineering tutorial, published November 18, 2026, on running a 180B-parameter mixture-of-experts model on a 32 GB MacBook with Transformers by streaming experts from disk. It is for engineers who run large open-weight models locally. Afterwards you will understand how llama.cpp, vLLM and Transformers offload experts and how the experimental experts_offload option is configured.
Is 180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers free?
180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers is free to access.
What level is 180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers for?
180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers is aimed at a advanced audience. Recommended background: Hands-on experience loading models with Hugging Face Transformers and PyTorch, Understanding of mixture-of-experts routing and basic weight quantization, To reproduce the main run: an Apple silicon Mac with 32 GB and Transformers installed from the PR branch.
How long does 180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers take?
Expect roughly One long-form article; reproducing the setup includes a ~5-hour expert conversion step. Most learners work through it at their own pace.
What will I learn from 180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers?
You'll learn: Compare where llama.cpp, vLLM and Transformers place MoE experts and what data moves per token; Read the llama.cpp --n-cpu-moe and vLLM --cpu-offload-params experts flags for offloading experts on limited VRAM; Load a model with a per-module device_map that sends experts to disk and dense layers to MPS; Design a bounded LRU expert cache with a shared slot pool and parallel disk reads; Prefetch experts by running the next layers' routers early on the current hidden states; Quantize experts offline to 3 bits and dense layers to 8 bits, and weigh the reported +0.2% perplexity cost; Diagnose why per-layer GPU sync overhead on Apple MPS blocks the last few percent of decode speed.
Topics
Sources
This page was written from 2 sources, 1 on domains other than huggingface.co.