Flash-MoE on MacBook: running 397B parameters on consumer hardware
A 397B-parameter model answering prompts on a 48GB laptop sounds like a demo with a catch. According to its author’s benchmarks, the main catch with Flash-MoE is speed, not functionality.
TL;DR
Flash-MoE streams Qwen3.5-397B-A17B from SSD and reports 4.36 tok/s at 4-bit quantization on a 48GB MacBook Pro M3 Max, with no Python and no framework. The trick is MoE sparse activation: only about 17B of 397B parameters are used per token, so the full model never needs to sit in RAM. Use it for batch jobs and privacy-sensitive work; stick with an API for real-time interaction. See model serving fundamentals for the production deployment context.
The conventional wisdom on running frontier-scale models locally goes like this: you need a server rack, serious VRAM, and probably a cloud bill anyway. Flash-MoE, built by Dan Woods (github.com/danveloper/flash-moe) with Claude Code in about 24 hours by his own account, does something that violates that assumption. It runs a 397 billion parameter model on a consumer laptop with an inference engine of roughly 7,000 lines of Objective-C and C plus 1,200 lines of Metal GPU shaders.
The project drew wide attention in March 2026, including a write-up on Simon Willison’s blog. Understanding why it works tells you more about MoE architecture than a hundred architecture diagrams. And understanding where it breaks tells you whether it belongs in your workflow.
Why MoE models are uniquely suited to consumer hardware
Qwen3.5-397B-A17B uses about 17 billion parameters per token, not 397 billion. The 397B figure describes total capacity, not per-token compute. This distinction is everything. A standard dense 70B model activates all 70B parameters for every token; Qwen3.5’s 397B model activates far fewer.
The architecture achieves this through expert routing. Each MoE layer holds 512 expert modules. A learned router picks 10 of them for each token, plus one shared expert that always runs; Flash-MoE cuts the routed count to 4. The other 508 routed experts sit idle for that token: present in the model weights, contributing nothing to its computation. Run this across 60 transformer layers and you get a model that is simultaneously enormous (in stored capacity) and lean (in per-token compute).
Flash-MoE exploits this asymmetry directly. Rather than loading the full 209GB model into RAM, it streams only the active experts for each layer from the NVMe SSD. The README reports 17.5 GB/s measured sequential reads on the test machine, and each 4-bit expert weighs roughly 6.75MB. Loading 4 experts takes about 1.5 ms per layer, or roughly 90 ms per token if every expert came from SSD. The non-expert weights take about 5.5GB (about 6GB in total), leaving roughly 42GB for the operating system’s page cache, which reaches about a 71% expert hit rate without any custom caching logic.
For comparison, running a 70B dense model at comparable quality requires fitting the entire weight matrix in memory or tolerating heavy swapping. MoE gives you the quality ceiling of a 397B model with the active-compute footprint of a 17B model. That asymmetry is what makes Flash-MoE structurally feasible, not a clever hack.
MoE routing per token in Flash-MoE (K=4 of 512 routed experts, plus the shared expert;
the model's default is 10 routed + 1 shared):
Token input
│
▼
Router network
│
├─── Expert 7 ◄── activated
├─── Expert 83 ◄── activated
├─── Expert 204 ◄── activated
├─── Expert 391 ◄── activated
│
└─── Expert 1..512 (remaining 508: skipped)
What Flash-MoE actually does differently
Flash-MoE bypasses every assumption baked into Python inference stacks: no PyTorch, no transformers library, no CUDA abstractions. It is roughly 7,000 lines of Objective-C and C plus 1,200 lines of hand-tuned Metal GPU shaders. The entire serving path runs on Apple Silicon without a Python interpreter in sight.
Three design decisions define the engine. First: SSD expert streaming. Rather than building a custom expert cache, the engine trusts the macOS page cache. With about 42GB of the laptop’s 48GB free after the engine loads, the page cache holds frequently accessed experts naturally. The 71% cache hit rate emerges without explicit management.
Second: FMA-optimized dequantization kernels. The 4-bit quantized weights must be dequantized before Metal can compute on them. Woods wrote custom FMA (fused multiply-add) kernels for this step that run 12% faster than the naive formulation, a measurable gain at 4.36 tok/s where every millisecond shows up in throughput.
Third: serial GPU→SSD→GPU pipeline. On Apple Silicon, SSD DMA transfers and GPU compute share the same unified memory bandwidth. Overlapping them causes contention, which Woods found across the 58 experiments logged in the repo. The serial pipeline (compute on GPU, stream from SSD, compute again) respects the hardware’s constraints instead of fighting them.
The model structure itself is unusual: 60 transformer layers split between 45 Gated DeltaNet layers and 15 full attention layers. Gated DeltaNet is a linear-attention layer (a delta-rule variant) whose cost grows linearly with context, while the periodic full-attention layers keep long-range recall. Qwen3.5-397B-A17B supports a 262,144-token context natively.
The real performance profile: tokens/sec, quality, memory pressure
At 4-bit quantization, Flash-MoE reports 4.36 tokens/second on an M3 Max MacBook Pro, with full tool calling. The README calls this the production configuration. That is the number to hold in your head. Everything else is context.
┌─────────────────────────────────────────────────────────┐
│ Flash-MoE reported performance (M3 Max 48GB) │
├──────────────────────┬──────────┬────────┬──────────────┤
│ Mode │ Tok/sec │ Disk │ Tool calling │
├──────────────────────┼──────────┼────────┼──────────────┤
│ 4-bit, FMA kernel │ 4.36 │ 209 GB │ Yes │
│ 4-bit, baseline │ 3.90 │ 209 GB │ Yes │
│ 2-bit │ 5.74 │ 120 GB │ No (JSON) │
│ 2-bit, peak 1 token │ 7.05 │ 120 GB │ No (JSON) │
└──────────────────────┴──────────┴────────┴──────────────┘
The 2-bit mode is where it gets interesting. A 30% speed gain sounds compelling (5.74 tok/s versus 4.36 tok/s) until you discover it breaks tool calling. At 2 bits the model writes \name\ instead of "name" in JSON output, which fails tool parsers. For anything touching agentic workflows, 2-bit is not usable. For pure text generation tasks (summarization, analysis, drafting) it may be worth evaluating.
Expert count is the other tuning knob. Flash-MoE routes each token to 4 experts plus the shared expert instead of the model’s default 10 plus shared. Woods reports that quality dropped off most sharply at 3 experts, which is why 4 is the floor. Going from 10 to 4 still changes the model’s behavior, and the project does not publish an independent quality benchmark, so evaluate on your own tasks before relying on it.
Memory pressure under load is lighter than you’d expect. The non-expert weights occupy about 5.5GB of RAM. The remaining memory serves as page cache for expert modules, with the OS warming the cache as requests come in. The first tokens are slower because their experts come from a cold SSD. After a few prompts, common experts stay cached and throughput settles near the reported figure.
When local makes sense vs. API
The decision is not “is 4.36 tok/s fast enough?” It is “what does your workload actually need?” These are different questions.
Local inference wins in two clear scenarios. The first is privacy. Some data genuinely cannot leave the machine: legal documents, medical notes, internal codebases. At 4.36 tok/s, generating a 2,000-token report takes about 8 minutes, and reading a long document in (prefill) adds time the README does not benchmark. For an overnight batch job on sensitive data, that is workable.
The second is batch work at modest volume. Cloud API pricing for Qwen3.5-397B runs on the order of dollars per million tokens (OpenRouter lists providers at varying rates). Once you own the hardware, the marginal cost per local token is small. But the ceiling is low: one laptop at 4.36 tok/s produces at most about 375,000 output tokens a day. If your volume is in that range and the work can wait, local is cheap. Beyond it, you need more machines or an API.
Short, latency-sensitive calls are a weaker case than they look. API round trips add network latency, but local prefill on a 397B model streamed from SSD is not fast either, so measure before assuming a laptop beats an API on short classification or extraction prompts.
This sounds great until you hit the interactive use case. A human reading at normal pace absorbs about 4-5 words per second, which maps roughly to 5-6 tokens per second. At 4.36 tok/s, Flash-MoE runs just below comfortable reading speed. For a conversation where you are watching tokens stream, the experience feels slightly slow. Hosted APIs running the same model on datacenter GPUs are much faster per stream and feel instantaneous by comparison.
flowchart TD
A[New inference workload] --> B{Data leave device?}
B -->|Cannot| C[Local: Flash-MoE]
B -->|Can| D{Latency requirement?}
D -->|Real-time interactive| E[API]
D -->|Batch / background| F{Volume?}
F -->|Fits one laptop's daily output| G[Local: cost wins]
F -->|High or unpredictable| H[API: simpler ops]
The operational cost of Flash-MoE is also real. It requires Apple Silicon with 48GB unified memory, a 209GB model download, Metal development tooling to build from source, and comfort with C compilation. This is not pip install territory. For teams already running Python inference stacks, the switching cost is non-trivial. For individual researchers or privacy-first workflows, it is a one-time setup. For a broader framework on when self-hosting pays off, see build vs buy for AI in 2026.
Agent workflows sit in a middle ground. Flash-MoE supports tool calling at 4-bit, so the agent deployment patterns you’d use are the same. But agents that call tools repeatedly and wait on responses are sensitive to per-step latency. A multi-step agent loop at 4.36 tok/s per step adds up. For agents running batch research overnight, this is irrelevant. For agents embedded in interactive products, it is a constraint.
The cost management framework for agents applies directly here: model routing based on task type. Use Flash-MoE locally for heavy analytical steps; route to an API for interactive, latency-sensitive steps. A hybrid approach is the honest answer for most workloads.
What to watch next
Flash-MoE started as one person’s quick project, and the experiment log in the repo shows how much was tried and discarded. Overlapping SSD reads with GPU compute hurt on this hardware because both share memory bandwidth, which is why the pipeline is serial. Smarter prefetching, such as predicting which experts the next layer will need, is still open.
Newer Apple chips with more memory bandwidth and faster storage should help. Community forks report higher speeds on M5-generation machines, though with more memory, so those numbers are not directly comparable to the 48GB M3 Max result.
The deeper implication is architectural. Qwen3.5-397B-A17B is not a special case; it reflects Alibaba’s bet that MoE is the right scaling architecture for open weights. If Alibaba, Mistral and others keep releasing frontier-class MoE models, the class of models runnable on consumer hardware via SSD streaming grows. Dense models are stuck at the memory wall. MoE models have a structural escape route.
FAQ
What is Flash-MoE and how does it work?
Flash-MoE is a C, Objective-C and Metal inference engine built by Dan Woods that runs Qwen3.5-397B-A17B on a MacBook Pro with 48GB unified memory. It streams only the active expert modules for each layer from the NVMe SSD, which its README measures at 17.5 GB/s sequential read, instead of loading the 209GB model into RAM. The non-expert weights take about 5.5GB of memory, and most of the rest serves as page cache for experts.
How fast is Flash-MoE on a MacBook Pro M3 Max?
At 4-bit quantization, the configuration its README calls production, Flash-MoE reports 4.36 tokens/second with full tool calling support. A 2-bit mode reaches 5.74 tok/s (about 30% faster) but breaks tool calling by generating malformed JSON. Its best single-token speed, 7.05 tok/s, was also measured in 2-bit mode. Hosted APIs running the same model on datacenter GPUs are much faster per stream.
Why can a 397B parameter model fit in 48GB of RAM?
It does not fit; it streams. Qwen3.5-397B-A17B is a Mixture-of-Experts model that uses only about 17B parameters per token. Flash-MoE keeps the roughly 5.5GB of non-expert weights in memory and reads the few experts each layer needs from SSD on the fly, with the OS page cache holding frequently used experts. The 209GB model lives on disk, not in RAM.
When does running Flash-MoE locally make more sense than using an API?
Local inference wins when you process sensitive data that cannot leave your machine, or for background jobs where a few tokens per second is acceptable. One laptop at 4.36 tok/s produces at most about 375,000 output tokens a day, so it suits modest volumes. An API wins for real-time interactive work where 4.36 tok/s feels sluggish.
Does Flash-MoE work on any Mac, or only specific models?
Flash-MoE targets Apple Silicon only: it needs Metal GPU shaders and fast internal SSD reads. The 48GB unified memory M3 Max is the tested configuration, and it currently runs one model, Qwen3.5-397B-A17B. The technique is also MoE-specific: it does not help dense models such as Llama 3 or Mistral Small, which have no expert routing to exploit.