HowdoyourunabigMoEmodelonasmallGPUwithexpertoffloading?

MoE offloading in llama.cpp is the reason a 16 GB or 24 GB card can run a 35B, 120B or even 177B mixture-of-experts model at usable speed. The short version: put everything on the GPU (-ngl all), then add --n-cpu-moe N and lower N until VRAM is nearly full. The rest of this page explains why that works, what the other flags do, how LM Studio, Ollama and vLLM compare, and what speeds developers actually report.

Why can MoE models run on a small GPU at all?

MoE models run well on a small GPU because each token only touches a small slice of the weights, and the biggest slice, the experts, is cheap enough to compute on the CPU. In a mixture-of-experts layer, a router picks a few experts per token and ignores the rest. Qwen3.6-35B-A3B has 256 experts per layer across 40 layers and activates 8 routed experts plus 1 shared expert per token, so only 3B of its 35B parameters do work on any given token. gpt-oss-120b activates 5.1B of 117B.

That split maps neatly onto a desktop PC:

• On the GPU: attention weights, the KV cache, embeddings, the router and any shared expert. Every token uses these, and attention over a long context is the part that benefits most from GPU compute.
• In system RAM: the routed expert weights. They are most of the file size, but each token reads only the few experts it was routed to.
• Across PCIe: per layer, only the token’s activations travel between the two, not the weights.

One Hacker News commenter put numbers on how lopsided this is: by their count, the attention weights of DeepSeek R1 (685B) come to around 16 GB and those of Kimi K2 (1T parameters) to about 12 GB (HN, Aug 2025). That is a community calculation, but it explains why trillion-parameter MoE models are even discussed on single-GPU rigs.

A dense model has no such split. Every weight is used for every token, so moving layers to the CPU slows the whole model down. That is the core difference from pure CPU inference, which is covered in run LLM on CPU.

Which llama.cpp flags offload experts to CPU?

llama.cpp has three flags for expert offloading, all documented in the llama.cpp server README, and --n-cpu-moe is the one most people should use. Under the hood, --cpu-moe and --n-cpu-moe are shortcuts for tensor overrides: the source matches expert tensors with the regex \.ffn_(up|down|gate|gate_up)_(ch|)exps and pins them to the CPU buffer.

The tuning loop is short. Load with all layers on the GPU and a high N, check free VRAM, lower N by a few layers, repeat until the next step would run out of memory. Leave headroom for the KV cache at the context length you actually use, which grows with context (see local LLM hardware requirements for KV cache sizing).

llama-server -m Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
  -ngl all --n-cpu-moe 20 -fa on -c 32768 --jinja

The -ot route is for cases where whole layers are too coarse. In the llama.cpp gpt-oss guide thread, one user on a 12 GB RTX 3060 offloaded only the ffn_up_exps tensors of layers 20 and up with -ot "\.([2-9][0-9])\.ffn_up_exps.=CPU" and got about 67 tok/s on gpt-oss-20b at 16K context, slightly ahead of -ncmoe 2 at about 64 tok/s. Both setups are explained in more depth in the llama.cpp server guide.

How do you do it in LM Studio, Ollama and vLLM?

LM Studio supports expert offloading as a load setting, Ollama does not officially support it, and vLLM offers a general parameter offload that can target expert weights.

LM Studio. Version 0.3.23 (August 12, 2025) added “Force Model Expert Weights onto CPU” in the advanced load settings, and the release notes say it uses the same technology as llama.cpp’s --n-cpu-moe. LM Studio recommends leaving it off when the whole model fits in VRAM. It can be saved per model.

Ollama. There is no documented switch. The feature request, ollama/ollama#11772, has been open since August 7, 2025, with no maintainer reply visible on the issue. Commenters there report the gap: one user with a Ryzen 7950X, 64 GB RAM and an RTX 3090 measured gpt-oss-120b at about 8.5 tok/s in Ollama against 29.4 tok/s in llama.cpp with --n-cpu-moe 24. Another reported that setting LLAMA_ARG_CPU_MOE=1 with num_gpu 99 on Ollama 0.30.0 lifted Qwen3.6-35B-A3B on an RTX 5080 from 14.6 to 36.5 tok/s, but also needed GGML_CUDA_NO_PINNED=1 to avoid an out-of-memory error. That is an undocumented workaround from one report, so expect it to break between releases. If you need this today, running llama.cpp directly is the reliable path.

vLLM. vLLM has no MoE-specific flag. Its OffloadConfig offers --cpu-offload-gb (GiB per GPU to offload) and --cpu-offload-params, a filter on parameter names where experts matches tensors like mlp.experts.w2_weight. A prefetch backend (--offload-backend prefetch, --offload-prefetch-step) loads offloaded layers ahead of use. This is weight offloading for a serving engine rather than llama.cpp-style CPU compute of experts, so it suits a multi-user server that is a few GB short of fitting, not a single developer squeezing a 120B model onto one consumer card.

How much system RAM and bandwidth do you need?

You need enough RAM to hold whatever does not fit in VRAM plus your OS and apps, and the speed of that RAM, not the CPU core count, sets the generation ceiling. A practical rule: model file size minus usable VRAM, plus 8 GB or more of headroom.

Bandwidth matters because each generated token has to read every active expert held in RAM. A rough upper bound is RAM bandwidth divided by the bytes of active expert weights per token. For example, dual-channel DDR4-3200 has a theoretical peak of 51.2 GB/s (2 channels x 8 bytes x 3,200 MT/s). If about 1.5 GB of a model’s active weights per token live in RAM, the CPU side alone caps out near 34 tok/s before any other overhead. Real numbers land well below such ceilings.

Community data backs the “bandwidth, not cores” point:

1. A user in the gpt-oss guide thread with an RX 7900 XT, Ryzen 9 5900X and dual-channel DDR4-3200 named RAM and PCIe speed as the likely bottleneck after their sweep.
2. In an InsiderLLM test, a thread sweep on an RTX 3090 with a 6-core i7-8086K peaked at 6 threads (about 16.1 tok/s) and got slightly worse at 12 (about 15.6).
3. The same test ran an 82 GB quant of Qwen3.8-Flash-Next (about 177B total, 6B active) on a machine with 62 GB of RAM by memory-mapping the file from NVMe, at about 22.6 tok/s on the 3090. So it can work when the model is larger than RAM, but it leans on a fast SSD.

What speeds do developers report?

Developers report usable speeds on 12 to 24 GB cards, roughly 10 to 85 tok/s with experts offloaded, depending on the model, with a large gap between generation and prompt processing. All figures below are community reports, not our tests, and come from different builds, quants and settings.

Two patterns stand out. First, speed climbs steadily as N drops, so the last few layers you fit on the GPU are worth the effort: on the 7900 XT, going from all experts on CPU to -ncmoe 8 nearly doubled decode speed. Second, prompt processing drops much harder than decode. In the Giles Thomas sweep, prompt speed fell from 3,360 tok/s with nothing offloaded to 1,008 tok/s at -ncmoe 12, while decode only fell from 140 to 85. A Hacker News user running Qwen3.5-122B-A10B with --n-cpu-moe 32 posted 18.5 tok/s prompt processing and 13.9 tok/s generation in llama-batched-bench and called prompt processing “absolutely punishing” (HN, Apr 2026). For coding agents that send large contexts, that prefill cost is often what you feel most.

Another HN commenter said offloading experts with --n-cpu-moe while filling all available VRAM can give about a 50% boost on models like gpt-oss-120b (HN, Oct 2025). For Qwen coding models specifically, see the Qwen Coder local guide.

When is a dense model the better choice?

A dense model is the better choice when it fits entirely in VRAM at the context you need, or when your workload is dominated by long prompts. Offloading trades speed for capacity, and the trade only pays when the bigger model is meaningfully better for your task.

Pick a dense model, or a smaller MoE that fits fully on the GPU, when:

• It fits. A 27B dense model that sits fully in 24 GB will usually beat a partly offloaded MoE on prompt processing, which matters for agents and code review over large files.
• Your RAM is slow or small. Single-channel memory or 16 GB of system RAM leaves little room for experts.
• You serve several users. Offloaded experts make every concurrent request wait on the same RAM bandwidth.

Pick an offloaded MoE when the larger model’s quality is the point, when you mainly generate rather than ingest long prompts, and when you have at least 32 GB, ideally 64 GB, of fast dual-channel RAM.

FAQ