Howdoesspeculativedecodingworkinllama.cpp,anddoesitspeedupcoding?

What is speculative decoding?

Speculative decoding is a way to get more than one token out of each forward pass of a large model without changing what it outputs. Speculative decoding in llama.cpp works like this: a cheap drafter guesses the next few tokens, the target model scores all of them in a single batch, and it keeps the longest prefix that matches what it would have produced anyway, plus one token of its own at the first mismatch.

It works because of a hardware fact the llama.cpp docs state directly: computing n tokens in a batch, as in prompt processing, is cheaper than computing them one by one. Single-user decoding is limited by memory bandwidth, so checking four positions costs roughly what generating one does.

The technique comes from Leviathan, Kalman and Matias (ICML 2023), who reported a 2x to 3x speedup on T5-XXL “with identical outputs.” That guarantee is the point: a bad draft costs time, never quality. With greedy sampling the output is token-for-token the same; with sampling, llama.cpp’s probabilistic mode uses rejection sampling so the distribution is unchanged.

Why does it help more on code than on prose?

Speculative decoding helps more on code because code is predictable, and the speedup is almost entirely a function of how often the drafter guesses right. Closing brackets, repeated identifiers, boilerplate imports and the unchanged lines of a file you asked the model to edit are all easy for a small model, or even a plain lookup, to predict.

LM Studio’s own benchmark shows the gap clearly. On an RTX 3090 Ti with the llama.cpp engine, the same feature gave very different results by prompt (LM Studio 0.3.10):

• “Quicksort in Python (code only)”, Qwen2.5-32B with a 0.5B draft: 21.84 to 45.15 tok/s, 2.07x.
• “Explain the Pythagorean theorem”, Llama 3.1 8B with a 1B draft: 1.36x.
• “Plan a 1 day trip to DC”, same pair: 1.05x.

The LM Studio docs put it the same way: drafts get rejected more often on open-ended creative text. Edit-style coding work, where the model reprints most of a file, is the best case of all, and it is what the draftless n-gram modes are built for.

How do you enable speculative decoding in llama-server?

You enable speculative decoding in llama-server with --spec-type, plus a draft model (-md) if the type needs one. The flags changed in 2026: the server README, fetched Oct 28, 2026, marks --draft-max and --draft-min as removed in favor of --spec-draft-n-max and --spec-draft-n-min. Many guides still show the old names. This page covers only the speculation flags; for everything else, see the llama.cpp server guide.

Classic draft model, for example a coder pair:

llama-server -m Qwen2.5-Coder-32B-Instruct-Q4_K_M.gguf \
  -md Qwen2.5-Coder-0.5B-Instruct-Q8_0.gguf \
  --spec-type draft-simple --spec-draft-n-max 8 -ngl 99 -ngld 99

MTP, using the prediction heads shipped inside the main model’s GGUF (example from PR #22673):

llama-server -hf ggml-org/Qwen3.6-27B-MTP-GGUF \
  --spec-type draft-mtp --spec-draft-n-max 2

Draftless n-gram lookup, aimed at refactors and long repeats:

llama-server -m model.gguf --spec-type ngram-mod \
  --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64

These are the main types the docs list at fetch time:

Types can be combined as a comma-separated list, and when a draft model is mixed with a draftless method, the draftless one takes precedence. --spec-default turns on ngram-mod with default settings. The default draft length is 3 tokens.

In LM Studio, which runs llama.cpp and MLX engines, you pick a draft model in the Speculative Decoding section of the sidebar (Power User mode or higher), or pass draft_model in an API request. If you’re choosing between the two tools generally, see LM Studio vs Ollama.

Which draft model should you pair with your main model?

Pair your main model with a much smaller model from the same family that uses the same tokenizer. llama.cpp enforces this. The compatibility check in common/speculative.cpp requires the same vocab type and the same BOS/EOS tokens, rejects vocab sizes that differ by more than 128 tokens (SPEC_VOCAB_MAX_SIZE_DIFFERENCE), and compares token text across the shared range. A failed check logs “the target and draft vocabs are not compatible.”

Pairs that sources name as working:

LM Studio’s docs give a rough size ceiling: up to a 1B draft for a 7B target, 3B for 14B, 7B for 32B, and nothing for a 3B target. Below roughly that ratio the draft is too expensive relative to the work it saves. If you run the Qwen coder models locally, the 0.5B and 1.5B coder siblings are the obvious first try.

For models that ship MTP heads, skip the separate draft model. It saves a download and a second set of weights, and the heads were trained on the target itself.

What do developers report in practice?

Developers report solid gains on code with matched pairs or MTP, and clear losses when acceptance drops. Every figure below is a community or vendor report, not our measurement, and each states its hardware.

• The original server PR. When Georgi Gerganov added speculation to llama-server, his code-generation run with Qwen2.5-Coder-32B Q4_K_M and a 0.5B draft went from 29.18 to 46.65 tok/s on an RTX 3090, and from 8.73 to 14.74 tok/s on a Tesla P40 (PR #10455). He warned that repeated runs on the same prompt look inflated because the draft context is reused, so only first-run numbers count.
• MTP on consumer GPUs. In the MTP PR thread, one user on an RTX 5090 with Qwen3.6-27B q4_0 reported 63.72 to 105.47 tok/s at about 80% acceptance. Another, on an RTX 3090 plus 3060 running Q6_K, went from 22.97 to 42.45 tok/s at 76% acceptance, but prompt processing fell from 1315 to 665 tok/s and VRAM use rose by about 2.5 GiB (PR #22673).
• Mac Studio, coder pair. On r/LocalLLaMA, an M2 Ultra user found Qwen2.5-32B-Coder nearly doubled in speed with the 1.5B coder as draft, from about 80 ms to 44 ms per token, and found 1.5B faster than 0.5B. The same user saw QwQ-32B get slower with either draft (r/LocalLLaMA).
• When acceptance collapses. In a Gemma 3 thread, one commenter whose speed dropped had an acceptance rate of about 3%, and a llama.cpp maintainer replied that such a low rate explains the slowdown. Others in the thread with acceptance above 0.80 saw large gains (r/LocalLLaMA).
• A careful negative result. A developer benchmarking Qwen3.6-35B-A3B UD-Q4_K_XL on one RTX 3090 found a 0.8B external draft cut decode from 115.7 to about 30 tok/s, and later found that draft was never vocabulary-compatible. On the same setup, MTP at n-max 2 gave +22.7% and DFlash +26.3%, while ngram-cache lost 19% (thc1006 on GitHub, discussed on HN).
• Line endings matter for n-gram modes. In the ngram-mod PR, a Devstral user’s acceptance went from 0.11 to 0.83 (2.32 to 29.79 tok/s) after switching a file from CRLF to LF, because models emit LF (PR #19164).

For how to measure decode speed consistently before and after, see tokens per second for local LLMs.

When does speculative decoding slow things down?

Speculative decoding slows things down when the drafter is wrong too often or costs too much relative to the target. Watch for these cases:

1. Low acceptance. Below roughly 40%, the docs suggest a shorter draft, a higher --spec-draft-p-min, or a different draft model. At 3 to 5% it is a net loss.
2. A fast target. A small dense model or an MoE with only a few billion active parameters already decodes quickly, so a separate draft model adds proportionally more cost. The 35B-A3B result above is the example.
3. Prose and creative writing. Acceptance drops, as LM Studio’s 1.05x trip-planning run shows.
4. Prompt-heavy work. MTP slowed prefill by about half in the report above. For agents that send long contexts and get short replies, prefill can dominate.
5. Parallel slots. At merge, the MTP PR listed parallel decoding as supported but not yet optimized.
6. Tight VRAM. A draft model and its KV cache need memory that might otherwise hold more context or more layers.

The fix in every case is the same: run your own prompts with and without --spec-type, read the draft acceptance rate line llama-server prints, and keep speculation only where it wins. The repo’s SPEED-Bench client under tools/server/bench/speed-bench compares a baseline run against a speculative run.

FAQ