refusal rate
Subscribe

KV cache quantization stores attention keys and values at 8 or 4 bits for longer context: memory math, llama.cpp, Ollama, LM Studio, vLLM flags, quality data.

[ fig. 01 — explainer ]48×5
[ fig. 01 — explainer ]16×5

Short answer

KV cache quantization stores the attention keys and values a model keeps for every token at lower precision, usually q8_0 or q4_0 instead of f16. q8_0 halves the cache and, in published llama.cpp and Ollama measurements, costs almost nothing in quality. q4_0 cuts it to about 28% but is model-dependent and riskier at long context. Turn on q8_0 when context, not weights, is what runs you out of VRAM.

Default cache type
f16 (llama.cpp, Ollama)
q8_0 / q4_0 size vs f16
about 53% / 28%
Devstral Small 2, 128k, f16
21.5 GB of KV cache
vLLM option
--kv-cache-dtype fp8

Key takeaways

  1. KV cache bytes = 2 x full-attention layers x KV heads x head dim x tokens x bytes per value. From the published configs, Devstral Small 2 24B needs 21.5 GB at 128k with an f16 cache; Qwen3.8 27B needs 8.6 GB.
  2. q8_0 stores 1.0625 bytes per value (34-byte blocks of 32) and q4_0 stores 0.5625, so they cut the f16 cache to about 53% and 28%.
  3. Published measurements put q8_0 close to lossless on Qwen and Llama-class models; Gemma 4 and q4_0 on both K and V show real divergence.
  4. In llama.cpp a quantized V cache requires flash attention, and Ollama only applies OLLAMA_KV_CACHE_TYPE when flash attention is on.
  5. The K cache is more sensitive than the V cache, so a q8_0 K with a q4_0 V is the usual compromise when q8_0 on both is not enough.

KV cache quantization is the setting that decides how many tokens of context fit next to your model, and for long-context coding it is usually the cheapest memory you can buy back. A Hacker News commenter running Qwen3.8 27B at Q4_K_M for coding said they settled on a q8_0 cache because it was “the only way” to fit the model plus 100k tokens into 24 GB, and still wondered what it cost them in quality. This page answers both halves: how much memory the cache takes, computed from real config.json files, and what the published measurements say about the quality trade-off. Sources were fetched on 2026-10-28.

What is the KV cache and why does it eat VRAM?

The KV cache is the store of attention keys and values a transformer keeps for every token already in context, so it does not recompute them for each new token. It grows linearly with context length, and unlike the weights it is not shrunk by your GGUF quant choice. The size for one sequence:

KV bytes = 2 x full-attention layers x KV heads x head dim x tokens x bytes per value

The 2 is one key and one value. At the default f16, each value is 2 bytes. Quantized types store blocks of 32 values with one f16 scale: ggml’s block_q8_0 is 34 bytes per 32 values (1.0625 bytes each) and block_q4_0 is 18 bytes per 32 (0.5625 bytes each). So q8_0 is about 53% of f16 and q4_0 about 28%, slightly worse than the “half” and “quarter” the Ollama FAQ rounds to.

Two things multiply the cache beyond this formula:

  • Parallel slots. Each concurrent sequence keeps its own cache. Ollama’s FAQ says memory scales with parallel requests times context length.
  • Architecture. Hybrid models only cache on their full-attention layers, and sliding-window layers cap their cache at the window. Read the model’s config before assuming every layer counts.

For weight quantization, which is a different decision, see GGUF quantization explained.

How much memory does the KV cache take at long context?

At 128k tokens the cache alone can be as big as a 24 GB card. The two models below are both current coding models, picked because their architectures differ. Numbers come from each model’s published config.json:

ModelLayers that cacheKV headsHead dimf16 per token
Devstral Small 2 24B40 of 40 (full attention)8128160 KiB
Qwen3.8 27B16 of 64 (full_attention_interval 4)425664 KiB

Multiplying out, in GB (10^9 bytes), single sequence:

Model, cache type8k32k128k
Devstral Small 2, f161.345.3721.47
Devstral Small 2, q8_00.712.8511.41
Devstral Small 2, q4_00.381.516.04
Qwen3.8 27B, f160.542.158.59
Qwen3.8 27B, q8_00.291.144.56
Qwen3.8 27B, q4_00.150.602.42
[ fig. 01 — KV cache size by context and cache type, computed from each model's Hugging Face config.json (excludes weights and runtime buffers) ]
Bar chart: at 128k tokens the KV cache for Devstral Small 2 24B is 21.5 GB at f16, 11.4 GB at q8_0 and 6.0 GB at q4_0, while Qwen3.8 27B needs 8.6, 4.6 and 2.4 GB, so quantizing the cache matters most for full-attention models at long context

The takeaway: at 8k the cache is small enough to ignore, and quantizing it saves well under a gigabyte. At 128k on a full-attention model, q8_0 frees about 10 GB, which is the difference between fitting a 24B model on one card and not. Qwen3.8’s hybrid layout already does most of that work, which is why it reaches 128k on 24 GB with room to spare. Devstral’s card says it has a 256k window; at f16 that window alone would be 43 GB. The hardware requirements guide adds the weights to these figures.

How do you enable KV cache quantization in llama.cpp, Ollama, LM Studio and vLLM?

Every major runtime exposes it as a load-time setting, and changing it means reloading the model.

RuntimeSettingTypesNotes
llama.cpp-ctk / --cache-type-k, -ctv / --cache-type-vf32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1K and V set separately; quantized V needs flash attention
OllamaOLLAMA_KV_CACHE_TYPE env var on the serverf16, q8_0, q4_0Global for all models; needs flash attention
LM StudioK and V cache quantization in the llama.cpp engine’s load settings; llamaKCacheQuantizationType and llamaVCacheQuantizationType in the SDKllama.cpp typesAdded in 0.3.7 (llama.cpp runtime 1.9.0+); V setting needs flash attention
vLLM--kv-cache-dtypeauto, fp8, fp8_e4m3, fp8_e5m2FP8 only in the docs; calibrated scales recommended

llama.cpp, from the server README:

bash
llama-server -m model.gguf -c 65536 -fa on -ctk q8_0 -ctv q8_0

Ollama, per its FAQ, set on the server process rather than per request:

bash
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve

vLLM, from its quantized KV cache docs:

bash
vllm serve <model> --kv-cache-dtype fp8

The vLLM docs warn that some layer types, such as sliding-window attention, are more sensitive, and offer --kv-cache-dtype-skip-layers sliding_window to keep those at native precision. Without calibration all scales are set to 1.0; the docs recommend calibrating with llm-compressor “for maximum accuracy”. In LM Studio, an SDK bug where the K and V cache keys were ignored was fixed in lmstudio-python 1.4.1, so update before you rely on scripted loads. Where each tool sets the context length itself is covered in how to set context length, and the full server flag set in the llama.cpp server guide.

Does a q8_0 or q4_0 KV cache hurt output quality?

q8_0 costs very little on most models that have been measured; q4_0 costs more, and how much depends on the model and on whether you also quantize K. The published evidence:

  • llama.cpp, 2024. The PR #7412 table measured perplexity and KL divergence against f16 weights and cache. q8_0 on both K and V moved perplexity from 6.2322 to 6.2344 (KLD 0.0010). q4_0 on V only reached 6.2518 (KLD 0.0050); q4_0 on both reached 6.4384 (KLD 0.0365). The author concluded there “seems to be no significant quality loss from using q8_0”, and that K is more sensitive than V.
  • Ollama’s implementer, 2024. Sam McLeod, who added the feature to Ollama, measured Qwen 2.5 Coder 7B at Q6_K: perplexity 8.3891 with f16 cache, 8.3934 with q8_0 (blog). He described q4_0 as “some noticeable quality reduction” in earlier tests.
  • localbench, April 2026. A KL-divergence study over about 250,000 tokens (coding, chat, tool calling, science, non-Latin scripts, long documents) found both Qwen 3.6 models under 0.04 at q8_0 and 0.087 to 0.117 at q4_0 (localbench). Gemma 4 was far more sensitive: 0.108 (31B) and 0.377 (26B A4B MoE) at q8_0. Qwen’s q4_0 damage concentrated in long documents (KL 0.581) and tool calling.
  • llama.cpp rotation, April 2026. PR #21038 applies a Hadamard rotation to Q, K and V before caching to reduce outliers. Its numbers: gpt-oss-20b on AIME25 went from 2.0% to 21.7% with a q4_0 cache and from 31.7% to 37.1% with q8_0, against 37.9% at f16. Results from llama.cpp builds before that change, including the 2024 figures above, are not comparable for q4_0.
  • vLLM FP8. In a vLLM issue, one contributor reported Qwen2.5-7B-Instruct dropping from 73% to 51% on 100 math problems with uncalibrated FP8, while a per-token-head INT8 prototype matched the baseline. A worse 3% result used a flag another contributor called obsolete, so read it as one user’s test of one older model without calibration data.

So “q8_0 is lossless” is true for the models that have been measured, not a law. Gemma-class models and agent workloads that lean on tool calls deserve a check before you commit.

Why does KV cache quantization need flash attention?

It needs flash attention because the quantized V path only exists in the flash-attention kernels. Without flash attention, llama.cpp stores V transposed for the standard matrix-multiply path (attn_v_trans is set to !flash_attn in llama-model.cpp), and 32-value quantized blocks do not fit that layout. The source makes it explicit: if V is quantized and flash attention is set to auto, llama.cpp logs that it is “enabling flash_attn since it is required for quantized V cache”; if you disabled it, loading fails. A quantized K cache alone does not have that requirement.

Ollama goes further and ties the whole setting to flash attention: the FAQ says the cache “can be quantized to significantly reduce memory usage when Flash Attention is enabled”, and McLeod’s write-up says that without it Ollama falls back to f16. LM Studio’s docs carry the same note for the V cache setting. The practical rule is to turn flash attention on whenever you quantize the cache, then confirm from the startup log that it stayed on.

What do developers report in practice?

Practitioners mostly run q8_0 on both K and V for coding and avoid q4_0 unless memory forces it. These are individual community reports, not tests run by this site.

  • q8_0 as the default. One Hacker News commenter put it bluntly: q8 KV is “good enough”, don’t bother with q4, and posted a llama-server line with --cache-type-k q8_0 --cache-type-v q8_0 -c 65536.
  • 128k on one 24 GB card. An October 2026 commenter runs Qwen3.8 27B at UD-Q4_K_XL with a 131k context, q8_0 for K and V, and a speculative draft model on an RX 7900 XTX, and says it “just barely fits” in 24 GB. Another got Ollama to a 180k context in 23 GB on a 3090 with flash attention on and OLLAMA_KV_CACHE_TYPE=q8_0.
  • Speed at depth. A user running a large MoE at 64k tokens of depth reported prompt processing at 57.4 tokens/sec with a q8_0 cache, against 13.1 at the same depth in their other run. One report on one machine, but it suggests the smaller cache can help speed at depth, not only capacity.
  • Engine differences. A commenter argued that llama.cpp’s q8_0 should work better than vLLM’s FP8, pointing to the INT8 issue above.
  • Long agent sessions. In the r/LocalLLaMA thread for the localbench results, readers took the Qwen 27B numbers as good news for q8_0 coding but flagged agentic sessions past about 30k tokens as the case the benchmark does not settle.

A June 2026 test of Qwen2.5-Coder-7B on an RTX 5060 Ti found the opposite of “free memory” at small contexts: at 8k, q4_0 saved only 112 MB more than q8_0, and its output had 8.3% text similarity to the f16 baseline versus 81.6% for q8_0 (InventiveHQ). Similarity is a blunt measure and the llama.cpp build is not stated, but the arithmetic point holds: at short context there is little to save.

Should you turn it on?

Turn on q8_0 when context is what pushes you over your VRAM, and leave f16 alone when it is not.

  1. If the model plus your working context fits at f16, keep f16. At 8k the savings are a few hundred megabytes.
  2. If long context does not fit, switch K and V to q8_0 with flash attention on. This is the setting with the most measurement behind it.
  3. If q8_0 is still not enough, try q8_0 K with q4_0 V before q4_0 on both, since K is the more sensitive half.
  4. Re-run a few of your own coding prompts at temperature 0 against the f16 output, especially long files and tool-call sequences, before you keep a 4-bit cache.

FAQ

Does KV cache quantization reduce model size?

No. It only shrinks the per-token cache that grows with context. The weights file and its quant level are unchanged.

Is q8_0 KV cache better than a smaller weight quant?

Usually, if your problem is context. q8_0 cache measured close to f16 on Qwen and Llama-class models, while dropping a weight quant level costs quality on every token. The localbench study notes the two losses stack.

Can I set different cache types for K and V in Ollama?

No. OLLAMA_KV_CACHE_TYPE sets one global type for both. llama.cpp (-ctk, -ctv) and LM Studio set them separately.

Does KV cache quantization slow down generation?

It depends on the backend. The InventiveHQ test measured 81.8 tokens/sec at f16 and 76.4 at q8_0 on an RTX 5060 Ti, but a 42% drop with q4_0 on an M3 Max. Check speed on your own hardware.

What does vLLM use instead of q8_0?

FP8. vLLM’s documented options are fp8, fp8_e4m3 and fp8_e5m2, with calibrated scales recommended over the default scale of 1.0.

Sources (10)
  1. llama.cpp server README: --cache-type-k, --cache-type-v, --flash-attn (ggml-org/llama.cpp)
  2. FAQ: How can I set the quantization type for the K/V cache?, Ollama docs
  3. Quantized KV Cache, vLLM docs
  4. LLMLoadModelConfig (llamaKCacheQuantizationType, llamaVCacheQuantizationType), LM Studio docs
  5. PR #21038: rotate activations for better quantization, ggml-org/llama.cpp (Apr 2026)
  6. PR #7412: CUDA quantized KV cache demo with PPL and KLD table, ggml-org/llama.cpp (2024)
  7. Bringing K/V context quantisation to Ollama, Sam McLeod (Dec 2024)
  8. Gemma 4 and Qwen 3.6 with q8_0 and q4_0 KV cache: KL divergence results, localbench (Apr 2026)
  9. Devstral-Small-2-24B-Instruct-2512 config.json, Hugging Face (Mistral AI)
  10. Qwen3.8-27B config.json, Hugging Face (Qwen)

Keep reading