WhatdoGGUFquantizationnameslikeQ4_K_Mmean?

What is GGUF?

GGUF is a single-file binary format for running a model with GGML-based engines such as llama.cpp. The GGUF spec calls it “a file format for storing models for inference with GGML and executors based on GGML”, built for fast loading, with mmap support and no external files. It was developed by Georgi Gerganov, the author of llama.cpp, according to the Hugging Face docs.

The point that matters for naming: unlike tensor-only formats such as safetensors, GGUF stores both the tensors and a standardized set of metadata. Tokenizer, architecture and the quantization type of each tensor travel in the one file. That is why you download a single Model-Q4_K_M.gguf and it just loads.

GGUF is a container, not a compression method. “Quantization” is what happens to the tensors inside it, and the suffix in the file name tells you which scheme was used.

What does each part of Q4_K_M mean?

Q4_K_M is three fields: Q4 is the base bit width, K is the block format, and M is the tensor mix. The names come from llama.cpp’s k-quants work, introduced in pull request #1684.

Two things make the headline number misleading. First, scales cost bits. The Hugging Face quantization table lists Q4_K at 4.5 bits per weight, Q5_K at 5.5, Q6_K at 6.5625 and Q3_K at 3.4375, all above their nominal width. Second, the M and L presets deliberately spend extra bits on a few tensors. In the original k-quants design, Q4_K_M uses Q6_K for half of the attention value (attention.wv) and feed-forward down (feed_forward.w2) tensors and Q4_K for the rest. Q3_K_L raises those same tensors to Q5_K instead.

The exact mix is a property of llama.cpp’s quantizer, so it can differ between model architectures and versions. Treat the PR as the design, and the file’s tensor viewer on Hugging Face as the ground truth for a specific download.

The real average shows up in file sizes. For Llama 3.1 8B, the llama-quantize README lists these:

Hardware sizing, meaning how much VRAM or RAM these files need once you add context, is a separate question covered in local LLM hardware requirements.

What are IQ quants and imatrix?

IQ quants (also called i-quants) are a newer family that pushes quality at 4 bits and below, and an imatrix is the calibration data they and the k-quants can use. The two are related but not the same thing.

An importance matrix is computed by running calibration text through the model. In pull request #4861, the author describes averaging the squared activations per tensor column over that data, then using them “as importances (weights) in a weighted RMSE minimization when quantizing the tensor.” In plain terms, weights that see large activations are rounded more carefully. The imatrix README shows the workflow: generate an imatrix file from a text dataset such as wiki.train.raw, then pass it to llama-quantize. It also advises against using the matrix for output.weight and token_embd.weight.

The IQ types use different internals:

• IQ2_XXS and friends. Pull request #4773 added a 2-bit type (2.0625 bits per weight) that picks from 256 grid points taken from the E8 lattice, guided by an importance matrix. It borrows ideas from the QuIP# paper.
• IQ4_NL. Pull request #5590 added a 4-bit type that maps values through a 16-entry non-linear lookup table. It matches Q4_0 in size, and the PR reports it is much better than Q4_0 on perplexity and nearly on par with Q4_K_S. Its main purpose is tensors whose column counts are not multiples of 256, which rules out k-quants.
• IQ4_XS, IQ3_S, IQ3_XXS and so on. The Hugging Face table lists these at 4.25, 3.44 and 3.06 bits per weight, all with super-blocks of 256 weights and importance-matrix-based weights.

One practical consequence: the quality of an IQ file depends on the imatrix its publisher used, and calibration data is a choice the publisher made. Look for a model card that says what it used.

How much quality do you lose at each level?

Quality loss is small from Q8_0 down to about Q5_K_M, noticeable at Q3, and steep at Q2. The cleanest published numbers are in the k-quants pull request, which reports wikitext perplexity for LLaMA 7B (lower is better). The F16 baseline is 5.9066.

The percentages are our arithmetic from the PR’s table. Two caveats apply. This is an old model and wikitext, so it shows the shape of the curve, not what happens on a current 2026 model. And perplexity measures next-token prediction on one text set, not task accuracy.

A second measure is how far the quantized model’s output distribution drifts from the F16 model. Pull request #4739 reports KL divergence to F16 on Llama-2 7B: 0.0226 for Q4_K_S and 0.0089 for Q5_K_S, with the fp16 top token matched on 92.9% and 95.2% of positions respectively. The step from 4 to 5 bits cuts the drift by more than half. The PR measured only the S variants, so it doesn’t speak to M.

Is code more sensitive to quantization than chat?

Developers say it is, but the rigorous evidence is thin. In an r/LocalLLaMA thread on Qwen3.6, summarized by InsiderLLM (May 2026), the original poster compared Ollama’s default Q4_K_M with a bartowski Q6_K_L on llama.cpp and called the coding gain huge. Commenters said Q4 followed long rule lists worse and broke tool-call JSON and diff formatting in agent sessions, and that Q5_K_M recovered most of it. One saw Q5 and Q6 as roughly equal on the 35B MoE. The caveats are real: the switch also changed runtime and calibration, and a follow-up said Ollama had been truncating context.

Against that anecdote, we found no rigorous GGUF-specific study that compares several quantization levels on code, so the honest answer is mixed and incomplete.

What exists:

• Q4_K_M on code, mixed. Afrin et al. (2026) tested six methods, including GGUF at Q4_K_M, on CodeLlama-7B and Qwen2.5-Coder-7B across McEval and CoderEval. In their tables, GGUF’s pass@1 moved between -0.05 and +0.12 against the FP16 baseline across ten model-benchmark pairs, and the significant degradations they report involve QuIP# and BitsAndBytes, not GGUF. They tested only 4-bit.

That is one study at one bit width, so it supports only a narrow claim: Q4_K_M does not usually wreck pass@1 on 7B code models. It does not show that code is immune, and a single changed token in a function can break it, so a perplexity number is a weak proxy. The Unsloth docs make a related point: they call KL divergence to the full-precision model the better gold standard than perplexity, and warn that calibrating on Wikipedia-style text can make a quant look better on Wikipedia-based tests through overfitting. If code quality matters to you, run your own test on your own repository, and compare two quant levels of the same model before committing.

What do developers report in practice?

Developers mostly report that Q4 feels fine in chat and starts to slip in long, instruction-heavy coding sessions, but every report changes more than one variable. The Qwen3.6 thread is covered above; two more follow. These are community reports, not our tests.

• Unsloth Dynamic files. In Hugging Face discussion #74 on Unsloth’s Qwen3.8-27B GGUFs (Aug 2026), users on llama.cpp and LM Studio called UD3-IQ4_XS good value on 16 GB of VRAM, but also reported crashes with Q5_K_XL in llama.cpp, more system-call errors from UD-Q6_K than from standard Q6_K, and worse 2-bit coding output than an earlier preview. Unsloth said the files were revised after release, so reports from different days may not match.
• Lower is not always worse. A Hacker News poster on an M4 Pro with 24 GB running LM Studio (Aug 2026) noted that a Q2 Qwen3.6 variant sometimes beat a Q3 variant, so bit count alone did not rank quality. Code reading and summarization only, on one machine.

Two lessons repeat. The recipe matters as much as the bit count: an imatrix-calibrated Q4 from a careful publisher can beat a plain one, and Unsloth’s own docs describe its Dynamic files as choosing a quant type per layer. And runtime problems can look like quantization damage, so change one variable at a time before blaming the quant.

Which quant should you download?

On 16 GB of VRAM, one user in the Unsloth Qwen3.8-27B Hugging Face thread called UD3-IQ4_XS good value, while the Qwen3.6 coding-agent reports above lean toward Q5 or Q6 when memory allows. Download Q4_K_M by default, step up to Q5_K_M or Q6_K if the file fits with room to spare, and go below Q4 only when memory forces you to. For agentic coding, the community reports above lean toward the step up.

• Q4_K_M. The common default. About 0.9% perplexity cost in the original 7B data at under a third of F16’s size.
• Q5_K_M or Q6_K. Pick these when you have headroom and want the closest match to the original weights. The step from Q4_K_M to Q5_K_M is cheap insurance for code work.
• Q8_0. Close to the original weights, but at 7.95 GiB for an 8B model it costs almost twice Q4_K_M’s 4.58 GiB. It rarely pays for itself.
• IQ4_XS, IQ3_S. Consider them when Q4_K_M just misses your memory budget. Check the publisher’s imatrix notes.
• Q2_K and IQ2. Last resort. In the data above, Q2_K added about 15% perplexity on a 7B model.

Larger models tolerate quantization better. The PR’s 13B column shows Q4_K_M at 5.3002 against F16’s 5.2543, about +0.9%, while Q2_K was +11.4% (5.8545), so a bigger model at a lower quant often beats a smaller model at a higher one on the same memory. That is the usual reason to run a low-bit large model, and it applies to abliterated releases too: see running abliterated models in Ollama for how those GGUFs are usually published. If you’re on a Mac, GGUF vs MLX covers the other major format.

FAQ