refusal rate
Subscribe

Flash attention in llama.cpp: what -fa on/off/auto does, which backends support it, why quantized KV cache needs it, and the Ollama and LM Studio switches.

[ fig. 01 — explainer ]48×5
[ fig. 01 — explainer ]16×5

Short answer

Flash attention in llama.cpp is a fused attention kernel that computes the same result as standard attention without writing the full score matrix to VRAM. Since August 2025 the -fa flag defaults to auto, which turns it on when your backend can run it. Leave it on: it shrinks the compute buffer at long context, is required for a quantized V cache, and in llama.cpp's own CUDA and Vulkan scoreboards it is never slower.

llama.cpp flag
-fa on | off | auto (default auto)
Env variable
LLAMA_ARG_FLASH_ATTN
Ollama override
OLLAMA_FLASH_ATTENTION=1 or 0
RTX 4090, pp512 with FA (CUDA scoreboard)
+23% vs off

Key takeaways

  1. Flash attention changes how attention is computed, not what it computes. It is exact, so output quality does not change.
  2. llama.cpp's -fa takes on, off or auto, and auto has been the default since PR #15434 (August 2025). Auto turns it off only when the kernel would land on a different device than the layer.
  3. Without it, llama.cpp allocates an F32 score tensor of context x batch x heads. For Qwen2.5-Coder 32B at 32k context that is 2.7 GB of scratch memory.
  4. A quantized V cache only exists on the flash-attention path, so llama.cpp forces -fa on (or refuses to load) when you set -ctv to q8_0 or q4_0.
  5. Ollama and LM Studio now enable it automatically on supported GPUs; OLLAMA_FLASH_ATTENTION=1 and LM Studio's load toggle are overrides.

Flash attention in llama.cpp is the -fa switch, and on any build from the last year you probably do not need to touch it: it defaults to auto and turns itself on when your backend supports it. It still matters: it decides how much VRAM a long context costs, a quantized V cache requires it, and on some older builds it was slower. A developer who benchmarked -fa on an RTX 5060 Ti in June 2026 found identical speed and VRAM with the flag on and off and concluded it was already on by default. This page explains why that happens, what the flag actually does, and when it is worth checking. Sources were fetched on 2026-11-02.

What is flash attention?

Flash attention is a way of computing exact attention in small tiles that stay in the GPU’s fast on-chip memory, so the full matrix of attention scores is never written to VRAM. The math is unchanged. What changes is memory traffic.

Standard attention multiplies the queries by every key to get a score for each query/key pair, runs softmax over those scores, then multiplies by the values. For a context of N tokens that score matrix is N x N per head, and it has to be written to and read back from VRAM. The original FlashAttention paper (Dao et al., 2022) argued that this memory traffic, not the arithmetic, is the bottleneck. It splits Q, K and V into blocks, loads one block at a time into on-chip SRAM, and keeps a running maximum and running sum so softmax can be computed incrementally. The paper calls it an “exact attention algorithm” that needs fewer reads and writes to high-bandwidth memory. FlashAttention-2 reworked how the work is split across GPU threads and reported about 2x the speed of the first version, reaching 50 to 73% of an A100’s peak FLOPs.

[ fig. 01 — Standard attention vs flash attention, after Dao et al. 2022 and llama.cpp's llama-graph.cpp ]
Diagram: standard attention writes a full context-by-batch score matrix to VRAM before softmax, while flash attention streams K and V tiles through on-chip memory with a running max and sum, so only the output is written back

llama.cpp does not use the Dao-AILab library. It has its own FLASH_ATTN_EXT operation with kernels per backend, which is why support and speed differ by backend.

In llama.cpp’s own graph code the difference is concrete. With flash attention off, llama-graph.cpp builds a kq tensor with ggml_mul_mat(k, q), kept in F32 because “this op tends to require high floating point range”, then applies softmax and multiplies by V. With it on, the whole sequence is one ggml_flash_attn_ext call. That kq tensor has one F32 value per cached token, per token in the batch, per attention head. Worked out for Qwen2.5-Coder 32B (40 query heads, from its config.json) at llama.cpp’s default physical batch of 512 tokens:

ContextScore tensor without flash attention
8k0.67 GB
32k2.68 GB
128k10.74 GB

That is scratch memory for one layer at a time, reserved at load time for the worst case, and it comes on top of the weights and the KV cache. Avoiding that round trip was the point from the start: in the original llama.cpp flash attention PR, merged in April 2024, the project’s maintainer wrote that “just avoiding the extra read/write of KQ will bring us close to the optimal performance”.

Is flash attention on by default in llama.cpp?

Yes. Since PR #15434, merged in August 2025, -fa defaults to auto. The server README lists the flag as:

ValueWhat it means
auto (default), also -1Enable if the backend can run it for this model, otherwise disable
on, also enabled, true, 1Force on
off, also disabled, false, 0Force off

The environment variable is LLAMA_ARG_FLASH_ATTN. Before that PR, -fa was a bare switch you had to add. It now takes a value, so old scripts that pass -fa followed by another flag need updating:

bash
llama-server -m model.gguf -c 32768 -fa on

How auto decides, from llama-context.cpp on 2026-11-02: when the context is created, llama.cpp reserves a test graph and checks where each flash-attention node would run. If any of them would be scheduled on a different device than its layer, which happens when that backend’s kernel does not support the model’s head size or cache type, it logs Flash Attention not supported, set to disabled and falls back. Three cases override it:

  • Quantized V cache. If -ctv is a quantized type and -fa is auto, llama.cpp logs “enabling flash_attn since it is required for quantized V cache”. If you set -fa off, the model fails to load.
  • Tensor split mode. --split-mode tensor also forces it on and errors if you disabled it.
  • Grok. The Grok architecture forces it off.

So the startup log is the place to check. Look for the flash_attn line in the context parameters and any “not supported, set to disabled” warning. The llama.cpp server guide covers the rest of the flags that appear in that block.

Which backends support flash attention?

Every major llama.cpp backend now has a flash-attention kernel. What differs is which head sizes and cache types it accepts. From each backend’s supports_op code on 2026-11-02:

BackendFlash attentionLimits that make auto fall back
CUDA (NVIDIA)YesHead sizes 40, 64, 72, 80, 96, 112, 128, 256, plus specific GQA layouts at 192, 320, 512 and 576. K/V types f32, f16, bf16, q4_0, q4_1, q5_0, q5_1, q8_0
ROCm / HIP (AMD)YesCompiled from the same ggml-cuda sources, so the same kernel selection
Metal (Apple)YesHead sizes 32 to 576 from a fixed list; K and V must use the same type; bf16 only on GPUs with bfloat support
VulkanYesHead sizes divisible by 8; needs cooperative matrix 2 or subgroup shuffle and vote support; K/V types include f16, bf16, q8_0, q4_0 and iq4_nl
SYCL (Intel)YesA oneDNN path for Xe2 was merged in July 2026
CPUYesGeneric kernel; auto enables it when the whole model runs on CPU

The Vulkan row is where history matters. In a September 2024 llama.cpp issue, a user on an RX 6800 XT saw generation drop from 63.33 to 32.69 tokens/s with flash attention on. A maintainer replied that Vulkan did not support it then, so the operation ran on the CPU. Advice to “turn off -fa on AMD” mostly dates from that period. Vulkan has had native kernels since, and the Vulkan scoreboard now lists an FA-on table. If you are setting up an AMD card, the local LLM on AMD GPU guide covers ROCm versus Vulkan.

Why do you need it for a quantized KV cache?

You need it because the quantized V cache only exists on the flash-attention path. Without flash attention, llama.cpp stores V transposed so it can use a plain matrix multiply (attn_v_trans is set to !flash_attn in llama-model.cpp), and 32-value quantized blocks do not fit that layout. That is why -ctv q8_0 forces -fa on.

The quality and memory trade-offs of q8_0 versus q4_0 are covered in KV cache quantization. For this page, two practical points:

  • Use q8_0 or q4_0 if you want a type every GPU backend’s flash-attention kernel lists. Other types can push the operation off the GPU.
  • Quantized-cache decode speed depends on the backend’s kernels. A September 2026 Hacker News post benchmarking a different engine against llama.cpp said 8-bit K with 4-bit V hurt llama.cpp’s decode speed, so they used a 16-bit cache. A July 2026 commenter on Strix Halo found the stock Vulkan and ROCm kernels re-dequantize the q8 cache redundantly and measured large gains from a patched branch. Both are single reports, but they explain why a quantized cache can save memory and still cost speed on some hardware.

How do you turn on flash attention in Ollama and LM Studio?

Both now turn it on by themselves on supported hardware. The settings are overrides.

Ollama. The FAQ says Ollama “uses Flash Attention automatically when the selected backend and devices support it”. Set OLLAMA_FLASH_ATTENTION=1 on the server process to force it on, or 0 to force it off. In Ollama’s source (ml/device.go), supported devices are CPU, Metal, ROCm, Vulkan, and CUDA GPUs of compute capability 6.0 or newer, excluding 7.2. Ollama also only applies OLLAMA_KV_CACHE_TYPE when flash attention is on.

bash
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve

Older guides, and a March 2026 Hacker News comment, say Ollama disables flash attention by default. That matched older releases. On a current build, check the server log instead of assuming.

LM Studio. Flash Attention is a toggle in the model load settings for the llama.cpp engine, and flashAttention in the SDK load config. Version 0.3.31 (November 4, 2025) made it the default for CUDA engines, and 0.3.32 (November 19, 2025) extended that to Vulkan and Metal. The docs note that the V cache quantization setting “requires Flash Attention to be enabled”. Where each tool sets context length is in how to set context length.

Does flash attention make generation faster?

A little for generation, more for prompt processing, and the gap grows with context. llama.cpp’s community scoreboards run llama-bench -fa 0,1 on Llama 2 7B Q4_0, with a 512-token prompt and 128 generated tokens. Selected rows:

GPU, backendPrompt (pp512) off / onGeneration (tg128) off / on
RTX 4090, CUDA11,993 / 14,771 t/s (+23%)186.2 / 189.0 t/s (+1.5%)
RTX 5090, CUDA14,073 / 14,970 t/s (+6%)290.0 / 300.4 t/s (+3.6%)
RTX 3090, CUDA5,175 / 5,560 t/s (+7%)158.2 / 161.9 t/s (+2.4%)
RTX 4090, Vulkan9,452 / 10,830 t/s (+15%)188.0 / 190.1 t/s (+1.1%)
RX 7900 XTX, Vulkan3,727 / 3,890 t/s (+4%)182.6 / 190.9 t/s (+4.5%)

Sources: CUDA scoreboard, Vulkan scoreboard. These are short-context community submissions, so they understate the long-context effect. Generation at batch size 1 is bound by reading the weights, which flash attention does not touch, so its gains are small. Tokens per second explained covers that ceiling.

What do developers report in practice?

People running long contexts treat -fa on as a fixed part of their command line; complaints mostly concern older builds or specific backends. These are individual community reports, not tests run by this site.

  • Long context on one card. A commenter running a Q5_K_M model on an RX 7900 XTX with llama.cpp reported 81k context with flash attention on and 8-bit K/V caches, at 625 t/s prompt processing and 30 t/s generation, in 24 GB.
  • Ollama, same idea. A single-3090 user fit a 180k context in 23 GB with OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0.
  • “Already on” results. The June 2026 InventiveHQ test above saw 81 to 83 t/s on an RTX 5060 Ti and 51 to 53 t/s on an M3 Max with the flag on or off, at 4k, 16k and 32k. The likeliest explanation is the auto default, though the CUDA build number is not stated.
  • Model-specific bugs. In August 2025 one Hacker News user said GPT-OSS-20B became “dumb as rocks” with flash attention or a quantized cache in their setup. It is a single early report, but it shows why a new architecture is worth a quick A/B with -fa off.

Should you enable it?

Yes, leave it on, which on a current build means leaving the default alone.

  1. On llama.cpp builds from late 2025 onward, keep -fa auto and read the startup log to confirm it resolved to enabled.
  2. Use -fa on explicitly when you quantize the V cache or use --split-mode tensor, so a config error fails loudly.
  3. Try -fa off only when you are debugging: garbled output on a new architecture, or a backend where llama-bench -fa 0,1 shows it slower on your card.

FAQ

Does flash attention change model output or quality?

No. FlashAttention is an exact algorithm. It reorders the computation, so tiny floating-point differences are possible, but it is not an approximation like sliding-window or sparse attention.

Is -fa the same as --flash-attn?

Yes. Both take on, off or auto, and so does the LLAMA_ARG_FLASH_ATTN environment variable. Since the default became auto, a bare -fa with no value is no longer the way to enable it.

Does flash attention work with Vulkan in llama.cpp?

Yes, on current builds, for head sizes divisible by 8 and on GPUs with the needed subgroup or cooperative-matrix features. The 2024 reports of it halving Vulkan speed came from before Vulkan had a native kernel.

Do I need flash attention for KV cache quantization?

For the V cache, yes. llama.cpp enables it automatically when -ctv is quantized and refuses to load if you forced it off. Ollama applies OLLAMA_KV_CACHE_TYPE only when it is on.

Sources (10)
  1. llama.cpp server README: -fa, --flash-attn [on|off|auto] (ggml-org/llama.cpp)
  2. PR #15434: llama: use FA + max. GPU layers by default, ggml-org/llama.cpp (Aug 2025)
  3. llama-context.cpp: auto flash attention resolution and quantized V cache check, ggml-org/llama.cpp
  4. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, Dao et al. (arXiv 2205.14135)
  5. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning, Dao (arXiv 2307.08691)
  6. Performance of llama.cpp with CUDA (scoreboard, FA on and off), ggml-org/llama.cpp discussion #15013
  7. Performance of llama.cpp with Vulkan (scoreboard, FA on and off), ggml-org/llama.cpp discussion #10879
  8. FAQ: How can I enable Flash Attention?, Ollama docs
  9. LM Studio 0.3.31 and 0.3.32 release notes: Flash Attention on by default for CUDA, then Vulkan and Metal
  10. LLMLoadModelConfig: flashAttention, llamaVCacheQuantizationType, LM Studio docs

Keep reading

[ fig. 02 — explainer ]18×8

What is KV cache quantization, and should you turn it on?

Nov 2, 2026 · Explainer

KV cache quantization stores the attention keys and values a model keeps for every token at lower precision, usually q8_0 or q4_0 instead of f16. q8_0 halves the cache and, in published llama.cpp and Ollama measurements, costs almost nothing in quality. q4_0 cuts it to about 28% but is model-dependent and riskier at long context. Turn on q8_0 when context, not weights, is what runs you out of VRAM.

Read
[ fig. 03 — how-to ]18×8

How do you run and configure llama.cpp's llama-server?

Nov 2, 2026 · How-to

llama-server is the HTTP server that ships with llama.cpp. Start it with `llama-server -m model.gguf` (or `-hf user/repo`) and it serves OpenAI- and Anthropic-compatible endpoints plus a web UI on 127.0.0.1:9931. Start it with no model and it becomes a router that loads and unloads several GGUF models on demand. It has no authentication until you pass --api-key.

Read
[ fig. 04 — explainer ]18×8

How many tokens per second do you need from a local LLM, and how do you measure it?

Nov 2, 2026 · Explainer

Tokens per second for a local LLM is two numbers. Prompt processing (prefill) is compute-bound and decides how long you wait before the first token. Generation (decode) is memory-bandwidth-bound and caps out near bandwidth divided by model size. Developers commonly treat 5 to 10 tokens/s as readable for chat and 20 to 30 for interactive coding, while agents with long prompts live or die on prefill speed.

Read