HowmanytokensperseconddoyouneedfromalocalLLM,andhowdoyoumeasureit?

What does tokens per second actually measure?

Tokens per second for an LLM measures two different phases, and a single number hides which one you’re looking at. The llama-bench README names them directly:

• Prompt processing (pp), or prefill. The model reads your whole prompt in batches. This is matrix-matrix work, so it is compute-bound and runs fast: thousands of tokens/s on a modern GPU.
• Text generation (tg), or decode. The model produces one token at a time, and each new token needs a full pass over the weights. This is memory-bound and runs far slower: tens to low hundreds of tokens/s.

The gap is large. On the llama.cpp CUDA scoreboard, an RTX 4090 running Llama 2 7B at Q4_0 posts about 11,993 tokens/s for pp512 and 186 tokens/s for tg128, a ratio of roughly 64 to 1. When someone on a forum says “I get 40 tok/s”, they almost always mean generation. When an agent feels slow, it is often prefill.

What you feel as a user maps onto the two numbers like this:

• Time to first token ≈ prompt tokens ÷ prompt processing speed.
• Time to finish ≈ time to first token + output tokens ÷ generation speed.

Why is generation speed bound by memory bandwidth?

Generation speed is bound by memory bandwidth because producing each token means streaming essentially every active weight from memory to the compute units, and at batch size 1 the arithmetic is trivial next to the data movement. That gives a usable rule of thumb: maximum tokens/s ≈ memory bandwidth ÷ model size in memory. For a mixture-of-experts model, use the size of the active parameters, not the file.

The llama.cpp scoreboards let you check that rule against real runs, because every entry uses the same model: Llama 2 7B Q4_0, which llama-bench reports as 3.56 GiB (3.82 GB). Bandwidth figures for the NVIDIA cards come from the spec tables on Wikipedia’s RTX 30, RTX 40 and RTX 50 series pages; the Apple figures are the ones listed in the Apple Silicon thread.

Three things fall out of the table:

1. The ceiling is a ceiling. None of these six beats bandwidth ÷ size, and most land between 55 and 75 percent of it. Use the formula to set expectations, then discount by about a third.
2. Bandwidth predicts generation, not prefill. The M2 Ultra has nearly as much bandwidth as an RTX 3090 and generates at 60 percent of its speed, but its pp512 is about a quarter of the 3090’s. Prefill tracks compute.
3. Bigger models scale down linearly. A 27B model at Q4 is roughly four to five times the size of this 7B, so expect roughly a fifth of these generation numbers. An HN commenter’s RTX 3090 result of 41 tokens/s on Qwen3.6-27B at UD-Q4_K_XL fits that estimate (community report).

How the weights get smaller, and what each quant costs in quality, is covered in GGUF quantization explained.

How many tokens per second is good enough for coding tasks?

How many tokens per second is good enough depends on whether you read the output, wait on it, or queue it, and for coding the answer is usually set by prefill rather than generation. There is no benchmark for “usable”, so the thresholds below are community opinions, not measurements.

In an r/LocalLLaMA thread asking for the minimum speed people tolerate for local agent work, one reply put the floor around 25 to 30 tokens/s for coding and another called 20 tokens/s usable (thread). Several replies argued the question was framed wrong: a user running Qwen3.6-27B in opencode said prompt processing ate most of the time once context passed about 50k tokens. That matches the arithmetic. A coding agent resends its system prompt, tool definitions and file contents on every turn, so a 30,000-token prompt at 500 tokens/s of prefill is a minute before the first output token, whatever the generation speed.

The opposite view is also worth stating. One Hacker News commenter argued that if you queue long-running tasks and spend the day reviewing, a model at 5 tokens/s running around the clock produces more than you can review. Speed matters most when you work serially: ask, wait, review, ask again.

For autocomplete, the output is a few dozen tokens, so time to first token is the whole experience. A fast generation number on a slow-prefill machine still gives you laggy completions once the editor sends a large file context.

How do you measure it with llama-bench and Ollama?

Measure with llama-bench when you want a clean, repeatable number, and with ollama run --verbose when you want the speed of the setup you actually use. Both report prefill and generation separately.

llama-bench. The defaults are pp512 and tg128, repeated 5 times, with the mean and standard deviation in tokens/s. The README notes that tokenization and sampling time are excluded.

# Baseline: 512-token prompt, 128 generated tokens, all layers on GPU
llama-bench -m model.gguf -p 512 -n 128 -ngl 99

# Generation speed after the KV cache is already filled (realistic for long chats and agents)
llama-bench -m model.gguf -p 0 -n 128 -d 0,8192,32768 -fa on

# A combined run: a 4,096-token prompt followed by 256 generated tokens
llama-bench -m model.gguf -pg 4096,256

# Compare quants or offload settings in one pass; output as JSON for scripts
llama-bench -m a-Q4_K_M.gguf -m a-Q8_0.gguf -ngl 99 -o json

The flags that matter most, from the README: -p and -n set prompt and generation lengths, -pg runs both in sequence, -d prefills the KV cache with that many tokens before each test, -ngl sets GPU layers, -fa toggles flash attention (on, off or auto), -ctk/-ctv set KV cache types, and -r sets repetitions. Every list-valued option runs all combinations, so one command can sweep a grid.

Ollama. Add --verbose to ollama run. The CLI source prints load duration, prompt eval count and duration, a prompt eval rate in tokens/s (prefill), and an eval rate in tokens/s (generation) after each reply.

ollama run <model> --verbose

From the API, the Ollama docs give the formula: tokens/s = eval_count / eval_duration × 10^9, since durations are in nanoseconds. The same response carries prompt_eval_count and prompt_eval_duration for prefill, and prompt_eval_cached_count for prompt tokens served from the cache.

To calculate tokens per second by hand for any other server, time the first token and the last token, then divide output tokens by the gap between them. Dividing by total wall time mixes prefill into the figure and understates generation speed.

Why are your numbers lower than the ones on Reddit?

Your numbers are probably lower because the posted ones were measured at empty context, with the whole model on the GPU, on a smaller quant or a different model. Check these before blaming the hardware:

• Context depth. Generation slows as the KV cache fills, since each token also reads the cache. The README’s own example on Qwen2 7B shows tg128 dropping from 120.6 to 116.7 tokens/s with only 512 tokens of prefilled context; at tens of thousands of tokens the drop is larger. Run -d at your real context length.
• Partial offload. One layer left on the CPU is expensive. In the README’s offload sweep on Llama 7B Q4_0, 34 of 35 layers on the GPU generated 71.8 tokens/s; all 35 gave 131.7. If the model doesn’t fit, see local LLM hardware requirements for how to size it.
• Quant and model size. Generation scales with bytes read per token, so Q8_0 is noticeably slower than Q4 on the same card: on the 40-core M4 Max in the Apple Silicon thread, Llama 2 7B generates 54 tokens/s at Q8_0 against 83 at Q4_0. MoE models break the size rule in your favor: the same HN commenter measured 153 tokens/s on Qwen3.6-35B-A3B versus 41 on the dense 27B on one RTX 3090.
• Speculative decoding and MTP. Some posted numbers include draft-model or multi-token-prediction speedups. One HN commenter reported tg128 of 81 tokens/s for Qwen3.8 27B UD-Q4_K_M on an RTX PRO 6000 in plain llama-bench, and about 140 tokens/s with MTP on a separate benchmark (community report). Compare like with like; speculative decoding in llama.cpp explains the mechanism.
• Thinking tokens. Reasoning models spend output tokens you don’t see as the answer. The tokens/s can be fine while the reply still feels slow.
• CPU runs. On the CPU, thread count changes generation speed, and more threads isn’t always faster. The CPU guide covers that sweep.

What do developers report in practice?

Developers report that generation numbers mostly track bandwidth, and that prefill is the number people learn to care about later. These are community reports, not our tests, and most don’t state every setting.

The Mac report deserves a note. The commenter quoted an article showing time to first token on Apple silicon that had to be charted on a log scale, because prefill on the M4 Air was so far behind a discrete GPU. Apple machines hold their own on generation thanks to wide memory buses, and other HN threads repeat that most of the time in agentic coding goes to prompt processing, which is where they fall behind. If you plan to run agents on a Mac, benchmark with -pg or -d at your real prompt length, not with tg128 alone.

FAQ