WhatarethehardwarerequirementsforalocalLLMforcoding?

Local LLM for coding hardware requirements come down to one sum: weights, plus KV cache, plus a little overhead, and that sum decides whether you need an 8, 16 or 24 GB GPU. This page shows the arithmetic once, with real model files and real architecture numbers, so you can answer it for any model. Figures below were fetched from Hugging Face and llama.cpp documentation on 2026-10-26 and 2026-10-27. The examples are Alibaba’s current Qwen models (Qwen3.5 9B, Qwen3.8 27B, Qwen3.6 35B-A3B) plus one older Qwen2.5-Coder model as a contrast, because their configs and GGUF sizes are public. For which model to pick, see the best local LLM for coding.

How do you estimate memory for a model?

Memory for a model is its weights plus its KV cache plus runtime overhead, and the weights part is simple: parameters times bits per weight, divided by 8.

The catch is that “4-bit” is not 4.0 bits. GGUF quant types mix precisions across tensors, so the average is higher. llama.cpp’s quantize README measures it on Llama 3.1 8B:

Worked example: Hugging Face lists about 27.8 billion parameters for Qwen3.8 27B. At Q4_K_M, 27.8 x 4.89 / 8 = 17.0 GB. The published Unsloth GGUF is 16.46 GB, so the formula lands about 3% high. Part of that is the vision tower included in the parameter count, and part is that Unsloth’s “UD” quants mix bit widths per tensor. Use the real file size when you can; use the formula when you can’t.

Weights below come from the file sizes in the Unsloth Qwen3.8-27B GGUF repo and the matching repos for the other two models, not from the formula. Sizes are decimal gigabytes.

The Qwen3.x rows use Unsloth’s “UD” variants for Q4_K_M and Q5_K_M, so they are not byte-identical to a plain llama.cpp quant. The last row is from Qwen’s own GGUF repo.

How much does context length add (KV cache)?

Context length adds a KV cache that grows linearly with every token in the window. How fast it grows depends on how many layers use full attention, and that changed with the current Qwen generation. The formula:

KV bytes = 2 x full-attention layers x KV heads x head dim x tokens x bytes per value

The 2 is for keys and values. Default storage is f16, which is 2 bytes per value, per the llama.cpp server’s --cache-type-k default. The numbers come from each model’s config.json. Qwen3.5, 3.6 and 3.8 are hybrid models: their configs set full_attention_interval to 4, so only every fourth layer keeps a growing KV cache and the rest use linear attention. Head dim is 256 for all three.

The older model, with head dim 128, shows what “every layer caches” costs: four times the 27B per token, from a similar-sized dense model. The linear-attention layers do keep a fixed-size state that does not grow with context. That state is not sized here, so the 1 GB overhead allowance below has to cover it.

Multiply by tokens and you get the cache. Totals add it to the Q4_K_M weights from the table above (128k means 131,072 tokens, within the 262,144 each Qwen3.x config lists as its maximum):

The totals exclude overhead. This page budgets about 1 GB for compute buffers and the CUDA context. That is an editorial planning allowance, not a measured figure, so check your own runtime.

You can shrink the cache. llama.cpp lets you store K and V as q8_0, q4_0 and other types with -ctk and -ctv. Since q8_0 stores roughly 8.5 bits per value against 16 for f16, expect about half the cache. Quality impact varies by model and has not been measured here. One developer using a 12 GB RTX 4070 reports about the same decode speed with a q8_0 cache at 32k as with f16 at 4k, from one run per setting and no perplexity number, so treat that as a single data point and test on your own code.

What hardware does a local LLM for coding need at 8, 12, 16, 24 and 32 GB?

You can run a 9B model on 8 GB, the 27B on 24 GB with room for context, and a 35B MoE at the 24 to 32 GB tier. One developer with a 16 GB RTX 5070 Ti ran the 27B at 2-bit with 128k context and a 4-bit KV cache and reported 50 to 60 tokens/sec (Hacker News); the table below puts that setup in its “tight” cell. Using the totals above plus the 1 GB allowance:

The 4-bit cache figure assumes q4_0 stores roughly 4.5 bits per value, about 0.28 of f16, so 128k on the 27B costs about 2.4 GB. Coding agents use context quickly, so if you work at 32k or more, read the 32k column, not the 8k one. When a model does not fit, the failure is either an out-of-memory error or a silent slowdown. On Linux, llama.cpp’s CUDA unified memory option “allows swapping to system RAM instead of crashing when the GPU VRAM is exhausted”, which keeps the process alive but is exactly the slow path you were trying to avoid.

If you’re picking a quant and wondering what you lose by going lower, GGUF quantization explained covers the quality side.

What do developers report on 16 GB and 24 GB cards?

Community reports match the arithmetic: the 27B fits on 16 GB only at 2-bit or 3-bit, and context, not weights, is the constraint. These are single-user reports from Hacker News, not tests run for this page, and they have not been reproduced.

• A developer with a 16 GB RTX 5070 Ti ran the 2-bit UD-Q2_K_XL at 128k context with a 4-bit KV cache and reported 50 to 60 tokens/sec (HN). That is the “tight” cell in the table: 9.83 GB of weights plus about 2.4 GB of cache.
• Another with an RTX 5080 (16 GB) reported 46 tokens/sec at UD-Q3_K_XL (HN). A third, on a 2016 Pascal GPU with 16 GB, works at roughly 100k context with a q8 cache but had to drop to Q2 for usable speed (HN).
• A commenter on a launch thread argued that 4-bit quants leave too little context on 16 GB, and that agents with heavy prompts can burn 30k tokens before doing any work (HN). That is why the 8k column understates real coding use.
• Spilling to system RAM hurts. One user with a 20 GB RTX 3080 running UD-Q4_K_XL with the KV cache offloaded to RAM saw 10 tokens/sec at first, dropping to 5 at 50k context (HN).

What changes on Apple Silicon unified memory?

On Apple Silicon the CPU and GPU share one memory pool, so you size against total RAM minus what macOS holds back, not a separate VRAM figure. Apple’s M4 Pro supports up to 64 GB at 273 GB/s and the M4 Max up to 128 GB at up to 546 GB/s, which is why a 64 GB Mac can hold Qwen3.8 27B at Q8_0 with a 32k context (31.2 GB, if you raise the GPU limit) where a 24 GB GPU cannot.

The GPU can’t use all of it by default. A llama.cpp discussion reports a default reserve of about a third of memory (a quarter above 32 GB) and describes raising the limit with sudo sysctl iogpu.wired_limit_mb=<mb>, which does not persist across reboots. Those figures date from 2023 threads, so check the recommendedMaxWorkingSetSize value in llama.cpp’s Metal startup output instead of trusting a percentage. Don’t hand the GPU everything: a maintainer advised against 100%.

Capacity is the strength; bandwidth is the weakness. The 546 GB/s ceiling on a Max chip is set by the chip, and token generation is generally limited by memory bandwidth, so check a bandwidth figure for any GPU you compare against; a model that fits can still answer slowly. For MLX versus GGUF on a Mac, see GGUF vs MLX.

Is CPU offload fine, and what about MoE models?

CPU offload is fine for MoE models and painful for dense ones. A dense 27B reads all its weights for every token, so any layer parked in system RAM runs at RAM speed. A mixture-of-experts model like Qwen3.6 35B-A3B stores about 36B parameters but, per its config, routes each token to 8 of 256 experts, so only a small fraction is read per token.

The memory bill is for the full 36B, but each token touches a fraction of it. llama.cpp exposes two flags for this, documented in the server README:

# keep every expert tensor in system RAM, everything else on the GPU
llama-server -m Qwen3.6-35B-A3B-UD-Q4_K_M.gguf -ngl all --cpu-moe -c 32768

# keep only the first N layers' experts on the CPU, the rest on the GPU
llama-server -m Qwen3.6-35B-A3B-UD-Q4_K_M.gguf -ngl all --n-cpu-moe 20 -c 32768

With --cpu-moe, the GPU holds attention weights and the KV cache, so a card in the 8 to 12 GB range can plausibly serve this model if you have about 24 GB of free system RAM for the 22.13 GB file. Developers report it works. One says Qwen3.6 35B-A3B runs well at Q4 on about 32 GB of combined RAM and VRAM by offloading most experts, and that “it doesn’t hurt performance too much” compared with dense models (HN). A 12 GB RTX 4070 owner reports about 34 tokens/sec decoding a 35B MoE with --cpu-moe and a q8_0 cache at 32k context (dev.to), though the post doesn’t state the quant. I haven’t measured speed, and you shouldn’t expect GPU-only throughput. Raise --n-cpu-moe only until the model loads, then lower it as VRAM allows. For choosing between runtimes, see Ollama vs llama.cpp vs vLLM.

Which GPU should a developer buy in 2026?

Buy the most VRAM you can afford on a current-generation NVIDIA card, then check memory bandwidth. VRAM capacity sets which models load; bandwidth sets tokens per second. Prices move too fast to print here, so these are the criteria:

• Capacity tiers matter more than model names. The jumps that change what you can run are 8, 12, 16, 24 and 32 GB. NVIDIA lists the RTX 5070 at 12 GB and the 5070 Ti at 16 GB, the RTX 3090 at 24 GB, and the RTX 5090 at 32 GB on a 512-bit bus.
• 24 GB is the step where the 27B runs at Q4. Below that, you are on 3-bit quants or MoE offload.
• Check power and case. The RTX 5090’s listed total graphics power is 575 W.
• System RAM is your MoE budget. If you plan to use --cpu-moe, 32 GB of RAM is a floor, not a luxury.
• Skip splitting across cards unless you must. llama.cpp’s docs warn that peer access can cause crashes or corrupted output on some motherboard and BIOS settings.

Rule of thumb from the tables: 8 GB is the entry point for a 9B coder; 16 GB runs the 27B only at 2-bit or 3-bit; 24 GB is the entry point for the 27B at Q4 and a usable context; 32 GB is where long context on the 27B stops being a compromise.

FAQ