Howdoyourunandconfigurellama.cpp'sllama-server?

The llama.cpp server, llama-server, is a single C/C++ binary that loads a GGUF model and exposes it over HTTP. This page is a working reference: install, the flags grouped by job, router mode for several models, the endpoints, and how to lock it down. Every flag and default below is quoted from the server README on master as fetched on 2026-10-28. The project moves weekly, so run llama-server --help on your own build before trusting any default, including ours.

What is llama-server, and when should you use it instead of Ollama?

Use llama-server when you want exact control over the GGUF file, the flags and the endpoints, and Ollama when you want a model catalog and one-word pulls. Both serve an OpenAI-style API on your machine. The difference is how much they decide for you. The full runtime trade-off lives in llama.cpp vs Ollama vs vLLM.

Community sentiment tracks that split. In an r/LocalLLaMA thread from December 2025, one user said they moved to llama.cpp because it was “not as easy to use but works better”, another cited Intel Arc support and speed, and a third went back to Ollama for its simpler interface (thread).

How do you install and start llama-server?

Install a prebuilt package, then point llama-server at a GGUF file or a Hugging Face repo. The install docs list one route per OS:

• macOS and Linux: brew install llama.cpp. The formula updates automatically with releases.
• Windows: winget install llama.cpp.
• Any OS: conda-forge (conda install -c conda-forge llama.cpp, with CUDA, Vulkan or Metal builds), or the binaries on the releases page.
• From source: build only the server target with CMake.

# build from source (binary lands in ./build/bin/llama-server)
cmake -B build
cmake --build build --config Release -t llama-server

# start with a local file
llama-server -m ./models/qwen-coder-Q4_K_M.gguf

# or download from Hugging Face (defaults to the Q4_K_M quant if the repo has one)
llama-server -hf ggml-org/GLM-4.7-Flash-GGUF:Q4_K_M

On Windows the same flags apply to llama-server.exe. The README’s Docker image is ghcr.io/ggml-org/llama.cpp:server (and :server-cuda). In a container you must pass --host 0.0.0.0, or the port mapping reaches nothing.

Once it’s running, open the address in a browser for the built-in web UI, or check GET /health. It returns 503 while the model loads and {"status": "ok"} when it’s ready.

Note the port. The current default is 127.0.0.1:9931. Up to at least tag b6000 the default was 8080, and most blog posts, editor docs and Docker examples still use 8080. If a client can’t connect after an upgrade, this is the first thing to check. Pass --port explicitly and the question goes away.

Which llama-server flags matter most?

About fifteen flags cover nearly every setup. Most of the rest are sampling, RoPE and CPU-affinity knobs you can leave alone. Each flag also has an environment-variable form (LLAMA_ARG_*), which is handy in Docker and systemd units.

Three of these trip people up most.

Context. -c 0 no longer means a small fixed window. With --fit on, the server takes the largest context that fits the free memory at load time. That is convenient on a dedicated GPU and surprising on a shared one. Pin -c if you need headroom, and see context length settings for how much memory each step costs.

Slots. -np splits work across parallel slots. When you set it explicitly, the context is divided between slots unless the KV cache is unified. GitHub issue #17989 shows how easily this confuses people: a user passed --parallel 1, still got 4 slots (the log read setting n_parallel = 4 and kv_unified = true), and closed the issue with a linked fix. If you depend on the slot count, read the startup log or GET /slots rather than assuming.

Offload. Mixture-of-experts models have their own CPU-offload flags (--cpu-moe, --n-cpu-moe), covered in MoE offloading in llama.cpp. Draft-model flags are in speculative decoding in llama.cpp.

How do you serve several models from one llama-server?

Start llama-server without -m and it runs in router mode: one process that loads and unloads models on demand and forwards each request to a child llama-server per model. The README calls this “Using multiple models”. It answers the old complaint that llama.cpp serves one model at a time. One Hacker News commenter corrected a project README that still said so in October 2026 (HN).

# serve every GGUF in a folder (multi-file models go in their own subfolder)
llama-server --models-dir ./models --models-max 2 -c 32768 -np 1

# or define per-model settings in an INI file
llama-server --models-preset ./my-models.ini

How it behaves, per the README:

• Sources. The router finds models in three places: the Hugging Face cache (anything you fetched with -hf), --models-dir, and --models-preset. Restart the router after adding files to the cache.
• Routing. POST endpoints route by the "model" field in the JSON body. GET endpoints such as /props route by a ?model= query parameter.
• Limits. --models-max caps loaded models (default 4, 0 = unlimited) and evicts the least recently used. --models-memory-margin N also unloads idle models until the new one fits with N MiB left free per device. A model that is serving a request is never unloaded.
• Presets. In the INI file, keys are flag names without dashes (n-gpu-layers = 123, c = 4096), a [*] section holds shared defaults, and command-line flags win over both. Host, port and API key belong to the router and are stripped from presets.
• Autoload. Models load on first request unless you pass --no-models-autoload or add ?autoload=false per request.

Router mode doesn’t add VRAM. One HN commenter described a coding agent that, on its own initiative, spawned a subagent on a lighter model. Their machine fits one model at a time, so llama-server ended up thrashing between the two models (HN). If your harness can pick models, either give it one model name or set --models-max 1 and expect reload pauses.

Which API endpoints does llama-server expose, and how do you secure them?

llama-server speaks three API dialects on one port, and none of them is authenticated until you set a key. The main routes:

To expose the server beyond localhost, do these four things in order:

1. Set a key. Use --api-key (or --api-key-file, one key per line). Clients then send Authorization: Bearer <key>, or x-api-key on the Anthropic route. A bad key gets an OpenAI-style 401 Invalid API Key.
2. Bind deliberately. --host 0.0.0.0 listens on every IPv4 interface. You can pass a comma-separated list or a .sock path instead to bind only where you need it.
3. Tighten CORS. By default the server reflects any Origin with credentials allowed. The README recommends setting --cors-origins to your frontend’s origin on a LAN, and an API key plus a reverse proxy for anything public. TLS is available natively only if you build with -DLLAMA_OPENSSL=ON and pass --ssl-key-file/--ssl-cert-file. Otherwise terminate TLS at the proxy.
4. Leave agent tools off. --tools, --agent and the MCP proxy flags are marked experimental: “do not enable in untrusted environments”. They expose file read/write and shell execution over the API.

A common pattern among HN commenters is to keep the server on a private network rather than the internet. One runs llama-server on a stronger machine on the LAN, reaches it over Tailscale, and keeps the coding agent itself in a Docker container with a single workspace mount (HN).

How do you connect editors and coding agents to llama-server?

Point any OpenAI-compatible client at http://<host>:<port>/v1, with any non-empty key if you didn’t set one. The README’s own examples use base_url="http://localhost:9931/v1" and api_key="sk-no-key-required".

curl http://127.0.0.1:9931/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LLAMA_API_KEY" \
  -d '{"model": "qwen-coder", "messages": [{"role": "user", "content": "Write a Python retry decorator"}]}'

Some practical notes:

• The model field. In single-model mode the server answers whatever model name you send. In router mode the name selects the model, so set -a aliases or preset names that match what your editor sends.
• Tool calling. Agent harnesses need tool calls parsed from the chat template. --jinja is on by default now, but older builds and some blog commands pass it explicitly. Tool use on /v1/messages requires it.
• Claude Code and other Anthropic clients. Use the /v1/messages route. The setup is covered in using a local LLM with Claude Code.
• Tracking token use. One HN commenter points agent users to GET /slots for up-to-date per-session token counts when a harness reports context use vaguely (HN).

What do developers report in practice?

The recurring pattern in public reports: people pin the flags that defaults get wrong for their workload, and the bugs they hit are usually multi-GPU or slot accounting, not the API. These are community reports, not our tests.

• A long-context, single-user coding setup. One HN commenter posted their full command for Qwen 3.8 27B (Unsloth UD-Q4_K_XL) at 131k context: --ctx-size 131072 --parallel 1 --cache-type-k q8_0 --cache-type-v q8_0 --host 0.0.0.0 --port 8080, plus a draft model for speculative decoding (HN). Three choices stand out: one slot so the whole window serves one agent, a q8_0 KV cache instead of the f16 default, and an explicit port.
• Row split on old multi-GPU rigs. A user running a 32B Q8_0 model across three Tesla P100s got garbled output from the second chat turn onward with --split-mode row. Removing the flag fixed the output but cut speed sharply. Disabling Intel VT-d (IOMMU) in the BIOS fixed it, after a contributor pointed at GPU peer-to-peer access (issue #16517). The default layer split avoids this.
• Slot counts that don’t match the flag. See issue #17989 above. Verify with /slots.
• Model thrash under agents. See the router section above. One loaded model plus an agent that calls a second model means reloads on every switch.

FAQ