Howmuchdoesaself-hostedLLMcostcomparedwithanAPI?

When does self-hosting an LLM beat an API?

A developer who measured electricity on an RTX 3090 found that a dense 27B model cost about €0.71 per million output tokens in power alone, more than the roughly €0.55 they cited for a hosted Flash-class API, and the rest of this page works out why. In short, self-hosted LLM cost beats an API when the GPU is busy most of the time, or when the API you would otherwise pay is a frontier model priced at $2 or more per million input tokens. Self-hosting loses when your traffic is light and bursty, or when the same open model is already sold per token at a few cents.

That gives three rules of thumb, each worked out below:

• Versus a frontier API, one owned GPU wins at under a million tokens a day, provided the open model is good enough for the job.
• Versus the same open model on an API, one developer almost never breaks even. A team might, and a batch workload usually does.
• Utilization is the variable. A rented GPU bills by the hour whether or not you send a request.

The model used throughout is Qwen3-Coder-30B-A3B, an Apache 2.0 mixture-of-experts coder with about 30.5B total and 3.3B active parameters (model card). It is the only coding model for which this page found a published, serving-grade throughput benchmark and a per-token API price on the same day. Treat the totals as illustrations of the method, not as a quality-equivalent comparison across models.

What goes into the cost of self-hosting?

The cost of self-hosting is hardware (bought or rented), electricity, idle time and your own time. Only the first is large for a single GPU.

1. Hardware. Buy a card and spread the price over its useful life, or rent by the hour. This page assumes $2,000 for an RTX 5090 and 36 months, an assumption taken from the MSRP the Knoop and Holtmann paper uses, because NVIDIA’s own product page does not publish a price. The PC around the card is not counted.
2. Electricity. The RTX 5090 has a 575 W total graphics power. At the EIA’s August 2026 US residential average of 18.16 cents per kWh, that is about $0.10 an hour at full draw (commercial: 14.47 cents).
3. Idle draw. A box left on all month still pays. The calculations below borrow a 50 W idle assumption from a community cost write-up, which comes to about $6.60 a month.
4. Ops time. Patching drivers, restarting a crashed server, watching a queue. A shared server needs this; a laptop does not.

For memory sizing, see how much VRAM a coding LLM needs. A 30B MoE at 4-bit fits a 24 GB card, which is why it is a common first target.

What do GPU rentals and APIs charge?

Rented GPUs run from about $0.34 to $4.29 an hour for the cards below, and the same open model sells per token for a small fraction of what a frontier model charges. All prices are USD, checked October 27, 2026 unless marked.

RunPod’s page is dated October 8, 2026 and the community column came from its structured data; Lambda’s is a current list price. Together AI lists a dedicated HGX H100 at $5.49 per GPU-hour.

Anthropic prices come from its pricing page, which also lists a 50% Batch API discount ($1 and $5 for Sonnet 5.5). The Qwen prices come from the OpenRouter endpoints listing, which showed two providers that day. Providers reprice often, so rerun the numbers before you commit. The cheapest open-model rows also show why per-token API vendors are the real competitor to your own GPU.

How fast is a self-hosted model, really?

One user on one stream gets a few hundred tokens a second; a server handling many requests at once gets roughly twenty times that from the same card. This gap is the whole economic story.

• Single stream. One community report (May 2026) measured about 231 tokens/s on an RTX 5090 running Qwen3-Coder-30B-A3B at Q5_K_M through Ollama, with a 32K context. That is steady-state generation on a short completion, and the author notes long prompts add a delay before the first token.
• Batched serving. Cloudrift ran vLLM with 4-bit AWQ on the same model and reported the figures below, using 1,000 input and 1,000 output tokens per request and high concurrency. The test was sponsored by a GPU host, and it is synthetic traffic.

The mechanism is not specific to this model. A small community repo measured vLLM on a 6 GB laptop GPU serving a 0.5B model and saw throughput climb from 129 tokens/s with one request in flight to 1,916 with sixteen, with p95 latency flat near 1.1 seconds. Decoding is limited by memory reads, so extra requests are nearly free until the batch gets large. For tools that do the batching, see Ollama vs llama.cpp vs vLLM.

What does the arithmetic look like in three cases?

In all three cases a frontier API costs the most, the open model on an API costs the least, and self-hosting lands between them unless the workload is a batch. Every input is stated so you can swap in your own.

Case 1: solo developer, owned RTX 5090. Assume 3M input and 0.5M output tokens per working day for 22 days, so 66M input and 11M output a month. Hardware is $2,000 over 36 months, idle draw is 50 W all month, and the card runs at its full 575 W for 30 hours of actual generation and prefill (an assumption).

Own-GPU cost is about four times lower than Sonnet 5.5 and about eight times higher than the cheapest open-model API. If you already own the card for gaming or work, the hardware line is sunk and the comparison improves.

Case 2: 10-developer team, shared server. Ten times the load: 660M input and 110M output a month. Two rented RTX 5090s for the 176 working hours in a month cost $243 at the community rate or $419 at secure. Add 4 hours of ops at an assumed $100 an hour and the total is $643 to $819.

Rental still loses to the open-model API by a wide margin. It beats Sonnet 5.5 by roughly 3x, again only if the open model is good enough. One caveat points the other way: the Cloudrift test used 1,000-token prompts, while agentic coding sessions carry much longer contexts, so budget more GPU than the benchmark suggests. A fuller team setup is covered in self-hosting an LLM for a dev team.

Case 3: batch workload. Take 20 million requests a month at 1,000 input and 1,000 output tokens each, such as classifying or rewriting documents with no latency requirement. At Cloudrift’s 4,570 output tokens/s that is 20 billion output tokens, or about 1,216 GPU-hours on one RTX 5090.

Here self-hosting wins by a factor of 2.4 to 8 even against the cheapest open-model API, and the margin survives a 50% haircut on throughput. The benchmark was synthetic and used a 4-bit quant, so validate on your own prompts first.

What do developers report in practice?

Community write-ups agree with the arithmetic: self-hosting is cheap per token at high load and not cheap at all for a single user.

• The RTX 3090 electricity measurement mentioned at the top, run with Ollama, covered a dense 27B Gemma model. A 26B mixture-of-experts model in the same write-up came in at about €0.27 per million output tokens. The author counted only marginal GPU energy.
• A personal cost breakdown (updated August 2026) assumed a $1,800 RTX 4090, an 8B model and 500,000 generated tokens a day. It found no break-even against a budget API at that volume, and about 20 months against a flagship-priced one.
• The Knoop and Holtmann paper argues the opposite, reporting self-hosted electricity-only costs of $0.001 to $0.04 per million tokens on consumer Blackwell cards. Read its assumptions first: 30 million tokens a day, small models of 8B to 27B, a US commercial power price of $0.12 per kWh, and tables that disagree with each other on some API throughput figures.

The pattern: results flip depending on model size (dense versus MoE), batching, and what you compare against. No community report found a single-developer case that beat a hosted open model on cost alone.

What hidden costs and benefits does the headline math miss?

The headline math misses quality, privacy, availability and your own time, and each can outweigh a $50 monthly difference.

• Model quality. A 30B open model is not Claude Sonnet 5.5. If the cheaper model needs more retries or review, the savings shrink. See local LLM vs Claude for coding.
• Privacy and control. Self-hosting keeps code on hardware you control and lets you pin a model version. Providers can change models or policies without notice, including refusal behavior (see why LLMs over-refuse).
• Rate limits. APIs throttle; your own GPU does not, until it saturates.
• Ops. A rented GPU bills while a crashed server sits idle. Autoscaling to zero helps, but adds engineering time.
• Idle waste. A GPU used for four hours a day is billed or depreciated for twenty-four.

How do you estimate your own break-even?

Compute monthly cost both ways, then find the utilization at which they cross. The formulas use only inputs you can look up.

API   = M_in x P_in + M_out x P_out            (M in millions of tokens)
Own   = hardware / months + kW x hours x $/kWh + idle_kW x 730 x $/kWh + ops
Rent  = $/hr x hours billed + ops
Break-even utilization u* = ($/hr) / (requests_per_hr_at_saturation x API_cost_per_request)

For the batch case, a rented RTX 5090 at 4,570 tokens/s serves about 16,450 requests of 1,000 output tokens an hour. The table shows the share of that capacity you must actually use, around the clock, before renting wins.

For an owned card, divide its fixed monthly cost (about $62 for a 5090 with idle draw) by your API cost per million tokens. In the solo mix above, that works out to about 620M tokens a month versus SiliconFlow, about 290M versus Bedrock and about 20M versus Sonnet 5.5. Those are the volumes at which the card pays for itself, and they are far above what one interactive user generates.

FAQ