Private LLM or cloud API: how to work out which one is cheaper for you

Fixed cost against variable cost

A self-hosted or private LLM on your own GPU server is mostly a fixed cost, paid whether the hardware is busy or idle, while a cloud API is a variable cost that scales with the tokens you send.

Occasional and irregular use is a variable-cost case, which favours an API; steady, high and predictable throughput is a fixed-cost case, which favours your own hardware. The cases that need a decision sit between the two, and there the arithmetic has to be done properly.

Costs that price comparisons leave out

Comparisons usually set the price of a server against the price of a million tokens and go no further. The four costs below are why real deployments cost more than such a comparison shows.

1. Idle capacity. For a server used only in office hours, idle time is most of the week. A GPU server is paid for whether or not it is answering requests, and internal workloads are almost never evenly distributed: they follow working hours in one time zone, with a dip at lunchtime. A server used eight hours a day on weekdays is busy roughly 24% of the week, 40 of 168 hours. The effective cost per token is the hardware cost divided by the tokens you put through it, not by the tokens it could serve at full load.

2. Power and cooling. A loaded GPU server draws kilowatts, and even idle it draws hundreds of watts. In most of Europe that electricity cost is not a small one, and the cooling is sized for the peak whatever the utilisation. If the rack needs containment or more airflow to take the load, that is an additional facilities project; the power and cooling arithmetic is in a separate article.

3. Engineering time. Someone has to keep the model serving stack running, apply driver and framework updates, deal with an inference server that stops responding at night, and re-run benchmarks when a better model is released. Business cases often set this cost to zero, but it recurs for as long as the platform runs.

4. Model churn. With an API, the current model comes with the price, and the provider retires older versions on its own schedule. With your own hardware you decide when to change models, and each change means re-testing everything downstream. That control is valuable in a regulated process, where a fixed model does not change under your evaluations; if you want to stay on current models, the re-testing becomes a recurring cost.

How to calculate the break-even with your own numbers

The calculation needs four inputs. All four can be measured, and it is worth measuring them before any hardware conversation.

Tokens per interaction. Take fifty real requests from the use case you intend to run. Measure input and output separately: they price differently on every API and they load the GPU differently. A retrieval-augmented question with six document chunks in the prompt is a very different shape from a chat message.

Interactions per day, and their distribution. The hardware is sized by how the requests are spread over the day more than by their total. Two hundred requests spread across the day and two hundred arriving in the ten minutes after a morning report are the same total but need very different hardware.

Concurrency at the peak. This is the number of requests that overlap, and it sets the GPU memory you need, because every simultaneous conversation holds its own KV cache; the formulas are in how much VRAM an LLM needs.

Latency tolerance. The question is whether a user is waiting for the answer. An interactive assistant and an overnight batch job over a document archive are sized very differently, and the batch job is where owned hardware has the clearest advantage: it can run at night and fill the idle hours that otherwise raise your cost per token. Compare it with the provider’s batch or asynchronous rate, though, not with the interactive rate: as of October 2026, OpenAI, Anthropic, Google and Mistral each publish a lower rate for batch processing on their pricing pages.

With those four inputs you can calculate both sides. The API side is tokens times the published rate. The owned side is the total cost of the server over its life, plus power, plus a realistic allowance for engineering time, divided by the tokens you will put through it. We publish no prices here, because API rates and hardware prices both change. Put your current hardware quote and your current API spend into the same spreadsheet. For the owned side you also need the tokens per second the server delivers on your model: the published results in our RTX PRO 6000 benchmarks article give the order of magnitude, and a load test with your own model and prompt lengths gives the figure to divide by.

If you would rather have the owned side worked out for you, our Private AI/ML service includes a TCO calculation against cloud GPUs before anything is bought. Tell us your current API spend and the model you use.

When data residency decides before cost

For a good number of European organisations the cost comparison never happens, because a constraint settles it earlier:

Personal data leaving the EU or EEA needs a transfer mechanism under Chapter V of the GDPR. Whether a given API meets that depends on where it runs and under whose contract. Some providers offer processing in EU regions or zero-data-retention terms, in some cases only on request or for approved customers. As of October 2026 the EU-US Data Privacy Framework still covers certified providers, while an appeal against it is pending at the Court of Justice of the EU, so read the provider’s actual terms rather than assuming. Some sectors have their own supervisory expectations on top. Contracts with your own customers can forbid such transfers, so check them before choosing an API.

Internal policy can rule out an API without any law involved: pricing models, unreleased designs, deal pipelines and source code often may not leave the company’s network.

When one of these applies, size the smallest configuration that carries the load. By our users-per-card rule, as applied in our company-size sizing guide, one 96 GB RTX PRO 6000 holds about 19 conversations of gpt-oss-120b at a 32K context with a 16-bit cache, so the number of cards follows from how many conversations run at the same moment.

If residency settles it, our Private AI/ML service deploys the models so that nothing goes to public APIs unless you enable that for a specific task. Describe the workload and its users in the form below, and we size the configuration for it.

Which workloads run privately and which on an API

A common split runs along data classes:

We recommend prototyping on an API to find out whether the use case is real, measuring the four inputs above while you do it, and buying hardware only for the workloads that turn out to have steady volume or a residency requirement. When hardware is bought before the use case is proven, it can end up busy for an hour or two a day, under 10 per cent of the time.

What we do

We supply the hardware for the owned side: NVIDIA DGX Spark, RTX PRO Blackwell and H200 NVL GPUs, and AI servers built to order, such as the four-card RTX PRO 6000 Server Edition inference node. Our engineering partner Vixen.UNO builds and supports the platform on it, under one European contract with Eurokommerz. If the model has to run on-premise under your control, private AI/ML describes how that engagement runs; if EU hosting is enough, the same engagement can run the models on dedicated hardware in a Tier III data centre in Lithuania. If your numbers point to an API, we will say so rather than quote a server that would sit idle most of the day.

FAQ