Private LLM or cloud API: how to work out which one is cheaper for you

Two options that are less different than they look

The choice is usually framed as private versus cloud. In cost terms it is simpler than that: you are choosing between a variable cost that scales exactly with use, and a fixed cost that does not care whether you use it.

That framing tells you the answer immediately for the two extremes. Occasional, bursty, unpredictable use is the definition of a variable cost problem: an API. Steady, high, predictable throughput is the definition of a fixed cost problem: your own hardware. Everything interesting is in the middle, and the middle is where the arithmetic has to be done properly.

The four cost lines nobody counts

Comparisons usually put the price of a server against the price of a million tokens and stop. The four lines below are what make real deployments cost more than the spreadsheet said.

1. Idle capacity. This is the largest by a wide margin. A GPU server is paid for whether or not anyone is asking it anything, and internal workloads are almost never evenly distributed: they happen during working hours, in one time zone, with a lunch dip. A box used eight hours a day on weekdays is available roughly 24% of the week. The effective cost per token is the hardware cost divided by the tokens you actually put through it, not by the tokens it could theoretically serve.

2. Power and cooling. A loaded GPU server draws kilowatts, and even idle it draws hundreds of watts; that is a real line on a real electricity bill, and in most of Europe it is not a small one, while the cooling is sized for the peak whatever the utilisation. If the rack needs containment or more airflow to take the load, that is a facilities project on top; we wrote out the power and cooling arithmetic separately.

3. Engineering time. Somebody keeps the model serving stack running, applies driver and framework updates, handles the night the inference server stops responding, and re-benchmarks when a better model is released. This is the line that is always zero in the business case and never zero in reality.

4. Model churn. An API gives you the current model as part of the price. Owning hardware means you decide when to move, and moving means re-testing everything downstream. That is a feature if you need stability: a fixed model that does not change under your evaluations is worth a lot in a regulated process, and a cost if you want to stay current.

The arithmetic, on your numbers

Four inputs. All four are measurable in a fortnight, and the exercise is worth doing before any hardware conversation.

Tokens per interaction. Take fifty real requests from the use case you actually intend to run. Measure input and output separately: they price differently on every API and they load the GPU differently. A retrieval-augmented question with six document chunks in the prompt is a very different shape from a chat message.

Interactions per day, and their distribution. Not the total, the shape. Two hundred requests spread across the day and two hundred arriving in the ten minutes after a morning report are the same total and completely different hardware.

Concurrency at the peak. How many of those overlap. This sets the GPU memory you need, because every simultaneous conversation holds its own KV cache; the formulas are in how much VRAM an LLM needs.

Latency tolerance. Whether a user is watching. An interactive assistant and an overnight batch job over a document archive have nothing in common in sizing terms, and the batch job is where owned hardware looks best: it can run at night and fill the idle hours that otherwise destroy your cost per token. Compare it against the provider’s batch or asynchronous rate, though, which is typically about half the interactive price, not against the interactive rate.

With those four you can compute both sides honestly: the API side is tokens times published rate; the owned side is total cost of the box over its life, plus power, plus a realistic engineering allowance, divided by the tokens you will genuinely put through it. We deliberately publish no prices here: API rates and hardware prices both move, and a number in an article ages badly. Put your current quote and your current API bill into the same sheet.

When residency decides it first

For a good number of European organisations the cost comparison never happens, because a constraint settles it earlier:

Personal data leaving the EU or EEA needs a transfer mechanism under Chapter V of the GDPR, and whether a given API satisfies that depends on where it runs and under whose contract: the large providers now offer EU-region processing and zero-data-retention terms, and the EU-US Data Privacy Framework covers certified providers, so read the actual terms rather than assuming. Some sectors have their own supervisory expectations on top. Some contracts with your own customers forbid it outright, and that clause is often already signed, sitting in a contract nobody re-reads.

There is also a category of data where the question is not legal but commercial: pricing models, unreleased designs, deal pipelines, source code. Plenty of organisations are simply unwilling to have that leave their own network, and that is a legitimate position that needs no further justification.

When one of these applies, the useful question stops being “which is cheaper” and becomes “what is the smallest thing that does the job”. That is a much easier question, and the answer is often smaller than people expect: a single professional card handles a great deal of internal work.

The answer most estates land on

In practice the common outcome is neither of the pure options. It is a split by data class:

The sequence that works: prototype on an API to find out whether the use case is real, measure the four inputs above while you do it, and buy hardware only for the workloads that turned out to have steady volume or a residency requirement. Buying first and finding the use case afterwards is how GPUs end up at 6% utilisation.

What we do

Eurokommerz supplies the hardware: NVIDIA DGX systems, RTX PRO Blackwell and H200 NVL cards, custom AI servers, and our engineering partner Vixen.UNO designs and runs the platform on top, under one European contract with us. If you want the model inside your own perimeter, private AI/ML describes how that engagement runs; if EU hosting is enough, EU cloud covers Tier-3 capacity in the EU.

We will also tell you when the honest answer is an API. A quote for a server you will use 6% of the time is not a sale we want.

FAQ