Private AI sizing request: the ten facts we need to size an AI server for your LLM users

What a private AI sizing request should contain

A private AI sizing request needs ten facts, and none of them is a GPU model number. We ask for the task in one sentence, the number of people and how many of them work at the same moment, the documents, a model preference if you have one, the length of a typical conversation, the response time you expect, the room, the servers and virtualisation you already run, your security constraints and the date you work towards. From these we work out the GPU memory, the card count and the server around the cards, and we reply within one business day. An estimate is enough wherever a figure is still open.

Readers who want the arithmetic find it in how many users one RTX PRO 6000 can serve and in the worked sizing example for a 70B model. The checklist in hardware terms, from processor cores to PDU outlets, is in what to specify when buying an AI server.

AI server sizing questionnaire in ten facts

our own list; the sections below explain the middle column, and the typical answers are those of the example request further down.

We size AI servers and workstations from these ten answers free of charge; send them through the form below or to request@eurokommerz.at, estimates included, and we reply with a proposed configuration.

People and concurrency, the figure that sizes the GPUs

A GPU serves the requests for which tokens are being generated at this moment. A hundred employees with the assistant open in a browser tab produce far fewer simultaneous generations, and we found no published ratio between headcount and requests in flight. For that reason the request should carry two numbers: the people who get access, and your estimate of how many work with the system at the same time.

Where you have no estimate, we start from the example values of our sizing guide by company size: 40 per cent of employees active in the busiest hour, 6 requests an hour each, 30 seconds per request and a peak factor of 2. That gives about 4 requests in flight per 100 employees, so about 20 for 500 employees and about 80 for 2,000. These are example values and no measurement. A pilot replaces them with the busy-hour peak from its own gateway logs; what a private AI pilot should measure lists the figures to collect.

How the model, the documents and the conversation length size the AI server

Two things occupy GPU memory: the weights of each model, fixed once the model and its precision are chosen, and the KV cache, which grows with every open conversation and with its length. Our sizing rule gives both together 0.9 × the memory the driver reports, less 3 GiB per card. By our estimate that is 25.7 GiB on a 32 GB card, 40.0 GiB on a 48 GB card and 83.0 GiB on a 96 GB card. The figures for 32 and 48 GB are scaled from the 95.6 GiB the driver shows on the 96 GB card; that scaling is an assumption, and the driver on the delivered card decides.

our estimates by the rule above, as published in our sizing guides: concurrent conversations of 8,192 tokens at full length with an FP8 KV cache, which takes 0.625 GiB (Qwen3-14B), 1.0 GiB (Qwen3-32B) and 1.25 GiB (Llama 3.3 70B) per conversation. Memory arithmetic, no speed measurement.

The same 48 GB card holds 39 conversations with a 14B model and 8 with a 32B model, and a 70B model in FP8 fits on a single card only at 96 GB, which is why the model question comes early. If you have no preference, say so. The configuration we propose then names the model size it was calculated for; the selection of the model itself belongs to the technical assessment of our Private AI/ML service. Current models and their memory are listed in LLM hardware requirements by model, their terms in open LLM licences for company use.

An assistant that answers from documents runs an embedding model, a reranker and a vector index beside the chat model. The volume sizes the NVMe storage for the index and the documents. Prompts that carry retrieved pages are longer than plain chat questions, which leads to the fifth fact: the KV cache of a conversation grows in proportion to the tokens it holds, so a full conversation of 32,768 tokens takes four times the cache of one of 8,192. The formula is in how much VRAM an LLM needs.

Response time in the words of your users

Memory decides how many conversations fit; memory bandwidth and the load decide how fast each answer appears. For the first reply we need no figure in tokens per second: tell us whether people read the answer while it is written, or whether results may be collected later. As a reference for live chat, silent reading at 238 words per minute corresponds to about 5.3 tokens per second, a figure our users-per-card guide takes from a 2019 meta-analysis in the Journal of Memory and Language.

No measured token rate is published for the RTX PRO 5000 with 48 GB, so the speed of a proposed build is confirmed in a load test or in the pilot.

Room, existing servers and security constraints

We check the rack, the power and the airflow before we quote, so the request should name the rack or the office the system goes into, the circuit and the cooling. A 230 V circuit fused at 16 A carries 3.68 kW, and we plan it at 80 per cent, which is 2.94 kW. By our estimate a workstation with two RTX PRO 5000 48 GB cards draws about 1.1 kW at the wall, a 2U server with four RTX PRO 4500 Server Edition cards about 1.4 kW and a server with eight of them about 2.2 kW. The estimate adds the board power of the cards and the rated power of the processors (one of 350 W in the workstation, two of 280 W in the servers) and puts 15 per cent on top; a measurement on the delivered system confirms it. Our site survey for GPU servers lists what to note in the room.

Existing servers and virtualisation change the card and the licence. If virtual machines are to share a card through NVIDIA vGPU, the card must be on NVIDIA's vGPU list. Of the cards named here that is the RTX PRO 4500 Server Edition, from vGPU release 20.0, and the RTX PRO 6000 Server Edition, from 19.0; the RTX PRO 5000 with 48 GB is not on the list. Compute profiles for virtual machines are licensed per GPU with NVIDIA AI Enterprise, which our article on NVIDIA AI Enterprise on vSphere covers. A vLLM stack on bare metal needs no NVIDIA AI Enterprise licence to run. We also build on a chassis or with parts you already own, after a compatibility check of the platform, power and cooling.

On security, our service has three deployment options: on your servers, where data never leaves your network; on dedicated hardware in a Tier III data centre in Lithuania; or a hybrid in which public APIs are enabled only by your explicit decision and stay visible in the query log. For the request, three statements are enough: whether anything may leave the network, how access to documents is organised today, and whether queries and answers must be logged. If the documents contain personal data, say so, because your side then decides whether a data protection impact assessment is needed. Technical detail can wait until an NDA is signed.

The date you work towards

Name the date that matters to you and say whether the first step is a pilot or production. We give no general duration for a project; quantities and lead time are confirmed in writing with the quote. If you expect growth, say so: the path by card count is described in from pilot to production, and the order of the work in the steps of an own AI server project.

An example sizing request, written out

The request below is invented and contains everything we need.

Subject: Sizing request for an internal assistant on our documents

1. We are a mechanical engineering company and want an assistant that answers staff questions from our manuals, service reports and contracts.
2. We have 180 employees. About 120 would get access, and we estimate that 6 to 8 work with it at the same moment in the busiest hour. We have no measurements.
3. The documents are about 40,000 PDF and Office files, roughly 60 GB, in German and English.
4. We have no model preference. It should be an open model whose licence allows company use.
5. A typical question is short, and the answer should rest on two or three pages of a document.
6. Answers should begin within a few seconds and appear at reading speed.
7. The system goes into our server room: one 19-inch rack with free units, 230 V circuits fused at 16 A, air cooling.
8. Our other systems run on a vSphere cluster. The AI server may run on bare metal beside it.
9. Nothing may go to public services. Access follows our departments, some documents contain personal data, and we need a log of queries.
10. We want to start with one department as a pilot. The board decides on the company-wide step in March 2027.

This is how we read it. The reading is an illustration and no offer, and every figure in it is our estimate. The example values give 180 employees about 7 requests in flight, so the calculation uses the upper figure of the author's own range, 8. The task needs a chat model with an embedding model, a reranker and an index. Short questions with two or three pages fit a declared context of 8,192 tokens.

For the pilot, a workstation with two RTX PRO 5000 Blackwell 48 GB cards has 80.0 GiB by our rule. A 14B model in FP8 runs as one copy per card with room for 78 conversations of that length in an FP8 KV cache, and a 32B model in FP8 split over both cards holds 48. These counts are for the chat model alone; the embedding model and the reranker take memory of their own. The build is described on our page about the workstation with RTX PRO 5000 48 GB cards, the card in the 48 GB and 72 GB comparison. For the rack, a 2U server with four RTX PRO 4500 Blackwell Server Edition cards has 102.8 GiB. Four copies of the 14B model hold 64 conversations; the 32B model fits only when it is split over two cards, and two such pairs hold 38. Details are in the specifications of the Server Edition and on the page about the server with four of these cards. Both builds stay below 1.5 kW at the wall and fit one 16 A circuit.

A split model exchanges data between the cards over PCIe: NVIDIA's datasheet for the RTX PRO 5000 lists no NVLink, and the Server Edition has no NVLink connector. We found no consistent published measurement of a split model on four RTX PRO 4500 cards, so the pilot measures the speed.

What we send back

The first answer is a proposed configuration: the cards and their count, processors, memory, storage and the power the system needs, with the calculation that led to it. This sizing is free of charge. Configuration and quote reach you within one business day; the quote states price, lead time and warranty terms in writing, with invoicing from Austria. Every server is assembled and tested before delivery and comes with manufacturer warranty; operating system, drivers, CUDA and a container runtime are installed on request.

If the request is about more than a server, the first call of our Private AI/ML service is free as well: we go through your process and data, and you leave with two or three possible solution scenarios. The technical assessment that follows is a paid stage with the solution architecture, the model and GPU selection and a pilot plan with metrics, and its price is fixed before work begins. A TCO calculation against cloud GPUs comes before the purchase.

You can order the hardware alone, the setup of the models and the assistant alone, or both together: the system delivered with everything installed and configured; write to us and we suggest the best solution for your tasks and budget. Where a formal tender takes the place of an e-mail, the questions for the whole project are in Private AI RFP: what to ask vendors.

FAQ