From AI pilot to production: scaling private AI hardware from one card to eight

What changes in the hardware from AI pilot to production

Between an AI pilot and production the hardware gets a different job. A pilot, often called a proof of concept (PoC), runs one model for a handful of testers on one or two cards. Production serves many requests at the same moment, keeps running when a server fails, holds more than one model, is monitored and backed up, may need licences the pilot did not need, and draws power that the room has to supply. A workable path grows by card count: one or two cards for the pilot, four cards for a department, eight cards or two four-card nodes for the whole company. The figures measured in the pilot decide which step is the right size.

summary of the sections below; licence wording from NVIDIA's NIM FAQ and NVIDIA's AI Enterprise licensing guide (last updated 2 September 2026), read 12 November 2026.

What the pilot must measure for LLM production sizing

A pilot should hand over three figures. The first is tokens per request: how long the prompts are, retrieved passages included, and how long the answers. The second is concurrency, the peak of requests in flight in the busiest hour. The third is the context length you will declare in production, because every count below is a count of conversations at the full declared length.

vLLM publishes the load figures itself: requests in flight (vllm:num_requests_running), the queue (vllm:num_requests_waiting), the share of KV cache in use (vllm:kv_cache_usage_perc) and the time to first token, read at the 95th percentile. Scale the measured peak to the user count of the next step. In the example of our rollout plan for 1,000 employees, 190 users with a peak of 12 requests in flight predict about 34 for 540 users and about 63 for 1,000. Arrival rate, token lengths and answer quality carry over from the pilot; latency carries over only on the same GPU type, engine version and settings, as our guide to what a private AI pilot should measure explains.

Without pilot data, sizing has to start from assumptions. Our guide by company size assumes that 40 per cent of employees are active in the busiest hour and send 6 requests an hour, that each request takes 30 seconds, and it applies a peak factor of 2. That comes to about 4 requests in flight per 100 employees. These figures are example values, and the logs of the pilot replace them.

Scaling LLM infrastructure by card count

Every count in this section follows one rule: weights and KV cache together get 0.9 × the memory the driver reports, less 3 GiB per card. By this estimate a 48 GB card has 40.0 GiB, a 32 GB card 25.7 GiB and a 96 GB card 83.0 GiB. The driver shows 95.6 GiB on the 96 GB card; for the smaller cards we scale from that value, which is an assumption, and the driver on the delivered card decides.

our estimates by the rule above, as published in our guides to these builds: concurrent conversations of 8,192 tokens at full length with an FP8 KV cache (0.625, 1.0 and 1.25 GiB per conversation; weights 15.2, 32.0 and 67.7 GiB). TP = tensor parallelism, one copy of the model split over that many cards. The counts are memory arithmetic and say nothing about throughput.

Memory does not pool across cards. A model that fits on one card runs as one copy per card with no traffic between the cards, and a failed card takes out only its own copy. A model larger than one card is split by the serving engine over PCIe, and every split costs traffic between the cards. Fewer copies hold more conversations, because each copy stores its own weights. The order is therefore: quantise first, then add copies behind a load balancer, and split only when the model does not fit.

The pilot on one or two cards

A workstation with one RTX PRO 5000 Blackwell 48 GB card carries a 14B or a 32B model in FP8 for a pilot team, and a second card doubles the copies or lets a 32B model serve 48 conversations as a pair. The card is described in our 48 GB and 72 GB comparison, the build on the page about the workstation with RTX PRO 5000 48 GB cards. A pilot that needs a 70B model in FP8 is better started on 96 GB cards: its 67.7 GiB of weights fit one 96 GB card and no smaller RTX PRO Blackwell card, and split over two 48 GB cards they leave room for only 9 conversations. The starter configuration on our AI servers page, two RTX PRO 6000 Blackwell Server Edition cards with two AMD EPYC processors and 384 GB of memory, has 166.1 GiB by our rule and, counted as in the table (8,192 tokens, FP8 KV cache), holds 24 conversations of that model as two copies or 78 as one copy split over both cards. Our worked sizing example for a 70B model follows this case from the weights to the wall socket, and which GPU for a first AI project compares the starting points.

A department on four cards

Four RTX PRO 4500 Blackwell Server Edition cards, passive single-slot cards with 32 GB and 165 W each (specifications), fit a 2U server. Four copies of the 14B model hold 64 conversations. At the example values 1,000 employees produce a peak of about 40 requests in flight, so the build has room to spare for them; our page on the server with four RTX PRO 4500 Server Edition cards gives the details. For larger models the step is the inference node of our AI servers page: four RTX PRO 6000 Server Edition cards with 332.2 GiB by our rule, which by the same count hold 48 conversations of the 70B model as four copies, 156 as two pairs and 211 as one copy over four cards.

The company on eight cards or two nodes

Eight RTX PRO 4500 Server Edition cards hold 128 conversations of the 14B model as eight copies, more than the peak of about 80 that the example values give for 2,000 employees; the server with eight of these cards is the build. The 70B model runs there only as split copies. By default vLLM disables its custom all-reduce on more than two GPUs that are connected only by PCIe, so a pair keeps the faster path and a split over four or eight cards does not. We found no consistent published measurement of tensor-parallel scaling on four or eight RTX PRO 4500 or 5000 cards, so the counts are memory figures and say nothing about speed. Which models these servers suit is the subject of H200 NVL alternatives for LLM inference. The other route to company size is two four-card nodes, which the next section describes.

You can order the server alone, the setup of the models and the assistant alone, or both together: the server delivered with everything installed and configured; send us the figures from your pilot and we suggest the best solution for your tasks and budget.

The second node and availability in production

Production needs a target in writing. Google's book "Site Reliability Engineering" calls it a service level objective, "a target value or range of values for a service level that is measured by an SLI", and its availability table shows what each target allows.

Google, "Site Reliability Engineering" (O'Reilly, 2017), Appendix A "Availability Table", read 12 November 2026; the table assumes no planned downtime.

A single server has to stop for a driver update or a repair, and the target decides how often and for how long that may happen. If one node alone reaches 99 per cent and the two fail independently, both are down at the same moment for 0.01 × 0.01 = 0.0001 of the time, so by our estimate the pair reaches 99.99 per cent. A shared power feed, a shared switch or an update applied to both at once breaks that assumption. The figure shows the arithmetic for a target of your own; it is no service level that we or a maker state.

The sizing is N+1 at the peak: each node runs every model of the service and carries the whole peak alone. In the example of our guide to LLM high availability with two nodes, 2,000 employees produce a peak of 80 requests in flight with gpt-oss-120b at a 32K context. Each of two nodes with four RTX PRO 6000 cards holds about 186 conversations with two copies split over card pairs and 76 with one copy per card (our estimates with a 16-bit KV cache). That guide adds: "Throughput at 80 concurrent requests is not measured, so ask for a load test before you order." The GPUs are only part of it. The embedding model, the reranker, the vector index, the chat front end with its database and the gateway each need a second instance. A failover loses the requests in flight on the failed node, and the surviving node starts without the prefix cache. Updates then run one node at a time. Our rollout guide puts the moment plainly: add the second server before the whole company depends on the assistant. Whether one eight-card server or two four-card servers is the better purchase is weighed in one 8-GPU server or two 4-GPU servers, and the power supplies inside each node in PSU redundancy in GPU servers.

More than one model, monitoring and backup

Production rarely stays with one model. An assistant that answers from documents adds an embedding model and a reranker, and departments ask for a second chat model. Each model is its own vLLM instance, on whole cards, on MIG instances or on shares of one card, as serving several models on one GPU server shows. Add up the memory of every model before the card count is fixed.

Monitoring becomes a duty with thresholds: the four vLLM figures named above, read in the busiest hour, and a review that decides on the next server early, because capacity on your own premises is a purchase. Our guides to LLM monitoring and GPU capacity planning cover both.

Backup changes as soon as the platform holds something unique. Base weights can be downloaded again only while that revision exists; adapters and fine-tuned weights exist nowhere else. A vector store needs its own consistent backup, for example pg_dump or pg_basebackup for pgvector. A running virtual machine with a passthrough GPU cannot be snapshotted. The restore test compares the SHA-256 values of the weight files, runs a fixed evaluation set and checks the top results of fixed queries. Backing up an AI server has the procedure.

NVIDIA AI Enterprise at the step to production

A pilot can run on development terms. NVIDIA's NIM FAQ, posted by NVIDIA in its developer forum on 11 September 2024 and read on 12 November 2026, lets members of the Developer Program use NIM microservices for research, development and experimentation on "up to 16 GPUs on any infrastructure". The same FAQ says: "Using NIM in production requires an NVIDIA AI Enterprise license." It counts as production "any use of NIM for purposes other than development, testing, research or evaluation". NVIDIA's licensing guide, last updated on 2 September 2026, states that "NVIDIA AI Enterprise software is licensed on a per-GPU basis", so a four-card node that serves NIM to staff needs four licences.

Not every production stack needs the licence. A vLLM stack on RTX PRO cards runs without NVIDIA AI Enterprise, compute vGPU profiles in virtual machines need it per GPU, and each H200 NVL card includes a five-year subscription. Our article on NVIDIA AI Enterprise licensing sets out the cases. Where licences are needed, they come on the same invoice as the hardware.

Power and room at each stage

our estimates as published in our guides to these builds: board power of the cards plus processors plus 15 per cent (workstation: one 350 W processor; RTX PRO 4500 servers: two processors of 280 W; RTX PRO 6000 servers: two processors of 200 W with two cards, of 280 W with four and of 360 W with eight); a 230 V circuit fused at 16 A carries 3.68 kW and is planned at 80 per cent, 2.94 kW. A measurement on the delivered system confirms the estimate.

By these estimates the pilot workstation, the servers with RTX PRO 4500 Server Edition cards and the two-card RTX PRO 6000 server stay within one 16 A circuit. From four 600 W cards the feed has to be planned. For an eight-card node Dell's datasheet gives about 8 kW as an indicative figure, so plan the feed on the maker's figure for the exact configuration. By our estimates a 10 kW rack takes one eight-card server with 600 W cards, two four-card servers or up to five two-card servers, and a 400 V 32 A three-phase feed delivers about 22 kW, of which we plan 17.7 kW. The site survey for GPU servers lists what to check in the room, and GPU rack power and cooling explains the airflow.

We check the rack, the power and the airflow before we quote; tell us the rack position and the feed, and we size the next step to them. The first facts to send are listed in our private AI sizing request, and where the order of purchases over a budget year matters, the AI hardware budget for 2027 continues from here.

FAQ