A private AI pilot: what to measure, how to measure it, and how the numbers size production hardware

Two jobs of a pilot: quality and sizing data

A private LLM pilot has two jobs: to show whether the model answers real questions well enough, and to produce the numbers that size production hardware. Both need the metrics, and a pass mark for each, written down before the first user logs in. In our private AI/ML service, that list is part of the paid technical assessment, together with the solution architecture and the model and GPU selection. Tell us which process you want to pilot and our team will help.

Not every number carries over to production. Demand carries over, meaning how often requests arrive, how long prompts and answers are and how much context they need, counted in the model’s own tokens. Quality carries over if production runs the same model at the same precision. Latency and throughput do not, because they depend on the GPU, the engine version and the serving settings used for the measurement.

Which numbers to collect during the pilot

Count requests rather than users. The figure that sizes memory is concurrency: requests being processed at the same instant, each holding its own KV cache on the GPU. Next to it, record the arrival rate at the busiest hour, the prompt and answer length of every request as a distribution (median and 95th percentile, not a mean) and the context each request needs, prompt plus answer. RAG prompts carry retrieved passages, chat prompts the conversation so far. vLLM publishes the raw material in Prometheus format at /metrics on its API port.

vLLM 0.30.0, released 22 September 2026; definitions from its source code and metrics design documentation. The metric names are the same in 0.31.0 of 5 October 2026; check /metrics again after an upgrade (0.32.0 was released on 18 October 2026). Run 0.31.0 or later, which fixes the security advisory GHSA-h3rc-6mm3-gc2m, preferably the current release, 0.33.0 of 2 November 2026 (as of November 2026), whose CUDA 13.0 builds need an R580 or newer driver. Counters appear with the suffix _total; histograms add _bucket, _sum and _count.

The histograms are coarse: token counts are bucketed in a 1, 2, 5 series (1,000, 2,000, 5,000 tokens and so on) and inter-token latency from 10, 25, 50, 75 and 100 ms upwards, and Prometheus notes that a percentile’s error “is limited by the width of the bucket”; keep exact per-request counts in the gateway log. Gauges are sampled only at each scrape, so short peaks are missed. The increase of vllm:e2e_request_latency_seconds_sum over the busiest hour, divided by 3,600, does not have that gap: by our arithmetic, it is the mean number of requests in flight in that hour, queued ones included and cancelled ones not. A lasting queue or any preemption means the pilot reached the limit of its hardware: demand was higher than the card could serve.

Latency metrics: TTFT, ITL, TPOT and end-to-end latency

Tools define latency differently; note the definition with every figure.

Time to first token (TTFT) is, in NVIDIA’s NIM benchmarking guide, the time “from query submission to the first received token”, including queuing, prefill and network latency; longer prompts raise it. vLLM’s histogram starts later, when tokenisation begins in its frontend, so it leaves out the network and any gateway in front.

Inter-token latency (ITL) is, for NVIDIA, the average time between consecutive tokens, “also known as time per output token (TPOT)”, and “tools differ on whether TTFT is included in the average”. AIPerf excludes it and computes end-to-end latency less TTFT, divided by the output tokens less one. vllm bench serve reports that per-request average as TPOT and every gap between outputs as ITL, as vLLM’s server histogram does; the individual gaps show stalls that the average hides. Per-user speed approaches 1 ÷ ITL for long answers, and AIPerf reports it that way: 50 ms is 20 tokens per second.

End-to-end latency runs from submission to the last token. Errors need their own count: failures, timeouts and cancellations at the gateway, and the finished_reason label on vLLM’s request counter, where length marks an answer cut off at its token or context limit. A request the client cancels does not reach that counter in vLLM 0.30.0.

Targets come from the use case: put them in the pilot plan, and both load-test tools below report goodput, the requests per second that meet every target. For reference, MLPerf® Inference v6.1 holds Llama 2 70B to 2 s to first token and 200 ms per output token in its server scenario, and to 450 ms and 40 ms in its interactive one, at the 99th percentile. Example targets for chat, RAG and agents are in our article on LLM latency targets, and dashboards for these metrics in LLM monitoring with vLLM metrics.

The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

Quality decides the model and the precision

Quality is measured with an evaluation set: real questions from its future users, each with an accepted answer and, for RAG, the passage it should come from; our RAG guide suggests fifty to a hundred. Score the retrieval hit rate, how often the right passage is among those retrieved, separately from the answers: a low hit rate is a search problem that more GPU capacity does not fix. Count the answers that ended on length too, because truncated answers read as wrong ones.

The same set decides the precision, which has the largest effect on the hardware. NVIDIA’s NVFP4 checkpoint of Llama 3.3 70B takes 39.8 GiB against 67.7 GiB in FP8, and its model card reports MMLU at 81.1 against 83.3 for BF16; only the evaluation set, run in the production precision of weights and KV cache, shows whether that matters. Since vLLM 0.32.0 the KV cache can also be stored in NVFP4 on RTX PRO 6000 cards, about 4.5 bits per value instead of FP8’s 8, and that precision needs the same test. Run it again on the production stack: vLLM’s documentation warns that by default it trades reproducibility for performance, and that even with the right settings results reproduce only on the same hardware and the same vLLM version.

Load tests: vllm bench serve and AIPerf

Metrics show what the users did; a load test shows what a given GPU does at a chosen load.

vllm bench serve comes with vLLM. Its default dataset is random prompts, 1,024 tokens in and 128 out; --dataset-name custom replays your own prompts, one JSON line per request with a prompt field, and --custom-output-len sets each answer’s token limit, 256 by default; --ignore-eos runs every answer to it. --max-concurrency caps the requests in flight; without it the default request rate, inf, sends everything at once. The report gives TTFT, TPOT and ITL (mean, median, 99th percentile), throughput and peak concurrency, counted per second; --goodput counts the requests that met millisecond targets for TTFT, TPOT or end-to-end latency.

AIPerf is NVIDIA’s current tool: GenAI-Perf’s documentation says it “is being phased out” and tells users “For new performance benchmarking needs, please use AIPerf instead”. It installs with pip install aiperf (release 0.13.0 as of October 2026) and runs as aiperf profile against OpenAI-compatible endpoints, with --streaming for TTFT and ITL. --concurrency holds a fixed number of requests in flight. --custom-dataset-type single_turn with --input-file replays JSONL lines with text and an optional output_length, that request’s token limit, so each pilot prompt is capped at its measured answer length. --goodput takes targets such as TTFT and ITL.

NVIDIA’s benchmarking guide prefers concurrency to request rate “for most benchmarks”, since with a rate “outstanding requests can grow without bound” once arrivals outrun the system. Sweep from one request to the measured peak and beyond, and record the engine version and settings with each result: in vLLM a smaller max_num_batched_tokens gives better ITL, a larger one better TTFT.

Which hardware to pilot on

Our first-project comparison covers the choice of GPU. For measurement, one rule follows from the hardware: generating tokens at low batch is bound by memory bandwidth and processing prompts by compute, so latency measured on one GPU type does not carry over to another.

A 128 GB DGX Spark suits functional and quality pilots: its unified memory holds Llama 3.3 70B in FP8 with room for the cache. At 273 GB/s, however, its decode ceiling for that model is 3.9 tokens per second, against 22.6 on an RTX PRO 6000 Server Edition and 68.0 on an H200 NVL (bandwidth divided by the 70.6 GB each token reads, by our arithmetic). One RTX PRO 6000 has the GB202 GPU of an RTX PRO 6000 production node, but the Workstation and Max-Q editions run at 1,792 GB/s and the Server Edition at 1,597 GB/s, so a workstation pilot overstates a rack node’s decode speed by about 12 per cent. Pilot on the H200 NVL when production will be H200 NVL: it runs FP8 but has no FP4 arithmetic, and NVIDIA lists its NVFP4 checkpoint of Llama 3.3 70B for Blackwell.

On average, requests in flight equal the arrival rate times the time each request takes, so a slower pilot shows more of them for the same traffic. Carry over the arrival rate and the token lengths, not the concurrency, and run the load test on the production GPU type before scaling out.

We supply all three GPU types discussed here, so the pilot and production can run on the same type. Write to us with the arrival rate and token lengths your pilot measured, and we size the production hardware from them.

From pilot numbers to production hardware

Take our 70B worked example as the pilot’s result: Llama 3.3 70B in FP8 with an FP8 KV cache (--kv-cache-dtype fp8), and demand that, scaled to the production user count, reaches 30 requests in flight at the busiest moment, each within 8,192 tokens (example values). The weights are NVIDIA’s checkpoints, 67.7 GiB in FP8 and 39.8 GiB in NVFP4; the cache is 2 × 80 layers × 8 key/value heads × 128 values × 1 byte, 160 KiB per token or 1.25 GiB per session, 2.5 GiB with a 16-bit cache. We plan as in our guide to users per RTX PRO 6000: 90 per cent of what the driver reports (95.6 GiB on a 96 GB card, 140.4 GiB on an H200 NVL), less about 3 GiB of overhead per card, less the weights; vLLM’s default has been 0.92 since release 0.20.0, CUDA graphs included.

Our arithmetic on the basis above, from NVIDIA’s bandwidth figures, checkpoint sizes and the model’s configuration: RTX PRO 6000 Server Edition at 1,597 GB/s, two- and four-card rows in FP8, sessions of a full 8,192 tokens with an FP8 KV cache. Decode ceiling for one stream: bandwidth ÷ 70.6 GB read per token in FP8, 40.6 GB in NVFP4; a split model runs at 22.6 as pipeline stages, up to 45.2 with tensor parallelism before the traffic between the cards. The RTX PRO 6000 has no NVLink, so a split copy exchanges data over PCIe, and vLLM disables its custom all-reduce for more than two GPUs without NVLink, which favours two pairs over one four-way copy.

Against 30 sessions, one card in FP8 falls short at 12, or 6 with a 16-bit cache; two copies on two cards hold 24, because each stores the weights again, while one copy split across both holds 78. One card in NVFP4 holds 34, if the evaluation set passed in NVFP4, and the H200 NVL holds 44. Real sessions rarely fill 8,192 tokens, and the 3 GiB of overhead is our estimate; vLLM prints the exact KV cache size and maximum concurrency at startup, as that guide shows.

A 40 ms per-token target, the interactive limit of MLPerf® Inference, is 25 tokens per second. That is above the 22.6 ceiling of a Server Edition card reading all FP8 weights for each token, so one card, two copies and a pipeline split all miss the target before any load is applied. Speculative decoding, which vLLM offers to cut inter-token latency in memory-bound workloads at low to medium load, can change that. Ceilings are upper bounds no system reaches, so the last step is the load test: the pilot’s prompts at 30 concurrent requests on the candidate, with the targets as goodput.

What we supply

We supply the DGX Spark Founders Edition (128 GB), the RTX PRO 6000 Blackwell in all three editions and the H200 NVL across the EU with manufacturer warranty, on one EU contract and invoice. They come on their own or in AI servers built to order, where the starter configuration with two RTX PRO 6000 Server Edition cards is listed for pilots and RAG assistants, and the inference node with four RTX PRO 6000 Server Edition cards for production inference for several teams. If you want the pilot planned and run for you, that is our private AI/ML service, delivered by our engineering partner Vixen.UNO under a contract with Eurokommerz: the first call is free of charge and ends with two or three scenarios, the paid technical assessment produces the pilot plan and its metrics at a price fixed before work begins, and the pilot’s results decide the scaling, with support under an agreed SLA.

FAQ