A private AI pilot: what to measure, how to measure it, and how the numbers size production hardware

A pilot is a measurement, not a demo

A private LLM pilot has two jobs: to show whether the model answers real questions well enough, and to produce the numbers that size production hardware. Both need the metrics, and a pass mark for each, written down before the first user logs in. On our private AI/ML service that list belongs to the paid technical assessment by our engineering partner Vixen.UNO, with the solution architecture and the model and GPU selection.

Not every number carries over. Demand does: how often requests arrive, how long prompts and answers are and how much context they need, counted in the model’s own tokens. Quality does, if production runs the same model at the same precision. Latency and throughput do not: they belong to the GPU, the engine version and the serving settings they were measured with.

The numbers to collect

Count requests, not people. The figure that sizes memory is concurrency: requests being processed at the same instant, each holding its own KV cache on the GPU. Next to it, record the arrival rate at the busiest hour, the prompt and answer length of every request as a distribution (median and 95th percentile, not a mean) and the context each request needs, prompt plus answer. RAG prompts carry retrieved passages, chat prompts the conversation so far. vLLM publishes the raw material in Prometheus format at /metrics on its API port.

vLLM 0.30.0, released 22 September 2026; definitions from its source code and metrics design documentation. Counters appear with the suffix _total; histograms add _bucket, _sum and _count.

The histograms are coarse: token counts are bucketed in a 1, 2, 5 series (1,000, 2,000, 5,000 tokens and so on) and inter-token latency from 10, 25, 50, 75 and 100 ms upwards, and Prometheus notes that a percentile’s error “is limited by the width of the bucket”; keep exact per-request counts in the gateway log. Gauges are sampled only at each scrape, so short peaks slip through, whereas the increase of vllm:e2e_request_latency_seconds_sum over the busiest hour, divided by 3,600, is the mean number of requests in flight in that hour, queued ones included and cancelled ones not, by our arithmetic. And a lasting queue or any preemption means the pilot hit its own limit: demand was higher than the card let through.

Latency, defined exactly

Tools define latency differently; note the definition with every figure.

Time to first token (TTFT) is, in NVIDIA’s NIM benchmarking guide, the time “from query submission to the first received token”, including queuing, prefill and network latency; longer prompts raise it. vLLM’s histogram starts later, when tokenisation begins in its frontend, so it leaves out the network and any gateway in front.

Inter-token latency (ITL) is, for NVIDIA, the average time between consecutive tokens, “also known as time per output token (TPOT)”, and “tools differ on whether TTFT is included in the average”. AIPerf leaves it out: end-to-end latency less TTFT, divided by the output tokens less one. vllm bench serve reports that per-request average as TPOT and every gap between outputs as ITL, as vLLM’s server histogram does; the average hides stalls, the gaps show them. Per-user speed approaches 1 ÷ ITL for long answers, and AIPerf reports it that way: 50 ms is 20 tokens per second.

End-to-end latency runs from submission to the last token. Errors need their own count: failures, timeouts and cancellations at the gateway, and the finished_reason label on vLLM’s request counter, where length marks an answer cut off at its token or context limit. A request the client cancels does not reach that counter in vLLM 0.30.0.

Targets come from the use case: put them in the pilot plan, and both load-test tools below report goodput, the requests per second that meet every target. For reference, MLPerf® Inference v6.0 holds Llama 2 70B to 2 s to first token and 200 ms per output token in its server scenario, and to 450 ms and 40 ms in its interactive one, at the 99th percentile.

The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

Quality decides the model and the precision

Quality is measured with an evaluation set: real questions from its future users, each with an accepted answer and, for RAG, the passage it should come from; our RAG guide suggests fifty to a hundred. Score the retrieval hit rate, how often the right passage is among those retrieved, separately from the answers: a low one is a search problem no GPU fixes. Count the answers that ended on length too, because truncated answers read as wrong ones.

The same set settles the precision, which moves the hardware most. NVIDIA’s NVFP4 checkpoint of Llama 3.3 70B takes 39.8 GiB against 67.7 GiB in FP8, and its model card reports MMLU at 81.1 against 83.3 for BF16; only the evaluation set, run in the production precision of weights and KV cache, shows whether that matters. Run it again on the production stack: vLLM’s documentation warns that by default it trades reproducibility for performance, and that even with the right settings results reproduce only on the same hardware and the same vLLM version.

Load tests: vllm bench serve and AIPerf

Metrics show what the users did; a load test shows what a given GPU does at a chosen load.

vllm bench serve comes with vLLM. Its default dataset is random prompts, 1,024 tokens in and 128 out; --dataset-name custom replays your own prompts, one JSON line per request with a prompt field, and --custom-output-len sets each answer’s token limit, 256 by default; --ignore-eos runs every answer to it. --max-concurrency caps the requests in flight; without it the default request rate, inf, sends everything at once. The report gives TTFT, TPOT and ITL (mean, median, 99th percentile), throughput and peak concurrency, counted per second; --goodput counts the requests that met millisecond targets for TTFT, TPOT or end-to-end latency.

AIPerf is NVIDIA’s current tool: NVIDIA called it “the designated successor to GenAI-Perf” in September 2026, and GenAI-Perf’s documentation says it “is being phased out”. It installs with pip install aiperf (release 0.12.0 as of September 2026) and runs as aiperf profile against OpenAI-compatible endpoints, with --streaming for TTFT and ITL. --concurrency holds a fixed number of requests in flight, --custom-dataset-type single_turn with --input-file replays JSONL lines with text and an optional output_length, that request’s token limit, so each pilot prompt is capped at its measured answer length, and --goodput takes targets such as TTFT and ITL.

NVIDIA’s benchmarking guide prefers concurrency to request rate “for most benchmarks”, since with a rate “outstanding requests can grow without bound” once arrivals outrun the system. Sweep from one request to the measured peak and beyond, and record the engine version and settings with each result: in vLLM a smaller max_num_batched_tokens gives better ITL, a larger one better TTFT.

Which hardware to pilot on

Our first-project comparison covers the choice; for measurement, the physics sets one rule: generating tokens at low batch is bound by memory bandwidth and processing prompts by compute, so latency measured on one GPU type does not carry over to another.

DGX Spark suits functional and quality pilots: its 128 GB of unified memory holds Llama 3.3 70B in FP8 with room for cache, but at 273 GB/s its decode ceiling for that model is 3.9 tokens per second, against 22.6 on an RTX PRO 6000 Server Edition and 68.0 on an H200 NVL (bandwidth divided by the 70.6 GB each token reads, by our arithmetic). One RTX PRO 6000 has the GB202 GPU of an RTX PRO 6000 production node, but the Workstation and Max-Q editions run at 1,792 GB/s and the Server Edition at 1,597 GB/s, so a workstation pilot overstates a rack node’s decode speed by about 12 per cent. Pilot on the H200 NVL when production will be H200 NVL: it runs FP8 but has no FP4 arithmetic, and NVIDIA lists its NVFP4 checkpoint of Llama 3.3 70B for Blackwell.

On average, requests in flight equal the arrival rate times the time each request takes, so a slower pilot shows more of them for the same traffic. Carry over the arrival rate and the token lengths, not the concurrency, and run the load test on the production GPU type before scaling out.

From pilot numbers to production hardware

Take our 70B worked example as the pilot’s result: Llama 3.3 70B in FP8 with an FP8 KV cache (--kv-cache-dtype fp8), and demand that, scaled to the production user count, reaches 30 requests in flight at the busiest moment, each within 8,192 tokens (example values). The weights are NVIDIA’s checkpoints, 67.7 GiB in FP8 and 39.8 GiB in NVFP4; the cache is 2 × 80 layers × 8 key/value heads × 128 values × 1 byte, 160 KiB per token or 1.25 GiB per session, 2.5 GiB with a 16-bit cache. We plan as in our guide to users per RTX PRO 6000: 90 per cent of what the driver reports (95.6 GiB on a 96 GB card, 140.4 GiB on an H200 NVL), less about 3 GiB of overhead per card, less the weights; vLLM’s default has been 0.92 since release 0.20.0, CUDA graphs included.

Our arithmetic on the basis above, from NVIDIA’s bandwidth figures, checkpoint sizes and the model’s configuration: RTX PRO 6000 Server Edition at 1,597 GB/s, two-card rows in FP8, sessions of a full 8,192 tokens with an FP8 KV cache. Decode ceiling for one stream: bandwidth ÷ 70.6 GB read per token in FP8, 40.6 GB in NVFP4; a split model runs at 22.6 as pipeline stages, up to 45.2 with tensor parallelism before the traffic between the cards.

Against 30 sessions, one card in FP8 falls short at 12, or 6 with a 16-bit cache; two copies on two cards hold 24, because each stores the weights again, while one copy split across both holds 78. One card in NVFP4 holds 34, if the evaluation set passed in NVFP4, and the H200 NVL holds 44. Real sessions rarely fill 8,192 tokens, and the 3 GiB of overhead is our estimate; vLLM prints the exact KV cache size and maximum concurrency at startup, as that guide shows.

A 40 ms per-token target, the interactive limit of MLPerf Inference, is 25 tokens per second: above the 22.6 ceiling of a Server Edition card reading all FP8 weights for each token, so one card, two copies and a pipeline split fail on speed before any load, unless speculative decoding, which vLLM offers to cut inter-token latency in memory-bound workloads at low to medium load, changes the picture. Ceilings are upper bounds no system reaches, so the last step is the load test: the pilot’s prompts at 30 concurrent requests on the candidate, with the targets as goodput.

What we supply

Eurokommerz supplies DGX Spark, the RTX PRO 6000 Blackwell in all three editions and the H200 NVL across the EU, with manufacturer warranty, on one EU contract and invoice. They come singly or in AI servers built to order, where a starter configuration with two RTX PRO 6000 Server Edition cards is listed for pilots and RAG assistants. The pilot itself is AI/ML Integration, delivered by the team of our engineering partner Vixen.UNO under a contract with Eurokommerz. As the private AI/ML page describes, a first call free of charge ends with two or three scenarios, the paid technical assessment delivers the pilot plan and its metrics at a price fixed before work begins, and the pilot’s results decide the scaling, with support under an agreed SLA.

FAQ