MLPerf® Inference v6.1 for PCIe GPU servers: eight RTX PRO 6000 Server Edition compared

What v6.1 shows for PCIe GPU servers

In MLPerf® Inference v6.1, published on 16 September 2026, the PCIe GPU servers with verified results that we cite are eight-card servers with the NVIDIA RTX PRO 6000 Blackwell Server Edition. On Llama 2 70B in the offline scenario, servers from MiTAC, Cisco, Dell and HPE reached 29,284.8 to 30,034.2 tokens per second, which is 3,661 to 3,754 tokens per second per card by our division. The choice of maker moved the result by less than 3 per cent; power, cooling and service decide the choice of chassis more than the benchmark does.

MLPerf® Inference: Datacenter v6.1, closed division, available category, benchmarks Llama 2 70B (llama2-70b-99.9) and gpt-oss-120b, offline and server scenarios, tokens per second, entries 6.1-0070, 6.1-0013, 6.1-0036, 6.1-0050 and 6.1-0082, each with eight RTX PRO 6000 Blackwell Server Edition; retrieved from mlcommons.org on 24 October 2026; results verified by MLCommons Association. "Not cited" means we did not verify a value for that cell. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

MLCommons' supplemental notes to the round mention further RTX PRO systems: Cisco servers with RTX PRO 4500 cards, four-card Server Edition nodes in Cisco's UCS X-Series and a four-card system from TTA, and Fujitsu's PRIMERGY CDI with eight cards in an external PCIe box. We leave them out of the tables because we have not matched their entry IDs to MLCommons' results table.

How MLPerf Inference measures a server

MLCommons runs MLPerf Inference in two divisions. In the closed division every submitter runs the same model to the same accuracy target, so results can be compared; the open division allows other models. The available category holds systems that are available at publication, as opposed to the preview and research categories. All entries in this article are closed division, available category.

Each benchmark runs in fixed scenarios. Offline hands the system all requests at once and measures throughput without a latency limit. Server sends requests at random intervals and reports the highest throughput at which the 99th percentile of the time to the first token and of the time per output token stays within the benchmark's limits. Interactive is the server scenario with tighter limits.

MLCommons MLPerf Inference rules, datacenter benchmark table, read on 24 October 2026 on the rules repository's main branch; MLCommons' Inference Datacenter page (v6.1). The limits apply at the 99th percentile.

The v6.1 Llama 2 70B results we cite are on the 99.9 per cent accuracy target, written llama2-70b-99.9, while the v6.0 results on our older benchmark pages are on the 99 per cent target, llama2-70b-99. The two are separate benchmarks, which matters when results from both rounds are put side by side below.

For a buyer, the server scenario is the one that resembles a shared service with many users, and the interactive one is closer to a chat where each answer should stream fast. A limit of 200 ms per output token at the 99th percentile means that 99 per cent of output tokens came at a pace of five tokens per second or faster; 40 ms means 25 or faster.

Llama 2 70B on four makers' eight-card servers

The four Llama 2 70B results come from four different chassis, with different processors, memory and PCIe layouts, and the same eight cards.

MLPerf® Inference: Datacenter v6.1, closed division, available category, Llama 2 70B (llama2-70b-99.9), offline and server scenarios, entries as listed, retrieved from mlcommons.org on 24 October 2026; results verified by MLCommons Association. Per card and the ratio are our division of the system results, not MLPerf metrics.

Two things follow. The spread between the highest and the lowest offline result is 2.6 per cent, so the card sets the throughput and the chassis changes it little. The server scenario costs only 2 to 3 per cent on this benchmark, which means these servers keep their throughput when requests arrive at random, as long as an output token may take up to 200 ms at the 99th percentile.

Card count changes little per card. In v6.0, HPE's two-card DL385 Gen11 reached 3,451 tokens per second per card on the 99 per cent variant, the four-card version with its cards capped at 450 W 3,340 and HPE's eight-card DL380a Gen12 3,739, by our division of entries 6.0-0052, 6.0-0054 and 6.0-0047 (MLPerf® Inference: Datacenter v6.0, closed division, available category, llama2-70b-99, offline, retrieved from mlcommons.org on 24 September 2026, results verified by MLCommons Association). A buyer can therefore plan throughput per card and choose the card count by memory, failure domain and power feed.

Two servers appear in both rounds. HPE's DL380a Gen12 with eight cards reached 29,908.4 tokens per second on the 99 per cent variant in v6.0 (entry 6.0-0047) and 29,284.8 on the 99.9 per cent variant in v6.1 (entry 6.1-0050), 2 per cent less on the stricter target. Dell's XE7740 went from 27,034.8 (entry 6.0-0022) to 29,796.7 (entry 6.1-0036), 10 per cent more, on the stricter target too. Both v6.0 values are MLPerf® Inference: Datacenter v6.0, closed division, available category, llama2-70b-99, offline, retrieved from mlcommons.org on 24 September 2026, results verified by MLCommons Association; the comparison across benchmarks is ours. Software and tuning change from round to round, and a result is a snapshot of one submitter's stack. We build eight-card servers with the RTX PRO 6000 Server Edition on chassis whose makers list the card; ask us which chassis fits your rack and power feed, and our team replies with a configuration.

gpt-oss-120b: the mixture-of-experts benchmark

gpt-oss-120b is a mixture-of-experts model that reads only a part of its weights per token, and its v6.1 benchmark asks reasoning questions in maths, science and code. Three eight-card Server Edition systems have results we cite.

MLPerf® Inference: Datacenter v6.1, closed division, available category, gpt-oss-120b, offline and server scenarios, tokens per second, entries as listed, retrieved from mlcommons.org on 24 October 2026; results verified by MLCommons Association. Per card is our division of the system result, not an MLPerf metric.

The spread is 3.2 per cent, and the server scenario kept 93.9 per cent (Nebius) and 95.6 per cent (HPE) of the offline result. The Nebius entry is a cloud instance, an eight-card node on Nebius' virtualised platform, according to MLCommons' supplemental notes, and it ran as fast as the bare-metal servers. The model's weights are about 65 GB in MXFP4, so one copy fits a single 96 GB card, and an eight-card server can run it as eight copies or split it; the result alone does not show which layout each submitter chose. The tokens per second are lower than on Llama 2 70B because the benchmarks differ in questions, answer lengths and latency limits; a token rate from one benchmark says nothing about another.

Llama 3.1 8B and Whisper on HPE's DL380a Gen12

HPE's entry 6.1-0050 also covers a small model and speech recognition, two tasks that fit one card many times over.

MLPerf® Inference: Datacenter v6.1, closed division, available category, Llama 3.1 8B and Whisper, scenarios as listed, entry 6.1-0050 (HPE ProLiant Compute DL380a Gen12, eight RTX PRO 6000 Blackwell Server Edition), retrieved from mlcommons.org on 24 October 2026; results verified by MLCommons Association. Per card is our division of the system result, not an MLPerf metric.

Under the interactive limits of 500 ms to the first token and 30 ms per token, the server kept 78 per cent of its offline throughput on the 8B model. This is the clearest figure in v6.1 for what tight per-answer speed costs on these cards. Our guide to a server for a customer service chatbot uses models of this size, and our contact centre AI article applies the 8B summary result.

What the results mean when you buy a server

MLPerf results are measurements of tuned systems on fixed tasks. They are a good check of three things: that a given server model runs the cards at their rated speed, that the throughput holds under latency limits, and how two cards compare under the same rules. They do not tell you how many of your employees one server serves, because that depends on your model, context length and how many requests run at the same moment.

For the user count, we size by memory first. Our users-per-card rule takes 90 per cent of the memory the driver reports, minus 3 GiB per card, minus the weights, and divides the rest by the cache per conversation; our guide to how many users one RTX PRO 6000 serves explains it. MLPerf then shows whether the compute keeps up with that many sessions.

Read a result with these questions: is it the same benchmark and accuracy target as the one you compare it with; which scenario; how many cards; and does the entry describe a system you can buy in that form. An entry with ten cards in a server whose maker lists eight is a published result, not an orderable configuration, as our RTX PRO 6000 benchmarks note for v6.0.

We supply servers with two, four or eight RTX PRO 6000 Server Edition cards on platforms from the makers above; send us your models, context length and peak requests, and our team replies with a configuration and the results that apply to it.

NVLink and HBM for comparison: DGX B300

NVIDIA's DGX B300 is not a PCIe server: its eight B300 GPUs sit on one NVLink baseboard with HBM3e. In v6.1 it reached 109,277 tokens per second on Llama 2 70B offline and 106,949 in the server scenario, and 106,637 on gpt-oss-120b offline and 100,630 in the server scenario (MLPerf® Inference: Datacenter v6.1, closed division, available category, llama2-70b-99.9 and gpt-oss-120b, entry 6.1-0071, retrieved from mlcommons.org on 24 October 2026, result verified by MLCommons Association).

By our division of the node results, that is 3.6 times the best eight-card PCIe result on Llama 2 70B (entry 6.1-0070) and 6.8 times on gpt-oss-120b (entry 6.1-0082). The DGX is a different class of machine with a different power and cooling budget; our comparison of the DGX B300 and an eight-card RTX PRO 6000 server sets out when each fits. We supply the DGX B300 per project, after a check of the company, the project and the intended use.

What v6.1 does not have

The round has gaps that matter for buyers of PCIe servers.

MLPerf® Inference: Datacenter, closed division, available category, Llama 2 70B (llama2-70b-99), offline scenario, tokens per second: v6.0 entries 6.0-0021, 6.0-0052 and 6.0-0054 and v5.0 entry 5.0-0018, retrieved from mlcommons.org on 24 September 2026; results verified by MLCommons Association. The 450 W cap of 6.0-0054 is from MLCommons' run logs as cited in our RTX PRO 6000 benchmarks.

The H200 NVL comparison therefore rests on v6.0: in that round eight H200 NVL in Dell's XE7740 led eight RTX PRO 6000 Server Edition in the same chassis on Llama 2 70B, as our H200 NVL benchmarks work out. The L40S comparison rests on v5.0, an older round, and our L40S benchmarks give the context. For a two-card server, the v6.0 result of 3,451 tokens per second per card, by our division, is close to the per-card figures of the eight-card servers in v6.1, on a different accuracy target.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build AI servers to order with two, four or eight RTX PRO 6000 Server Edition cards, RTX PRO 4500 Server Edition or H200 NVL with NVLink bridges, assembled and tested before delivery, with manufacturer warranty on every component, on one EU contract and invoice. Our guide to 2, 4 or 8 RTX PRO 6000 Server Edition per server compares the sizes, and NVIDIA RTX PRO Server explained covers the designs behind the entries above.

FAQ