NVFP4 KV cache on the NVIDIA RTX PRO 6000 Blackwell Server Edition: accuracy and users

What an NVFP4 KV cache changes on the RTX PRO 6000 Server Edition

An NVFP4 KV cache stores the keys and values of every open conversation in 4-bit floating point with one FP8 scale per 16 values, so it needs 56 per cent of the memory of an FP8 cache. On the NVIDIA RTX PRO 6000 Blackwell Server Edition that turns 12 concurrent 8K sessions of Llama 3.3 70B per card into 21, by our users-per-card rule. The first published measurement, whose capacity and speed tests ran on this card, found an accuracy cost of a fraction of a point to 1.6 points; SGLang marks the feature experimental, and vLLM added it for these cards in release 0.32.0 of 18 October 2026.

Once a model is loaded, the KV cache decides how many users fit on a card. Weight formats are covered in our guide to FP8, NVFP4, MXFP4 and INT4, and the general memory budget in how much VRAM an LLM needs. This article covers the one decision those pages leave open: whether to switch the cache dtype from FP8 to NVFP4 on RTX PRO 6000 Server Edition servers, what it buys and what it costs. All versions and figures were read on 24 October 2026.

Bytes per token: FP8 against NVFP4

NVFP4 uses the same layout for the cache as for weights. Each value is an E2M1 number, one sign bit, two exponent bits and one mantissa bit. Every block of 16 values shares an FP8 E4M3 scale, and the tensor has one FP32 scale on top. The LMSYS post that introduced the format in SGLang counts it per block: 8 bytes of packed data and 1 byte of scale for 16 values, against 16 bytes in FP8. That is a ratio of 0.5625, and LMSYS gives the idealised capacity gain as about 1.78 times "before workspace and other overheads".

For a model with full attention in every layer, the cache per token is 2 × layers × KV heads × head dimension × bytes per value. Llama 3.3 70B has 80 layers, 8 KV heads and a head dimension of 128, which gives 163,840 values per token.

Llama 3.3 70B geometry and the BF16 and FP8 figures as in our VRAM guide; NVFP4 = FP8 × 0.5625 by our arithmetic from the format as described in the LMSYS post of 16 September 2026, read 24 October 2026.

The 128K column shows where the format matters most. A single 128K conversation with an FP8 cache needs 20 GiB, more than the 15.3 GiB that remain on one card beside FP8 weights of Llama 3.3 70B. With an NVFP4 cache it needs 11.25 GiB and fits. Long-context work, agents with large tool outputs and document review are the cases where the cache, not the number of users, sets the limit, as our guide to long-context LLM hardware shows.

The format has one detail to keep in mind when reading published results. The LMSYS experiments used a global FP32 scale of 1.0, without calibration, and the authors write that proper calibration may narrow the accuracy gap. vLLM's documentation describes the same choice for the FP8 cache: without calibration "all quantization scales are set to 1.0", and it recommends calibrating against a dataset with llm-compressor for the highest quality.

Sessions per card by our users-per-card rule

Our users-per-card rule takes 90 per cent of the 95.6 GiB the driver reports, less 3 GiB per card for activations and the runtime, less the weights. On one RTX PRO 6000 Server Edition that leaves 83.0 GiB for weights and cache. A session is a request holding a full 8,192-token context.

our users-per-card rule (0.9 × 95.6 GiB, less 3 GiB per card, less the weights); weights from NVIDIA's Llama 3.3 70B checkpoints, 67.7 GiB in FP8 and 39.8 GiB in NVFP4; 1.25 GiB per 8K session with an FP8 cache, 0.70 GiB with NVFP4. The FP8-cache figures are those of our users and four-card guides.

These counts are a floor, as in our guide to users per RTX PRO 6000: PagedAttention allocates blocks on demand, and most sessions never fill their declared context. They also assume the full 1.78 times. The one published measurement on this card came out lower. On Qwen3.8-27B at 32K context, LMSYS raised the concurrency on one Server Edition card from 44 requests with an FP8 cache to 70 with NVFP4, 1.59 times, and at 160K from 10 to 15. LMSYS states its 1.78 times before workspace and other memory overheads and writes that actual usable capacity depends on workload and configuration. Read the cache size vLLM or SGLang prints at start-up rather than multiplying by 1.78; our guide to vLLM KV cache size and maximum concurrency explains those log lines.

We supply servers with two, four or eight RTX PRO 6000 Server Edition cards; send us your model, context length and number of users, and our team replies with a configuration sized for an FP8 cache and the headroom an NVFP4 cache would add.

Published accuracy: what the LMSYS measurement found

The post "Accelerating Long-Context and Agentic Inference with NVFP4 KV Cache" appeared on the LMSYS blog on 16 September 2026, written by the SGLang, Qwen and NVIDIA teams. Its accuracy runs on Qwen3.5-397B-A17B used eight RTX PRO 6000 Blackwell Server Edition cards, the first such measurement on this card that we found; for the Qwen3.8-27B accuracy runs the post names no GPU, while its speed runs on that model used one Server Edition card. Both cache configurations used the same FP8 weights; only the cache dtype changed.

LMSYS blog, 16 September 2026, FP8 weights, global FP32 scale 1.0, GPU for these runs not named in the post; AIME 2025 in thinking mode. The post gives the GSM8K and GPQA-Diamond results as differences. Read 24 October 2026.

The second model was Qwen3.5-397B-A17B in FP8 on eight Server Edition cards with TP=8. There the NVFP4 cache gave one fewer correct GSM8K answer, about 0.08 points, and identical totals on GPQA-Diamond and AIME 2025. The authors limit their own conclusion: two models and a small set of tasks do not establish a general rule, and they recommend validation per task.

vLLM's pull request 46963, which brought the NVFP4 cache to SM8x and SM12x, adds a warning from the other end of the scale. On GSM8K over all 1,319 questions, Qwen3-0.6B scored 0.384 with a BF16 cache and 0.174 with NVFP4. Qwen3-4B on 100 questions scored 0.920 and 0.910 in two BF16 runs and 0.880 in both NVFP4 runs. A maintainer called the 0.6B drop quantisation noise on a very small model. The pattern in these numbers is consistent: the larger models lost little, the smallest a great deal. A company assistant on a 70B-class or large mixture-of-experts model is closer to the LMSYS case; a small model for classification or extraction needs its own test before the cache dtype changes.

Speed: decode, prefill and agent workloads

The capacity gain also shows up in throughput, because a larger resident cache lets the engine batch more requests. LMSYS measured Qwen3.8-27B with 1,024 output tokens per request on one Server Edition card.

LMSYS blog, 16 September 2026, Qwen3.8-27B, FP8 weights, one RTX PRO 6000 Blackwell Server Edition; the post gives the same-concurrency gain as a range over the three lengths; at 1M the NVFP4 run used a static memory fraction of 0.75 against 0.90 for FP8. Read 24 October 2026.

At the same concurrency, NVFP4 decoding was faster even though each value must be converted. The SM120 decode kernel described in the post takes BF16 queries and NVFP4 keys and values and multiplies in BF16, so it reads less memory per token. Prefill went the other way by a small margin: mean time to first token was 0.20 to 0.40 per cent longer at matched concurrency, which the authors attribute mainly to the quantisation and dequantisation overhead, and 0.74 to 10.72 per cent shorter under load driven by capacity, a figure that, as they note, includes admission and queueing time and cannot be credited to faster prefill alone. The authors note that decode gains are not the same as end-to-end speedups.

The post also contains an agent benchmark, AgentX, on Qwen3.5-397B-A17B: comparable results up to a concurrency of 8, and above 12 the FP8 run lost throughput while NVFP4 kept rising. That sweep ran on eight RTX 6000D cards, a different GPU from the Server Edition, so treat it as an indication for agent loads, not as a figure for our servers.

Engine support in October 2026

LMSYS blog of 16 September 2026; vLLM release notes v0.32.0 of 18 October 2026 (pull requests 46963 and 60452) and pull request 46963; TensorRT-LLM quantisation matrix, documentation 1.4.0rc1, last updated 19 October 2026. All read 24 October 2026.

SGLang. The LMSYS post lists SM12x and SM100/SM103 as supported and more architectures as planned. On the model side it supports grouped-query attention and sparse MLA, with sparse GQA in progress. Its example server pairs the cache with --prefill-attention-backend flashinfer and --decode-attention-backend trtllm_mha, and the FP8 baseline used --kv-cache-dtype fp8_e4m3. The results were measured on one pinned SGLang commit; a release that includes the code is the safer base for production.

vLLM. Pull request 46963, merged on 7 October 2026, enables --kv-cache-dtype nvfp4 on SM8x and SM12x through FlashInfer; before it, its author writes, the dtype could not be used on these GPUs at all, while SM100 already had a path. Release 0.32.0 shipped it with XQA decode on SM12x. The pull request lists what it refuses: head size 64, attention sinks and decode context parallelism. A user also reported much slower decoding with MTP speculative decoding on SM120; the author answered with pull request 60452, the XQA decode path on SM120 that shipped in 0.32.0, so that MTP can run in full CUDA graphs again. Whether that removes the slowdown on your model is for your own measurement. Check your model's head size and attention design against that list, and measure with your speculative decoding settings, before you switch.

TensorRT-LLM. NVIDIA's quantisation matrix gives sm120 its own row. It lists the FP8 KV cache but no NVFP4 KV cache, which it lists for the sm100/103 class. A deployment on TensorRT-LLM stays on FP8 for the cache on these cards. Our guide to SM120 software support has the wider version picture for all three engines.

H200 NVL and the other cards

The H200 NVL is Hopper, compute capability 9.0, and has no FP4 Tensor Cores. vLLM still accepts an NVFP4 cache there: pull request 60620, merged on 8 October 2026 and listed with the 0.32.0 cache changes, adds SM90 to the architectures that read NVFP4 through FlashInfer's fa2 path. Its author writes that SM90 lacks FP4 conversion instructions, so FlashInfer dequantises in software, as on SM8x, and that the change was not run on SM90 hardware. A reviewer ran it on an H200 and found prefill and decode within 0.3 per cent of a dequantised BF16 reference at head size 128, with GSM8K results mostly close to BF16 and one five-shot run notably lower. No throughput was measured. The LMSYS post does not cover Hopper.

For the H200 NVL the memory saving is the same 0.5625 ratio, and with 141 GB per card the cache is rarely the first limit: our rule gives 44 sessions of Llama 3.3 70B with FP8 weights and an FP8 cache on one card. Whether the software conversion costs speed has not been published. On the L40S and L4 (Ada, 8.9), vLLM's SM8x path applies; the original description of pull request 46963 listed SM89 as untested. Our comparison of RTX PRO 6000 and H200 NVL for LLM inference weighs the two cards on everything else.

How to adopt an NVFP4 cache on a production server

The steps follow from the published results and the engine notes above.

1. Keep the server sized for FP8. Plan card count and context length with an FP8 cache, as our sizing pages do. Treat the NVFP4 cache as headroom you can switch on after testing, and do not promise it to users as capacity.
2. Pin the engine. vLLM 0.32.0 or later, or an SGLang release that contains the NVFP4 cache code. Record the FlashInfer version the image carries.
3. Check model features. Head size, attention sinks, sliding windows, MLA and speculative decoding decide whether the engine accepts the dtype and how fast it decodes.
4. Run your evaluation set twice. Same weights, same prompts, FP8 cache against NVFP4 cache, with the metrics your users care about. Our guide to LLM evaluation before model upgrades describes the procedure. Give small models and long-context tasks the most attention.
5. Read the start-up log. Compare the GPU KV cache size in tokens between the two runs; the ratio on your model is the gain to plan with.
6. Load test at your context length. Measure time to first token and decode speed at the concurrency you expect, then keep the FP8 setting documented as the fallback.

Our engineering partner Vixen.UNO sets up vLLM and the model on servers we deliver within our Private AI/ML service; tell us which model and context length you plan, and we check the engine version and cache settings against the release notes before you order.

FAQ