Qwen3.8-Flash-Next NVFP4 on 2× NVIDIA RTX PRO 6000 Blackwell Server Edition

Two Server Edition cards for Qwen3.8-Flash-Next

Qwen3.8-Flash-Next runs on a server with two NVIDIA RTX PRO 6000 Blackwell Server Edition cards when you use NVIDIA's NVFP4 checkpoint and keep the model's 51B n-gram table in host memory. Each card then holds about 37.9 GiB of weights and about 45 GiB of cache, which by our users-per-card rule is room for about 157 concurrent conversations of 32,768 tokens. NVIDIA has tested the checkpoint only on B200 and B300, so plan a test run on the delivered system before you rely on it.

Our Qwen hardware guide sizes Qwen's own FP8 and BF16 checkpoints of Flash-Next on RTX PRO 6000 and H200 NVL. This page covers the NVIDIA checkpoint that the guide did not yet list, and what it changes for the entry server among our AI servers: two cards instead of four.

The model is a mixture of experts. NVIDIA's model card gives 125B parameters in total and 6B active per token, plus a 51B n-gram embedding and a 4B multi-token prediction (MTP) module. The metadata widget on the same page shows "120B params", which does not match the text; we use the card's text. The native context is 262,144 tokens, extensible to 1,000,000.

What NVIDIA's NVFP4 checkpoint contains

NVIDIA quantised the checkpoint with Model Optimizer 0.46.0 and did no fine-tuning. Only the routed experts of the language model are in NVFP4; the rest stays in higher precision.

NVIDIA model card and config.json of nvidia/Qwen3.8-Flash-Next-NVFP4 on Hugging Face, read on 24 October 2026. NVIDIA states that the MTP module and the n-gram table match the FP8 checkpoint "byte-for-byte".

The file list shows ten model shards of 79.0 GB in total and a separate file of 53.7 GB that holds the MTP module and the n-gram table, 132.7 GB or 123.6 GiB together. NVIDIA calls the checkpoint "approximately 2.7x smaller" than BF16. Against Qwen's FP8 checkpoint, which vLLM's recipe puts at 172.78 GiB, it saves about 49 GiB.

NVIDIA compares the NVFP4 build with the FP8 model, not with BF16. On GPQA Diamond it scores 91.5 against 92.0, on τ²-Bench Telecom 90.1 against 90.8 and on Terminal-Bench 2.1 82.9 against 83.3; on several others, such as AA-LCR and SciCode, it scores higher than FP8. NVIDIA gives single scores without a spread between runs, so run your own test set on both builds.

Where the 51B n-gram table goes

The n-gram table is the reason Flash-Next needs more memory than its 6B active parameters suggest. The config.json sets an n-gram vocabulary of 20,000,000 entries of 2,560 values. At one byte per value in FP8 that is 51.2 GB, or about 47.7 GiB, by our arithmetic; our Qwen guide rounds it to 47.5 GiB.

vLLM's Flash-Next recipe, updated on 13 October 2026, describes the table as "a 51B lookup memory" that "can be asynchronously offloaded to host memory". It states that vLLM enables the offload by default, that --engram-config '{"cpu_offload":true}' sets it explicitly, and that it "currently only runs on Nvidia devices". It asks for "at least 51 GB plus runtime headroom" of host memory. Our entry configuration has 384 GB of DDR5, so the table takes a small part of it.

The recipe also states, for eight H100 cards, that "H100's 80GB/GPU is not enough headroom for the 51B N-gram/PLE embedding table", and its troubleshooting notes that plain TP=4 on four H100 80 GB runs out of memory at start-up without the offload. As in our Qwen guide, we read that as one copy of the table on every GPU. On that reading, a TP=2 copy on two Server Edition cards would need 47.7 GiB of table plus 37.9 GiB, half of the remaining 75.8 GiB, on each card: 85.6 GiB against the 83.0 GiB our rule allows. On two cards, the offload is required.

The recipe describes the offload for Qwen's checkpoints. NVIDIA's NVFP4 card does not mention it. Since the table in the NVIDIA repository is the same FP8 table, we expect the same path to apply, but this is one of the points to confirm in the test run.

Memory and conversations per layout

Our users-per-card rule takes 90 per cent of the 95.6 GiB the driver reports and subtracts 3 GiB per card, which leaves 83.04 GiB per RTX PRO 6000 Server Edition. What remains after the weights is cache. The derivation is in our guide to how many users one RTX PRO 6000 can serve.

Flash-Next keeps a full cache only in 12 of its 48 layers. Each has two KV heads of 256 values, 12 KiB per token in FP8. The sparse-attention indexer adds a 128-wide key per layer, which we count at up to 1.5 KiB per token, and the 36 linear-attention layers hold 48 value heads of 128 × 128 values in float32, 108 MiB per conversation whatever its length. Split over two cards, each card holds one KV head, its own copy of the indexer key and half of the linear-attention state: about 0.29 GiB per card for a conversation of 32K, by our arithmetic.

Our estimates by the users-per-card rule, not measurements; layout = number of cards × checkpoint, with the n-gram table in host RAM or on the GPUs; concurrent conversations of 32,768 tokens at full length with an FP8 KV cache and FP8 indexer cache, as vLLM's recipe sets. Weights from the file list of nvidia/Qwen3.8-Flash-Next-NVFP4 (132.7 GB) and the 172.78 GiB of Qwen's FP8 checkpoint in vLLM's recipe, read on 24 October 2026. The FP8 row is from our Qwen hardware guide.

The recipe's commands set --max-num-seqs 256, and it advises keeping that value when a Mamba-cache error appears at start-up. On two cards that limit, not memory, caps the number of 8K conversations: by memory, two cards would hold about 400 of them.

These are requests in flight at the same moment, not named users. With the example values of our guide to a private ChatGPT server by company size, 500 staff produce about 20 requests in flight at the busiest moment, so 157 conversations cover several thousand staff, as far as memory goes. The speed each user sees at that load is what the test run measures.

We supply this server with two RTX PRO 6000 Server Edition cards and set up the model on request; send us the context length and the peak requests you expect, and our team replies with a layout and the host memory it needs.

Software checks on SM120 before you buy

The RTX PRO 6000 is a Blackwell card of compute capability 12.0, the SM120 family; B200 and B300 are SM100-class chips with different kernels. NVIDIA's card lists B200 and B300 as supported and tested, names vLLM as the engine and uses --tensor-parallel-size 8, --quantization modelopt and --trust-remote-code in its commands. It requires Transformers 5.16.0 or later and names a vLLM commit rather than a release; MTP speculative decoding also needs a vLLM pull request (#55513) until it is merged.

vLLM's recipe states that "Stable ≥0.30.0 runs the model" and that it is "tuned and verified on nightly". Its tested systems are GB300, H200, H100 and MI355X; it mentions an NVFP4 variant only for one GB200 and has no RTX PRO command. Three facts from vLLM's release notes matter for these cards:

1. NVFP4 experts. vLLM 0.30.0 of 22 September 2026 made "W4A4 NVFP4 over weight-only kernels on SM120/121" the default (pull request #55170). A community report on the vLLM forum of April 2026 described the NVFP4 expert kernels failing to initialise on four RTX PRO 6000 cards of an unstated edition, with a fall-back to weight-only Marlin kernels, so the vLLM version decides which path runs.
2. FP8 blocks in the MTP experts. vLLM 0.30.0 lists a blockwise FP8 tuning for SM12x aimed at GB10, and vLLM's Qwen3.8-27B recipe states for SM120 cards that "The block-scaled FP8 checkpoint needs no workaround". TensorRT-LLM's quantisation matrix (1.4.0rc1) lists no FP8 block scaling for SM120, so this model is a vLLM or SGLang job on these cards.
3. Remote code. The NVIDIA command loads remote code. vLLM 0.31.0 fixed the security advisory GHSA-h3rc-6mm3-gc2m for --trust-remote-code, so run 0.31.0 or later.

StorageReview's GPU roundup of 14 October 2026, with vLLM 0.27.1, found that the FP4 advantage of Blackwell cards "has not reached the serving stack yet" on a dense NVFP4 model. Releases since then have changed the defaults, which is one more reason to measure on the version you deploy. Our guide to Blackwell RTX and SM120 software support tracks these changes.

Splitting the model over two PCIe cards

The Server Edition has no NVLink, so the two cards exchange partial results over PCIe Gen5. With exactly two GPUs, vLLM keeps its custom all-reduce, provided the peer-to-peer test at start-up passes; on more than two PCIe-only GPUs it is off by default. vLLM 0.30.0 (release notes of 22 September 2026) added an opt-in FlashInfer PCIe IPC all-reduce for servers without NVLink (#53576). Our guide to RTX PRO 6000 without NVLink explains both.

For mixture-of-experts models, expert parallelism is the other way to split. vLLM's recipe runs eight GPUs as TEP8, tensor parallelism with expert parallelism, because, for the FP8 checkpoint, "plain TP8 is incompatible with its 128-wide quantization blocks", and NVIDIA's MTP command adds --enable-expert-parallel. On two cards both TP=2 and expert parallelism are worth trying in the test run. In a two-socket server, nvidia-smi topo -m shows whether the pair sits under one processor, and our guide to P2P, IOMMU and ACS for NCCL covers the BIOS and kernel settings.

One card for development

With the table in host memory, the NVFP4 weights come to about 75.8 GiB on the GPU, which one Server Edition card holds with about 7.2 GiB left, room for about 13 conversations of 32K by our rule. vLLM's recipe mentions single-GPU use of the NVFP4 variant on a GB200, not on an RTX PRO card. A developer can try prompts, tools and the MTP setting this way, while a service for a team belongs on two cards.

Server, host memory and power

The entry configuration among our AI servers is two RTX PRO 6000 Server Edition cards, dual AMD EPYC, 384 GB of DDR5 ECC and two NVMe drives. NVIDIA's configuration guide for NVIDIA-Certified Systems asks for at least twice the GPU memory in system memory, 384 GB for two 96 GB cards, and the n-gram table needs at least 51 GB of it. Our guide to how much RAM a GPU server needs explains the rule, and KV cache offloading to CPU memory and NVMe covers the other use of host memory.

The checkpoint takes 132.7 GB on disk; keeping Qwen's FP8 checkpoint beside it for comparison adds about 186 GB. With two 16-core EPYC 9135 and the cards at 600 W, we estimate about 1.8 kW at the wall, inside the 2.94 kW we plan per 230 V 16 A feed, as our page on the two-card Server Edition server works out. A measurement on the delivered system confirms the figure.

Licence: NVIDIA Open Model License and Qwen Community License

NVIDIA's card states that the repository is governed by the NVIDIA Open Model License, with the Qwen Community License 1.0 of the base model as additional terms. Clause 2 of the Qwen licence reads: "If the licensee or any of its affiliates conducts a Model as a Service or AI Work Assistant business, the licensee shall obtain a separate license from Qwen before Using the Software or its derivative works for any commercial purpose." It exempts internal use that does not make the software, its outputs or its model capabilities available to any third party.

The licence defines Model as a Service as "giving a third party access to language model inference or fine-tuning (e.g., via API or a hosted endpoint)" and an AI Work Assistant as "an independent AI-powered product primarily designed for AI-assisted coding or office productivity". Products with more than 100 million monthly active users or US$20 million of monthly revenue must display the model name. As read on 24 October 2026, the text has no territorial restriction and names no governing law. Whether a clause applies to your company is for your legal counsel to confirm.

When four cards or H200 NVL fit better

Four RTX PRO 6000 Server Edition cards hold the NVFP4 build with the table on every GPU, about 62 conversations of 32K, which avoids the dependency on host memory, or about 245 with the offload. They suit longer contexts or a second model beside Flash-Next, as our page on 4× RTX PRO 6000 Server Edition shows. Two H200 NVL cards with an NVLink bridge run Qwen's FP8 checkpoint with the table in host memory and keep about 60.7 GiB per card for cache, by our Qwen guide; they have no FP4 arithmetic. Our list of NVFP4 checkpoints for the RTX PRO 6000 collects the other models published in this format.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build AI servers to order with two to eight RTX PRO 6000 Server Edition cards and the host memory the model needs, assembled and tested before delivery, on one EU contract and invoice with manufacturer warranty on every component. The configuration and ordering steps are on our page to buy a 2× RTX PRO 6000 Server Edition server. Setting up Qwen3.8-Flash-Next with retrieval and access control is part of our Private AI/ML service; tell us how you plan to use the model and we propose the test run.

FAQ