GLM-5.3-Flash on 4× NVIDIA RTX PRO 6000 Blackwell Server Edition: NVFP4, users and power

Four Server Edition cards are enough in NVFP4

A server with four NVIDIA RTX PRO 6000 Blackwell Server Edition cards runs GLM-5.3-Flash for a department when it uses NVIDIA's NVFP4 checkpoint. The 190.4 GiB of weights leave 35.4 GiB per card by our users-per-card rule, room for about 51 concurrent conversations of 32,768 tokens with a 16-bit cache, or about 100 with the FP8 cache that NVIDIA's checkpoint sets. Z.ai's FP8 checkpoint also fits the four cards, but it leaves 6.6 GiB per card, about nine such conversations; for FP8 at scale, four H200 NVL or eight Server Edition cards are the build.

GLM-5.3-Flash is the mid-size model of Z.ai's GLM-5.3 generation: 320B parameters with 18B active, multimodal input and a context of 1,048,576 tokens, under the MIT licence. Our family page on GLM hardware requirements sizes GLM-5.3, GLM-5.3-Flash and GLM-4.x in FP8 for 1, 20 and 100 users. This page does the NVFP4 sizing for one four-card server, the layout over PCIe and the software status on these cards.

Our estimate by the users-per-card rule (83.04 GiB per Server Edition, 123.36 GiB per H200 NVL), 0.69 GiB of 16-bit cache per conversation of 32,768 tokens held on every card of a tensor-parallel copy, as in our GLM guide; FP8 cache counted at half. Checkpoints: sums of the safetensors files of zai-org/GLM-5.3-Flash and nvidia/GLM-5.3-Flash-NVFP4, read on 24 October 2026.

The model and its two checkpoints

Z.ai's model card states "320B total parameters and just 18B active parameters" and a "hybrid architecture combining sparse and linear attention"; the Hugging Face metadata shows 321B. The config.json names the architecture Glm5NextForConditionalGeneration with 45 layers. Of these, 34 use linear attention (KDA) and 11 use DeepSeek-style sparse attention with a compressed latent of 512 values. Layers 0 to 2 are dense, the other 42 are MoE layers with 288 routed experts, eight of them active per token, and one shared expert. A vision tower of 24 layers and one multi-token-prediction layer complete the checkpoint.

Hugging Face file lists of zai-org/GLM-5.3-Flash and nvidia/GLM-5.3-Flash-NVFP4, summed on 24 October 2026; FP8 format from the quantization_config in config.json; GiB conversion by our arithmetic. vLLM's recipe gives "about 306 GiB" for the FP8 checkpoint.

NVIDIA's NVFP4 card is dated 9 September 2026, made with Model Optimizer 0.47.0 and tested on "NVIDIA Blackwell GB200". Its hf_quant_config.json quantises the routed experts and the dense MLP of the first three layers to NVFP4 in blocks of 16 and sets an FP8 KV cache. It keeps the attention of all 45 layers, the shared experts, the routers, the embeddings, the output layer and the vision tower unquantised. The card's accuracy table puts NVFP4 next to BF16: 0.9211 against 0.9217 on GPQA Diamond and 0.763 against 0.7688 on MMMU Pro. The NVFP4 card states the MIT licence, like Z.ai's LICENSE file, and mentions no other model licence; NVIDIA's repository has no separate LICENSE file.

Why the cache sits on every card

The cache per conversation is small, and every card of a tensor-parallel copy holds all of it. The 11 sparse-attention layers store one latent of 512 values per token (kv_lora_rank 512, no separate RoPE part). In 16-bit that is 11 KiB per token, plus at most 2.75 KiB for the 128-wide key of the sparse indexer, so 0.43 GiB for a conversation of 32K by our arithmetic. The 34 KDA layers keep a fixed state of 64 heads of 128 × 128 values; vLLM 0.31.0 lists "FlashKDA keeping the recurrent state in FP32", which makes 136 MiB per conversation, divided over the cards with the heads.

vLLM's blog on decode context parallelism of 7 August 2026, which our DeepSeek hardware guide quotes, states that under tensor parallelism "the latent KV cache is replicated in full across every TP rank". Our config arithmetic therefore gives at most about 0.46 GiB per card for each 32K conversation. Our GLM guide works from Z.ai's stated reduction "by 4.44× versus GLM-5.3" and arrives at about 0.69 GiB; we plan with the larger figure, so the counts in this page are on the safe side.

The replication changes how to scale. Eight cards with one TP=8 copy of the NVFP4 checkpoint leave 59.2 GiB per card, about 85 conversations, while two TP=4 copies on the same eight cards leave 35.4 GiB per card in each copy, about 102 in total. For this model, more copies beat a wider split. The method behind the rule is in how much VRAM an LLM needs.

NVFP4 on four Server Edition cards

Split over four cards, the NVFP4 checkpoint puts 47.6 GiB on each, and our rule leaves 35.4 GiB of the 83.04 GiB per card for the cache. With 0.69 GiB per 32K conversation that is about 51 concurrent conversations; with the FP8 cache that NVIDIA's checkpoint sets and vLLM applies with --kv-cache-dtype fp8, about 100 by our estimate. With our config figure of 0.46 GiB, the 16-bit count would be about 77. These are requests in flight at once, far fewer than the number of people who use the service; our guide to private ChatGPT server sizing by company size turns headcount into that number.

NVIDIA's vLLM settings in the card are tensor parallelism of four, data parallelism of one, expert parallelism switched on, an FP8 KV cache, at most 32 concurrent sequences and --gpu-memory-utilization 0.90, the same 90 per cent our rule takes. The limit of 32 sequences is NVIDIA's test setting on GB200; on four Server Edition cards, raise it only as far as the cache in the start-up log allows. The 6.6 GiB that the FP8 checkpoint leaves per card is why we do not recommend FP8 on four of these cards beyond a single team.

We supply the four-card Server Edition server and set up GLM-5.3-Flash on it; send us your context length and the number of requests at peak, and our team replies with the checkpoint, layout and conversation figures we would plan with.

Tensor and expert parallelism over PCIe

The RTX PRO 6000 Server Edition has no NVLink, so the four cards exchange data over PCIe Gen5. Under tensor parallelism every layer ends in an all-reduce; by default vLLM disables its custom all-reduce on more than two PCIe-only GPUs and uses NCCL, and release 0.30.0 (22 September 2026) added an opt-in FlashInfer PCIe IPC all-reduce for systems without NVLink. With expert parallelism, which NVIDIA's command switches on, each token travels to the cards that hold its eight experts and back. Both cross the same links, and in a two-socket server some paths also cross the link between the processors.

Keep each TP=4 copy inside one server and run more copies for more users, as the table shows. nvidia-smi topo -m shows the path between every pair of cards. Our article on RTX PRO 6000 without NVLink compares the layouts, and P2P, IOMMU and ACS on Linux covers the settings that make peer-to-peer work.

Software status on SM120 in October 2026

The RTX PRO 6000 is compute capability 12.0 (SM120). For GLM-5.3-Flash, vLLM's release notes of 0.31.0, 5 October 2026, list "NoPE sparse MLA on the FlashInfer SM120 backend", the attention type of its 11 sparse layers, together with KDA and SM90 fixes; 0.32.0 of 18 October 2026 adds an FP8 cache path that gives the model "up to 31% more output throughput" on SM90, the H200 generation. vLLM 0.30.0 had already made native W4A4 NVFP4 kernels the default on SM120, as our SM120 software guide records. vLLM issue 53963, opened on 26 August 2026 for this model on the RTX PRO 6000 ("no SM120 sparse-MLA path for rope-free MLA"), was still open on 24 October 2026, with user reports in October of problems with MTP drafts on the SM120 backend; test with and without MTP speculative decoding.

vLLM's GLM-5.3-Flash recipe, updated 8 October 2026, names vLLM 0.29.0 or later, FlashInfer 0.6.17 or newer "for NoPE sparse MLA", and says the NVFP4 variant "requires NVIDIA Blackwell GPUs". Its NVIDIA example runs FP8 at TP=4 on one GB200 tray; it has no RTX PRO 6000 profile. NVIDIA's own container tag in the NVFP4 card is built for arm64 and CUDA 13.0, the GB200 platform, so an x86 server with Server Edition cards needs a standard vLLM image of 0.31.0 or later. Its SGLang command selects TensorRT-LLM attention backends for GB200, and vLLM lists its own trtllm-gen attention kernels for compute capability 10.x only, so check that the backend you choose supports SM120. Read the backend vLLM chooses in the start-up log before you sign off the server.

H200 NVL or RTX PRO 6000 Server Edition for GLM-5.3-Flash

The two four-card servers suit different checkpoints. Four H200 NVL have 141 GB of HBM3e per card at 4.8 TB/s and a four-way NVLink bridge. They hold Z.ai's FP8 checkpoint with 46.9 GiB per card, about 68 conversations of 32K, and vLLM 0.32.0's FP8 cache path for SM90 applies to them; the recipe, updated before that release, still states that "Hopper does not support FP8 KV cache for this model", which is why we count the H200 NVL with the 16-bit cache. vLLM's recipe ties NVFP4 to Blackwell, so on the H200 NVL plan with FP8. Four Server Edition cards run NVFP4 natively at 1,597 GB/s per card and hold about 51 conversations by the same cache figure.

• One department, NVFP4 accepted: four RTX PRO 6000 Server Edition.
• FP8 quality, more users, NVLink between the cards: four H200 NVL, as our page on the four-card H200 NVL server describes.
• Room for growth on Blackwell: eight Server Edition cards as two NVFP4 copies, about 102 conversations, or FP8 at TP=8, about 64.
• Two cards: not enough for either checkpoint; a smaller model such as GLM-4.7-Flash fits one card, as our GLM guide shows.

The card comparison of 4× RTX PRO 6000 or 4× H200 NVL covers the hardware in general, and 300B MoE models on four RTX PRO 6000 sets GLM-5.3-Flash against Beam-1.0, DeepSeek-V4-Flash, Qwen3.8-Flash-Next and MiMo-V2.6-Flash on the same server.

Power, host memory and storage

With the four cards at 600 W and two AMD EPYC 9355 at 280 W each, we estimate about 3.4 kW at the wall, or about 2.7 kW with the cards capped at 450 W: cards and processors plus 15 per cent for memory, drives, fans and conversion losses. At 600 W that is above the 2.94 kW we plan per 230 V 16 A feed, so the rack position needs 32 A or three-phase; the cap brings it below. Four H200 NVL at 600 W give the same estimate. A measurement on the delivered system confirms or replaces these figures, and our page on 600, 450 or 300 W for the Server Edition shows what the cap costs.

NVIDIA's certified-systems guide asks for at least twice the GPU memory as host memory: 768 GB of DDR5 ECC for four Server Edition cards, the inference node on our AI servers page. Storing both checkpoints takes 533 GB of NVMe; the BF16 weights, which vLLM's recipe puts at roughly twice the FP8 weight memory, add more if you fine-tune.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build AI servers to order with four or eight RTX PRO 6000 Server Edition cards or four H200 NVL, assembled and tested before delivery, on one EU contract and invoice, with manufacturer warranty on every component handled through us. Quantities and lead time are confirmed in writing with the quote. GLM-5.3-Flash, retrieval and the serving stack are set up through our Private AI/ML service; tell us what the model should do and we propose the server and checkpoint.

FAQ