1T-parameter models for 100 users: a DGX B300 or two servers with eight H200 NVL

What 100 users of a 1T model need

A hundred people working with a model of 750B to 1.6T parameters at the same moment need more than the weights. For conversations of 32K tokens, one server with eight H200 NVL reaches that number by memory for Kimi K2.6 and DeepSeek-V4-Pro, and for Muse Spark Open in NVFP4 if the engine runs it weight-only, by our users-per-card rule. GLM-5.3 needs two such servers run as one copy, or a DGX B300 with two pipeline stages, and Muse Spark Open in BF16 needs two servers or a DGX B300.

Our model guides end at this point: their 100-user cells read "over 8" for some of these models, because they size one server. This page takes the next step and compares two servers with eight H200 NVL each, 16 cards in four NVLink domains of four, with one NVIDIA DGX B300, eight B300 GPUs in one NVLink domain. We build the H200 NVL servers to order and supply the DGX B300 per project. The model figures come from the Hugging Face file lists, which we summed on 24 October 2026, and from our guides to Kimi K2, GLM, DeepSeek-V4-Pro and Muse Spark Open.

Conversations of 32,768 tokens at full length by our users-per-card rule, 16-bit cache unless stated. One server: TP=4 × PP=2, except Kimi K2.6, where the figure is the cache that vLLM's Kimi-K2-Thinking recipe measured on eight H200 with decode context parallelism (DCP). Two servers: one copy as TP=4 × PP=4. DGX B300: TP=8 unless stated, about 270 GB per GPU read cautiously as 10⁹ bytes. Weights: sums of the Hugging Face safetensors files, read on 24 October 2026.

Why the cache type moves the answer

The weights of these models differ by a factor of three, but the cache per conversation differs by more. Kimi K2.6 and GLM-5.3 store one compressed latent vector per layer, 2.14 GiB and 3.06 GiB per 32K conversation in 16-bit by our guides' counts. DeepSeek-V4-Pro compresses further, to about 0.30 GiB. Muse Spark Open stores keys and values for 16 KV heads, about 2.36 GiB by our count, because 46 of its 62 layers keep only a 2,048-token window.

Where the cache lives matters as much as its size. vLLM's blog post of 7 August 2026 states: "Under normal TP there is nothing to split by head, meaning the latent KV cache is replicated in full across every TP rank." For Kimi, GLM and DeepSeek, each card of a tensor-parallel layout therefore holds every conversation's whole cache, and the spare memory of one card sets the count. Pipeline stages divide it, because each card keeps only its own layers' cache, and DCP splits each conversation by token. Muse Spark Open has 16 KV heads, so tensor parallelism divides its cache among the cards and their memory adds up. Our guide to long-context LLM hardware explains these cache types in more detail.

This is why a DGX B300 with almost twice the memory of one H200 NVL server does not automatically hold twice the users. Under TP=8, GLM-5.3 leaves about 135 GiB on each B300 GPU, room for 72 conversations with an FP8 cache, while two H200 NVL servers in four pipeline stages leave 79.4 GiB per card but each card holds only a quarter of the layers. Layout is a configuration choice on either platform, so plan the layout together with the hardware.

Two servers with eight H200 NVL: one copy or two

Two servers can run the model in two ways. As two copies, each server holds the whole model and the users split between them: twice as many as one server holds. As one copy, the model is cut into pipeline stages across both servers, which frees memory on every card: the weights of GLM-5.3 fall from 88 GiB to 44 GiB per card, and the spare memory rises from 35.4 to 79.4 GiB.

Our estimates by the users-per-card rule for GLM-5.3 in FP8 (703.7 GiB), 3.06 GiB per 32K conversation in 16-bit and 1.88 GiB in FP8 as in our GLM guide; each stage holds at most 20 of the 78 layers. If the engine reserved 7.4 GiB per card more than our rule, as vLLM's Kimi-K2-Thinking recipe measured on eight H200, the one-copy layout would hold about 92 and 149.

vLLM's documentation for multi-node serving says: "Set tensor_parallel_size to the number of GPUs per node and pipeline_parallel_size to the number of nodes." On the H200 NVL, each server holds two NVLink domains of four cards joined over PCIe, so we keep tensor parallelism inside a domain and give each domain its own pipeline stage, four stages over two servers, as our guide to the 8× H200 NVL server does for one server. Between the two servers only the activations at the stage boundary travel, once per step. The same documentation warns that if NCCL reports [send] via NET/Socket, it "used a raw TCP socket, which *is not* efficient for cross-node tensor parallelism", and it shows [send] via NET/IB/GDRDMA as the sign of GPUDirect RDMA. Each server of our training node has 400G networking for this link.

The trade-off is availability. Our comparison of one 8-GPU server or two 4-GPU servers shows that a model split across two hosts stops when either host fails, and the same holds here. Two copies keep half the capacity through a failure, a reboot or a driver update. For Kimi K2.6 that is the obvious choice: one server already holds 174 conversations with DCP in vLLM's measurement, so two copies give 348 with full redundancy. For GLM-5.3 the one-copy layout is the only way to reach 100 users on two servers. We build both layouts and size them per model; send us the model, the context length and the peak conversations, and our team replies with the figures for each.

DeepSeek-V4-Pro and Muse Spark Open

DeepSeek-V4-Pro leaves only 19.4 GiB per card on one server, but its cache is small. With two pipeline stages, one per NVLink domain, our rule gives about 126 conversations of 32K, or about 78 if the engine reserves as much as the Kimi recipe measured. One server reaches 100 users on paper; two servers or a DGX B300 reach it with margin. At the 384K output length that DeepSeek's card recommends for its higher reasoning levels, one server holds about 10 conversations, which is where a second server pays off.

Muse Spark Open is the model where the card generation matters most. Its NVFP4 checkpoint runs natively on Blackwell, so eight RTX PRO 6000 Server Edition hold it with about 71 conversations of 32K with a 16-bit cache, or about 142 with an FP8 cache, by our rule, as our Muse Spark Open guide shows. On the H200 NVL it runs only weight-only, a path neither Meta nor vLLM documents for this model. The BF16 checkpoint, 1,531.4 GiB, runs natively on Hopper but needs two H200 NVL servers.

One DGX B300: memory, FP4 and NVLink in one system

A DGX B300 has eight Blackwell Ultra GPUs with 288 GB of HBM3e each, of which about 270 GB is exposed, as the accelerator name B300-SXM-270GB in MLPerf® entries shows. Read cautiously as 10⁹ bytes, our rule gives 223.3 GiB per GPU and 1,786.5 GiB per system. It computes NVFP4 and the FP4 experts of DeepSeek-V4-Pro natively, and NVLink joins all eight GPUs, so tensor parallelism over eight does not cross PCIe or a network.

The published throughput figures show the difference that memory alone does not. On Llama 2 70B in the offline scenario, the DGX B300 reached 109,277 tokens per second, against 32,004.0 for a server with eight H200 NVL in the round before (MLPerf® Inference: Datacenter, closed division; v6.1 entry 6.1-0071, NVIDIA DGX B300, retrieved from mlcommons.org on 24 October 2026; v6.0 entry 6.0-0021, Dell PowerEdge XE7740 with eight H200 NVL, benchmark llama2-70b-99, retrieved from mlcommons.org on 24 September 2026; results verified by MLCommons Association). By our division, which is not an MLPerf metric, one DGX B300 delivered 3.4 times the throughput of one H200 NVL server and 1.7 times that of two. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

We supply the DGX B300 per project. Before we can quote, the distribution asks for the company, the project and the intended use, as our guide to buying a DGX B300 in the EU describes.

Power, rack and licences per option

H200 NVL servers: our training node and our wall estimate of about 6.3 kW per server (eight cards at 600 W, two AMD EPYC 9555, plus 15 per cent), to be confirmed by a measurement on the delivered system. DGX B300: NVIDIA's user guide, datasheet and data-centre guide as quoted in our DGX B300 articles; NVIDIA AI Enterprise terms from NVIDIA's licensing guide as quoted there and in our H200 NVL guide.

Two H200 NVL servers can stand in two racks on separate feeds, and each needs a 400 V three-phase position, as our guide to GPU server power on 16 A, 32 A and three-phase shows. A DGX B300 concentrates the load: NVIDIA's data-centre guide estimates a peak of 19.7 kW per AC system and plans two systems per rack at 39.4 kW peak, with active rear-door heat exchangers recommended for high density. If your room tops out at 20 kW per rack, the two servers fit where the DGX would need facility work. We check rack, feed and airflow before we quote; describe your rack positions and their power limits, and we say which option your room takes.

Which option for which case

• Kimi K2.6 for 100 users: one server with eight H200 NVL and DCP holds them; a second server as a second copy adds redundancy.
• GLM-5.3 for 100 users: two H200 NVL servers as one copy with an FP8 cache, or a DGX B300 with two pipeline stages; two separate copies are not enough.
• DeepSeek-V4-Pro at long reasoning lengths: two servers or a DGX B300; one server holds 100 users at 32K only on paper.
• Muse Spark Open in NVFP4: eight RTX PRO 6000 Server Edition, the native path on PCIe, or a DGX B300; in BF16, two H200 NVL servers or a DGX B300.
• Several of these models, or throughput above all: a DGX B300, which delivers more tokens per second per system and holds every model in this table in one NVLink domain.

The licences differ too. DeepSeek publishes V4-Pro under the MIT licence; Kimi K2.6 has a Modified MIT License with a display clause for very large services; GLM-5.3 has its own licence with a security review clause for large Model-as-a-Service operators; Muse Spark Open's licence requires a separate agreement with Meta for Model-as-a-Service providers above US$20 million revenue. Our model guides quote each clause, and your legal department decides whether it applies.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build servers with eight H200 NVL and two 4-way NVLink bridges, two AMD EPYC processors, 2.3 TB of DDR5 ECC, eight NVMe drives and 400G networking, as one server or as a pair for one model, assembled and tested before delivery, on one EU contract and invoice with manufacturer warranty on every component; the ordering steps are on our page to buy an 8× H200 NVL server. The DGX B300 we quote per project. Splitting a model across servers, the serving platform and capacity planning are part of our Private AI/ML service, and our guide to GPU capacity planning for an LLM platform shows how to measure the peak before you order.

FAQ