One 600 GB LLM on two servers with four NVIDIA RTX PRO 6000 Blackwell Server Edition each

Two four-card servers for one 600 GB model

A model of about 600 GB, such as Kimi-K2.6 or Kimi K2.7-Code in INT4, runs across two servers with four NVIDIA RTX PRO 6000 Blackwell Server Edition cards each. By our users-per-card rule the eight cards keep 664.3 GiB for weights and cache; the 554.3 GiB of weights leave 13.8 GiB per card. vLLM splits the model into two pipeline stages, one per server, with tensor parallelism over the four cards inside each, and only the activations at the boundary between the stages cross the network.

The layout works, with three costs: an RDMA network between the servers, a cluster runtime that has to start both halves together, and a failure domain that covers both hosts. It suits a site where an eight-card chassis does not fit the rack power or the room, or where two four-card servers already exist. Our comparison of one 8-GPU server and two 4-GPU servers covers the general choice, and our guide to a GPU inference cluster without InfiniBand the network patterns; this page works through one model of this size on that pair.

Which models of about 600 GB fit two four-card servers

Our users-per-card rule takes 90 per cent of the 95.6 GiB the driver reports and subtracts 3 GiB per card, 83.0 GiB per card. Four cards give 332.2 GiB, two servers 664.3 GiB.

Hugging Face file lists (Kimi K2.7-Code, Muse Spark Open NVFP4 and GLM-5.3 NVFP4 summed on 24 October 2026; the others from our model guides, read 9 to 24 October 2026); left per card by our rule, our arithmetic.

Kimi-K2.6 and K2.7-Code share the same architecture and the same checkpoint size of 595.2 GB, so the numbers below apply to both. GLM-5.3 in NVIDIA's NVFP4 version has more room per card, and our article on GLM-5.3 NVFP4 on eight Server Edition cards sizes it; the memory figures are the same on two servers. Muse Spark Open in NVFP4 keeps a grouped-query cache that, unlike Kimi's latent cache, divides across all eight cards in this layout: about 71 conversations of 32K with a 16-bit cache by our estimate, as our Muse Spark Open guide counts for one eight-card server, and vLLM's recipe for the model lists "TP + PP" among the layouts NVIDIA supports.

vLLM layout: TP=4 inside each server, two pipeline stages

vLLM's documentation on parallelism and scaling, updated on 15 October 2026, gives the rule: "Set tensor_parallel_size to the number of GPUs per node and pipeline_parallel_size to the number of nodes." For two four-card servers that is --tensor-parallel-size 4 --pipeline-parallel-size 2. Each server holds half of the layers. The four cards in a server split every layer and exchange partial results over PCIe, as in a single four-card server; our comparison of TP=2, TP=4 and replicas without NVLink covers that traffic.

vLLM runs this either on a Ray cluster or with its own multiprocessing backend. For the second, the documentation starts the same vllm serve command on both servers with --nnodes 2, --node-rank 0 or 1 and --master-addr set to the first server, and adds --headless on the second. It asks that "every node provides an identical execution environment, including the model path and Python packages", so both servers carry the same image and a local copy of the 595 GB checkpoint at the same path.

Tensor parallelism over all eight cards across both servers is the other option in the documentation, which warns that "Efficient tensor parallelism requires fast internode communication". It would send two all-reduces per layer, 61 layers for Kimi K2, across the network for every token. NVIDIA's own evaluation of its GLM-5.3 NVFP4 checkpoint used "tensor parallelism 8 across two 4-GPU nodes" on B300 hardware, and the card does not describe the network. Over Ethernet between two Server Edition servers we plan with pipeline stages.

What crosses the network, and how much

In a pipeline split, each token's hidden state and residual cross the stage boundary. Our guide to clusters without InfiniBand works this out for DeepSeek-V3.2 in vLLM, and Kimi K2 has the same DeepSeek-V3 text architecture with a hidden size of 7,168 in its config.json: two tensors of 7,168 values in BF16, 28 KiB per token.

Our arithmetic at line rate, without protocol overhead or switch latency: 28,672 bytes per token (2 × 7,168 × 2 bytes); 65,536 × 28,672 bytes = 1.88 GB; 32 × 28,672 bytes = 0.92 MB.

The volume is modest. A 200 Gb/s link carries 25 GB/s, against 64 GB/s in each direction for one PCIe 5.0 x16 slot inside a server, and a long agent prompt costs tens of milliseconds at the boundary. What the network adds is a fixed delay on every decode step, since each token passes through both servers in turn. A pipeline split also does not make one request faster than one server could make it: by our reading, the second server adds memory and lets vLLM keep more requests in flight, while each token still runs through all layers in sequence.

NVIDIA's configuration guide for NVIDIA-Certified Systems, last updated on 30 September 2026, sets the floor for the fabric: "Minimum 200 Gbps for multi-node inference", with "multiple nodes are connected with high-speed networking (either InfiniBand or RoCE)". It also asks that "NICs and NVMe drives should be placed within the same PCIe switch or root complex as the GPUs". For two four-card servers we fit one or two 200 or 400 GbE ports per server for the east-west link, beside the 25 GbE ports for users and model pulls.

We build both servers to order with the NICs for the stage link and check the PCIe placement before we quote; send us the model, the context and your rack feeds, and our team replies with the configuration for one or two servers.

Conversations on the pair, against one eight-card server

Pipeline stages also divide the cache. Each card keeps the cache only for the layers of its own stage, so a Kimi conversation of 32K, 2.14 GiB in 16-bit, costs about 1.09 GiB per card in two stages of 31 and 30 layers, against 2.14 GiB on every card under tensor parallelism over eight. Our Kimi K2 hardware requirements guide derives the cache from config.json.

Our estimates by arithmetic, not measurements: Kimi-K2.6 or K2.7-Code in INT4, 13.8 GiB of cache per card, conversations at full length, whole numbers rounded down; FP8 cache at 576 bytes per layer instead of 1,152, one byte per latent value, as vLLM 0.32.0 sizes a dense MLA cache and as in our Kimi K2.7-Code guide. DCP: decode context parallelism, which splits each conversation's cache over the cards by token.

By memory the pair matches one eight-card server in the same two-stage layout. The difference is decode context parallelism. vLLM's blog of 7 August 2026 describes it for latent-attention models such as Kimi K2.6, with the condition that the tensor-parallel size is at least the DCP size; inside one eight-card server, TP=8 with DCP 8 raises the figure to about 51 conversations of 32K by our estimate. Across two servers, DCP could only run within each four-card stage, and the blog does not discuss it together with pipeline parallelism, so we do not count on it for the pair. Treat all these figures as upper bounds: on eight H200, vLLM's Kimi recipe left about 7.4 GiB per card less cache than our rule.

Failure domain, start-up and security

Both servers carry one model, so the loss of either host stops it, as the loss of a single eight-card server would. A firmware update, a driver change or a kernel reboot on one server also stops the service, and both servers have to run the same vLLM version. A service that must keep running through a host failure needs a second copy of the model elsewhere; our guide to high availability for LLM inference on two nodes describes replicas behind a load balancer for models that fit one server.

The stage link belongs on its own network. vLLM's documentation asks to set VLLM_HOST_IP to an address on a private network and states that traffic on it is unencrypted, so keep untrusted parties off that network. Before the first load test, check that NCCL uses RDMA: start vLLM with NCCL_DEBUG=TRACE and look for [send] via NET/IB/GDRDMA in the log, while [send] via NET/Socket means plain TCP, which the documentation calls "not efficient for cross-node tensor parallelism".

Each server keeps its own copy of the checkpoint on local NVMe, 595 GB, plus room for the next version, since vLLM loads the weights for its stage on each node. Our guide to GPU server acceptance testing describes the load test to run on the pair before go-live.

Power and rack: why two servers instead of one

Rack power is the usual reason for this layout. Our estimate for a four-card server is about 3.4 kW at the wall with the cards at 600 W and about 2.7 kW with them capped at 450 W, as our four-card Server Edition server guide works out. An eight-card server comes to about 6.3 kW.

Our estimates: cards, two processors and 15 per cent for the rest, as in our sibling guides; planning at 80 per cent of the circuit, 2.94 kW at 230 V 16 A and 5.89 kW at 230 V 32 A. A measurement on the delivered system confirms the figures.

Two capped four-card servers each fit a single-phase 16 A feed per side and can sit in two racks, which also spreads the heat. The cap costs some speed; our guide to RTX PRO 6000 Server Edition power at 600, 450 and 300 W collects the published measurements. NVIDIA's configuration guide asks for system memory of twice the GPU memory, 768 GB per four-card server, the same 1,536 GB in total as one eight-card server. What doubles is the platform: two chassis, two pairs of processors, two management controllers and twice the network ports.

When one eight-card server is the better choice

If one rack position offers three-phase power and the room takes about 6.3 kW, one eight-card server is simpler for a single copy of a 600 GB model. It needs no RDMA fabric and no cluster runtime, the stage boundary stays on PCIe inside the host, and decode context parallelism over eight cards becomes possible, which raises the Kimi figure from about 12 to about 51 conversations of 32K by our estimate. Our overview of large LLMs on eight Server Edition cards sizes the eight-card case. If the two four-card servers will later run separate models, the pair keeps that flexibility; for a single large model with many users, eight H200 NVL in one server hold more cache, as our guide to an eight-card H200 NVL server shows.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build AI servers to order with four or eight RTX PRO 6000 Server Edition cards, with 25, 100, 200 or 400G NICs and out-of-band management, assembled and tested before delivery, on one EU contract and invoice with manufacturer warranty on every component. Quantities and lead time are confirmed in writing with the quote. Our page to buy a 4× RTX PRO 6000 Server Edition server lists the four-card variants. Setting up vLLM across both servers, the stage network and the model is part of our Private AI/ML service, with engineering by our partner Vixen.UNO.

FAQ