AI agent server for 100K-token contexts: 4 or 8 RTX PRO 6000 Server Edition or H200 NVL

Which server ten AI agents with 100K context need

For about ten LLM agents that work all day with contexts above 100,000 tokens, the KV cache decides the size of the server, more than the model does. With Qwen3-235B-A22B in FP8 at tensor parallelism of four, four NVIDIA RTX PRO 6000 Blackwell Server Edition cards hold the model and 12 agents at 100,000 tokens, but only 9 at 131,072, by our sizing rule with an FP8 cache. For ten agents at 128K with room to spare, take eight Server Edition cards as two TP=4 copies, which hold 18, or four H200 NVL joined by a four-way NVLink bridge, which hold 23.

The figures in this article are our estimates, not measurements, for a request we see often: a lab or a development team wants about ten agents working through documents, code or data all day, on an open mixture-of-experts model served with TP=4.

The task in plain words

An agent works in steps. It calls the model, runs a tool such as a search, a script or a database query, adds the result to its context and calls the model again. After a few hours of work an agent can carry 100,000 tokens or more: the task, the documents it read, the tool output and its own notes. Our guide to GPU sizing for AI agents explains why one agent task sends many times the tokens of a chat.

The GPU keeps a working memory for every token of every open agent, the KV cache. For a chat assistant with 8,000-token conversations it is small; for ten agents at 128,000 tokens it is 1.3 million tokens held at once, which on this model takes more GPU memory in 16-bit than the model itself. Every new token also reads the whole cache, so long contexts slow each step. That is why the decision for this task is about memory per agent and bandwidth, and the card count follows from it.

Four options at a glance

Agents at full context length per server, Qwen3-235B-A22B-Instruct-2507 in FP8, FP8 KV cache (16-bit cache in brackets); 100K = 100,000 and 128K = 131,072 tokens. Our estimate by the sizing rule below, not a measurement.

With an FP8 cache, eight Server Edition cards and four H200 NVL both carry ten agents at 128K with room to grow. If your evaluation shows that the model needs a 16-bit cache, only the H200 NVL configurations hold ten agents at 128K. Four Server Edition cards carry ten agents at 100K with an FP8 cache and little margin, and they are the right start when the contexts stay at that length.

Why long context makes the KV cache the sizing driver

For a model with full attention in every layer, the cache per token is 2 × layers × KV heads × head dimension × bytes per value. The config.json of Qwen3-235B-A22B-Instruct-2507-FP8 on Hugging Face gives 94 layers, 4 KV heads and a head dimension of 128, so a token takes 188 KiB in 16-bit and 94 KiB in FP8. Qwen's model card gives a native context of 262,144 tokens, and its vLLM example uses --tensor-parallel-size 4.

Our arithmetic from the model's config.json, read on 24 October 2026. The FP8 checkpoint is 236 GB on Hugging Face, 219.8 GiB.

Our sizing rule takes 0.9 of the memory the driver reports, less 3 GiB per card, less the weights: 83.0 GiB usable per RTX PRO 6000 and 123.4 GiB per H200 NVL. Four Server Edition cards give 332.2 GiB, which leaves 112.4 GiB of cache beside the weights; four H200 NVL give 493.4 GiB and leave 273.6 GiB. Divided by the cache per agent, that gives the table above. The FP8 cache is set in vLLM with --kv-cache-dtype fp8; check answer quality with it on your own evaluation set before you size on it.

TP=4 suits this model. With four KV heads, each card holds one head, so the cache divides evenly. On eight cards, one TP=8 instance would not hold more agents than two TP=4 copies: vLLM's context-parallel documentation states that once TP exceeds the KV heads, the cache "will be duplicated for tp_size / H times". Decode context parallelism (--decode-context-parallel-size) can remove that duplication and, by our arithmetic, roughly double the agents of a single TP=8 instance, and vLLM's Qwen3 usage guide asks for --enable-expert-parallel with TP=8 on FP8 MoE checkpoints. Test that layout in the pilot before you count on it. Our guide to RTX PRO 6000 without NVLink explains what TP=4 sends over PCIe.

How long context slows each agent

Decoding reads the active weights and the whole cache of every agent in the batch for each new token. Qwen3-235B-A22B activates 22B parameters per token, about 22 GB in FP8, and each agent adds its own cache. In a batch, the agents' tokens route to different experts, so a step reads more weights than one token does; our arithmetic counts only the 22 GB, so each figure below is an upper bound by our estimate. Dividing the memory bandwidth by the bytes read per step gives a ceiling per agent.

Our ceiling arithmetic: 1,597 GB/s per RTX PRO 6000 Server Edition and 4.8 TB/s per H200 NVL (NVIDIA), FP8 weights and FP8 cache, TP=4; each value is an upper bound by our estimate. Real speeds are lower: all-reduce traffic, attention compute and prefill of new tool results are not counted.

The pattern matters more than the figures: at 100K the cache of ten agents adds about 24 GB per card to every decoding step, against about 2 GB at 8K, and the H200 NVL's bandwidth shows most at long context. A published measurement points the same way. In a Level1Techs forum build log of July 2026, four RTX PRO 6000 Max-Q at 300 W ran DeepSeek-V4-Flash with SGLang at TP=4 at 697 output tokens per second with 2,048-token inputs at 80 concurrent requests, and 39 tokens per second with 65,536-token inputs at 24.

Prefill is the other cost. Each step adds a tool result that the model has to compute, and a cache miss means recomputing the whole context. vLLM's prefix caching keeps earlier steps reusable, as our guide to long-context LLM hardware explains, so the share of hits is one of the first numbers to measure.

We build inference servers with four or eight GPUs for workloads like this; send us the model, the context length per agent and the number of agents, and our team replies with the cache budget, a configuration and a quote.

Four or eight RTX PRO 6000 Server Edition

The inference node on our AI servers page carries four RTX PRO 6000 Server Edition cards with two AMD EPYC processors, 768 GB of DDR5 ECC memory and four NVMe drives. For this task it holds the FP8 model and ten agents up to 100K, as one TP=4 instance. Everything runs through one copy: a card fault or a hang stops all agents, and the all-reduces of TP=4 run over PCIe through NCCL.

Eight cards run two TP=4 copies of the model, each on one processor socket, behind a load balancer with session affinity so that each agent returns to the copy that holds its cache. That doubles the agents to 18 at 128K, and one copy keeps working while the other is serviced. Servers with eight cards are built to order and follow the same platform rules: two sockets, at least six processor cores per GPU and memory at twice the GPU memory, as NVIDIA's certified-systems guide sets.

Four or eight H200 NVL with NVLink

The H200 NVL has 141 GB of HBM3e at 4.8 TB/s, three times the bandwidth of the RTX PRO 6000 Server Edition, and NVIDIA's H200 page lists two- or four-way NVLink bridges at 900 GB/s per GPU against 128 GB/s for PCIe Gen5. Four H200 NVL on one four-way bridge run the TP=4 instance with its all-reduces on NVLink, and hold 23 agents at 128K with an FP8 cache or 11 with a 16-bit one. The card runs FP8 natively but has no FP4 arithmetic, which does not matter for this FP8 checkpoint, and NVIDIA includes a five-year NVIDIA AI Enterprise subscription with it.

Eight H200 NVL form two four-way domains, two TP=4 copies with NVLink inside each, for about 46 agents at 128K. That is the training node on our AI servers page: eight H200 NVL with NVLink bridges, two AMD EPYC processors, 2.3 TB of DDR5 ECC, eight NVMe drives and 400G networking. It also leaves room for fine-tuning work beside the agents. Our comparison of RTX PRO 6000 and H200 NVL for LLM inference covers the cards in more detail.

Why the Server Edition and not the Max-Q in this rack

The RTX PRO 6000 Max-Q Workstation Edition is the workstation card: 300 W, active cooling, and in NVIDIA's words "optimized for dense workstation configurations (up to four GPUs)". The Server Edition is passive and, per NVIDIA, "designed for multi-GPU server deployments requiring passive cooling". Its product brief SP-12355-001 (June 2025) specifies the airflow it needs at the heat sink, from 43 CFM at 25 °C inlet to 125 CFM at 45 °C, the airflow a GPU server's fans are built to deliver.

Many server makers do not list the Max-Q for their GPU servers. Lenovo's product guide LP2364 lists it for two ThinkSystem servers and marks all others as unsupported, while Supermicro names it in the GPU list of the SYS-422GA-NRT without a Max-Q count (the eight cards it states are Server Edition). The Server Edition is also the edition on NVIDIA's vGPU list, and its power limit, 300 W to 600 W by the brief, can be set out of band so that it persists across driver reloads. The Max-Q has more memory bandwidth on paper, 1,792 against 1,597 GB/s, which would raise the decode ceilings above by about 12 per cent. For an agent server that runs all day in a rack, a chassis the maker has validated with this card matters more. Four cards at 600 W draw 2.4 kW and eight draw 4.8 kW before the host; at the wall, with two AMD EPYC 9355 or two EPYC 9555 processors and 15 per cent for the rest of the server, that is about 3.4 kW and 6.3 kW by our estimate. Plan the feed on the server maker's power-supply rating, as our article on GPU rack power and cooling explains.

Host memory and KV cache offloading

NVIDIA's certified-systems guide, updated on 30 September 2026, sets system memory at a minimum of twice the GPU memory: 768 GB for four RTX PRO 6000 Server Edition, 1,536 GB for eight, 1,128 GB for four H200 NVL and 2,256 GB for eight, which we build as 1,152 GB and 2.3 TB. For agents, that memory has a second job.

Agents pause while their tools run, and the engine may evict the cache of a waiting agent to make room for others. KV cache offloading keeps those blocks in CPU memory, so the agent resumes without recomputing 100,000 tokens of prefill. vLLM's offloading guide, dated 22 October 2026, says the connector "extends the prefix cache by offloading completed KV blocks to slower but larger tiers", and recommends a CPU tier larger than the aggregate GPU KV cache, because "a smaller CPU tier just mirrors what the GPU already holds and adds no hit rate."

For four Server Edition cards with 112.4 GiB of GPU cache, a CPU tier of 256 GiB holds about 21 agent contexts at 128K in FP8, by our arithmetic. Of 768 GB, about 715 GiB, that leaves room for the page cache of the 219.8 GiB checkpoint and about 240 GiB for the operating system, the agents' tools and a retrieval index. Each card reloads its quarter of a 128K context, about 3.2 GB, in no less than about 50 ms at the 64 GB/s that PCIe 5.0 x16 carries each way. With eight cards, plan the 1,536 GB that the NVIDIA rule gives and a tier per copy. The options and flags are in our guide to KV cache offloading to CPU memory and NVMe, and host sizing in our guide to CPU, RAM and storage for a 4-GPU server.

Our engineering partner Vixen.UNO deploys vLLM and agent workflows on-premise as part of our Private AI/ML service; describe how long your agents run and how long their tools pause, and we size GPU, host memory and NVMe together.

What to measure in a pilot

A pilot on the planned model with recorded agent tasks turns these estimates into your numbers. Our guide to what to measure in a private AI pilot covers the method; for long-context agents, these are the figures that decide four or eight cards.

1. Context per call at the last step of a task, and how it grows over a working day, from the gateway log.
2. Prefix cache hit rate, from vllm:prefix_cache_hits divided by vllm:prefix_cache_queries.
3. Cache pressure: vllm:kv_cache_usage_perc near 1 and any rise in vllm:num_preemptions, which means requests lost their cache.
4. Time to first token per step (vllm:time_to_first_token_seconds) and inter-token latency (vllm:inter_token_latency_seconds) at the peak hour.
5. Answer quality with an FP8 cache against a 16-bit cache on your own evaluation set.
6. Tool pause lengths, which tell you how much offload memory keeps agents warm.
7. Stability of the TP=4 instance over several days, with the engine release and driver you plan to run.

Our guide to vLLM monitoring metrics explains how to collect these with Prometheus.

FAQ