Server for coding agents: 2, 4 or 8 RTX PRO 6000 Server Edition for 20 to 100 developers

Which server 20 to 100 developers need for coding agents

For 20 to 100 developers who use coding agents on open models, a server with two, four or eight NVIDIA RTX PRO 6000 Blackwell Server Edition cards covers the range, chosen by how many agents generate at the same time and how long their contexts grow. Two cards running Qwen3-Coder-Next carry at most about 35 agent sessions of 64K tokens at a planned 50 tokens per second each, four cards about 70 and eight cards about 140, an upper bound by our estimate that a pilot confirms; an FP8 cache roughly doubles each figure.

A team that wants the largest Qwen coding model, Qwen3-Coder-480B-A35B, needs eight cards for at most 13 to 26 sessions, or eight H200 NVL for 34 to 69. Our guide to a private coding assistant covers the model choice, licences, completion models, IDE extensions and the first tier from DGX Spark to one or two cards. This page starts where that one stops: a development department of 20 to 100 people, where several dozen agents work at once and the question is the server.

The task in plain words

A coding agent works through a task in steps. It reads files, runs a build or the tests, reads the output and calls the model again with everything it has seen so far. After an hour of work the prompt of each call carries tens of thousands of tokens. The model keeps a working memory for every token of every open session, the KV cache, and every new token it writes reads the whole cache of its session again.

Two numbers follow for the server. The first is how many sessions fit beside the model in GPU memory. The second is how many of them can generate at once before each one slows below a useful speed, because every session adds the bytes of its cache to each step. The smaller of the two decides how many developers the server carries.

Developers are not sessions. Many people use agents only part of the day, and some run two agents in parallel. Our rule counts one generating session per developer at the busy hour, the upper bound for a team where every developer works with an agent. The gateway log of a pilot gives the real share, and a lower share lets the same server carry more people.

Context length per session: 64K, 128K or 256K

Ollama's documentation asks for at least 64,000 tokens of context for agents and coding tools, and the Qwen3-Coder models have a native context of 262,144 tokens. We size at 64K (65,536 tokens) and at 128K (131,072), the lengths at which agents on large repositories usually run, and leave 256K for the few sessions that need it.

The cache per token is 2 × layers with full attention × key/value heads × head dimension × bytes per value, with the numbers from each model's config.json on Hugging Face. Models that use full attention in only part of their layers need much less.

Repository sizes from Hugging Face, read on 24 October 2026 (gpt-oss-120b: the 15 safetensors files in the root). Cache in BF16 by our arithmetic from each config.json; an FP8 cache halves it. Qwen3-Coder-Next adds up to about 80 MiB of linear-attention state per session.

Qwen3-Coder-480B-A35B has 62 layers with 8 key/value heads of dimension 128, which gives 248 KiB per token. Qwen3-Coder-Next, an 80B model with 3B active parameters, keeps a per-token cache only in every fourth of its 48 layers, with 2 key/value heads of dimension 256, 24 KiB per token. gpt-oss-120b uses a 128-token sliding window in 18 of its 36 layers and full attention in the other 18, with 8 key/value heads of dimension 64, and its context ends at 131,072 tokens. All four are published under Apache 2.0.

Two limits: memory and speed per session

Our sizing rule takes 90 per cent of the memory the driver reports, less 3 GiB per card, less the weights: 83.0 GiB usable per RTX PRO 6000 and 123.4 GiB per H200 NVL. Two cards give 166.1 GiB, four 332.2 GiB and eight 664.3 GiB. Divided by the cache per session, the room beside the weights gives the sessions that fit in memory.

Speed is the second limit. Each decoding step reads the active weights and the cache of every session in the batch, so the bandwidth of a card divided by the bytes read per step gives a ceiling per session. We count the active weights of one token per step; in a batch of a mixture-of-experts model the sessions route their tokens to different experts and a step reads more, so each ceiling is an upper bound by our estimate, and so is every session count at 50 tokens per second derived from it. We plan that ceiling at 50 tokens per second, two and a half times the 20 tokens per second implied by the 50 ms per token that our guide to LLM latency targets gives as an example for agent steps, because real decoding stays well below the ceiling. For Qwen3-Coder-Next on a pair, the memory would hold 57 sessions of 64K, but at 57 the ceiling falls to about 32 tokens per second; at 35 sessions it stays at 50.

Sessions per server by our sizing rule and our ceiling arithmetic, BF16 cache (FP8 cache in brackets; gpt-oss-120b with BF16 cache only); "planned" is the smaller of the two limits, and where memory is the smaller one the speed column repeats it (on eight H200 NVL the ceiling at 34 or 69 sessions of 64K is still about 63 tok/s). 1,597 GB/s per RTX PRO 6000 Server Edition and 4.8 TB/s per H200 NVL (NVIDIA); active weights read per step: 3 GB for Coder-Next, about 5 GB for gpt-oss-120b, 35 GB for Coder-480B. Estimates, not measurements; the counts at 50 tok/s rest on ceilings that are an upper bound by our estimate.

The ceilings leave out the all-reduce traffic between cards and, more importantly, prefill: each agent step adds the output of a tool or a build, which the model has to compute before it answers. Prefix caching, on by default in vLLM, keeps the earlier part of the session reusable while its cache stays in memory, so the first figure to measure in a pilot is the share of prefix cache hits. Our guide to long-context LLM hardware explains the effect.

How many developers each configuration carries

With one generating session per developer at the busy hour, the table translates into team sizes. These are the ranges we propose before a pilot, at 64K contexts unless stated; where speed sets them, they are an upper bound by our estimate.

Our estimates from the table above, one generating session per developer at the busy hour. An FP8 cache is set with --kv-cache-dtype fp8 in vLLM; check answer quality with it on your own tasks before you size on it.

For 20 to 40 developers on Qwen3-Coder-Next, two cards are the start; for 40 to 100, four cards, with the FP8 cache taking them to 100 if your evaluation accepts it; at 100 developers with 128K contexts, eight cards. The 480B model changes the scale: it needs eight cards for its weights alone and then carries 13 to 26 sessions of 64K on eight RTX PRO 6000 Server Edition, 34 to 69 on eight H200 NVL, so it suits a team that needs its quality for a subset of tasks, next to a smaller model for the rest.

We build these servers with two, four or eight RTX PRO 6000 Server Edition or eight H200 NVL; send us your developer count, the share using agents and the planned context, and our team replies with a configuration and its sessions.

Two cards: one pair for a team of 20 to 35

The entry server on our AI servers page carries two RTX PRO 6000 Server Edition cards with two AMD EPYC processors, 384 GB of DDR5 ECC memory and two NVMe drives. Qwen3-Coder-Next in FP8 runs on it as one TP=2 instance, as its Hugging Face card shows with --tensor-parallel-size 2 and --tool-call-parser qwen3_coder. The 2 key/value heads split one per card, so the cache divides evenly.

The server draws about 1.8 kW by our estimate and runs on one 230 V 16 A feed. Its limit is redundancy: one instance serves everyone, and a card fault stops all agents. gpt-oss-120b fits the same pair with at most 24 sessions at the planned speed, an upper bound by our estimate, or runs as two single-card copies with 9 sessions of 64K each in memory, if the team prefers two independent instances.

Four cards: two pairs for 40 to 100 developers

The inference node carries four RTX PRO 6000 Server Edition cards with 768 GB of DDR5 ECC and four NVMe drives, at about 3.4 kW by our estimate. Four cards run Qwen3-Coder-Next best as two pairs behind a load balancer with session affinity, so that each agent returns to the copy that holds its cache. A single TP=4 instance would duplicate the cache of a model with two key/value heads, and by default vLLM disables its custom all-reduce for more than two GPUs without NVLink, as our coding assistant guide explains; vLLM 0.30.0 (release notes of 22 September 2026) added an opt-in FlashInfer PCIe IPC all-reduce for systems without NVLink (#53576). Our article on the RTX PRO 6000 without NVLink covers what a pair sends over PCIe.

Two pairs also give the department a working server while one pair is updated. The same node can instead run one pair with Qwen3-Coder-Next and one with gpt-oss-120b, so that teams compare the models on their own repositories.

Eight cards: 100 developers or the 480B model

Eight cards are the step for a department of about 100 whose agents work all day, or for Qwen3-Coder-480B-A35B. Its FP8 checkpoint of 482 GB, 449 GiB, leaves 215 GiB of cache room on eight RTX PRO 6000 Server Edition by our rule: 13 sessions of 64K with a 16-bit cache, 27 in FP8. With 8 key/value heads, TP=8 places one head per card, so nothing is duplicated. vLLM's Qwen3 usage guide asks for --enable-expert-parallel with TP=8 on FP8 checkpoints of these MoE models, and the model card advises reducing the context, for example to 32,768, if the server runs out of memory.

Eight H200 NVL with NVLink bridges, the training node on our AI servers page with 2.3 TB of DDR5 ECC and 400G networking, hold the same model with 34 sessions of 64K, 69 with an FP8 cache, and three times the bandwidth per card. The bridges join four cards each, so a TP=8 instance still crosses PCIe between the two NVLink domains. Both eight-card servers draw about 6.3 kW by our estimate with two AMD EPYC 9555 processors, and Dell's datasheet gives about 8 kW per eight-card node as an indicative figure, so plan the 400 V three-phase feed on the server maker's figure, as our GPU server power guide sets out; host memory follows NVIDIA's certified-systems guide at twice the GPU memory, 1,536 GB for eight RTX PRO 6000 Server Edition.

A published throughput figure for these cards

The MLPerf® Inference v6.1 results include gpt-oss-120b, one of the coding models in the table. Eight RTX PRO 6000 Server Edition in an HPE ProLiant Compute DL380a Gen12 produced 15,256 tokens per second offline and 14,581.7 tokens per second in the server scenario, which holds latency bounds per request. By our division, not an MLPerf metric, that is about 1,823 tokens per second per card in the server scenario. Other eight-card Server Edition systems reached 15,508.4 (Cisco UCS C845A M8) and 15,738.5 (Nebius) offline on the same benchmark.

A standard benchmark does not reproduce agent sessions on your code base, so the figure says what the cards deliver on this model in a standard test, not how many developers they carry. MLPerf® Inference v6.1, data centre, closed division, gpt-oss-120b, offline and server scenarios, entries 6.1-0050, 6.1-0013 and 6.1-0082, retrieved from mlcommons.org on 24 October 2026, results verified by MLCommons Association. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

What to measure in a pilot before you order

A pilot with recorded agent tasks from your own repositories replaces our assumptions with your figures. Our guide to what a private AI pilot should measure has the method; for coding agents these are the numbers that decide between two, four and eight cards.

1. Sessions generating at the same time at the busy hour, from the gateway log, against the number of developers.
2. Prompt length at the last step of a long task, which sets 64K or 128K.
3. Prefix cache hit rate, from vllm:prefix_cache_hits divided by vllm:prefix_cache_queries.
4. Time to first token and inter-token latency per step at the busy hour.
5. Task success on your own repositories with the FP8 cache against the 16-bit cache, and with Qwen3-Coder-Next against the 480B model.
6. Preemptions in vllm:num_preemptions, which show agents losing their cache.

Our engineering partner Vixen.UNO deploys coding models on-premise with access control and logging as part of our Private AI/ML service; describe your repositories and how your developers work with agents, and we set up the pilot and size the server from its figures.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build AI servers to order with two, four or eight RTX PRO 6000 Server Edition, the passive card for rack servers in continuous operation, or with eight H200 NVL with NVLink bridges, assembled and tested before delivery. Operating system, drivers, CUDA and a container runtime are installed on request. Every component carries the manufacturer warranty, and the order runs on one EU contract and invoice.

FAQ