Mistral Medium 3.5 hardware requirements: two H200 NVL or two RTX PRO 6000 Server Edition

Mistral Medium 3.5 on two GPUs: which pair

Mistral Medium 3.5 needs two GPUs: its 124.4 GiB of FP8 weights miss one H200 NVL by about 1 GiB by our users-per-card rule. Two NVIDIA H200 NVL cards on an NVLink bridge are the pair we recommend for a service, with three times the cache and three times the generation speed of two NVIDIA RTX PRO 6000 Blackwell Server Edition cards. Two Server Edition cards run the model for a pilot or a handful of users, and four of them hold more conversations than two H200 NVL (37 against 22 at 32K) but reach only about two thirds of their speed per stream.

Our estimates: users-per-card rule, conversations at full length; ceiling = bandwidth of both cards ÷ 133.6 GB, an upper bound. Card data from NVIDIA as in our model and card guides; link bandwidths as NVIDIA gives them, totals over both directions. Wall power from our two-card server pages: cards at 600 W plus two AMD EPYC 9135 (Server Edition pair) or two EPYC 9355 (H200 NVL pair), plus 15 % for the rest of the server.

The rest of this page explains each row, then gives a recommendation per use case. The family overview, with Mistral Small 4, Devstral and Large 3, is our guide to Mistral hardware requirements.

What Mistral publishes for Medium 3.5

The model card on Hugging Face describes "a dense 128B model with a 256k context window", with text and image input, and states that it "supersedes all our previous coding models, namely Devstral, across all benchmarks" and "replaces Devstral 2 in our coding agent Vibe".

Mistral AI, model card, config.json, LICENSE and file list of Mistral-Medium-3.5-128B and its EAGLE draft on Hugging Face, read on 24 October 2026. 133.6 GB is the sum of the three model safetensors files, 133,606,164,352 bytes.

The repository holds the weights twice, as Hugging Face files and as Mistral's consolidated files of the same size, so a download is about 267 GB; one copy goes into GPU memory. The config.json sets weight_block_size to null with static activation scales, which means one FP8 scale per tensor. The vision encoder and the output head stay in BF16.

The card's vLLM and SGLang examples run the model with --tensor-parallel-size 8 and --gpu_memory_utilization 0.8, on vLLM nightly. Mistral's release post, as quoted in our Mistral guide, calls self-hosting "possible on as few as four GPUs". Neither states a context length or load, so we size from memory and bandwidth; Mistral's own smallest figure for self-hosting is four GPUs.

Why one H200 NVL is not enough

Our users-per-card rule takes 90 per cent of the memory the driver reports and subtracts 3 GiB per card for CUDA graphs and activations: 0.9 × 140.4 − 3 = 123.4 GiB per H200 NVL and 0.9 × 95.6 − 3 = 83.0 GiB per RTX PRO 6000 Server Edition. The weights are 124.4 GiB, so one H200 NVL is short by 1.07 GiB before a single conversation.

A higher gpu_memory_utilization would load the model on one card with a few GiB to spare, but that cache would hold less than one conversation of 32K, too little for a service. The derivation of the rule is in our guide to how much VRAM an LLM needs.

KV cache: 352 KiB per token

A dense model with grouped-query attention keeps keys and values for every layer. For Medium 3.5 that is 88 layers × 2 × 8 KV heads × 128 values, 352 KiB per token in 16-bit or 176 KiB with an FP8 cache, set with --kv-cache-dtype fp8 in vLLM. One conversation of 32K takes 11 GiB in 16-bit or 5.5 GiB in FP8. With tensor parallelism over two or four cards, the eight KV heads divide evenly, so each card holds its share of every conversation.

Our estimate by the users-per-card rule: one copy split with tensor parallelism, concurrent conversations at full length; cache from config.json. With the EAGLE draft loaded, about 2.9 GiB less per copy.

These are requests in flight at the same moment, far fewer than the staff who use the service. Coding agents fill long contexts, though, and at 128K with an FP8 cache two H200 NVL hold five conversations and two Server Edition cards one. We supply both pairs and the four-card versions; send us your context length and peak requests in flight, and our team replies with the configuration and these figures for your case.

Speed: a dense model reads 133.6 GB per token

Every generated token reads all the weights of a dense model once, so per stream, memory bandwidth sets the ceiling. The RTX PRO 6000 Server Edition has 1,597 GB/s and the H200 NVL 4,800 GB/s, three times as much; both cards are rated up to 600 W.

Our estimate: combined bandwidth ÷ 133.6 GB of weights, an upper bound that ignores KV-cache reads and the exchange between the cards.

Real rates are lower, and the gap grows with the number of cards, because every layer waits for the cards to exchange results. Batching many users raises the total but leaves the per-user ceiling where it is. For a mixture-of-experts model the same card gives far more, since it reads only the active experts per token; for a dense 128B model, the H200 NVL's HBM3e decides the user experience. We found no published measurement of Medium 3.5 on either card as of 24 October 2026. Our comparison of RTX PRO 6000 and H200 NVL for LLM inference shows the same arithmetic for 70B models.

NVLink bridge or PCIe for a dense split

Tensor parallelism splits every layer and, per the Megatron-LM paper, needs two all-reduces per transformer layer. For the 88 layers of Medium 3.5 that is 176 all-reduces per generated token, each carrying the 12,288-wide activation vector, 24 KiB in BF16, so about 4.1 MiB per token for a TP=2 pair by our estimate. The volume is small; the waiting is what costs time, 176 times per token.

On two H200 NVL the 2-way NVLink bridge carries this at 900 GB/s per GPU. On two Server Edition cards it runs over PCIe Gen5 at 128 GB/s, and the latency per exchange matters more than the bandwidth: Google reported that working PCIe peer-to-peer cut the latency between RTX PRO 6000 Server Edition cards from 12.68 to 0.53 µs, as our article on one model on several GPUs over PCIe or NVLink cites. vLLM keeps its own custom all-reduce for two PCIe cards, and disables it by default for more than two, so a four-card Server Edition split falls back to NCCL unless the opt-in FlashInfer PCIe IPC all-reduce that vLLM 0.30.0 added for systems without NVLink (release notes of 22 September 2026, #53576) is switched on.

For the H200 NVL pair, the server maker has to list the 2-way bridge for the exact server, and NVIDIA's brief asks for bridged cards under the same CPU, as our guide to the H200 NVL NVLink bridge explains.

Speculative decoding with Mistral's EAGLE draft

Mistral publishes an EAGLE draft model for Medium 3.5, a single 3.07 GB file. The card's vLLM example sets "method": "eagle" and "num_speculative_tokens": 3 in --speculative_config, with a max_model_len of 65,536 for the draft. The draft proposes several tokens, and the large model checks them in one pass, which reads the 133.6 GB of weights once for several tokens. For a dense model bound by bandwidth that is the most direct way to raise speed per user on either pair.

The card gives no acceptance rate or speed-up and points to an SGLang cookbook for benchmarks, so test it on your own prompts. Our guide to speculative decoding on-premise explains the settings.

Software: FP8 per tensor on both cards

Per-tensor FP8 is the simplest FP8 format for the engines. The H200 NVL computes FP8 natively on Hopper. On the RTX PRO 6000 Server Edition, compute capability 12.0, TensorRT-LLM's quantisation matrix lists FP8 per tensor for sm120, and vLLM supports FP8 from compute capability 8.9, as our guide to Blackwell RTX (SM120) software support sets out. This is easier than block-scaled FP8 checkpoints such as Mistral Large 3, whose 128 × 128 blocks TensorRT-LLM does not list for sm120.

We found no NVFP4 version of Medium 3.5 from Mistral as of 24 October 2026; the NVFP4 releases on its Hugging Face page are for Small 4 and Large 3. We therefore size the FP8 release only.

Devstral 2 hardware requirements and the licence

Devstral 2, Mistral's 123B dense coding model, has the same 88-layer layout and sizes almost the same, as our Mistral guide shows. Medium 3.5 now takes its place in Mistral's own coding agent, so for a new coding server we size Medium 3.5. Our article on a server for coding agents covers the workload.

Both models come under a "Modified MIT License". Its second section reads: "You are not authorized to exercise any rights under this license if the global consolidated monthly revenue of your company (or that of your employer) exceeds $20 million (or its equivalent in another currency) for the preceding month." Above that line, the licence names two routes: a commercial licence from Mistral AI, "which Mistral AI may grant you at its sole discretion", or Mistral's hosted services. At an even monthly revenue that line lies at about USD 240 million a year, so a larger company should settle the licence before it buys hardware for the model. Whether the clause applies to your company is a question for your legal counsel.

Which configuration for which use

• Pilot or a few users: two RTX PRO 6000 Server Edition. The model runs, seven conversations of 32K fit with an FP8 cache, and about 24 tokens per second is the ceiling per stream. The same server runs other models later.
• A department's assistant or coding service: two H200 NVL with the 2-way bridge, 22 conversations of 32K and a ceiling three times higher. Our page on the two H200 NVL server has the configuration.
• Capacity on Blackwell cards: four RTX PRO 6000 Server Edition, 37 conversations of 32K, about 3.4 kW at the wall by our estimate, and a TP=4 split over PCIe. Our four-card Server Edition server page has the details.
• Twenty and more users at long contexts: four H200 NVL in one 4-way NVLink domain, 67 conversations of 32K with an FP8 cache; our four H200 NVL server page covers it, also at about 3.4 kW.

The H200 NVL also includes a five-year NVIDIA AI Enterprise subscription, which starts 90 days after the card ships to the server maker.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We supply the H200 NVL with its NVLink bridges and the RTX PRO 6000 Server Edition in AI servers built to order, assembled and tested before delivery, on one EU contract and invoice with manufacturer warranty on every component. Setting up Medium 3.5 with its EAGLE draft, retrieval and the platform around it is our Private AI/ML service.

FAQ