30B MoE models on 24 GB GPUs: NVIDIA L4 or RTX PRO 4000 Blackwell in one server

Which 30B mixture-of-experts models fit a 24 GB card

On one 24 GB card, most current 30B-class mixture-of-experts (MoE) models either do not fit or leave room for only one to three conversations, even in 4-bit. They become practical on two cards: a pair of RTX PRO 4000 Blackwell or L4 holds Nemotron 3.5 Lightning or Qwen3.6-35B-A3B in NVFP4 with dozens of conversations, so servers and workstations with four or eight 24 GB cards run them as TP=2 pairs. The one model in this class that fits a single card with several conversations is gpt-oss-20b.

Memory sets the limit. An MoE model with about 3B active parameters computes like a 3B model, but every one of its parameters sits in GPU memory, as our article on MoE total and active parameters explains. Qwen3.6-35B-A3B is a 37.5 GB checkpoint in FP8 and still 23.4 GB in NVFP4, against 24 GB on the card.

Hugging Face model cards and file lists (sum of the safetensors files; repository total for the Qwen FP8 and Lightning repositories, whose lists the page truncates), read on 24 October 2026. GLM-4.7-Flash: "30B-A3B" as on its card, 31B in the metadata.

The licences differ in detail. Apache 2.0 and MIT carry no use restriction. OpenMDW-1.1 requires a copy of the agreement and the copyright notices when you distribute the model, and NVIDIA's Lightning card states that the model is "ready for commercial use". The NVIDIA Nemotron Open Model License of Nemotron 3 Nano states that works "are commercially usable", as our Nemotron hardware guide quotes. Whether a clause applies to your company is for your legal department to confirm.

How much of 24 GB holds weights and cache

Our users-per-card rule takes 90 per cent of the memory the driver reports and subtracts 3 GiB per card for the CUDA context, CUDA graphs and activations. A 96 GB RTX PRO 6000 reports 95.6 GiB; scaled to 24 GB that is about 23.9 GiB, which gives 0.9 × 23.9 − 3 = 18.5 GiB for an RTX PRO 4000. The L4 has GDDR6, where ECC takes 6.25 per cent of capacity by NVIDIA's CUDA Best Practices Guide, so it has 17.2 GiB with ECC on and 18.5 GiB with ECC off, as our page on the 8× L4 server derives. Check the figure with nvidia-smi -q on your own card. The rule comes from our guide to how many users one RTX PRO 6000 can serve.

The cache per conversation is small for these models, because most of their layers keep no growing cache. Qwen3.6-35B-A3B has 40 layers, of which 10 use full attention with two KV heads of dimension 256; the other 30 are Gated DeltaNet layers with a fixed state. Nemotron 3 Nano and 3.5 Lightning have 52 layers: 23 Mamba-2, 23 MoE and 6 attention layers with two KV heads of dimension 128.

Our arithmetic from each model's config.json on Hugging Face, read on 24 October 2026: 2 × layers with a full cache × KV heads × head size × bytes; the fixed state is the float32 recurrent state of the linear or Mamba layers. Gemma 4 26B A4B from our Gemma hardware guide (an upper bound). gpt-oss-20b: 12 of 24 layers keep the full context, 8 KV heads of 64; the other 12 use a 128-token window.

One card: what fits and what does not

With these figures, one card leaves this many conversations of 32,768 tokens in flight beside the weights. Rows that need more than the usable memory are marked.

Our estimates by the users-per-card rule, not measurements: 18.5 GiB usable on the RTX PRO 4000 and 17.2 GiB on the L4 with ECC on, cache per conversation as in the table above, rounded down.

Two rows show the arithmetic. gpt-oss-20b leaves 18.5 − 12.8 = 5.7 GiB on an RTX PRO 4000, and 5.7 ÷ 0.75 gives 7 conversations. Nemotron 3 Nano in NVFP4 leaves 0.5 GiB, three conversations at 0.14 GiB. With ECC off, the L4 reaches the same 18.5 GiB and takes Gemma and Nano with one and three conversations.

The rule keeps a margin, and an engine started with a higher memory share and a short context can load Lightning or Qwen3.6 in NVFP4 on one card. That leaves no room for the 128K context that Qwen's FP8 card recommends "to preserve thinking capabilities", and no room for a second user at long context. NVIDIA's NIM support matrix, as our Nemotron guide reports, gives a minimum of 30 GB per GPU for Lightning in NVFP4 at TP1. Gemma 4 26B A4B has a further limit: NVIDIA's NVFP4 card states that vLLM currently works only with TP=1, so this checkpoint cannot be split over a pair.

Two, four or eight cards: pairs make these models work

A TP=2 pair splits every layer over two cards, and with two KV heads each card keeps one, so the cache per conversation does not grow. Two RTX PRO 4000 provide 37.0 GiB by our rule, two L4 34.4 GiB.

Our estimates by the users-per-card rule, not measurements: concurrent conversations of 32,768 tokens with an FP8 KV cache, ECC on for the L4. Last column: two pairs on four RTX PRO 4000 and four pairs on eight L4; for Qwen3.6 in FP8 one TP=4 copy on four RTX PRO 4000 and two TP=4 copies on eight L4, where its two KV heads are held twice.

For Lightning on two RTX PRO 4000: 37.0 − 20.1 = 16.9 GiB, and 16.9 ÷ 0.14 gives 121 conversations. Qwen3.6-35B-A3B in NVFP4 leaves 15.2 GiB on the same pair, 40 conversations at 0.37 GiB. In FP8, at 34.9 GiB, Qwen3.6 fills a pair of RTX PRO 4000 almost completely and exceeds a pair of L4; four cards hold it with room for about 57 conversations. GLM-4.7-Flash, published in BF16 only at 62.5 GB, needs four cards and leaves about 15.8 GiB on four RTX PRO 4000. It uses multi-head latent attention, whose compressed cache vLLM keeps whole on every card of a split copy, as our Kimi K2 guide explains for the same mechanism, so that is about two conversations of 32K with a 16-bit cache by our estimate.

These are requests in flight at the same moment, not named users. Our guide to a private ChatGPT server by company size uses about 20 requests in flight at the busiest moment for 500 staff and about 40 for 1,000. By those example values, two RTX PRO 4000 with Lightning cover a large company on memory alone, and Qwen3.6 in NVFP4 covers about 1,000 staff on one pair.

We supply the L4 and the RTX PRO 4000 and build servers and workstations with four or eight of them; send us the model, context length and peak requests in flight, and our team replies with a layout per card pair.

Formats: NVFP4 on Blackwell, FP8 and weight-only FP4 on Ada

The RTX PRO 4000 is a Blackwell card at compute capability 12.0 with fifth-generation Tensor Cores that compute FP4, so NVFP4 checkpoints run in FP4 arithmetic on it. The L4 is an Ada card at compute capability 8.9: it computes FP8 natively and has no FP4 Tensor Cores.

NVIDIA compute capability list and product pages; vLLM documentation and gpt-oss recipe as summarised in our FP4 article, read in October 2026.

On the L4, vLLM loads an NVFP4 checkpoint and falls back to weight-only execution through its Marlin kernels: the weights stay 4-bit in memory and the arithmetic runs in 16-bit, as our article on FP4 models on GPUs without FP4 Tensor Cores documents. The memory saving remains, so the pair tables above apply to the L4 as well. The cards themselves name other hardware. Qwen3.6-35B-A3B-NVFP4 lists "NVIDIA Hopper and NVIDIA Blackwell" and was tested on a GB300; the Gemma 4 26B A4B NVFP4 card lists Blackwell, tested on a B200; the Lightning card lists DGX Spark, GB200, H100 and H200, and Ampere "via W4A16". None names the L4 or the RTX PRO 4000, so a short load test of the exact checkpoint and engine version belongs in the plan. Our page on Blackwell RTX software support covers the engines on compute capability 12.0.

Qwen's FP8 checkpoint uses "fine-grained fp8 quantization with block size of 128". That format needs kernels for block-scaled FP8, which TensorRT-LLM's support matrix does not list for Ada, so confirm that your engine runs it on the L4 before you plan on it. The Nemotron 3 Nano FP8 card names the H100 and A100 as supported hardware and quantises the KV cache to FP8 as well. Our guide to FP8, NVFP4, MXFP4, INT4 and GGUF explains the formats themselves.

Speed: why 3B active parameters suit 24 GB cards

For one conversation, decoding is limited by how fast the card reads the weights that each token needs. A dense 32B model in FP8 reads about 34 GB per token; Nemotron 3 Nano in FP8, with 3.5B active parameters, reads about 3.5 GB plus its shared layers.

Our arithmetic, memory bandwidth (300 GB/s L4, 672 GB/s RTX PRO 4000) ÷ bytes read per token, active weights only. Upper bounds by our estimate: real decoding reaches a fraction, a pair loses time to the all-reduce over PCIe, and the dense row is from our four-card RTX PRO 4000 page.

In a batch, the tokens of different conversations route to different experts, so one step reads more than the active weights of one token, and the MoE figures are upper bounds by our estimate. The order holds: on cards with 300 or 672 GB/s, a model with 3B active parameters answers several times faster than a dense model of similar quality, which is the reason to pick this class for 24 GB cards.

Published measurements on 24 GB cards are scarce. StorageReview ran Gemma 4 26B A4B in NVFP4 with an FP8 cache on the 32 GB RTX PRO 4500 Blackwell Server Edition on 24 September 2026 and reported 1,613 output tokens per second at 32 concurrent requests with 256-token prompts and answers; it did not run the model on the L4, which has no FP4 hardware. We found no published result of these models on an RTX PRO 4000, so treat throughput as unknown until a test on your prompts. Our RTX PRO 4500 benchmarks collect what has been measured on the neighbouring card.

Eight L4 in a server or four RTX PRO 4000 in a workstation

The two cards belong in different machines. The L4 is a passive, low-profile, single-slot data-centre card at 72 W, so eight fit a 2U or 3U server whose maker lists them; Lenovo lists eight in its SR650 V3, as our 8× L4 server page shows. The RTX PRO 4000 is an actively cooled single-slot workstation card at 145 W; Dell's Precision 7960 configurator offers four, as our page on four RTX PRO 4000 in one workstation lists. Supermicro's SYS-322GA-NR datasheet, read on 24 October 2026, includes the RTX PRO 4000 Blackwell among the GPUs of that 3U server, and a server takes an active card only where its maker lists it.

NVIDIA product pages and datasheets; usable memory, conversations and power by our estimate as on our 8× L4 and 4× RTX PRO 4000 pages (cards plus processors plus 15 per cent).

The L4 server has more memory and four pairs, so it holds more conversations; each RTX PRO 4000 pair answers faster per token and computes NVFP4 natively. Both draw under the 2.94 kW we plan per 230 V 16 A feed. A measurement on the delivered system confirms the power estimate. Tell us whether the machine goes into a rack or under a desk, and we name the chassis that lists these cards.

When another card fits better

Where one card should hold one of these models with room for users, more memory per card is the fix. One RTX PRO 4500 Blackwell Server Edition holds 25.7 GiB by our rule: Lightning in NVFP4 with about 39 conversations of 32K and Qwen3.6-35B-A3B in NVFP4 with about 10, on one card, with MIG halves for small models, as our page on the 8× RTX PRO 4500 Server Edition server shows. One RTX PRO 6000 Server Edition with 83.0 GiB holds Qwen3.6-35B-A3B in FP8 with about 129 conversations of 32K.

The single-card comparison of the two slot-saving cards is in our article on single-slot GPUs: RTX PRO 4000 or RTX PRO 4500 Server Edition. For dense models of up to about 14B parameters on one 24 GB card, our guide to small language models on the L4 has the per-model tables, and the Qwen family is sized in our Qwen hardware guide.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We supply the L4 and the RTX PRO 4000 Blackwell as cards for servers and workstations you already run, and build servers with eight L4 and workstations with four RTX PRO 4000 to order, assembled and tested before delivery, with the operating system, NVIDIA driver, CUDA and a container runtime installed on request. Every component carries the manufacturer warranty, handled through us, on one EU contract and invoice; quantities and lead time are confirmed in writing with the quote. The ordering steps are on our pages to buy an L4 server and to buy an RTX PRO 4000 workstation.

FAQ