Llama 3.3 Nemotron Super 49B on NVIDIA L40S: two cards, or one newer card

Why one L40S is not enough for the FP8 checkpoint

Llama 3.3 Nemotron Super 49B v1.5 does not fit one NVIDIA L40S in FP8: the checkpoint is 52.0 GB (48.4 GiB), and a 48 GB L40S holds about 37.3 GiB of weights and cache by our users-per-card rule. Two L40S in a tensor-parallel pair run it with 26.2 GiB left for the KV cache, about 34 concurrent conversations of 8K tokens or 8 of 32K with an FP8 cache. One RTX PRO 6000 Blackwell Server Edition holds the same checkpoint alone with more room, and one RTX PRO 5000 72 GB with less.

This page is for companies that already run L40S servers, from the 2024 and 2025 builds, and want NVIDIA's reasoning model on them without new hardware, or want to know what one newer card would change. The figures are our estimates from NVIDIA's files on Hugging Face, read on 24 October 2026, with the arithmetic shown. Our Nemotron hardware guide sizes this model for the RTX PRO 6000, H200 NVL and DGX Spark; the L40S is not in its tables.

The model and its checkpoints

Hugging Face model card and file lists of nvidia/Llama-3_3-Nemotron-Super-49B-v1_5-FP8 and -NVFP4, read on 24 October 2026; sizes are our sums of the safetensors files.

NVIDIA's card describes the model as a "derivative of Meta Llama-3.3-70B-Instruct", reduced with Neural Architecture Search (NAS), released on Hugging Face on 25 July 2025 and "ready for commercial use". It reasons by default; adding /no_think to the system prompt switches reasoning off, with temperature 0.6 and top_p 0.95 recommended for reasoning and greedy decoding without. NVIDIA ran its evaluations with a 64K sequence length. Besides English and coding, the card lists German, French, Italian, Portuguese, Hindi, Spanish and Thai.

KV cache per conversation

NAS removed attention from some of the 80 layers and kept it in the others. The config.json gives 64 query heads with n_heads_in_group set to 8, so 8 KV heads, and a hidden size of 8,192, so 128 dimensions per head. One attention layer then stores 8 × 128 × 2 (key and value) × 2 bytes = 4 KiB per token in a 16-bit cache. 49 of the 80 layers have attention weights: our count of the attention weights that the model's weight index on Hugging Face lists per layer (layers 0 to 5, 8 to 10, 12 to 41, 52 and 71 to 79), the same figure as in our Nemotron guide. That gives 196 KiB per token, 1.53 GiB per 8K conversation and 6.1 GiB per 32K conversation in 16-bit, half that with an FP8 cache set by --kv-cache-dtype fp8.

That is a lot of cache for a 49B model. Nemotron 3 Super keeps attention in only 8 of its 88 layers and needs 4 KiB per token in FP8, as our Nemotron guide shows; this model needs 98 KiB in FP8. With reasoning on, the trace before each answer adds thousands of tokens, so a conversation reaches 32K sooner than a chat without reasoning would.

The L40S as a host for this model

The L40S is NVIDIA's Ada Lovelace data-centre card: 48 GB of GDDR6 with ECC at 864 GB/s, 350 W, passive cooling, PCIe 4.0 x16, no NVLink and no MIG. Its Tensor Cores compute FP8, with 733 dense teraFLOPS, so the FP8 checkpoint runs natively; they do not compute FP4.

Our users-per-card rule takes 90 per cent of the memory the driver reports and subtracts 3 GiB per card for CUDA graphs and activations. Scaled from the 95.6 GiB a 96 GB card reports, a 48 GB card shows about 47.8 GiB, and with ECC on, which NVIDIA's CUDA Best Practices Guide puts at 6.25 per cent of GDDR capacity, about 44.8 GiB. That leaves 37.3 GiB per card, or 40.0 GiB with ECC off, as our eight-L40S server page derives. The 48.4 GiB of FP8 weights exceed either figure.

Conversations per layout

Our estimate by the users-per-card rule: concurrent conversations at full length, 48.4 GiB of FP8 weights (28.9 GiB in the NVFP4 row), 196 KiB per token in 16-bit and 98 KiB in FP8. L40S rows with ECC on except the second; usable 61.5 GiB on the RTX PRO 5000 72 GB and 83.0 GiB on the RTX PRO 6000 Server Edition.

For the first row, two cards hold 2 × 37.3 = 74.6 GiB; less 48.4 GiB of weights that leaves 26.2 GiB, and 26.2 ÷ 0.77 is 34 conversations of 8K with an FP8 cache. Tensor parallelism splits the cache by its eight KV heads, so both cards carry half of every conversation.

These are requests in flight at the same moment, far fewer than the staff who use the service. A two-card L40S pair suits a team of a few dozen people asking short questions with reasoning switched off for simple tasks. For long documents or many reasoning sessions, the cache runs out first, and two pairs or a newer card are the next step. We size the layout on existing L40S servers as well as on new ones; send us the server model, its card count and your expected peak, and our team replies with the figures for each layout.

Tensor or pipeline parallelism over PCIe 4.0

The L40S exchanges data with its partner card over PCIe 4.0 x16. vLLM's parallelism guide, dated 15 October 2026, names this case: "if the GPUs on the node do not have NVLINK interconnect (e.g. L40S), leverage pipeline parallelism instead of tensor parallelism for higher throughput and lower communication overhead." Pipeline parallelism (PP=2) puts half of the layers on each card and passes activations once per step, while tensor parallelism (TP=2) needs an all-reduce in every layer but lets both cards work on every token, which shortens the time per token for a single user.

With four L40S the choice widens. vLLM's source disables its custom all-reduce because it is "not supported on more than two PCIe-only GPUs" and uses NCCL instead, so one TP=4 copy runs on NCCL by default, though it holds the most cache; two TP=2 pairs avoid NCCL and keep two independent copies. vLLM 0.30.0 (release notes of 22 September 2026) added an opt-in FlashInfer PCIe IPC all-reduce for systems without NVLink (#53576), which a TP=4 copy can try. In a two-socket server, a pair under the same processor avoids the link between the sockets; nvidia-smi topo -m shows the path between every two cards. Our guide to one model on several GPUs over PCIe or NVLink explains the options.

Speed per conversation, an upper bound

A dense model reads all its weights for every generated token, so the memory bandwidth sets a ceiling on the tokens per second of one conversation. By our estimate, with about 52 GB read per token in FP8:

Our estimate, an upper bound: memory bandwidth from NVIDIA's datasheets ÷ checkpoint size (52.0 GB FP8, 31.1 GB NVFP4); the TP=2 row assumes each card reads half the weights and ignores the all-reduce time over PCIe.

Real rates fall below these ceilings, and more so for the L40S pair, which waits for PCIe in every layer. The table still shows the point for a reasoning model: two L40S are not slower than one newer card for a single conversation, because together they read memory faster. With a trace of a few thousand tokens before each answer, 30 tokens per second means an answer after a minute or more, which is why reasoning off for routine tasks matters. We found no published measurement of this model on the L40S; our page on NVIDIA L40S benchmarks collects the published results for the card.

Software: vLLM and NIM on the L40S

The model card names NVIDIA Ampere and Hopper as recommended microarchitectures and an H100-80GB as test hardware; the Ada Lovelace L40S is not named there. Its vLLM example installs vLLM 0.9.2 and runs the FP8 checkpoint with --tensor-parallel-size=8, --max-model-len=65536, --quantization=modelopt, --enforce-eager and --trust-remote-code. On two L40S, set the tensor or pipeline size to 2 and the context to what your conversations need, since a 64K context per conversation takes 12.3 GiB of 16-bit cache by our arithmetic, almost half the free memory of the pair. Run a current vLLM release rather than the card's version; our page on vLLM security advisories lists which releases fix which advisories.

NVIDIA's NIM support matrix for LLMs, release 2.0.13, updated on 19 October 2026, lists "NVIDIA-L40S" among the verified GPUs for Llama-3.3-Nemotron-Super-49B-v1.5, next to the RTX PRO 6000 Blackwell Server Edition and the H200 NVL, with BF16, FP8 and NVFP4 profiles at tensor parallelism of 1, 2, 4 or 8. Its optimised profiles are NVFP4 at TP1 on B200 and FP8 at TP2 on H200. Production use of a NIM needs an NVIDIA AI Enterprise licence per GPU, which the L40S does not include; our guide to NVIDIA AI Enterprise licensing explains the terms, and vLLM serves the same checkpoints under its Apache 2.0 licence.

NVIDIA also publishes the model in NVFP4 at 31.1 GB, which would fit one L40S by memory with 8.4 GiB left. The L40S has no FP4 Tensor Cores, so an engine can at best unpack the 4-bit weights, and support for NVFP4 on Ada differs between engines and versions, as our guide to FP4 models on H200 NVL and Ada GPUs sets out. Test it on your server before you plan on it.

One newer card in place of the pair

RTX PRO 6000 Blackwell Server Edition. One card holds the FP8 checkpoint with 34.6 GiB of cache, 45 conversations of 8K with an FP8 cache, more than the two-L40S pair, and computes NVFP4 natively, which raises the cache to 54.1 GiB and the speed ceiling to about 51 tokens per second. It is passive like the L40S and draws 300 to 600 W against 700 W for two L40S. It needs a slot, a 16-pin power cable and a fan set that the server's maker lists for it, so an older L40S chassis is not automatically a host. Our pages on L40S vs RTX PRO 6000 Server Edition and the L40S server upgrade cover the comparison and the swap.

RTX PRO 5000 72 GB. One card holds the FP8 weights with 13.1 GiB of cache, 17 conversations of 8K with an FP8 cache. It is a 300 W workstation card with an active cooler, so in a rack it belongs only in a chassis whose maker lists it; it suits a development team that runs the model on workstations. Our comparison of the RTX PRO 5000 48 GB and 72 GB covers the two versions.

Keep the L40S pair when the server is in place, the load is a few dozen people and the budget for new cards is better spent elsewhere. An eight-card L40S server runs four pairs, about 136 conversations of 8K in total by our estimate, or two pairs of this model next to other workloads.

Licence: NVIDIA Open Model License and Llama 3.3

The card states that the model is governed by the NVIDIA Open Model License, with the Llama 3.3 Community License Agreement as additional information and the attribution "Built with Llama". The NVIDIA Open Model License, last modified on 24 October 2025, states: "Models are commercially usable." It ends your rights automatically if you bypass, disable or circumvent a safety guardrail of the model without a substantially similar guardrail, ends them on patent or copyright litigation over the model, and on distribution requires a copy of the agreement and a notice "Licensed by NVIDIA Corporation under the NVIDIA Open Model License."

The Llama 3.3 Community License, dated 6 December 2024, requires a copy of the agreement on redistribution, "Built with Llama" displayed prominently, and "Llama" at the beginning of the name of a derived model you distribute. A licensee whose products had more than 700 million monthly active users on the Llama 3.3 release date must request a separate licence from Meta under its Section 2. Its Acceptable Use Policy withholds the Section 1(a) rights for "any multimodal models included in Llama 3.3" from companies with their principal place of business in the EU, with an exception for end users; Nemotron Super 49B is a text model, so by its wording that clause does not reach it. Your legal counsel confirms how both licences apply to your company.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We supply the L40S, the RTX PRO 6000 Server Edition and the RTX PRO 5000 as cards for servers you run, after a compatibility check, or in AI servers built to order, assembled and tested before delivery, on one EU contract and invoice with manufacturer warranty on every component. NVIDIA AI Enterprise licences for NIM come on the same invoice. Setting up the model with retrieval and a gateway is our Private AI/ML service, delivered with our engineering partner Vixen.UNO; tell us which L40S servers you run today, and we say what the model needs on them.

FAQ