NVIDIA L40S: what the published benchmarks show, and where the card stands in 2026

The card in brief

NVIDIA announced the L40S in August 2023 as “a powerful, universal data center processor”. It is an Ada Lovelace card with 18,176 CUDA cores, 568 fourth-generation Tensor Cores, 142 third-generation RT cores and 48 GB of GDDR6 with ECC at 864 GB/s, rated at 91.6 TFLOPS in FP32 and 733 TFLOPS in dense FP8. The passive, dual-slot, full-length board has a 350 W maximum and connects over PCIe 4.0 x16, without NVLink or MIG. Three NVENC and three NVDEC engines handle AV1; the four DisplayPort 1.4a outputs are off by default. NVIDIA’s launch claims, up to 1.2 times the generative AI inference and up to 1.7 times the training performance of the A100, came without test conditions.

MLPerf® Inference results, divided per card

MLCommons publishes every MLPerf Inference result with the logs of the run. All results in this section are MLPerf Inference: Datacenter results in the closed division; the table shows the offline scenario, in which the system receives all samples at once and throughput is measured.

MLPerf Inference: Datacenter, closed division, offline scenario. Entry 5.0-0018: v5.0 (April 2025), Dell PowerEdge XE7745; entry 5.1-0012: v5.1 (September 2025), Cisco UCS C845A M8; both with eight L40S and NVIDIA’s TensorRT stack. Weights as MLCommons lists them: INT8 for ResNet-50 and RetinaNet, FP8 for SDXL and for Llama 2 70B in its 99 per cent accuracy variant. Retrieved from mlcommons.org on 24 September 2026, results verified by MLCommons Association; this applies to every MLPerf result in this article, with its entry ID in the table or the text. Per card is our division of the system result by its number of GPUs, not an MLPerf metric. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

Per card, by our division, object detection gave the same 815 images per second in Dell’s eight-card server in v5.0 and in Cisco’s eight-card server in v5.1, in the same division and scenario. Cisco also ran a two-card server in v5.0: its UCS C245 M8 (entry 5.0-0012) reached 57,832.5 samples per second offline on DLRM v2 recommendation, about 28,900 per card, and 1.4 images per second on SDXL with INT8 weights, about 0.7 per card, against 0.78 per card with FP8 weights in Dell’s eight-card server, in the same round, division and scenario. Our L4 comparison sets the v5.1 RetinaNet result of the eight-card Cisco server against a two-card L4 server (entry 5.1-0018): about 815 against 229 images per second per card, 3.6 times as much, by our division, in the same round, division and scenario; those runs measured no power.

In the server scenario of v5.0, where queries arrive at random and the 99th-percentile latencies must stay within 2 s to the first token and 200 ms per output token, Dell’s XE7745 ran Llama 2 70B at 3,201.1 tokens per second (entry 5.0-0018, closed division). Its log shows 0.47 s to the first token and 110 ms per output token at the 99th percentile and a mean of 99 ms per output token, about ten tokens per second for one answer by our arithmetic.

MLCommons’ result tables list further L40S entries, among them HPE’s eight-card ProLiant DL380a Gen12 in v5.0 and Red Hat’s runs with the open-source vLLM engine in v5.1 and v6.0 (April 2026); we keep to the Dell and Cisco results above. We leave out v4.0 and v4.1, whose result tables we could not open on 24 September 2026, and the v6.1 round of 16 September 2026.

Language models: bandwidth sets the pace

For a single request, speed is capped by how fast the card reads its weights, because for a dense model every new token reads all layer weights and the output layer once. The fullest single-card picture comes from NVIDIA’s tables for NIM, its packaged inference microservice (NIM 1.8, page updated 20 July 2026): Llama 3.1 8B Instruct on one L40S in a Supermicro server, with 1,000 input and 1,000 output tokens per request.

NVIDIA NIM LLM benchmarking tables, Llama 3.1 8B Instruct in FP8 on one L40S; throughput is NVIDIA’s total for all requests together, in output tokens per second by our check.

By our arithmetic the model reads 7.5 to 8 GB per token in FP8, its linear layers plus an output layer that may stay in BF16, so 864 GB/s caps one stream at roughly 110 tokens per second; NVIDIA measured 71, about two thirds of that. In BF16 the tables show 45 tokens per second against a ceiling of about 58. With 100 requests at once, each still gets about 29 tokens per second by our arithmetic. Between 200 and 250 requests the first token jumps from under 0.9 s to 5 s. Long prompts hit the limit sooner: with 20,000 input tokens the first token takes 1.3 s for one request and 27.5 s for 25, and total output stays near 290 tokens per second from 25 requests on. By our arithmetic each such request holds 1.3 GiB of KV cache even in FP8, so no more than about 20 fit at once, by the sizing method of our guide to concurrent users: 90 per cent of the card’s 48 GB, less about 3 GiB of overhead and the weights. The MLPerf figures use other datasets and settings and are not comparable with these.

What fits in 48 GB

A 70B model in FP8, about 68 GiB of weights for Llama 3.3 70B, does not fit one L40S; on this card it is a job for at least two cards over PCIe, split by tensor parallelism or by pipeline parallelism, which vLLM’s documentation suggests for higher throughput on GPUs without NVLink such as the L40S. In 4-bit a 70B model only just fits: an INT4 AWQ checkpoint of Llama 3.1 70B is about 39.8 GB, which by our sizing method leaves about 0.2 GB, and at best about 1.4 GB if the engine needs only 2 GB: room for one 8,192-token conversation at most, and only with an FP8 cache (1.34 GB, against 2.7 GB with a 16-bit cache). A 32B model in FP8 fits with room for users, and an 8B model in FP8 leaves more than 30 GB for cache. Ada has no FP4 arithmetic: TensorRT-LLM’s support matrix of 21 September 2026 lists per-tensor FP8, an FP8 KV cache and INT4 AWQ and GPTQ (W4A16 and W4A8) for the architecture, but not NVFP4, MXFP4 or block-scaled FP8.

Against one RTX PRO 6000 Server Edition, two L40S have the same 96 GB and, by our arithmetic, slightly more combined bandwidth, 1,728 against 1,597 GB/s, for 700 W against up to 600 W and in twice the slot width. But with tensor parallelism a model split across two cards exchanges data twice per layer over PCIe 4.0, and the pair has neither MIG nor FP4 arithmetic: NVFP4, which shrinks Llama 3.3 70B to about 40 GiB, runs on Ada only as weight-only compression in vLLM and not in TensorRT-LLM.

Image, vision and video

NVIDIA’s inference performance page (September 2026) lists one L40S and one RTX PRO 6000 Server Edition in the same Supermicro SYS-521GE-TNRT server model and TensorRT container. Where batch size and precision match:

NVIDIA, AI inference performance tables, TensorRT container 26.07, synthetic data, one GPU each; Swin models at sequence length 384; the video model is Stable Video Diffusion. Ratios are our division.

Per watt the gap is small: NVIDIA’s efficiency column gives the L40S 4.17 and 2.16 samples per second per watt on the two Swin models, against 4.58 and 2.52 for the RTX PRO 6000, 10 and 17 per cent more by our arithmetic. For image generation the same page gives the L40S 0.36 Stable Diffusion XL images per second at batch 1 and one Flux image every 12.5 s in FP8; the RTX PRO 6000 rows use batch 4 and FP4 respectively, so they are not comparable.

The L40S has three eighth-generation NVENC encoders with AV1 and, in NVIDIA’s codec matrix, no session limit, but no 4:2:2 H.264 or HEVC encoding, which the four ninth-generation encoders of the RTX PRO 6000 Server Edition add. For transcoding and camera streams, the L4, with two encoders, four decoders and four JPEG decoders at 72 W, gives more video engines per watt.

Virtual workstations: time slices, not partitions

The L40S is shared through time-sliced vGPU, on NVIDIA’s vGPU list from release 16.1; NVIDIA’s release notes for vGPU 20 (driver branch R595) on VMware vSphere, September 2026, cover it on VCF 9.0 and 9.1 and on ESXi 8.0 Update 3 P06 and later 8.0 updates, and allow mixed profile sizes on one card from ESXi 8.0 Update 3. The Q-series profiles, which need an NVIDIA RTX vWS licence, run from L40S-1Q to L40S-48Q: up to 32 VMs with 1 GB each, six with 8 GB (four when profile sizes are mixed) and one with the whole card. Compute profiles are licensed through NVIDIA AI Enterprise, which does not come with the card.

StorageReview showed in April 2025 what time slicing means under load. On a Dell PowerEdge R760 with one L40S, Proxmox VE and vGPU 18.0, one VM with L40S-48Q scored 21,147 points in the Cinebench 2024 GPU test. Four VMs with L40S-12Q running it at the same time scored 2,514 to 2,567 each, together less than half of that by our arithmetic, while one 12 GB VM running alone scored 15,133. In Blender, a VM fell to as little as 8 per cent of a single VM’s result. NVIDIA’s default best-effort scheduler balances cycles across the running vGPUs; its equal-share and fixed-share schedulers give each vGPU a defined share instead. Size seats for the concurrent peak; our MIG and vGPU guide has the seat counts and licences. NVIDIA validates each vGPU release for specific ESXi versions, so plan vSphere and vGPU updates together; our engineering partner Vixen.UNO carries out vSphere version and architecture updates within our VMware optimisation service.

Power, cooling and software

NVIDIA’s product brief gives 350 W as both the default and the maximum board power, names no minimum, and specifies one 16-pin auxiliary power connector. There is no fan: the heat sink accepts airflow from either side, the server’s fans must carry away 350 W per card, and NVIDIA rates it for 0 to 50 °C in operation. The board is dual-slot, 4.4 by 10.5 inches and 1,052 g without bracket and extenders; eight of them draw 2,800 W before processors and fans. The software floor in NVIDIA’s 2023 product brief is the R535 driver branch with CUDA 12.2, and vGPU 16.1 for virtualisation; R535 reached its end of life in June 2026, according to NVIDIA’s driver table of 9 September 2026, so a new installation starts on a supported branch such as R580 or R595. FP8 needs compute capability 8.9 or newer, which Ada has. With no NVLink, cards exchange data over PCIe 4.0 x16, 64 GB/s in both directions together.

Where it stands in 2026

We found no NVIDIA end-of-life notice for the L40S as of September 2026, and NVIDIA’s vGPU lifecycle page of 3 August 2026 lists it among the GPUs that “remain fully supported”; that page covers vGPU software support, not how long the card remains orderable. NVIDIA positions the RTX PRO 6000 Blackwell Server Edition as its successor, claiming “up to 5x higher large language model (LLM) inference throughput” without published test conditions; NVIDIA’s own image, vision and video tables above show 1.74 to 2.15 times the work per card, by our division. The RTX PRO 6000 Server Edition has twice the memory, 1.85 times the bandwidth, FP4, MIG and PCIe 5.0, at up to 600 W against 350 W.

The L40S remains the right card when the models fit in 48 GB (8B to 32B language models, vision, image generation), when the servers are already qualified for 350 W double-width cards, when the same hosts serve graphics and virtual workstations, and when an existing L40S estate grows with the same profiles and spares. It is the wrong card for a 70B model serving a team, for tenants that need hardware isolation, for models that need FP4 arithmetic and for high-density video transcoding.

What we supply

Eurokommerz supplies the NVIDIA L40S across the EU with manufacturer warranty, on an EU contract with EU invoicing, as single cards or in GPU servers built to order, including VDI and rendering nodes with RTX PRO Blackwell, L40S or L4 cards and NVIDIA vGPU licensing on the same invoice as the hardware. When a model outgrows 48 GB, we supply the RTX PRO 6000 Server Edition and the H200 NVL. For vGPU estates on vSphere, our engineering partner Vixen.UNO delivers VMware optimisation: it audits the estate and its licences, matches Broadcom editions to the real workloads and modernises vSphere in agreed maintenance windows, with support under an agreed SLA.

FAQ