NVIDIA L40S: what the published benchmarks show, and where the card stands in 2026

L40S specifications

NVIDIA announced the L40S in August 2023 as “a powerful, universal data center processor”. It is an Ada Lovelace card with 18,176 CUDA cores, 568 fourth-generation Tensor Cores and 142 third-generation RT cores. It carries 48 GB of GDDR6 with ECC at 864 GB/s and is rated at 91.6 TFLOPS in FP32 and 733 TFLOPS in dense FP8. The passive, dual-slot, full-length board has a 350 W maximum and connects over PCIe 4.0 x16, without NVLink or MIG. Three NVENC and three NVDEC engines handle AV1; the four DisplayPort 1.4a outputs are off by default. NVIDIA’s launch claims, up to 1.2 times the generative AI inference and up to 1.7 times the training performance of the A100, came without test conditions that we could find.

MLPerf® Inference results, divided per card

MLCommons publishes every MLPerf® Inference result with the logs of the run. All results in this section are MLPerf® Inference: Datacenter results in the closed division; the table shows the offline scenario, in which the system receives all samples at once and throughput is measured.

MLPerf® Inference: Datacenter, closed division, offline scenario. Entry 5.0-0018: v5.0 (April 2025), Dell PowerEdge XE7745; entry 5.1-0012: v5.1 (September 2025), Cisco UCS C845A M8; both with eight L40S and NVIDIA’s TensorRT stack. Weights as MLCommons lists them: INT8 for RetinaNet, FP8 for Llama 2 70B in its 99 per cent accuracy variant. Retrieved from mlcommons.org on 24 September 2026, results verified by MLCommons Association; this applies to every MLPerf® result in this article, with its entry ID in the table or the text. Per card is our division of the system result by its number of GPUs, not an MLPerf® metric. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

Per card, by our division, object detection gave about 815 images per second in Cisco’s eight-card server in v5.1, and Llama 2 70B about 435 tokens per second in Dell’s eight-card server in v5.0. Our L4 comparison sets the v5.1 RetinaNet result of the eight-card Cisco server against a two-card L4 server (entry 5.1-0018): about 815 against 229 images per second per card, 3.6 times as much, by our division, in the same round, division and scenario; those runs measured no power.

MLCommons’ result tables list further L40S entries and further benchmarks of the same systems; we keep to the two results above, whose entry IDs and values we have checked against MLCommons’ data. We leave out v4.0 and v4.1, whose result tables we could not open on 24 September 2026. We found no L40S result in the closed division of the v6.1 round, published on 16 September 2026.

Tokens per second on the L40S: NVIDIA NIM results

For a single request, speed is capped by how fast the card reads its weights, because for a dense model every new token reads all layer weights and the output layer once. The fullest single-card picture comes from NVIDIA’s tables for NIM, its packaged inference microservice (NIM 1.8 container, in NVIDIA’s NIM benchmarking guide 1.0.0, last updated 1 April 2026): Llama 3.1 8B Instruct on one L40S in a Supermicro server, with 1,000 input and 1,000 output tokens per request.

NVIDIA NIM LLM benchmarking tables, Llama 3.1 8B Instruct in FP8 on one L40S; throughput is NVIDIA’s total for all requests together, in output tokens per second by our check.

By our arithmetic the model reads 7.5 to 8 GB per token in FP8: its linear layers plus an output layer that may stay in BF16. At 864 GB/s that caps one stream at roughly 110 tokens per second; NVIDIA measured 71, about two thirds of that. In BF16 the tables show 45 tokens per second against a ceiling of about 58. With 100 requests at once, each request still gets about 29 tokens per second by our arithmetic. Between 200 and 250 requests, the time to first token jumps from under 0.9 s to 5 s.

Long prompts reach the limit sooner. With 20,000 input tokens, the first token takes 1.3 s for one request and 27.5 s for 25, and total output stays near 290 tokens per second from 25 requests on. By our arithmetic each such request holds 1.3 GiB of KV cache even in FP8, so no more than about 20 fit at once by the sizing method of our guide to concurrent users: 90 per cent of the memory the driver reports with ECC on (about 45 GiB), less about 3 GiB of overhead and the weights. The MLPerf® figures use other datasets and settings and are not comparable with these.

What fits in 48 GB

A 70B model in FP8, about 68 GiB of weights for Llama 3.3 70B, does not fit one L40S; on this card it is a job for at least two cards over PCIe, split by tensor parallelism or by pipeline parallelism, which vLLM’s documentation suggests for higher throughput on GPUs without NVLink such as the L40S. In 4-bit a 70B model only just fits. An INT4 AWQ checkpoint of Llama 3.1 70B is about 39.8 GB (37.1 GiB), which by our sizing method leaves about 0.4 GiB, or up to about 1.4 GiB if the engine needs only 2 GiB. That is room for one 8,192-token conversation at most, and only with an FP8 cache (1.25 GiB, against 2.5 GiB with a 16-bit cache). A 32B model in FP8 fits with room for users, and an 8B model in FP8 leaves more than 30 GB for cache. Ada has no FP4 arithmetic. TensorRT-LLM’s hardware support matrix (TensorRT-LLM 1.4.0rc1, last updated 19 October 2026) lists per-tensor FP8, an FP8 KV cache and INT4 AWQ and GPTQ (W4A16 and W4A8) for the architecture, but not NVFP4, MXFP4 or block-scaled FP8.

Against one RTX PRO 6000 Server Edition, two L40S have the same 96 GB and, by our arithmetic, slightly more combined bandwidth, 1,728 against 1,597 GB/s, for 700 W against up to 600 W and in twice the slot width. But with tensor parallelism a model split across two cards exchanges data twice per layer over PCIe 4.0, and the pair has neither MIG nor FP4 arithmetic: NVFP4, which shrinks Llama 3.3 70B to about 40 GiB, runs on Ada only as weight-only compression in vLLM and not in TensorRT-LLM.

We supply both options, two L40S cards or one RTX PRO 6000 Server Edition. Tell us your model and the number of users, and we reply with a written quote for the option that fits.

Image, vision and video

NVIDIA’s inference performance page (September 2026) lists one L40S and one RTX PRO 6000 Server Edition in the same Supermicro SYS-521GE-TNRT server model and TensorRT container. Where batch size and precision match:

NVIDIA, AI inference performance tables, TensorRT container 26.07, synthetic data, one GPU each; Swin models at sequence length 384; the video model is Stable Video Diffusion. Ratios are our division.

Per watt the gap is small: NVIDIA’s efficiency column gives the L40S 4.17 and 2.16 samples per second per watt on the two Swin models, against 4.58 and 2.52 for the RTX PRO 6000, 10 and 17 per cent more by our arithmetic. For image generation the same page gives the L40S 0.36 Stable Diffusion XL images per second at batch 1 and one Flux image every 12.5 s in FP8; the RTX PRO 6000 rows use batch 4 and FP4 respectively, so they are not comparable.

The L40S has three eighth-generation NVENC encoders with AV1 and, in NVIDIA’s codec matrix, no session limit, but no 4:2:2 H.264 or HEVC encoding, which the four ninth-generation encoders of the RTX PRO 6000 Server Edition add. For transcoding and camera streams, the L4, with two encoders, four decoders and four JPEG decoders at 72 W, gives more video engines per watt.

Virtual workstations on the L40S: time-sliced vGPU

The L40S is shared through time-sliced vGPU, on NVIDIA’s vGPU list from release 16.1; NVIDIA’s release notes for vGPU 20 (driver branch R595) on VMware vSphere, September 2026, cover it on VCF 9.0 and 9.1 and on ESXi 8.0 Update 3 P06 and later 8.0 updates, and allow mixed profile sizes on one card from ESXi 8.0 Update 3. The Q-series profiles, which need an NVIDIA RTX vWS licence, run from L40S-1Q to L40S-48Q: up to 32 VMs with 1 GB each, six with 8 GB (four when profile sizes are mixed), four with 12 GB on L40S-12Q and one with the whole card; our guide to the L40S vGPU profiles lists every type. Compute profiles are licensed through NVIDIA AI Enterprise, which does not come with the card.

StorageReview showed in April 2025 what time slicing means under load. On a Dell PowerEdge R760 with one L40S, Proxmox VE and vGPU 18.0, one VM with L40S-48Q scored 21,147 points in the Cinebench 2024 GPU test. Four VMs with L40S-12Q running it at the same time scored 2,514 to 2,567 each, together less than half of that by our arithmetic, while one 12 GB VM running alone scored 15,133. In Blender, a VM fell to as little as 8 per cent of a single VM’s result. NVIDIA’s default best-effort scheduler balances cycles across the running vGPUs; its equal-share and fixed-share schedulers give each vGPU a defined share instead. Size seats for the concurrent peak; our MIG and vGPU guide has the seat counts and licences. NVIDIA validates each vGPU release for specific ESXi versions, so plan vSphere and vGPU updates together; our engineering partner Vixen.UNO carries out vSphere version and architecture updates within our VMware optimisation service.

Power, cooling and software

NVIDIA’s product brief gives 350 W as both the default and the maximum board power and specifies one 16-pin auxiliary power connector; we found no minimum board power in it. The card has no fan. The heat sink accepts airflow from either side, the server’s fans must carry away 350 W per card, and NVIDIA rates it for 0 to 50 °C in operation. The board is dual-slot, 4.4 by 10.5 inches and 1,052 g without bracket and extenders; eight of them draw 2,800 W before processors and fans. The minimum software in NVIDIA’s 2023 product brief is the R535 driver branch with CUDA 12.2, and vGPU 16.1 for virtualisation; R535 reached its end of life in June 2026, according to NVIDIA’s driver table of 9 September 2026, so a new installation starts on a supported branch such as R580 or R595. FP8 needs compute capability 8.9 or newer, which Ada has. With no NVLink, cards exchange data over PCIe 4.0 x16, 64 GB/s in both directions together.

L40S in 2026: support status and when to choose it

We found no NVIDIA end-of-life notice for the L40S as of 5 November 2026, and NVIDIA’s vGPU lifecycle page, last updated on 29 October 2026, lists it among the GPUs that “remain fully supported”; that page covers vGPU software support, not how long the card remains orderable. NVIDIA compares the RTX PRO 6000 Blackwell Server Edition with it, and its datasheet says the Tensor Cores “deliver up to 5x the performance of the previous generation for LLM inference with support for FP4 precision” (the L40S has no FP4), for which we found no published test conditions; NVIDIA’s own image, vision and video tables above show 1.74 to 2.15 times the work per card, by our division. The RTX PRO 6000 Server Edition has twice the memory, 1.85 times the bandwidth, FP4, MIG and PCIe 5.0, at up to 600 W against 350 W.

The L40S remains the right card when the models fit in 48 GB (8B to 32B language models, vision, image generation), when the servers are already qualified for 350 W double-width cards, when the same hosts serve graphics and virtual workstations, and when an existing L40S environment is extended with the same profiles and spares. It is the wrong card for a 70B model serving a team, for tenants that need hardware isolation, for models that need FP4 arithmetic and for high-density video transcoding.

What we supply

We supply the NVIDIA L40S on one EU contract and invoice with manufacturer warranty, as single cards or in GPU servers built to order. For VDI and rendering we build nodes with RTX PRO Blackwell, L40S or L4 cards, with the NVIDIA vGPU licences on the same invoice as the hardware. When a model outgrows 48 GB, we supply the RTX PRO 6000 Server Edition and the H200 NVL. For vGPU environments on vSphere, our engineering partner Vixen.UNO delivers VMware optimisation: an audit of the environment and its licences, Broadcom editions matched to the real workloads, and vSphere modernisation in agreed maintenance windows, with support under an agreed SLA.

FAQ