What a server with 8× NVIDIA L40S runs in 2026: models, users, video and VDI

What eight 48 GB cards add up to

A server with eight NVIDIA L40S cards runs language models up to 70B in FP8, many small and mid-size models side by side, video analytics on hundreds of camera streams and up to 256 virtual desktops. By our users-per-card rule it holds about 298.4 GiB of weights and cache with ECC on, enough for two copies of Llama 3.3 70B in FP8 serving about 130 conversations at once, at about 3.9 kW for the whole server.

The L40S is NVIDIA's universal data-centre card of the Ada Lovelace generation: 48 GB of GDDR6 with ECC at 864 GB/s, 18,176 CUDA cores, 142 RT cores, FP8 Tensor Cores, 350 W, passive and dual-slot, on PCIe 4.0 x16. It has no NVLink and no MIG, so the eight cards do not pool their memory, and a model larger than one card is split over PCIe.

NVIDIA L40S product page, read on 24 October 2026; usable memory and sums are our arithmetic.

If you are planning a purchase, our page to buy an L40S server has the configuration, chassis and ordering steps. This page works out what eight cards do.

How much of 384 GB holds models and cache

Our users-per-card rule takes 90 per cent of the memory the driver reports and subtracts 3 GiB per card for CUDA graphs, activations and the runtime; the rest holds weights and KV cache. Scaled from the 97,887 MiB a 96 GB card reports, a 48 GB card shows about 47.8 GiB. NVIDIA's CUDA Best Practices Guide states that ECC on GDDR memory takes 6.25 per cent of capacity, which leaves about 44.8 GiB with ECC on. That gives 0.9 × 44.8 − 3 = 37.3 GiB per card and 298.4 GiB for eight, or 40.0 and 320.0 GiB with ECC off. Check the figure with nvidia-smi -q in the ECC mode you will run. The rule is derived in our guide to how many users one RTX PRO 6000 can serve.

With an FP8 KV cache, a conversation of 8,192 tokens takes 0.5625 GiB on Qwen3-8B, 0.625 GiB on Qwen3-14B, 1.0 GiB on Qwen3-32B and 1.25 GiB on Llama 3.3 70B, by the layers, KV heads and head size in each model's configuration. A 16-bit cache doubles these figures and halves the conversations.

Models and layouts on eight L40S

Eight cards give four ways to place a model: one copy per card, TP=2 pairs, TP=4 copies or one copy over all eight. Fewer copies leave more memory for conversations and cost more traffic over PCIe at every layer.

Our estimates by the users-per-card rule, not measurements: concurrent conversations of 8,192 tokens at full length with an FP8 KV cache, ECC on, in brackets ECC off. Weights are the Qwen and NVIDIA FP8 checkpoints and the INT4 AWQ checkpoint of 39.8 GB as in our sizing guides.

Two rows show the arithmetic. A TP=4 copy of Llama 3.3 70B has 4 × 37.3 = 149.2 GiB; less 67.7 GiB of weights that leaves 81.5 GiB, and 81.5 ÷ 1.25 gives 65 conversations per copy, 130 for two. One copy of Qwen3-14B leaves 37.3 − 15.2 = 22.1 GiB on its card, 35 conversations, 280 for eight cards.

A 32B model in FP8 is where 48 GB per card counts: it fits one L40S with room for five conversations, eight with ECC off, while a 32 GB card cannot hold it at all. Qwen's FP8 release of Qwen3-32B scales its weights in blocks of 128 by 128, a format TensorRT-LLM's support matrix does not list for Ada, as our L40S benchmark review notes, so confirm that your serving engine runs it on this card. Ada has no FP4 Tensor Cores: vLLM loads NVFP4 checkpoints on the L40S only as weight-only compression, and 4-bit models usually run as INT4 through AWQ or GPTQ. Our article on FP4 models on H200 NVL and Ada GPUs covers the formats.

Users per model for 500 to 2,000 staff

The conversations above are requests in flight at the same moment, not named users. With the example values of our guide to a private ChatGPT server by company size, 500 staff produce about 20 requests in flight at the busiest moment, 1,000 staff about 40 and 2,000 staff about 80.

By those values, two TP=4 copies of Llama 3.3 70B in FP8 with about 130 conversations cover 2,000 staff with headroom, and one copy on half the server, about 65, covers 1,000. With a 16-bit cache the same layout holds half. Eight copies of a 14B-class model hold several times the 80 requests of 2,000 staff, which suits a company that runs one assistant per department or many short requests from applications.

The eight cards also carry several models at once: an assistant on four cards, a coding model on two, and an embedding model and a reranker for RAG on one card each, each with its own memory and restart cycle. A failed card then takes one eighth of the GPU memory out of service. Our guide to serving several models on one GPU server covers the engine settings. We size this server per model; send us your models and the peak requests in flight, and our team replies with a layout and the conversations each model gets.

Splitting a model over PCIe 4.0

Every split copy exchanges partial results between its cards at every layer, here over PCIe 4.0 x16 at 64 GB/s in both directions together. vLLM's parallelism guide, updated on 15 October 2026, names the L40S as an example of GPUs without NVLink, and its advice for such GPUs is to "leverage pipeline parallelism instead of tensor parallelism for higher throughput and lower communication overhead". By default vLLM disables its custom all-reduce on more than two PCIe-only GPUs and uses NCCL. vLLM 0.30.0 (release notes, 22 September 2026) added an opt-in FlashInfer PCIe IPC all-reduce for systems without NVLink (#53576); the FlashInfer pull request behind it reports an all-reduce microbenchmark on eight L40S in BF16 where "All 72 points beat NCCL, from 1.22× to 5.92×", and we found no published end-to-end gain for a whole model.

For one stream, speed is capped by how fast the cards read the weights. Llama 3.3 70B in FP8 reads about 70.6 GB per token, so one L40S could reach about 12 tokens per second and an ideal TP=4 copy about 49 before any PCIe cost, by our arithmetic. Plan TP=4 and TP=8 for capacity, and in a two-socket server use nvidia-smi topo -m to keep a pair or a set of four cards under one processor. Our article on one model on several GPUs over PCIe or NVLink explains the options.

Published results for eight L40S

MLCommons publishes every MLPerf® Inference result with the logs of the run. The table shows two results with eight L40S, both in the closed division and the offline scenario, where the system receives all samples at once.

MLPerf® Inference: Datacenter, closed division, offline scenario. Entry 5.0-0018: v5.0, Dell PowerEdge XE7745 with eight L40S, Llama 2 70B; entry 5.1-0012: v5.1, Cisco UCS C845A M8 with eight L40S, RetinaNet. Retrieved from mlcommons.org on 24 September 2026, result verified by MLCommons Association. Per card is our division of the system result, not an MLPerf® metric.

The two entries come from different rounds and different makers, and the offline scenario measures throughput with all requests queued, not the speed of one answer. We found no L40S result in the v6.1 round of 16 September 2026.

The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

StorageReview measured one L40S with vLLM 0.27.1 in a Dell PowerEdge R770 and published the results on 14 October 2026: 995 output tokens per second on Ministral 3 14B Instruct in FP8 at 32 concurrent requests of 512 tokens in and out, and about 230 with 8,192 input tokens. Eight independent copies would reach at most about 7,960 and 1,840 tokens per second by our arithmetic, an upper bound from one card. Our L40S benchmark review collects all published figures.

Video analytics and transcoding on 24 engines

Each L40S has three NVENC and three NVDEC engines with AV1 encode and decode, so eight cards bring 24 of each. Our guide to a GPU server for video analytics derives a decode ceiling of about 90 H.264 or 164 H.265 streams at 1080p and 30 frames per second per card from NVIDIA's NVDEC application note, 720 or 1,312 for eight cards. These are upper bounds; NVIDIA states that the rate scales with the video clock.

NVIDIA DeepStream 9.2 documentation, performance of dGPU pretrained models with TensorRT, end to end with rendering off, read on 24 October 2026; eight-card sums are our arithmetic, not measurements.

Divided by 30, the PeopleNet sum equals about 297 cameras analysed at full frame rate before any headroom. At 10 analysed frames per second the inference work allows about three times as many, and the decoders become the limit, since they decode every frame of every stream. With H.264 cameras the ceiling of 720 streams decides; with H.265 it is higher. A test run with your own model and codec sets the final count. For transcoding farms, the L4 with two encoders and four decoders at 72 W gives more video engines per watt, as our L4 vs L40S comparison shows.

For generative media, NVIDIA's inference tables give one L40S 1.34 Stable Video Diffusion videos per minute. Our guide to video generation GPU requirements covers the larger video models, many of which need more than 48 GB.

Graphics, rendering and VDI on eight L40S

The L40S has 142 third-generation RT cores and 91.6 TFLOPS of FP32, 732.8 TFLOPS for eight cards, and NVIDIA names rendering, 3D graphics and Omniverse among its workloads. The card has no MIG, so virtual machines share it by time-sliced vGPU. NVIDIA's vGPU types reference of 29 September 2026 gives these maxima per card in equal-size mode.

NVIDIA Virtual GPU Types Reference (29 September 2026) and NVIDIA AI Enterprise Ada Lovelace vGPU types (2 September 2026), as listed in our L40S vGPU profiles guide; eight-card totals are our arithmetic.

These are maximum densities, and time slicing shows under load: in StorageReview's test of April 2025, four 12 GB VMs running Cinebench 2024 at once scored 2,514 to 2,567 each, against 21,147 for one VM with the whole card. Size seats for the concurrent peak and pilot with your own applications. The L40S runs on vGPU 19, the long-term branch NVIDIA supports until July 2028, and on vGPU 20. Q profiles need an RTX vWS licence, B profiles vPC and C profiles NVIDIA AI Enterprise; our guide to L40S vGPU profiles lists them all, and VDI GPU sizing scales hosts to larger estates.

Power, cooling and chassis

At 350 W per card, eight cards draw 2,800 W. With two AMD EPYC 9355 at a default TDP of 280 W each, cards and processors come to 3,360 W, and with 15 per cent for memory, drives, fans and conversion losses we estimate about 3.9 kW at the wall. A measurement on the delivered system confirms the estimate. We plan a 230 V 16 A feed at 80 per cent, 2.94 kW, so this server needs a 32 A single-phase feed (5.89 kW planned) or a three-phase position on each of its A and B sides. By the 160 CFM per kW of our guide to GPU rack power and cooling, it needs roughly 620 CFM of air at an 11 °C rise.

Dell XE7745 technical guide (Rev. A09, September 2026), Cisco UCS C845A M8 data sheet (24 August 2026), Puget Systems configurator, read on 24 October 2026.

Dell rates the XE7745's 3,200 W supplies at full power from 220.1 to 240 V and at 2,900 W from 200 to 220 V, and sets the inlet limit with L40S cards at 35 °C, against 30 °C for the same server with RTX PRO 6000 Server Edition cards. NVIDIA's certified-systems guide asks for at least six cores per GPU and twice the GPU memory in system memory, so 48 cores and 768 GB for eight cards; our guide to how much RAM a GPU server needs explains the rule. We check the feed, the inlet temperature and the chassis listing before we quote; tell us the rack position and its feed, and we name the chassis that fits.

When another configuration fits better

Four RTX PRO 6000 Server Edition cards hold Llama 3.3 70B in FP8 as four copies with 48 conversations in all, or as two TP=2 pairs with 156, without splitting a copy over four cards, and add MIG and FP4. Eight RTX PRO 4500 Server Edition cards give 256 GB at 1,320 W with 16 MIG halves, a trade our page on eight L40S or eight RTX PRO 4500 Server Edition weighs; the card pairs are compared in L40S vs RTX PRO 4500 Server Edition and L40S vs RTX PRO 6000 Server Edition. Training and fine-tuning of large models belong on H200 NVL with NVLink bridges. Where one server must not be a single point of failure, two four-card servers are the safer layout, as one 8-GPU server or two 4-GPU servers compares.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build AI servers to order with eight L40S cards in chassis whose makers list them, assembled and tested before delivery, on one EU contract and invoice with manufacturer warranty on every component, and install the operating system, drivers, CUDA and a container runtime on request. NVIDIA vGPU and AI Enterprise licences come on the same invoice. Models, RAG and the platform on top are our Private AI/ML service.

FAQ