What a server with 8× NVIDIA L4 runs: video analytics, small models, VDI and power

What eight 24 GB cards add up to

A server with eight NVIDIA L4 cards runs video analytics on several hundred camera streams, AV1 transcoding, small language models of up to about 14B parameters one copy per card, embedding and reranking models for RAG, and up to 192 light virtual desktops. By our users-per-card rule it holds about 137.6 GiB of weights and cache with ECC on, at about 1.3 kW for the whole server.

The L4 is NVIDIA's low-power Ada Lovelace card: 24 GB of GDDR6, 300 GB/s, FP8 Tensor Cores, 72 W, passive, single-slot and low-profile, on PCIe 4.0 x16 with 64 GB/s. It has no NVLink and no MIG, so the eight cards do not pool their memory and are shared between virtual machines by time-sliced vGPU.

NVIDIA L4 product page, read on 24 October 2026; NVIDIA marks its tensor figures "Shown with sparsity", half without. Usable memory and sums are our arithmetic.

If you are planning a purchase, our page to buy an L4 server has the configuration, chassis and ordering steps. This page works out what eight cards do.

How much of 192 GB holds models and cache

Our users-per-card rule takes 90 per cent of the memory the driver reports and subtracts 3 GiB per card for CUDA graphs, activations and the runtime; the rest holds weights and KV cache. Scaled from the 97,887 MiB a 96 GB card reports, a 24 GB card shows about 23.9 GiB. NVIDIA's CUDA Best Practices Guide states that ECC on GDDR memory takes 6.25 per cent of capacity, which leaves about 22.4 GiB with ECC on. That gives 0.9 × 22.4 − 3 = 17.2 GiB per card and 137.6 GiB for eight, or 18.5 and 148.0 GiB with ECC off. Check the figure with nvidia-smi -q in the ECC mode you will run. The rule is derived in our guide to how many users one RTX PRO 6000 can serve.

With an FP8 KV cache, a conversation of 8,192 tokens takes 0.5625 GiB on Qwen3-8B, 0.625 GiB on Qwen3-14B, 1.0 GiB on Qwen3-32B and 1.25 GiB on a Llama 70B model, by the layers, KV heads and head size in each model's configuration. A 16-bit cache doubles these figures and halves the conversations.

Small models and layouts on eight L4

Eight cards allow one copy per card, TP=2 pairs, TP=4 copies or one copy over all eight. On the L4 the choice is sharper than on larger cards, because a 14B model leaves little room for cache on one card and a 32B model does not fit at all.

Our estimates by the users-per-card rule, not measurements: concurrent conversations of 8,192 tokens at full length with an FP8 KV cache, ECC on, in brackets ECC off. Weights are the Qwen and NVIDIA FP8 checkpoints and the INT4 AWQ checkpoint of 39.8 GB, as in our sizing guides.

Two rows show the arithmetic. One copy of Qwen3-8B leaves 17.2 − 8.8 = 8.4 GiB on its card, 14 conversations, 112 for eight cards. A TP=2 pair of Qwen3-14B has 34.4 GiB; less 15.2 GiB of weights that leaves 19.2 GiB, 30 conversations per pair and 120 for four pairs, five times the count of eight single copies.

Qwen's FP8 releases scale their weights in blocks of 128 by 128, a format TensorRT-LLM's support matrix does not list for Ada, as our L40S benchmark review notes, so confirm that your serving engine runs the checkpoint on the L4. Ada has no FP4 Tensor Cores: 4-bit models usually run as INT4 through AWQ or GPTQ, which vLLM's quantisation table marks as supported on Ada. Our article on FP4 models on H200 NVL and Ada GPUs covers the formats, and our guide to small language models on the L4 sizes Gemma, Ministral and Nemotron models per card.

Users per model for 500 to 2,000 staff

The conversations are requests in flight at the same moment, not named users. With the example values of our guide to a private ChatGPT server by company size, 500 staff produce about 20 requests in flight at the busiest moment, 1,000 staff about 40 and 2,000 staff about 80.

By those values, eight copies of Qwen3-8B with about 112 conversations cover 2,000 staff, and four TP=2 pairs of Qwen3-14B with about 120 do the same with a stronger model. A 32B model in two TP=4 copies covers about 1,000 staff with headroom. The eight cards can also carry several models at once: an assistant on four cards, a classifier on two, and an embedding model and a reranker on one card each, each with its own memory and restart cycle. Our guide to serving several models on one GPU server covers the engine settings.

We size this server per model and per camera pipeline; send us your models and the peak requests in flight, and our team replies with a layout and the conversations each model gets.

Splitting a model over PCIe 4.0

Every split copy exchanges partial results between its cards at every layer, here over PCIe 4.0 x16 at 64 GB/s. vLLM's parallelism guide, updated on 15 October 2026, says that GPUs without NVLink, with the L40S as its example, should "leverage pipeline parallelism instead of tensor parallelism for higher throughput and lower communication overhead". By default vLLM disables its custom all-reduce on more than two PCIe-only GPUs and uses NCCL; vLLM 0.30.0 (release notes, 22 September 2026) added an opt-in FlashInfer PCIe IPC all-reduce for systems without NVLink (#53576), for which we found no published end-to-end result. The default favours TP=2 pairs under one processor and makes TP=8 a capacity layout.

For one stream, speed is capped by how fast the cards read the weights. Qwen3-8B in FP8 reads about 9.4 GB per token, so one L4 could reach about 32 tokens per second; Qwen3-14B about 18 on one card and 37 on an ideal TP=2 pair; Llama 3.3 70B in FP8, at about 70.6 GB per token, about 34 on an ideal TP=8 copy before any PCIe cost. These are ceilings by our arithmetic. Use nvidia-smi topo -m to keep a pair under one processor, and read our article on one model on several GPUs over PCIe or NVLink for the options.

Published results for the L4

MLCommons publishes MLPerf® Inference results with the logs of each run. We found no L4 result with eight cards in a recent round, and none at all in v6.1 of 16 September 2026. The v5.1 round has a two-card L4 system on RetinaNet object detection.

MLPerf® Inference: Datacenter v5.1, closed division, RetinaNet, offline scenario, entry 5.1-0018, retrieved from mlcommons.org on 24 September 2026, result verified by MLCommons Association. Per card is our division of the system result, not an MLPerf® metric.

At that rate per card, eight cards would reach about 1,834 images per second, an extrapolation by our arithmetic, not a result. We found no comparable published language-model result for the L4 in recent datacenter rounds.

The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

StorageReview measured one L4 with vLLM 0.27.1 in a Dell PowerEdge R770 and published the results on 14 October 2026: 364 output tokens per second on Ministral 3 14B Instruct in FP8 at 32 concurrent requests of 512 tokens in and out. Per rated watt that is 5.1 tokens per second, against 2.8 for the L40S and 6.7 for the RTX PRO 4500 Server Edition in the same test. Eight independent copies would reach at most about 2,900 tokens per second by our arithmetic, an upper bound from one card.

Video analytics on 32 decoders

A video analytics server is limited first by its hardware decoders and then by the inference per analysed frame. Each L4 has four NVDEC engines, so eight cards bring 32. Our guide to a GPU server for video analytics derives a decode ceiling of about 120 H.264 or 218 H.265 streams at 1080p and 30 frames per second per L4 from NVIDIA's NVDEC application note, 960 or 1,744 for eight cards. NVIDIA states that the rate scales with the video clock, so these are upper bounds.

NVIDIA's DeepStream 7.1 documentation has a measured run on one L4: 1080p files at 30 frames per second through a pruned ResNet18 TrafficCamNet detector on every frame, an IOU tracker and two ResNet18 classifiers, with NVIDIA's INT8 configuration file, on an AMD EPYC 7763 system.

NVIDIA DeepStream 7.1 documentation, performance, read on 24 October 2026; the eight-card column is our estimate, not a measurement: per-card streams × 8, planned at 70 per cent for peaks and growth, assuming processors and PCIe do not limit.

The processor is the catch with H.265. In NVIDIA's run, 81 H.265 streams on one card took 46.1 per cent of the EPYC 7763, and eight cards at that load would need about 3.7 times that processor by linear arithmetic, against about 64 per cent of it for 68 H.264 streams per card. Plan the processors from a test run with your cameras. Heavier models change the picture: NVIDIA publishes no PeopleNet or RT-DETR figure for the L4, so a pipeline with a larger detector needs its own test, or a larger card such as the L40S, for which NVIDIA's DeepStream 9.2 tables give 1,114 frames per second on PeopleNet with a tracker.

Two settings stretch the decoders further. DeepStream's inference interval property analyses one frame in three with interval=2, while the tracker still sees every frame, and skip-frames set to decode_key decodes key frames only, for scenes that change slowly. The decoders still decode every frame in the default mode, so size them for the cameras' full frame rate.

AV1 transcoding and video engines

For encoding, NVIDIA's L4 page states that "servers equipped with L4 allow hosting up to 1,040 concurrent AV1 video streams at 720p30 for mobile users", with the footnote "8x L4 AV1 low-latency P1 preset encode at 720p30". That is NVIDIA's figure for exactly this server size, about 130 streams per card, for a low-latency preset at 720p. The same page states that "L4 delivers 120X higher AI video performance than CPU-based solutions", measured with eight L4 against a two-socket server with Intel 8362 processors, in an end-to-end pipeline with CV-CUDA and TensorRT 8.6 against OpenCV 4.7 and PyTorch on the processors; that is a comparison with a CPU-only server, not with another GPU. Four JPEG decoders per card, 32 in all, serve cameras that send MJPEG or still images.

Virtual desktops on eight L4

The L4 has no MIG, so desktops share a card by time-sliced vGPU. NVIDIA's vGPU types reference of 29 September 2026 gives these maxima per card in equal-size mode; in mixed-size mode the 1 GB maximum falls to 16 per card.

NVIDIA Virtual GPU Types Reference (29 September 2026) for B and Q profiles and NVIDIA AI Enterprise 8.2 Ada Lovelace vGPU types (2 September 2026) for C profiles, as listed in our L4 vGPU profiles guide; eight-card totals are our arithmetic.

These are maximum densities. Twelve desktops on L4-2B share the card's two encoders, six per encoder, so remote-display encoding becomes a limit before memory does. B profiles need a vPC or vWS licence, Q profiles vWS and C profiles NVIDIA AI Enterprise. Our guide to L4 vGPU profiles lists every profile, and VDI GPU sizing scales hosts to larger estates.

Power, cooling and chassis

At 72 W per card, eight cards draw 576 W. With two AMD EPYC 9355 at a default TDP of 280 W each, cards and processors come to 1,136 W, and with 15 per cent for memory, drives, fans and conversion losses we estimate about 1.3 kW at the wall. A measurement on the delivered system confirms the estimate. That fits one 230 V 16 A feed, which we plan at 2.94 kW, and needs about 210 CFM of air at an 11 °C rise by the 160 CFM per kW of our guide to GPU rack power and cooling. A video server with larger processors for H.265 draws more.

Lenovo lists eight L4 in its 2U SR650 V3 and SR655 V3 and its 3U SR675 V3, and ten in the 2U SR650 V4; Dell lists up to sixteen in the 4U PowerEdge XE7745, with a 35 °C inlet limit for the card in its listed configuration. NVIDIA's certified-systems guide asks for at least six cores per GPU and twice the GPU memory in system memory, so 48 cores and 384 GB for eight cards. Tell us the rack position, its depth and its feed, and we name the chassis that fits.

When another configuration fits better

Where one answer must come faster than 300 GB/s allows, or a model needs 32 GB on one card, eight RTX PRO 4500 Server Edition cards give 256 GB at 800 GB/s per card, FP4 and 16 MIG instances at 1,320 W of card power, as our page on the 8× RTX PRO 4500 Server Edition server shows. For heavier vision models, VDI with CAD profiles and 32B models per card, the 8× L40S server has 48 GB per card at 2,800 W. For a 70B model serving many users, four RTX PRO 6000 Server Edition cards hold one copy per card without splitting it, and our L4 vs L40S comparison sets the two Ada cards side by side.

What we supply

You can buy this system from us: our team sizes the configuration to your workload and budget and replies within one business day. We build AI servers to order with eight L4 cards in chassis whose makers list them, assembled and tested before delivery, on one EU contract and invoice with manufacturer warranty on every component, and install the operating system, drivers, CUDA and a container runtime on request. NVIDIA vGPU and AI Enterprise licences come on the same invoice. Models, RAG and the platform on top are our Private AI/ML service.

FAQ