Serving YOLO and vision models in production: Triton, ONNX Runtime and TensorRT compared

Three layers of one serving stack

To serve a trained detection or classification model in production, most teams export it from PyTorch to ONNX, compile it into a TensorRT engine for the GPU it will run on, and put Triton Inference Server in front, which collects requests from many cameras or applications into batches. ONNX Runtime fits where the same model file must also run on CPUs, or where TensorRT does not support every layer of the model. TorchServe, the PyTorch serving project, has been archived and receives no security fixes.

The three names are often compared as if they were alternatives. They sit at different layers. TensorRT is NVIDIA's compiler and runtime: it turns an ONNX graph into an engine optimised for one GPU and one precision. ONNX Runtime runs ONNX graphs through execution providers, among them a CUDA provider and a TensorRT provider. Triton is a server: it loads models from a repository, exposes HTTP/REST and gRPC endpoints, batches requests and reports metrics, and its backends include TensorRT, ONNX Runtime, PyTorch, OpenVINO and Python.

This page is about vision models that are not language models: YOLO detectors, segmentation networks, classifiers and transformers such as RT-DETR. LLM serving with vLLM, SGLang and TensorRT-LLM is compared in our article on vLLM, SGLang, TensorRT-LLM and Ollama, and camera pipelines with hardware decoding in our guide to a GPU server for video analytics.

Our summary of NVIDIA's Triton and TensorRT documentation, the ONNX Runtime documentation and the TorchServe documentation, read on 25 October 2026.

TorchServe is archived: what that means for a running service

The TorchServe documentation carries the notice "Limited Maintenance: This project is no longer actively maintained." It continues: "While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed." The GitHub repository was archived by its owner on 7 August 2025 and is read-only.

A service that runs today keeps running. The risk grows with time: a flaw found in the server or in one of its dependencies stays open, and newer PyTorch and CUDA releases are not tested against it. For a model behind an internal network that is a planned migration, for a model reachable from production networks it is a security item. The model itself usually moves without retraining: the PyTorch checkpoint is exported to ONNX or TorchScript and served by Triton, which has backends for both.

ONNX Runtime: one model file, several execution providers

ONNX Runtime loads an ONNX file and assigns each part of the graph to an execution provider. Triton's ONNX Runtime backend documents that "When GPU is enabled for ORT, CUDA execution provider is enabled", and that with TensorRT switched on the CUDA provider runs the nodes TensorRT cannot. The TensorRT provider is switched on in the model configuration, with keys such as precision_mode set to FP16 and trt_engine_cache_enable; without them, its precision defaults to FP32 and its workspace to 1 GB.

Two points from the ONNX Runtime documentation matter in operations. "The first run (warmup) can be very long because building engine involves exhaustive profiling", so a container that starts without a cached engine answers late. The same page states that, with TensorRT, ONNX Runtime "delivers better inferencing performance on the same hardware compared to generic GPU acceleration". And cached engines "are not portable and optimized for specific Nvidia hardware": the documentation asks to delete old engine and profile files after a change of hardware, of the model, of the ONNX Runtime version or of the TensorRT version. Keep the engine cache on local storage per GPU type.

Versions move fast. Triton release 26.09 ships ONNX Runtime 1.28.0, and the project's release page listed 1.29.0 as its newest release on 25 October 2026, with NVFP4 work for compute capability 12.0 (SM120) in its notes. The requirements table on the TensorRT provider page lists CUDA and TensorRT versions per ONNX Runtime release; check it for the build you deploy.

TensorRT engines are built for one GPU and one version

NVIDIA's TensorRT documentation states it plainly: "By default, TensorRT engines are compatible only with the version of TensorRT used to build them" and "only compatible with the type of device where they were built". TensorRT records the compute capability in the plan, and "If they do not match, the plan will fail to deserialize." Triton's model repository guide adds that a TensorRT plan is specific to a GPU's CUDA compute capability and offers the cc_model_filenames property to keep one plan per capability in the same model directory.

NVIDIA CUDA GPUs page (compute capability; the H200 NVL is not listed separately from the H200) and NVIDIA product pages, read on 25 October 2026.

A fleet that mixes L4 cards in edge servers with RTX PRO 4500 Server Edition cards in the data centre therefore needs two engines per model. TensorRT offers two ways to relax this. A hardware-compatible engine built with the level AMPERE_PLUS runs on Ampere and newer GPUs, and SAME_COMPUTE_CAPABILITY on GPUs of the same capability; NVIDIA warns that such an engine "can have lower throughput and/or higher latency" and that the same-capability level generally performs better than AMPERE_PLUS. A version-compatible engine runs on later TensorRT releases, packs a copy of the runtime with every plan and "can be slower", and NVIDIA advises to "only deserialize trusted plans this way". Engines built with TensorRT 11 do not run on a TensorRT 10 runtime.

Ultralytics, whose YOLO models are the usual starting point, states the same for its format="engine" export: "TensorRT engines are hardware- and runtime-specific", and an .engine file should not be treated as a portable model format. For INT8 it calls it critical that the export runs on the device that will use the engine, with a calibration dataset. In practice, keep the ONNX file as the artefact under version control and build engines in the deployment pipeline, on the target card and with the TensorRT version of the serving container. We supply the L4 and the RTX PRO Server Editions for mixed fleets like this; tell us which cards your sites run, and we reply with the card types and the number of engines your models need.

Triton dynamic batching for vision models: repository, batches, instances

Triton loads models from a repository with one directory per model, a config.pbtxt and at least one numbered version directory. A TensorRT plan is named model.plan by default, an ONNX file model.onnx, a TorchScript file model.pt and a Python model model.py. For most TensorRT and ONNX models, Triton derives the configuration itself unless started with --disable-auto-complete-config. Ultralytics' Triton guide builds such a repository from a YOLO model exported to ONNX with dynamic=True, starts the container image nvcr.io/nvidia/tritonserver:26.02-py3 and calls it from Python with YOLO("http://127.0.0.1:8000/yolo", task="detect").

Dynamic batching is the main reason to put Triton in front of a vision model. The dynamic batcher "allows inference requests to be combined by the server, so that a batch is created dynamically", and dynamic_batching { } in the configuration turns it on with default settings, which "will create batches as large as possible up to the maximum batch size". max_queue_delay_microseconds lets the server wait a little for more requests. NVIDIA's procedure is to choose a maximum batch size, enable the batcher with defaults, measure latency and throughput with Perf Analyzer, and then raise the batch size or the delay until the latency budget is reached; the documentation adds that preferred batch sizes "should not be used for most models".

Instances set how many copies of a model run on each card. "By default, a single execution instance of the model is created for each GPU available in the system", and instance_group with count: 2 places two on each GPU. Ensembles chain models in the server: "An ensemble model represents a pipeline of one or more models", for example image preprocessing followed by a detector and a classifier, and each member model scales on its own. Triton's documentation names Business Logic Scripting (BLS) as the second way to build pipelines, with the logic written in a Python model. With model warmup configured, Triton reports a model as ready only after warmup has completed, which suits load balancers that check readiness.

NVIDIA Triton Inference Server release notes, release 26.09, page last updated 1 October 2026, read on 25 October 2026. The release supports compute capability 7.5 and later (Turing to Blackwell).

TensorRT's own documentation was at 11.4.0 on the same day, one minor release ahead of the container. An engine built with a standalone TensorRT 11.4 is therefore not one that Triton 26.09 loads; build it inside the same container image, or with the TensorRT version the container lists.

Published throughput on the cards we supply

Published figures for vision models on server cards are scarce, and few use the same model on several cards. NVIDIA's DeepStream 9.2 performance page measures whole video pipelines, "end-to-end performance of the entire video analytic application", with output rendering turned off, and runs the models through TensorRT. The RT-DETR rows are the closest to a plain detector service; our video analytics guide converts the same table into camera counts, while here it serves to compare the cards.

NVIDIA DeepStream 9.2 documentation, Performance, dGPU pretrained models, frames per second per GPU, input 640 × 640, FP16, TensorRT, read on 25 October 2026. SE = Server Edition. The page lists no L4 figures.

For the L4, the DeepStream 7.1 measurement in our video analytics guide is the published reference: 68 H.264 streams at 1080p and 30 frames per second with a light ResNet18 detector, a tracker and two classifiers on one card. Ultralytics publishes latencies for its own models on a T4, the L4's predecessor, with TensorRT 10 at 640 pixels: 1.7 ms per image for YOLO26n, 4.7 ms for YOLO26m and 11.8 ms for YOLO26x. At batch 1 and one request at a time that is about 590, 210 and 85 images per second, by our arithmetic; batching through Triton raises the throughput, and the gain has to be measured with your model.

Precision changes the figures as much as the card. Ultralytics' TensorRT page lists yolov8n.engine on an A100 at 0.52 ms in FP32, 0.33 ms in FP16 and 0.29 ms in INT8 per image at batch 1 on COCO validation. NVIDIA's DeepStream figures above and Ultralytics' Triton example both run in FP16; INT8 needs calibration and an accuracy check against your validation set before it goes into production.

Which card for a vision serving server

The load is images per second: cameras or clients times the frames each one sends for analysis. A detector at 640 pixels is small; YOLO26x has 55.7 million parameters by Ultralytics' table, so GPU memory goes to batches, several model copies and, in video pipelines, decoded frames. Size against the measured rate of your model on the target card, keep headroom for peaks, and treat the published figures above as an order of magnitude.

The L4 suits edge and branch servers and general-purpose hosts: 72 W, passive, low profile and single slot, 24 GB, powered from the PCIe slot, with four video decoders. Our guide to adding a GPU to an existing rack server covers fitting one into a host you already run. The RTX PRO 4500 Server Edition is a single-slot passive Blackwell card at 165 W with 32 GB, and eight of them serve eight independent models; our comparison of the L4 and RTX PRO 4500 Server Edition weighs the two. The RTX PRO 6000 Server Edition suits central servers with heavier models such as RT-DETR or open-vocabulary detectors, many cameras per card, or MIG instances for separate teams.

We supply the L4, L40S and the RTX PRO Server Editions, as cards for servers you run or in servers built to order; send us your models, image sizes and images per second, and our team replies with a card and server proposal.

Operating notes for the serving server

TensorRT's support matrix names R580 as the minimum driver branch for its CUDA 13.x packages, and the Triton release notes refer to NVIDIA's CUDA compatibility page for the driver a container needs. Our overview of NVIDIA driver branches and CUDA versions lists the branches. On Blackwell cards, software built without kernels for compute capability 12.0 does not run; our article on Blackwell sm_120 software support shows which releases include them.

Run Triton as a container with the NVIDIA Container Toolkit and pin each container to its cards; our guide to GPU assignment in Docker without Kubernetes shows how, and Kubernetes GPU sharing with MIG, time-slicing and MPS covers clusters. Rebuild TensorRT engines whenever the container's TensorRT version changes, and keep the engines next to the image tag they were built with. For the licence of the YOLO weights and code behind the service, see our guide to the Ultralytics YOLO licence; for training the models, our vision training server guide.

What we supply

We supply the NVIDIA L4, L40S, RTX PRO 4500 Server Edition and RTX PRO 6000 Server Edition for servers you already run, and build AI servers to order with them, assembled and tested before delivery, on one EU contract and invoice with manufacturer warranty on every component. On request the server arrives with the operating system, NVIDIA driver, CUDA and a container runtime installed. Quantities and lead time are confirmed in writing with the quote.

FAQ