vLLM, SGLang, TensorRT-LLM, llama.cpp and Ollama: which serving engine for which job

OpenAI-compatible API, versions, licences and ports

vLLM, SGLang, TensorRT-LLM, llama.cpp and Ollama all serve open-weight models behind an OpenAI-compatible HTTP API, so an IDE extension, a chat front end or an application written for that API can use any of them for the endpoints each one implements. They differ in how they batch requests, which 4-bit and 8-bit formats they run on which GPU, how they split a model across GPUs, and how they behave with their default settings. Release versions below are as of 5 November 2026; flags and defaults come from each project’s documentation, release pages and source as of 23 September 2026.

Project READMEs, documentation, source, PyPI and GitHub release pages, read 23 September 2026; versions rechecked on 5 November 2026. SGLang, trtllm-serve, llama-server and Ollama listen only on the local machine unless configured otherwise. Ollama’s own container image, however, sets OLLAMA_HOST=0.0.0.0:11434, and the documented docker run with -p 11434:11434 publishes the port on all host addresses, so a server set up that way exposes an API without authentication. Ollama before 0.35.0 is also affected by CVE-2026-103663 (GHSA-w8p2-phwr-px3r, published 8 October 2026): a path traversal in /api/pull lets anyone who reaches the API place a file outside the model store. It is rated critical (CVSS 9.4) where the Ollama service can write to /usr/lib/ollama, since the file then runs as root after a restart, and 6.9 otherwise. Run 0.35.0 or later and keep port 11434 off the network. vLLM listens on all interfaces unless --host is set; release 0.31.0 fixed a high-rated code-execution issue for servers started with --trust-remote-code, which vLLM’s advisory GHSA-h3rc-6mm3-gc2m of 6 October 2026 documents (CVSS 8.1, versions from 0.7.3 to before 0.31.0 affected; the 0.31.0 release notes do not mention it), and 0.32.0 fixed further lower-rated issues, so run 0.31.0 or later, preferably 0.33.0 of 2 November 2026, the current release as of November 2026, whose CUDA 13.0 builds need an R580 or newer driver, on a shared endpoint. Since 0.32.0, vLLM allows no CORS origins by default, so browser clients on another origin need --allowed-origins; it also needs Python 3.11 or later and treats max_tokens as an upper bound rather than a reservation. For TensorRT-LLM, NVIDIA published a security bulletin on 27 October 2026 (CVE-2026-65119, CVE-2026-24269, CVE-2026-65120, all rated High), fixed in 1.3.0rc25 and later; the bulletin lists 1.3.0rc24 as the affected version.

The documented quick-start commands are vllm serve Qwen/Qwen2.5-1.5B-Instruct; SGLang’s module with --model-path and a model name; trtllm-serve followed by a Hugging Face model name; llama-server -m with a GGUF file (the main README’s quick start now uses llama serve); and ollama serve, whose OpenAI-compatible routes sit under /v1.

How each one batches and caches

vLLM stores the key/value cache in fixed-size blocks with PagedAttention, which its 2023 paper describes as “inspired by the classical virtual memory and paging techniques in operating systems”, and schedules with continuous batching and chunked prefill. Prefix caching is on by default in current releases, so requests that begin with the same tokens reuse cached blocks.

SGLang also batches continuously over a paged cache and adds RadixAttention, which keeps prompt prefixes in a radix tree and reuses them across requests. Agents, retrieval pipelines and multi-turn chats produce exactly such shared prefixes.

TensorRT-LLM (NVIDIA’s current documentation writes it TensorRT LLM) uses in-flight batching and a paged KV cache with block reuse. PyTorch has been its default backend since release 1.0. The stable 1.2.1 still installs trtllm-build and accepts --backend tensorrt; the 1.3 and 1.4 pre-releases drop both, and NVIDIA’s documentation states that the TensorRT engine backend has been removed (its release notes list the change under release 1.2, although the setup file of release 1.2.1 still registers trtllm-build): PyTorch is the sole execution backend, and trtllm-serve loads a Hugging Face checkpoint with “no separate checkpoint-conversion or engine-build step”. Guides built around trtllm-build therefore describe a workflow that NVIDIA is retiring.

llama.cpp is a C/C++ engine for GGUF files, with integer quantisation from 1.5 to 8 bits and “CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity”: --n-gpu-layers sets how many layers go to the GPU, and --n-cpu-moe keeps the expert weights of the first N layers in system memory. llama-server batches continuously across a fixed number of slots.

Ollama adds model downloads and lifecycle management on top. In May 2025 it wrote that it “has so far relied on the ggml-org/llama.cpp project for model support” and announced a new engine built on the GGML tensor library, starting with multimodal models; its README still lists llama.cpp as its backend.

Which formats run on which GPU

FP8 arithmetic starts with Ada in both TensorRT-LLM’s and vLLM’s tables, and FP4 arithmetic needs Blackwell. The engines document this differently: TensorRT-LLM publishes a matrix per architecture, vLLM’s table stops at Hopper, SGLang names SM ranges, and vLLM’s release notes cover SM120 and SM121. On older generations vLLM and SGLang still load FP4 checkpoints through Marlin kernels, which keep the weights in 4-bit and compute in 16-bit. Our format guide explains the formats themselves.

TensorRT-LLM quantisation matrix, gpt-oss deployment guide and release notes (documentation site at 1.3.0rc29 as of 5 October 2026, ahead of the stable 1.2.1), vLLM quantisation hardware table, build file and 0.30.0 release notes, SGLang quantisation guide and NVIDIA’s DGX Spark playbooks, read 23 September 2026. Beta is NVIDIA’s label for TensorRT-LLM on DGX Spark, validated for a listed set of models. GGUF files run in llama.cpp and Ollama on all four generations; Ollama lists compute capability 8.9, 9.0, 12.0 and 12.1.

On Hopper, FP4 weights load but compute in 16-bit: TensorRT-LLM’s gpt-oss guide pairs MXFP4 weights with BF16 activations on the H200 through its Triton MoE backend, and TensorRT-LLM’s performance table lists gpt-oss-120b on an H200 as FP8. We supply the GPUs in the table above and build AI servers to order with them, such as the inference node with four RTX PRO 6000 Server Edition cards. Tell us your model and the number of users, and we propose a configuration.

Tensor, pipeline and expert parallelism across GPUs and servers

vLLM supports tensor, pipeline, data and expert parallelism (--tensor-parallel-size, --pipeline-parallel-size, --enable-expert-parallel); across nodes its documentation names Ray as the default runtime and also shows a multiprocessing set-up with --nnodes and --node-rank. Its guidance is tensor parallelism when a model fits one node, pipeline parallelism added across nodes, and pipeline rather than tensor parallelism on GPUs without NVLink, with the L40S as its example. That applies to every RTX PRO Blackwell card, none of which has NVLink.

SGLang takes --tp, --pp-size and --ep, and --nnodes spreads tensor parallelism over several machines. TensorRT-LLM takes --tp_size, --pp_size and --ep_size, and launches multi-node runs through trtllm-llmapi-launch under Slurm.

llama.cpp splits layers and KV cache across GPUs by default (--split-mode layer) and runs them as a pipeline. row splits the weights by rows, and tensor, marked experimental, splits weights and KV cache; both compute on the GPUs in parallel, and none keeps the model on one GPU. Across machines, the only option we found is an RPC backend that its README calls “fragile and insecure”, not to be run “on an open network or in a sensitive environment”. Ollama puts a model on one GPU if it fits there and otherwise spreads it across all of them; we found no multi-machine mode in its documentation. Our guide to one model on several GPUs covers the PCIe and NVLink side.

Concurrency limits, GPU memory, metrics and LoRA adapters

vLLM 0.30.0 source and documentation, SGLang server arguments and LoRA guide, trtllm-serve reference and LoRA guide, llama-server README and source, Ollama FAQ, context-length page and Modelfile reference, September 2026.

Ollama’s defaults matter most on a shared server. Each model processes one request at a time until OLLAMA_NUM_PARALLEL is raised, the server queues up to 512 requests before rejecting more, and every parallel slot adds context memory: “a 2K context with 4 parallel requests will result in an 8K context”. Its context-length page sets the default by VRAM: 4K below 24 GiB, 32K from 24 to 48 GiB and 256K from 48 GiB. Its FAQ still gives 4,096, so set OLLAMA_CONTEXT_LENGTH explicitly. llama-server’s README gives -np a default of -1, auto, and its server source then sets four slots that share one unified KV buffer. vLLM’s memory share rose from 0.9 to 0.92 in release 0.20.0 of April 2026, which also began counting CUDA graph memory inside it. The share applies per instance, which matters when two instances share a card. Our article on sharing one DGX Spark shows what these settings mean on one machine.

NVIDIA NIM: engines packaged by NVIDIA

NIM is NVIDIA’s packaged form of such an engine, with industry-standard APIs. NVIDIA’s NIM product page, updated on 28 August 2026, speaks of large language models supported by TensorRT-LLM, vLLM or SGLang. The documentation of NIM for LLMs 2.0.12, updated on 18 September 2026, says something different: NIM LLM “is built on vLLM”, and “The multi-backend container (vLLM, TensorRT-LLM, and others) is replaced by a dedicated vLLM container.” Its release notes for 2.0.13 updated the bundled vLLM to 0.28.0 and, with its support matrix, added model-specific containers, for GLM-5.2 among others, that use SGLang as the inference backend. Release 2.1.4, the newest in NVIDIA's release notes as of 5 November 2026 (updated 23 October 2026), moves to vLLM 0.30.0 and SGLang 0.5.20 and fixes 28 security issues in the vLLM and SGLang images. NVIDIA’s DGX Spark playbook describes an OpenAI-compatible interface and requires an NGC API key. Development and testing are free through NVIDIA’s Developer Program. For production, NVIDIA’s NIM FAQ, last modified on 6 August 2026, states: “Using NIM in production requires an NVIDIA AI Enterprise license.” NVIDIA’s NIM offerings page, updated on 15 October 2026, lists the NIM Certified Feature Branch “at no charge under the applicable license terms” and requires the subscription for the Production Branch, so have the quote state which offering a production server rests on. NIM fits where that licence is in place anyway, as the H200 NVL includes five years of it, and where a supported stack matters more than choosing the engine yourself; the open-source engines themselves need no AI Enterprise licence. Our NVIDIA AI Enterprise page covers the licence.

Which engine for which job

For one developer trying models on one GPU or a DGX Spark, use Ollama or llama.cpp: they handle GGUF files, simple downloads and CPU offload when a model is larger than GPU memory. Before others use it, raise OLLAMA_NUM_PARALLEL or move to a batching server.

A shared endpoint for a team or an application calls for vLLM or SGLang. Both batch continuously, reuse cached prefixes, export Prometheus metrics, serve several LoRA adapters at once and split models across GPUs and machines. With agents and retrieval, where prompts share long prefixes, that reuse matters most; for coding teams, see our guide to a private coding assistant.

We can deploy the endpoint for you on-premise with vLLM or NVIDIA AI Enterprise, as part of our Private AI/ML service. Write to us about the endpoint you plan; the first call is free of charge.

If you want NVIDIA’s published numbers and a documented format matrix, use TensorRT-LLM. NVIDIA measures its per-GPU performance tables with it and publishes a format matrix per architecture; on DGX Spark its support is labelled beta. vLLM and SGLang also run FP4 natively on RTX PRO Blackwell.

On the H200 NVL and other Hopper cards, serve FP8, or INT4 through AWQ or GPTQ. TensorRT-LLM’s matrix lists neither NVFP4 nor MXFP4 for Hopper, and where FP4 weights do load, in vLLM, SGLang or TensorRT-LLM’s gpt-oss path, the arithmetic runs in 16-bit.

For a supported stack under contract there is NIM, which for language models runs vLLM inside from release 2.0 and SGLang in some model-specific containers, according to its documentation.

We found no comparison of these engines that holds across models, GPUs and prompt lengths, and we quote none: test your own model with your own prompt lengths.

What we supply

We supply the GPUs these engines run on, from DGX Spark and the RTX PRO Blackwell workstation cards to the RTX PRO 6000 Server Edition, L4, L40S and H200 NVL, on their own or in AI servers built to order, on one EU contract and invoice and with manufacturer warranty. Tell us the engine, precision and context length you have chosen, and we configure the server for them, with the operating system, drivers, CUDA and a container runtime installed on request. Where NIM is part of the plan, the NVIDIA AI Enterprise licence goes into the same quote.

FAQ