vLLM, SGLang, TensorRT-LLM, llama.cpp and Ollama: which serving engine for which job
- vLLM 0.30.0, SGLang 0.5.20, TensorRT-LLM 1.2.1 and Ollama 0.34.2 were the current stable releases on 23 September 2026; with llama.cpp, all five are licensed under Apache 2.0 or MIT and serve an OpenAI-compatible API
- TensorRT-LLM’s support matrix lists NVFP4 and MXFP4 for RTX PRO Blackwell (sm120) but for neither Hopper nor Ada, where FP4 weights compute in 16-bit: through Marlin kernels in vLLM and SGLang and, for gpt-oss on the H200, Triton kernels in TensorRT-LLM
- TensorRT-LLM’s current 1.3 pre-releases and documentation drop the TensorRT engine backend, so trtllm-serve loads a Hugging Face checkpoint with no build step; the stable 1.2.1 still installs trtllm-build beside its default PyTorch backend
- Defaults decide how a shared server behaves: Ollama answers 1 request per model at a time, llama-server opens 4 slots and vLLM claims 92 per cent of GPU memory per instance
- For a team endpoint, vLLM or SGLang; for one user, GGUF files or CPU offload, llama.cpp or Ollama; for NVIDIA’s published tables and per-architecture format matrix, TensorRT-LLM
Five engines, one API
All five engines serve open-weight models behind an OpenAI-compatible HTTP API, so an IDE extension, a chat front end or an application written for that API can use any of them for the endpoints each one implements. What differs is how they batch requests, which 4-bit and 8-bit formats they run on which GPU, how they split a model across GPUs, and what they do before anyone changes a default. Versions, flags and defaults below come from each project’s documentation, release pages and source as of 23 September 2026.
| ENGINE | MAINTAINED BY | LICENCE | RELEASE, SEPT 2026 | SERVER COMMAND |
|---|---|---|---|---|
| vLLM | community project begun at UC Berkeley’s Sky Computing Lab | Apache 2.0 | 0.30.0, 22 September | vllm serve, port 8000 |
| SGLang | hosted by LMSYS, a non-profit | Apache 2.0 | 0.5.20, 18 September | python3 -m sglang.launch_, port 30000 |
| TensorRT-LLM | NVIDIA | Apache 2.0 | 1.2.1 stable (April); 1.3.0rc27, 17 September | trtllm-serve, port 8000 |
| llama.cpp | ggml-org | MIT | continuous numbered builds | llama-server, port 8080 |
| Ollama | Ollama | MIT | 0.34.2, 15 September | ollama serve, port 11434 |
Project READMEs, documentation, source, PyPI and GitHub release pages, read 23 September 2026. SGLang, trtllm-serve, llama-server and Ollama listen only on the local machine unless told otherwise. Ollama’s own container image, however, sets OLLAMA_, and the documented docker run with -p 11434:11434 publishes the port on all host addresses, so a server set up that way exposes an API without authentication. vLLM listens on all interfaces unless --host is set.
The documented starting points are short: vllm serve Qwen/Qwen2.5-1.5B-Instruct; SGLang’s module with --model-path and a model name; trtllm-serve followed by a Hugging Face model name; llama-server -m with a GGUF file (the main README’s quick start now uses llama serve); and ollama serve, whose OpenAI-compatible routes sit under /v1.
How each one batches and caches
vLLM stores the key/value cache in fixed-size blocks with PagedAttention, which its 2023 paper describes as “inspired by the classical virtual memory and paging techniques in operating systems”, and schedules with continuous batching and chunked prefill. Prefix caching is on by default in current releases, so requests that begin with the same tokens reuse cached blocks.
SGLang also batches continuously over a paged cache and adds RadixAttention, which keeps prompt prefixes in a radix tree and reuses them across requests. Agents, retrieval pipelines and multi-turn chats produce exactly such shared prefixes.
TensorRT-LLM (NVIDIA’s current documentation writes it TensorRT LLM) uses in-flight batching and a paged KV cache with block reuse. PyTorch has been its default backend since release 1.0. The stable 1.2.1 still installs trtllm-build and accepts --backend tensorrt; the current 1.3 pre-releases drop both, and NVIDIA’s documentation states that the TensorRT engine backend has been removed: PyTorch is the sole execution backend, and trtllm-serve loads a Hugging Face checkpoint with “no separate checkpoint-conversion or engine-build step”. Guides built around trtllm-build describe the path being retired.
llama.cpp is a C/C++ engine for GGUF files, with integer quantisation from 1.5 to 8 bits and “CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity”: --n-gpu-layers sets how many layers go to the GPU, and --n-cpu-moe keeps the expert weights of the first N layers in system memory. llama-server batches continuously across a fixed number of slots.
Ollama adds model downloads and lifecycle management on top. In May 2025 it wrote that it “has so far relied on the ggml-org/llama.cpp project for model support” and announced a new engine built on the GGML tensor library, starting with multimodal models; its README still lists llama.cpp as its backend.
Which formats run on which GPU
FP8 arithmetic starts with Ada in both TensorRT-LLM’s and vLLM’s tables, and FP4 arithmetic needs Blackwell. The engines document this differently: TensorRT-LLM publishes a matrix per architecture, vLLM’s table stops at Hopper, SGLang names SM ranges, and vLLM’s release notes cover SM120 and SM121. On older generations vLLM and SGLang still load FP4 checkpoints through Marlin kernels, which keep the weights in 4-bit and compute in 16-bit. Our format guide explains the formats themselves.
| GPU GENERATION | NVFP4 | MXFP4 | FP8 | INT4 AWQ, GPTQ |
|---|---|---|---|---|
| Ada, 8.9 (L4, L40S) | TensorRT-LLM: no. vLLM, SGLang: weight-only via Marlin | TensorRT-LLM: no. vLLM: weight-only via Marlin | TensorRT-LLM: per-tensor. vLLM: W8A8 | TensorRT-LLM: W4A16, W4A8. vLLM: yes |
| Hopper, 9.0 (H200 NVL) | as Ada | TensorRT-LLM: gpt-oss weights, BF16 compute. vLLM: weight-only via Marlin | TensorRT-LLM: per-tensor, block, rowwise. vLLM: W8A8 | as Ada |
| RTX PRO Blackwell, 12.0 | FP4 kernels in TensorRT-LLM, vLLM and SGLang | TensorRT-LLM: yes | TensorRT-LLM: per-tensor only. SGLang: block-scaled too | not in TensorRT-LLM’s sm120 row. vLLM: Marlin, built for 12.x |
| DGX Spark (GB10), 12.1 | vLLM; TensorRT-LLM beta | vLLM and TensorRT-LLM beta, for gpt-oss | vLLM, block-scaled; TensorRT-LLM beta | vLLM: AWQ INT4 in NVIDIA’s two-unit playbook |
TensorRT-LLM quantisation matrix, gpt-oss deployment guide and release notes (documentation site at 1.3.0rc28, ahead of the newest release on PyPI), vLLM quantisation hardware table, build file and 0.30.0 release notes, SGLang quantisation guide and NVIDIA’s DGX Spark playbooks, read 23 September 2026. Beta is NVIDIA’s label for TensorRT-LLM on DGX Spark, validated for a listed set of models. GGUF files run in llama.cpp and Ollama on all four generations; Ollama lists compute capability 8.9, 9.0, 12.0 and 12.1.
On Hopper, FP4 weights load but compute in 16-bit: TensorRT-LLM’s gpt-oss guide pairs MXFP4 weights with BF16 activations on the H200 through its Triton MoE backend, and TensorRT-LLM’s performance table lists gpt-oss-120b on an H200 as FP8.
Several GPUs and several servers
vLLM supports tensor, pipeline, data and expert parallelism (--tensor-parallel-size, --pipeline-parallel-size, --enable-expert-parallel), with Ray as the default runtime across nodes. Its guidance is tensor parallelism when a model fits one node, pipeline parallelism added across nodes, and pipeline rather than tensor parallelism on GPUs without NVLink, with the L40S as its example. That applies to every RTX PRO Blackwell card, none of which has NVLink.
SGLang takes --tp, --pp-size and --ep, and --nnodes spreads tensor parallelism over several machines. TensorRT-LLM takes --tp_size, --pp_size and --ep_size, and launches multi-node runs through trtllm-llmapi-launch under Slurm.
llama.cpp splits layers and cache across GPUs by default (--split-mode layer, pipelined), with row and an experimental tensor mode as alternatives. Across machines, the only option we found is an RPC backend that its README calls “fragile and insecure”, not to be run “on an open network or in a sensitive environment”. Ollama puts a model on one GPU if it fits there and otherwise spreads it across all of them; we found no multi-machine mode in its documentation. Our guide to one model on several GPUs covers the PCIe and NVLink side.
Concurrency, metrics and adapters
| ENGINE | CONCURRENCY LIMIT | MEMORY CLAIMED | METRICS | LORA SERVING |
|---|---|---|---|---|
| vLLM | --max-num-seqs | 92 per cent of GPU memory, per instance | Prometheus at /metrics | several adapters, --enable-lora |
| SGLang | --max-running-requests, unset by default | --mem-fraction-static, computed unless set | /metrics after --enable-metrics | several per batch, 8 by default |
| TensorRT-LLM | --max_batch_size | --kv_, 0.9 of memory left after weights | /metrics after enable_ | several adapters, chosen per request |
| llama-server | 4 slots when --parallel is auto | fits unset options to device memory | /metrics after --metrics | loaded with --lora, switched via API |
| Ollama | 1 request per model | context by VRAM: 4K, 32K or 256K | none found in its docs | ADAPTER in the Modelfile |
vLLM 0.30.0 source and documentation, SGLang server arguments and LoRA guide, trtllm-serve reference and LoRA guide, llama-server README and source, Ollama FAQ, context-length page and Modelfile reference, September 2026.
Ollama’s defaults surprise shared machines most. Each model processes one request at a time until OLLAMA_ is raised, the server queues up to 512 requests before rejecting more, and every parallel slot adds context memory: “a 2K context with 4 parallel requests will result in an 8K context”. Its context-length page sets the default by VRAM, 4K below 24 GiB, 32K from 24 to 48 GiB and 256K from 48 GiB, while its FAQ still gives 4,096, so set OLLAMA_ explicitly. llama-server shares one KV buffer across its four automatic slots. vLLM’s memory share rose from 0.9 to 0.92 in release 0.20.0 of April 2026, which also began counting CUDA graph memory inside it, and it applies per instance, which matters when two instances share a card. Our article on sharing one DGX Spark shows what these settings mean on one machine.
NVIDIA NIM: engines packaged by NVIDIA
NIM is NVIDIA’s packaged form of such an engine, with industry-standard APIs. NVIDIA’s NIM product page, updated on 28 August 2026, speaks of large language models supported by TensorRT-LLM, vLLM or SGLang; the documentation of NIM for LLMs 2.0.12, updated on 18 September 2026, differs: NIM LLM “is built on vLLM”, and “The multi-backend container (vLLM, TensorRT-LLM, and others) is replaced by a dedicated vLLM container.” NVIDIA’s DGX Spark playbook describes an OpenAI-compatible interface and requires an NGC API key. NVIDIA’s terms are free access for development and testing through its Developer Program and an NVIDIA AI Enterprise licence for production. NIM fits where that licence is in place anyway, as the H200 NVL includes five years of it, and where a supported stack matters more than choosing the engine yourself; the open-source engines themselves need no AI Enterprise licence. Our NVIDIA AI Enterprise page covers the licence.
Which engine for which job
One developer, one GPU or a DGX Spark, trying models: Ollama or llama.cpp, for GGUF files, simple downloads and CPU offload when a model is larger than GPU memory. Before others use it, raise OLLAMA_ or move to a batching server.
A shared endpoint for a team or an application: vLLM or SGLang. Both batch continuously, reuse cached prefixes, export Prometheus metrics, serve several LoRA adapters at once and split models across GPUs and machines. With agents and retrieval, where prompts share long prefixes, that reuse matters most; for coding teams, see our guide to a private coding assistant.
NVIDIA’s published numbers and a documented format matrix: TensorRT-LLM. NVIDIA measures its per-GPU performance tables with it and publishes a format matrix per architecture; on DGX Spark its support is labelled beta. vLLM and SGLang also run FP4 natively on RTX PRO Blackwell.
H200 NVL and other Hopper cards: FP8, or INT4 through AWQ or GPTQ. TensorRT-LLM’s matrix lists neither NVFP4 nor MXFP4 for Hopper, and where FP4 weights do load, in vLLM, SGLang or TensorRT-LLM’s gpt-oss path, the arithmetic runs in 16-bit.
A supported stack under contract: NIM, which for language models runs vLLM inside from release 2.0, according to its documentation.
We have not found a comparison of these engines that holds across models, GPUs and prompt lengths, and we quote none: test your own model with your own prompt lengths.
What we supply
Eurokommerz supplies the GPUs these engines run on across the EU with manufacturer warranty, from DGX Spark and the RTX PRO Blackwell workstation cards to the L4, L40S and H200 NVL, individually or in AI servers built to order. We configure the server for the engine, precision and context you choose, and supply NVIDIA AI Enterprise licences where NIM is part of the plan.
FAQ
Which serving engines expose an OpenAI-compatible API?
Why does Ollama answer only one request at a time?
Does TensorRT-LLM still build TensorRT engines?
Can an H200 NVL run NVFP4 or MXFP4 models?
How much GPU memory does vLLM claim by default?
Do these engines need an NVIDIA AI Enterprise licence?
Tell us which models you want to serve, how many people or applications will call them and which GPUs you have or plan. We will tell you which engine and precision suit that hardware and how much memory is left for the cache. We reply within one business day.
Talk to an expertWe reply within one business day