GitLab Duo Self-Hosted with vLLM: models, AI Gateway and GPUs for 50 to 500 developers

What GitLab Duo Self-Hosted needs

GitLab Duo Self-Hosted lets a GitLab Self-Managed instance run Duo features such as Code Suggestions, Chat and the Agent Platform on models served from your own GPU server, with vLLM as the serving platform GitLab documents. It needs the Premium or Ultimate tier with the GitLab Duo Enterprise add-on, a self-hosted AI Gateway on ordinary CPU servers and an OpenAI-compatible vLLM endpoint per model. For 50 to 500 developers, a server with two, four or eight NVIDIA RTX PRO 6000 Blackwell Server Edition cards covers the range by our sizing, with Qwen3.8-27B for agent sessions and a smaller model for code suggestions and chat.

All GitLab facts below are from GitLab's documentation as published for version 19.6 and read on 25 October 2026. GitLab changes this area with almost every release, so check the pages again before you order. For the general choice of coding models, IDE extensions and licences, see our guide to a private coding assistant; this page covers what is specific to GitLab.

Tier, add-ons and versions

GitLab's page lists "Tier: Premium, Ultimate" and "Offering: GitLab Self-Managed, GitLab Dedicated for Government". It is explicit about the add-on: "To use GitLab Duo features with GitLab Duo Self-Hosted, you must have the GitLab Duo Enterprise add-on", even for features that Duo Core or Duo Pro offer through GitLab's cloud gateway.

The version history matters when you plan an upgrade:

GitLab documentation "GitLab Duo Self-Hosted", "Configure GitLab Duo Self-Hosted" and "Offline deployment" (docs version 19.6), read 25 October 2026.

GitLab describes three ways to run it. In the fully self-hosted configuration "you deploy your own AI Gateway and use only supported LLMs in your infrastructure", and it "can operate in fully isolated networks". The hybrid configuration sends selected features to GitLab-managed models and needs internet access. With an online licence the instance contacts GitLab's licence and billing endpoints on port 443; GitLab states that "Inference data, including code inputs, model prompts, and model responses, does not leave the customer network". Billing differs between online and offline licences, and GitLab's sales team confirms the terms; we make no statement on them.

The AI Gateway: CPU servers, no GPU

The AI Gateway is GitLab's service between the instance and your models. GitLab's installation page says plainly: "A GPU is not needed for the AI Gateway." It runs as a Docker image or a Helm chart, listens on port 5052 for HTTP and 50052 for the Agent Platform's gRPC service, and its image tag must match the GitLab version: for GitLab vX.Y.*-ee use the latest self-hosted-vX.Y.*-ee tag.

GitLab's own scaling figures are the best guide for it:

• 2 vCPUs and 8 GB of RAM handle about 40 concurrent requests;
• 8 vCPUs and 32 GB handle up to 160 concurrent requests, for roughly 50 to 200 users;
• above 200 users, several such instances with autoscaling;
• as a planning metric, "7 RPS (requests per second) per 1000 active users".

Code completion with a self-hosted model routes through the GitLab instance, so the developers' IDEs do not need direct access to the gateway. Agent Platform flows run as jobs on your GitLab runners with GitLab's executor image, which the offline guide asks you to pin to the version your instance requests.

vLLM as the serving platform

GitLab's serving-platform page asks for vLLM "v0.18.1 or later" and says that for OpenAI-compatible platforms such as vLLM the endpoint URL "must be suffixed with /v1". In the admin area you add each deployment with a name, model family, endpoint, an optional API key and a model identifier of the form custom_openai/<model>.

Three settings decide whether it works in practice:

1. Tool calling. GitLab's model page requires --enable-auto-tool-choice and a matching --tool-call-parser; without them, requests with tools fail with the error that "auto" tool choice requires both flags. Our article on vLLM tool-calling parsers lists the parser per model family.
2. Context length. The Agent Platform trims the history to fit a 200,000-token context window, and the Code Review flow sends up to 1 MiB of merge-request context. Set --max-model-len to the context you want to support and size the KV cache for it.
3. Timeouts. The AI Gateway ends a single chat model request after 30 seconds by default (AIGW_DUO_CHAT__MODEL_REQUEST_TIMEOUT raises it), and the general request timeout is 120 seconds, adjustable from 60 to 600. A GPU server that is too small shows up as timeouts in Chat before anything else.

Set --api-key on vLLM and enter the same key in GitLab. It protects only the paths under /v1, /v2, /inference and /cohere (vLLM 0.32.0 security guide), so keep the engine on an internal network or behind an LLM gateway with keys and logging. Each Duo feature gets its own model from a drop-down, so code suggestions, Chat and the Agent Platform can run on different models and cards.

Which models GitLab supports

GitLab rates each model per feature: F for full, P for partial and L for limited functionality, which GitLab describes as unsuitable for the feature. Most of the list are cloud models. The open-weight models that fit a server with RTX PRO 6000 Server Edition cards are these:

ratings F / P / L from GitLab "Supported models and hardware requirements" (docs 19.6; Mistral Small 24B 2506 = Mistral Small 24B Instruct 2506; CODE = code completion and code generation, both rated alike for these four), read 25 October 2026. Weights: sums of the safetensors files on Hugging Face (Mistral-Small-3.2-24B-Instruct-2506, gpt-oss-120b, Mistral-Medium-3.5-128B) and the repository size of Qwen3.8-27B-FP8 (30.9 GB), read 25 October 2026.

GitLab also lists Llama 3.1 and 3.3 70B, Codestral 22B and very large models such as GLM-5.3, Kimi-K2.6 and MiniMax-M2.7, and a beta list of "compatible" models such as Gemma-4-31B-IT and Qwen3.6-35B-A3B for which GitLab gives no model-specific support.

Licences differ. gpt-oss-120b and Qwen3.8-27B are Apache 2.0 by their LICENSE files; Mistral Small 3.2 is tagged Apache 2.0 on its model card, and its repository has no LICENSE file. Mistral Medium 3.5 is under a "Modified MIT License" whose section 2 withdraws all rights if "the global consolidated monthly revenue of your company (or that of your employer) exceeds $20 million", so a company above that threshold has to request a commercial licence, which, by the same file, Mistral AI may grant at its sole discretion. Codestral 22B's licence restricts production use, as our coding-assistant guide explains.

GitLab's own hardware table is based on the A100 and older models, from one 40 GB A100 for a 7B model to two 80 GB A100 for Codestral 22B, and states that H100 and H200 are expected to work but untested. It does not list Blackwell cards, so the card counts below are our sizing, not GitLab's.

We supply RTX PRO 6000 Server Edition and H200 NVL servers built to order, with vLLM set up on request; send us your developer count and the Duo features you plan to use, and our team replies with a card layout and the models for each feature.

How many sessions one card holds

Our users-per-card rule takes 90 per cent of the memory the driver reports and subtracts 3 GiB per card: 83.0 GiB on one RTX PRO 6000 Server Edition. What the weights leave is KV cache for the requests in flight. The cache per token comes from each model's config.json.

Our estimates by the users-per-card rule. Cache per 32K tokens in 16 bits: Qwen3.8-27B 2.0 GiB on its 16 full-attention layers plus at most 0.14 GiB linear-attention state; Mistral Small 3.2 5.0 GiB (40 layers, 8 KV heads of 128); Mistral Medium 3.5 11.0 GiB (88 layers, 8 KV heads of 128); gpt-oss-120b 1.125 GiB. Memory arithmetic, not measured throughput.

Qwen3.8-27B is the efficient agent model here: only 16 of its 64 layers keep a full KV cache, so a 32K session costs about 2.1 GiB. Mistral Medium 3.5 is the only open model rated F for every feature, but as a dense 128B model with 88 full layers it needs 11 GiB per 32K session in 16 bits, and two cards hold three such sessions. On two H200 NVL with NVLink, 246.8 GiB usable by our rule, it holds 11 sessions of 32K, 22 with an FP8 cache.

For code suggestions the load is light. GitLab's planning figure of 7 requests per second per 1,000 active users gives 3.5 requests per second for 500 developers, and every second a request stays in flight adds 3.5 requests on the GPU. Mistral Small 3.2 on one card holds 30 requests of 8K tokens with a 16-bit cache by our rule, so it carries 500 developers' completions as long as the average request finishes within about 8 seconds. A pilot measures the real time.

Configurations for 50 to 500 developers

Agent sessions decide the server. Our rule from the guide to a server for coding agents counts one generating session per developer at the busy hour as the upper bound; GitLab's Duo usage analytics or the gateway log of a pilot give the real share, and a lower share lets the same server carry more people.

Our estimates by the users-per-card rule with one Qwen3.8-27B copy per agent card (47 sessions of 32K each with an FP8 KV cache, 25 with a 16-bit cache); wall power by our estimate with the cards at 600 W, two AMD EPYC processors and 15 per cent for the rest, as on our server pages. Plan feeds on the maker's figure.

About 50 developers: two cards. One card serves Code Suggestions and Chat with Mistral Small 3.2, the other runs Qwen3.8-27B for the Agent Platform with 47 sessions of 32K tokens on an FP8 cache, close to one per developer. With 128K contexts the agent card holds 13 sessions, so a team that runs long review flows needs a second agent card early. This is our entry configuration, described on our page to buy a two-card Server Edition server, and it runs from one 230 V 16 A feed.

About 150 developers: four cards. Three Qwen3.8-27B copies hold 141 agent sessions of 32K with an FP8 cache, and the first card can serve Chat with gpt-oss-120b instead, which GitLab also rates F for chat. The four-card Server Edition server has the parts and power figures.

300 to 500 developers: eight cards, or two servers. Seven agent cards hold 329 sessions of 32K with an FP8 cache, enough for 500 developers if about two thirds work with an agent at the busy hour. From this size, two four-card servers are worth considering instead of one eight-card server, so that Duo keeps running when one server is down; with one card for code and chat in each, their six agent cards hold 282 sessions of 32K with an FP8 cache, and an LLM gateway in front of both spreads the load. An eight-card server needs three-phase power, as our page on eight RTX PRO 6000 Server Edition for LLMs explains.

One model for everything. Mistral Medium 3.5 on all eight cards gives every feature GitLab's full rating, but holds 49 sessions of 32K in 16 bits (98 with an FP8 cache), a fraction of the Qwen layout, and its licence excludes companies above the revenue threshold. We would choose it only when a pilot shows that the smaller models fall short for your code.

Offline and air-gapped installation

GitLab's offline guide covers Premium and Ultimate on GitLab 18.9 or later with an offline cloud licence, for which you need an opt-out exemption from cloud licensing before purchase. The artefacts to move across the air gap are the AI Gateway image (about 0.6 GiB), the Agent Platform flows executor (about 0.5 GiB), the vllm/vllm-openai image at v0.18.1 or later (about 9 GiB) and the model weights, which vLLM reads from the file system; GitLab's example is Mistral Small 24B at about 48 GB. The inference host needs the NVIDIA driver, CUDA and the NVIDIA Container Toolkit. GitLab notes that its health check tests the gateway and licence but not inference, so test Chat and Code Suggestions directly after each upgrade.

The general procedure for an isolated GPU server, from mirrored packages to checksums, is in our guide to an air-gapped LLM server. On each GitLab upgrade, transfer the matching gateway and executor images; the model weights stay as long as you keep the models.

What we supply

We build GPU servers to order with two, four or eight RTX PRO 6000 Server Edition or H200 NVL cards, assembled and tested before delivery, with Linux, the NVIDIA driver, CUDA and a container runtime installed on request, and our engineering partner Vixen.UNO sets up vLLM and the models within our Private AI/ML service. Quantities and lead time are confirmed in writing with the quote, on one EU contract and invoice. GitLab licences and add-ons come from GitLab; we supply the hardware and the model serving behind them.

FAQ