A private LLM platform on Kubernetes: the stack layer by layer, and the GPUs it runs on

The stack, layer by layer

A private LLM platform on Kubernetes is a stack of layers, each with one job. The table lists them with the versions current on 23 October 2026.

Release notes and release histories of NVIDIA, vLLM, KServe, the Gateway API Inference Extension, llm-d and Kubeflow, read 24 September 2026 and again on 23 October 2026.

GPU nodes and the GPU Operator

The GPU Operator installs what a GPU node needs, each part as a container. Release 26.7.1 ships device plugin and GPU Feature Discovery v0.20.1, DCGM Exporter v4.6.1-4.8.4 and MIG Manager v0.15.1, and 26.7.2, the newest in NVIDIA’s release notes on 23 October 2026, only updates the DRA driver to v0.5.1; the Operator is open source under Apache 2.0, and its platform page lists Kubernetes 1.33 to 1.37 on Ubuntu 22.04, 24.04 and 26.04 LTS, Red Hat Enterprise Linux and Rocky Linux. Since 1.34 reached end of life on 27 October 2026, upstream Kubernetes maintains only 1.35, 1.36 and 1.37 (as of November 2026), so new clusters start at 1.35 or later. The operator’s support list names the RTX PRO 6000 and RTX PRO 4500 Server Editions, the H200 NVL, the L40S and the L4. We found no entry for the RTX PRO 4000 or 5000, or for the RTX PRO 6000 Workstation and Max-Q editions.

GPU Feature Discovery labels each node with its GPU model, so the language model’s Deployment can select RTX PRO 6000 or H200 NVL nodes through nvidia.com/gpu.product. A replica that spans several GPUs needs whole cards, because NVIDIA’s MIG user guide (updated 4 November 2026) describes the MIG support in NCCL 2.31 and later only as experimental (one rank per MIG instance, CUDA driver 13.0 or later, NCCL_MNNVL_ENABLE=0). Cards for small models can be split: MIG Manager takes a layout per device, so one card of a node can run MIG while the others stay whole, and the mixed strategy advertises each profile as its own resource, such as nvidia.com/mig-1g.24gb. Time-slicing has no memory or fault isolation and suits development, not a production endpoint. Our guide to GPU sharing covers the details.

Serving: vLLM, KServe or NIM

A vLLM server serves one model; vLLM’s FAQ says that serving several from one server “is not currently supported”, so each model becomes its own Deployment. vLLM’s Kubernetes page shows the plain form: the vllm/vllm-openai image, weights on a PersistentVolumeClaim, /health probes on port 8000 and a memory-backed /dev/shm, because vLLM “needs to access the host’s shared memory for tensor parallel inference”. It lists other ways to deploy vLLM, KServe and llm-d among them; our engine comparison covers the engines.

KServe 0.21.0, released on 25 September 2026, wraps this in an InferenceService. With the model format huggingface, its runtime uses vLLM by default and “automatically falls back to the standard Hugging Face backend” for models vLLM does not support. A storageUri such as hf:// or pvc:// names the weights, and the OpenAI routes sit under /openai/v1/. For advanced LLM features, KServe’s administrator guide points to LLMInferenceService, still at an alpha API version, which adds a Gateway API route, scheduling on prefix cache hits and load, prefill and decode disaggregation and multi-node serving; for standard LLM serving the guide still starts with InferenceService. Release 0.21.0 adds DRA resource claims in ServingRuntimes and KEDA autoscaling with scale-to-zero for LLM services, and moves the LLMInferenceService samples to API version v1alpha2. Kubeflow adds pipelines, training, notebooks and a model registry; its Community Distribution 26.03.1, the patch release of 15 June 2026, bundles KServe 0.18.0, three minor releases behind 0.21.0.

NVIDIA NIM is the NVIDIA AI Enterprise option. NIM for LLMs 2.0.13, which NVIDIA now calls NIM for Large Language Models and Vision Language Models, “is built on vLLM”: the earlier multi-backend container “is replaced by a dedicated vLLM container”. Its release notes for 2.0.13, updated on 19 October 2026, and its support matrix add model-specific containers, for GLM-5.2 among others, that use SGLang as the inference backend. The NIM Operator, at 3.1.2 of 12 August 2026, requires GPU Operator 24.3.0 or later; among its resources, NIMCache downloads models from NVIDIA NGC or Hugging Face to network storage, and NIMService creates the Deployment, the Service and, when enabled, an autoscaler. NVIDIA’s terms are free access through its Developer Program for research, development and testing on up to 16 GPUs, and an AI Enterprise licence per GPU in production. NVIDIA’s NIM offerings page, updated on 15 October 2026, lists the NIM Certified Feature Branch at no charge and requires the subscription for the Production Branch, so have the quote state which offering a production server rests on. The H200 NVL includes five years of the licence; we found no included subscription for the RTX PRO 6000 Server Edition or the L4. Our NVIDIA AI Enterprise page covers the licence.

We supply NVIDIA AI Enterprise licences on the same invoice as the GPUs and AI servers we build to order. Tell us how many GPUs will run NIM in production, and the licences go into the quote with the hardware.

Routing, access and security

Routing. The Gateway API Inference Extension is a Kubernetes project; its release v1.6.2 came out on 17 September 2026. It defines the InferencePool, stable at inference.networking.k8s.io/v1 since v1.0.0: a set of model server pods behind an endpoint picker. The picker reads vllm:num_requests_waiting, vllm:num_requests_running, vllm:kv_cache_usage_perc and the LoRA adapters in use on each replica and sends every request to the most suitable one; a plain Service ignores those signals. Since v1.6.0 of August 2026 the project itself ships only a lightweight endpoint picker “designed for conformance testing”: its full endpoint picker moved to the llm-d project, and for production the documentation points to llm-d-router or an endpoint picker of your own. llm-d reached its first stable release, 1.0.0, on 20 October 2026. KServe’s LLMInferenceService deploys such a scheduler itself. The gateway must implement the extension; the project lists Istio, Agentgateway and NGINX Gateway Fabric among the implementations.

Access. We found nothing on access control in the extension’s documentation. Keys per team, rate limits and the log of queries and answers come from the gateway’s policies or an OpenAI-compatible proxy, and that has to be the only way in. vLLM’s --api-key protects only paths under /v1, /v2, /inference and /cohere. Its security page, read on 23 October 2026, lists the routes the key does not cover: inference routes such as /invocations, /pooling, /classify, /score and /rerank, and control routes such as /pause, /resume and /update_weights, which “can alter model behavior”; /abort_requests is open only on servers started with --tokens-only. The page says not to rely on the key alone. Logging of queries and answers and protection against prompt injection are part of our private AI/ML service, which we deliver under one contract; the first call is free of charge. Write to us about the teams and models the gateway must serve.

Isolation. Pods accept all inbound connections until a network policy selects them, and the Kubernetes documentation warns: “Creating a NetworkPolicy resource without a controller that implements it will have no effect.” Run a network plugin that enforces policies and let only the gateway, the endpoint picker and Prometheus reach the serving port, since vLLM serves /metrics on the same port as its API. vLLM calls traffic between the nodes of a multi-node deployment “insecure by default” and says it must run on an isolated network. Kubernetes stores Secrets, such as the Hugging Face or NGC key, unencrypted in etcd by default, so enable encryption at rest. Pin images by digest: vLLM’s example uses the latest tag, which Kubernetes advises against in production, and --trust-remote-code, which trusts “remote code (e.g., from HuggingFace)”; leave it off unless the model needs it, and run vLLM 0.31.0 or later, preferably 0.32.0, the current release as of 23 October 2026; 0.31.0 fixed a high-severity advisory (GHSA-h3rc-6mm3-gc2m) for servers started with that flag.

Model storage: how weights reach a pod

KServe 0.20 documentation, NVIDIA NIM Operator documentation and the Kubernetes documentation on volume access modes, read 24 September 2026.

The network sets the floor for every download. Llama 3.3 70B in FP8 is 68 GiB, or 584 Gbit: at least 23 seconds at a 25 Gbit/s line rate and 5.8 seconds at 100 Gbit/s by our arithmetic, before anything loads into GPU memory. Each replica that starts on another node has to download the weights again, so local copies matter once autoscaling moves replicas.

RAG services beside the model

Retrieval adds an embedding model, a reranker, a vector database and ingestion jobs. vLLM serves embedding and reranking models with its pooling runner (--runner pooling) at /v1/embeddings and /v1/rerank, KServe’s Hugging Face runtime serves them under /openai/v1/, and NVIDIA’s NeMo Retriever embedding NIMs “expose an API compatible with OpenAI’s API standard”, with a reranking NIM beside them. vLLM’s /rerank and /score routes without the /v1 prefix are not covered by its API key.

By our arithmetic from Hugging Face model data, Qwen3-Embedding-8B (7.57 billion parameters) has 15.1 GB of weights in BF16, Qwen3-Reranker-8B 16.4 GB and Qwen3-Embedding-0.6B 1.2 GB. The vector index is sized in RAM. As 32-bit floats, a million chunks at 4,096 dimensions take 16.4 GB of raw vectors before any index structure, and 4.1 GB at 1,024 dimensions. Ingestion must carry each document’s access rights into the index so that retrieval filters by user, the first problem our RAG guide tackles; Kubeflow Pipelines can run it as a graph of containers.

Metrics and autoscaling

vLLM exports Prometheus metrics at /metrics, among them vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds, vllm:num_requests_running, vllm:num_requests_waiting and vllm:kv_cache_usage_perc, where 1 means a full cache. DCGM Exporter adds GPU metrics on port 9400, per card and per MIG instance. Our monitoring guide explains why GPU utilisation misleads, which also rules it out for autoscaling.

KEDA’s Prometheus scaler takes a query, a threshold and an activationThreshold; KEDA creates a Horizontal Pod Autoscaler, scales from zero to one itself and leaves one to N to the autoscaler. With KEDA’s default metric type, AverageValue, the query result is divided by the number of replicas before the comparison, so a query summed over a model’s replicas asks for one replica per threshold’s worth of requests. The waiting queue alone is a poor target, because it drops to zero once capacity suffices, and the autoscaler then shrinks the Deployment until the queue returns. Requests in flight, running plus waiting, are steadier, as long as the threshold stays below the number of requests one replica can run at once; with a higher threshold, requests queue before a replica is added. KServe’s own example scales on vllm:num_requests_running. Without KEDA, an HPA needs a custom or external metrics adapter; either way, its default downscale stabilisation window is 300 seconds.

On-premise, an autoscaler adds pods to the GPUs that already exist. A new replica starts only where a card of the right type is free and serves only after its weights arrive and load; after scaling to zero, the next request waits for all of that.

Which hardware carries which layer

KV cache left = 0.9 × driver-visible memory (95.6 GiB on the RTX PRO 6000, 140.4 GiB on the H200 NVL), less about 3 GiB of overhead, less the weights, the budget of our users-per-card guide (vLLM itself defaults to 0.92, unchanged in 0.32.0); other sizes by our arithmetic from Hugging Face model data; MIG profiles from NVIDIA’s MIG user guide.

The reference configurations on our AI servers page map onto this. The Private AI starter has two RTX PRO 6000 Server Edition cards. One can hold a 70B-class model in FP8 with 12 sessions of 8,192 tokens at 1.25 GiB each in an FP8 KV cache, by the rule in the note; the other can be split into four 1g.24gb instances for the embedder, the reranker and up to two more small models. The Inference node’s four cards serve several models, or one larger model across cards. Without NVLink, vLLM suggests pipeline parallelism for higher throughput, while tensor parallelism shares out each token’s weight reads and so speeds up a single user; above two PCIe cards vLLM leaves the all-reduce to NCCL, so for a model that fits two cards we would test two TP=2 instances against one TP=4 copy on your model. Our article on one model on several GPUs compares the layouts. On the Training node’s H200 NVL, a 1g.18gb MIG instance (16.5 GB on NVIDIA’s product page) fits the 0.6B embedder but not the 8B one, whose 15.1 GB of weights would leave almost nothing for vLLM’s working memory and cache. While that card runs MIG, plan NCCL training on the other cards, since NCCL supports MIG only experimentally, from release 2.31. The L4, at 72 W and low profile, adds a small-model GPU without MIG; the RTX PRO 4000 also has 24 GB, but we found no entry for it in the GPU Operator’s support list.

What we supply

We supply the GPUs and servers for each layer, with manufacturer warranty on one EU contract and invoice: the RTX PRO 6000 Server Edition or the H200 NVL for the language model, the L4 for small models, and AI servers built to order from the reference configurations above. On request the servers arrive with the operating system, drivers, CUDA and a container runtime installed, and NVIDIA AI Enterprise licences for NIM come on the same invoice. The platform itself is our private AI/ML service, built by our engineering partner Vixen.UNO under one contract with us: a Kubernetes-based platform with KServe and Kubeflow, private models with vLLM, Ollama or NVIDIA AI Enterprise, RAG assistants that respect each user’s access rights, and logging of queries and answers. It runs on your servers or in the Tier III data centres of Vixen.UNO’s data-centre partner Baltneta in Lithuania.

FAQ