A private LLM platform on Kubernetes: the stack layer by layer, and the hardware under it

The stack, layer by layer

A private LLM platform on Kubernetes is a stack of layers, each with one job. The table lists them with the versions current in September 2026.

Release notes and release histories of NVIDIA, vLLM, KServe, the Gateway API Inference Extension, llm-d and Kubeflow, read 24 September 2026.

GPU nodes and the GPU Operator

The GPU Operator installs what a GPU node needs, each part as a container. Release 26.7.1, the newest in NVIDIA’s release notes on 23 September 2026, ships device plugin and GPU Feature Discovery v0.20.1, DCGM Exporter v4.6.1-4.8.4 and MIG Manager v0.15.1; the Operator is open source under Apache 2.0, and its platform page lists Kubernetes 1.33 to 1.37 on Ubuntu 24.04. Its support list names the RTX PRO 6000 and RTX PRO 4500 Server Editions, the H200 NVL, the L40S and the L4. We found no entry for the RTX PRO 4000 or 5000, or for the RTX PRO 6000 Workstation and Max-Q editions.

GPU Feature Discovery labels each node with its GPU model, so the language model’s Deployment can select RTX PRO 6000 or H200 NVL nodes through nvidia.com/gpu.product. A replica that spans several GPUs needs whole cards, because NCCL is not supported with MIG. Cards for small models can be split: MIG Manager takes a layout per device, so one card of a node can run MIG while the others stay whole, and the mixed strategy advertises each profile as its own resource, such as nvidia.com/mig-1g.24gb. Time-slicing has no memory or fault isolation and suits development, not a production endpoint. Our guide to GPU sharing covers the details.

Serving: vLLM, KServe or NIM

A vLLM server “hosts one model at a time”, so each model becomes its own Deployment. vLLM’s Kubernetes page shows the plain form: the vllm/vllm-openai image, weights on a PersistentVolumeClaim, /health probes on port 8000 and a memory-backed /dev/shm, because vLLM “needs to access the host’s shared memory for tensor parallel inference”. It lists 13 other ways to deploy vLLM, KServe and llm-d among them; our engine comparison covers the engines.

KServe 0.20.0, released on 6 August 2026, wraps this in an InferenceService. With the model format huggingface, its runtime uses vLLM by default and “automatically falls back to the standard Hugging Face backend” for models vLLM does not support. A storageUri such as hf:// or pvc:// names the weights, and the OpenAI routes sit under /openai/v1/. For LLMs, KServe’s installation guide now points to LLMInferenceService, still at an alpha API version, which adds a Gateway API route, scheduling on prefix cache hits and load, prefill and decode disaggregation and multi-node serving. Kubeflow adds pipelines, training, notebooks and a model registry; its Community Distribution 26.03.1, the patch release of 15 June 2026, bundles KServe 0.18.0, two minor releases behind 0.20.0.

NVIDIA NIM is the NVIDIA AI Enterprise option, and NIM for LLMs 2.0.12, which NVIDIA now calls NIM for Large Language Models and Vision Language Models, “is built on vLLM”: the earlier multi-backend container “is replaced by a dedicated vLLM container”. The NIM Operator, at 3.1.2 of 12 August 2026, requires GPU Operator 24.3.0 or later; among its resources, NIMCache downloads models from NVIDIA NGC or Hugging Face to network storage, and NIMService creates the Deployment, the Service and, when enabled, an autoscaler. NVIDIA’s terms are free access through its Developer Program for research, development and testing on up to 16 GPUs, and an AI Enterprise licence per GPU in production; the H200 NVL includes five years of it, the RTX PRO 6000 Server Edition and the L4 none. Our NVIDIA AI Enterprise page covers the licence.

Routing, access and security

Routing. The Gateway API Inference Extension, a Kubernetes project whose release v1.6.2 came out on 17 September 2026, defines the InferencePool, stable at inference.networking.k8s.io/v1 since v1.0.0: a set of model server pods behind an endpoint picker that reads vllm:num_requests_waiting, vllm:num_requests_running, vllm:kv_cache_usage_perc and the LoRA adapters in use on each replica and sends every request to the best placed one; a plain Service ignores those signals. Since v1.6.0 of August 2026 the project itself ships only a lightweight endpoint picker “designed for conformance testing”: its full endpoint picker moved to the llm-d project, and for production the documentation points to llm-d-router or an endpoint picker of your own. KServe’s LLMInferenceService deploys such a scheduler itself. The gateway must implement the extension; the project lists Istio, Agentgateway and NGINX Gateway Fabric among the implementations.

Access. We found nothing on access control in the extension’s documentation. Keys per team, rate limits and the log of queries and answers come from the gateway’s policies or an OpenAI-compatible proxy, and that has to be the only way in. vLLM’s --api-key protects only paths under /v1, /v2, /inference and /cohere; /pause and /abort_requests answer without it, and vLLM’s security page says not to rely on the key alone. Logging of queries and answers and protection against prompt injection are part of the AI/ML Integration service, delivered by the team of our engineering partner Vixen.UNO.

Isolation. Pods accept all inbound connections until a network policy selects them, and the Kubernetes documentation warns: “Creating a NetworkPolicy resource without a controller that implements it will have no effect.” Run a network plugin that enforces policies and let only the gateway, the endpoint picker and Prometheus reach the serving port, since vLLM serves /metrics on the same port as its API. vLLM calls traffic between the nodes of a multi-node deployment “insecure by default” and says it must run on an isolated network. Kubernetes stores Secrets, such as the Hugging Face or NGC key, unencrypted in etcd by default, so enable encryption at rest. Pin images by digest: vLLM’s example uses the latest tag, which Kubernetes advises against in production, and --trust-remote-code, which trusts “remote code (e.g., from HuggingFace)”; leave it off unless the model needs it.

Model storage: how weights reach a pod

KServe 0.20 documentation, NVIDIA NIM Operator documentation and the Kubernetes documentation on volume access modes, read 24 September 2026.

The network sets the floor for every download. Llama 3.3 70B in FP8 is 68 GiB, or 584 Gbit: at least 23 seconds at a 25 Gbit/s line rate and 5.8 seconds at 100 Gbit/s by our arithmetic, before anything loads into GPU memory. Each replica that starts on another node pays it again, so local copies matter once autoscaling moves replicas.

RAG services beside the model

Retrieval adds an embedding model, a reranker, a vector database and ingestion jobs. vLLM serves embedding and reranking models with its pooling runner (--runner pooling) at /v1/embeddings and /v1/rerank, KServe’s Hugging Face runtime serves them under /openai/v1/, and NVIDIA’s NeMo Retriever embedding NIMs “expose an API compatible with OpenAI’s API standard”, with a reranking NIM beside them. vLLM’s /rerank and /score routes without the /v1 prefix are not covered by its API key.

By our arithmetic from Hugging Face model data, Qwen3-Embedding-8B (7.57 billion parameters) has 15.1 GB of weights in BF16, Qwen3-Reranker-8B 16.4 GB and Qwen3-Embedding-0.6B 1.2 GB. The vector index is sized in RAM: as 32-bit floats, a million chunks at 4,096 dimensions hold 16.4 GB of raw vectors before any index structure, and 4.1 GB at 1,024. Ingestion must carry each document’s access rights into the index so that retrieval filters by user, the first problem our RAG guide tackles; Kubeflow Pipelines can run it as a graph of containers.

Metrics and autoscaling

vLLM exports Prometheus metrics at /metrics, among them vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds, vllm:num_requests_running, vllm:num_requests_waiting and vllm:kv_cache_usage_perc, where 1 means a full cache. DCGM Exporter adds GPU metrics on port 9400, per card and per MIG instance. Our monitoring guide explains why GPU utilisation misleads, which also rules it out for autoscaling.

KEDA’s Prometheus scaler takes a query, a threshold and an activationThreshold; KEDA creates a Horizontal Pod Autoscaler, scales from zero to one itself and leaves one to N to the autoscaler. With KEDA’s default metric type, AverageValue, the query result is divided by the number of replicas before the comparison, so a query summed over a model’s replicas asks for one replica per threshold’s worth of requests. The waiting queue alone is a poor target: it drops to zero once capacity suffices, and the autoscaler then shrinks the Deployment until the queue returns. Requests in flight, running plus waiting, are steadier, as long as the threshold stays below the number of requests one replica can run at once; with a higher threshold, requests queue before a replica is added. KServe’s own example scales on vllm:num_requests_running. Without KEDA, an HPA needs a custom or external metrics adapter; either way, its default downscale stabilisation window is 300 seconds.

On premises an autoscaler adds pods, not GPUs: a new replica starts only where a card of the right type is free and serves only after its weights arrive and load, and after scaling to zero the next request waits for all of that.

Which hardware carries which layer

KV cache left = 0.9 × driver-visible memory (95.6 GiB on the RTX PRO 6000, 140.4 GiB on the H200 NVL), less about 3 GiB of overhead, less the weights, the budget of our users-per-card guide (vLLM 0.30.0 itself defaults to 0.92); other sizes by our arithmetic from Hugging Face model data; MIG profiles from NVIDIA’s MIG user guide.

The reference configurations on our AI servers page map onto this. The Private AI starter’s two RTX PRO 6000 Server Edition cards can hold a 70B-class model in FP8 on one card, 12 sessions of 8,192 tokens at 1.25 GiB each in an FP8 KV cache by the rule in the note, and split the other into four 1g.24gb instances for the embedder, the reranker and up to two more small models. The Inference node’s four cards serve several models, or one larger model across cards, where vLLM advises pipeline rather than tensor parallelism without NVLink. On the Training node’s H200 NVL, a 1g.18gb MIG instance, 16.5 GB on NVIDIA’s product page, fits the 0.6B embedder but not the 8B one, whose 15.1 GB of weights would leave almost nothing for vLLM’s working memory and cache; and while that card runs MIG it cannot join NCCL training. The L4, at 72 W and low profile, adds a small-model GPU without MIG; the RTX PRO 4000 also has 24 GB, but we found no entry for it in the GPU Operator’s support list.

What we supply

Eurokommerz supplies the GPUs and servers under these layers with manufacturer warranty, on one EU contract and invoice: RTX PRO 6000 Server Edition and H200 NVL for the language model, L4 for small models, and AI servers built to order from the reference configurations, with operating system, drivers, CUDA and a container runtime installed on request; NVIDIA AI Enterprise licences for NIM come on the same invoice. The platform on top is AI/ML Integration, delivered by the team of our engineering partner Vixen.UNO: a Kubernetes-based platform with KServe and Kubeflow, private models with vLLM, Ollama or NVIDIA AI Enterprise, RAG assistants that respect each user’s access rights, and logging of queries and answers. It runs on your servers or in the Tier-3 data centres of Vixen.UNO’s data-centre partner Baltneta in Lithuania, and the contract is with Eurokommerz.

FAQ