New GPU server setup: from BMC and BIOS to the first model in vLLM, step by step

GPU server setup guide: the order of steps

A new GPU server reaches a working model endpoint in ten steps, and the order matters more than any single setting: the BMC before the BIOS, the BIOS before the operating system, the driver before the Container Toolkit, and a health baseline before the first user. This runbook follows a server with four NVIDIA RTX PRO 6000 Blackwell Server Edition cards, the inference node on our AI servers page, from the rack to a vLLM endpoint and links each step to the guide that covers it in depth.

Our runbook; tools and checks from the NVIDIA documentation cited in each section below, read on 25 October 2026.

Steps 5 to 9 are the same on every server in a fleet, and our guide to automating GPU server setup with Ansible turns them into roles. The checks at delivery itself, serial numbers and a long load run, are in our acceptance testing guide.

Before power-on: feeds and GPU power cables

Four RTX PRO 6000 Server Edition cards at 600 W draw 2.4 kW on their own, and the whole server about 3.4 kW at the wall by our estimate (cards, two EPYC 9355 processors and 15 % for the rest); at a 450 W cap per card it is about 2.7 kW. A 230 V 16 A feed loaded to 80 % carries 2.94 kW, so the server needs a 32 A single-phase or a three-phase feed, one for each side of its redundant power supplies, sized on the server maker's figure. Our guide to 16 A, 32 A and three-phase feeds does the arithmetic for larger servers.

Check one detail inside the chassis before the lid goes back on. The Server Edition reads its power cable's sense pins, and a cable strapped for 450 W sets both the maximum and the default limit to 450 W; nvidia-smi cannot raise the limit above what the cable signals, as our page on Server Edition power at 600, 450 and 300 W explains. Step 6 confirms the result in software.

BMC first: network, accounts, firmware list

The BMC is the server's own management computer, reachable even when the operating system is down, and it is the first thing to configure. Put its port on the management network, replace the factory credentials with named accounts, and confirm that the remote console and virtual media work, because the operating system install, a Secure Boot key enrolment and every BIOS change later run through them.

Then record the firmware: BIOS, BMC and the vBIOS of every card as the BMC reports them, with the server's name and date. Identical cards bought together should report identical versions. Whether an update is due, and in which order, is the subject of our firmware update guide; the BMC's Redfish interface also feeds the out-of-band checks in our BMC and Redfish monitoring guide.

BIOS settings for a multi-GPU server

Four firmware settings decide whether the cards work together, and their names differ between board makers, so look for the function rather than the label.

• Above 4G decoding and Resizable BAR. Each GPU needs a large address window. NVIDIA's Server Edition product brief gives the card a BAR1 of 128 GiB in its default display-off mode, so both settings stay on.
• IOMMU. NCCL's guide states that on bare-metal Linux, CUDA and the driver do not support peer-to-peer transfers with the IOMMU in translated mode. A host that runs only containers can use passthrough mode or turn the IOMMU off; a host that passes cards to virtual machines needs it on.
• ACS. With ACS on, peer traffic goes through the processor's root complex, which NCCL's guide associates with slow transfers or hangs. Virtual machines need ACS, so the same split applies.
• Secure Boot. Decide now. It can stay on with signed modules, and the choice decides the driver route in step 5.

Each of these trades protection for speed, so record the settings in the build documentation and agree them with whoever owns host security. Our guide to P2P, IOMMU and ACS on RTX PRO 6000 servers gives the kernel parameters and the tests.

Ubuntu or RHEL, then the NVIDIA driver

For a server installed in late 2026, Ubuntu 24.04 LTS, RHEL 9 and RHEL 10 appear on every NVIDIA support list, CUDA, the GPU Operator and NVIDIA AI Enterprise included, as our comparison of Linux for GPU servers shows. Install it with current updates before the driver, since the kernel modules are built or chosen for the running kernel.

The driver must use the open kernel modules. NVIDIA's driver README states that "Blackwell and later are only supported by the open kernel modules", and every RTX PRO Blackwell card is a Blackwell GPU. On Ubuntu that means Canonical's -server-open packages, signed for Secure Boot, or NVIDIA's own repository with nvidia-open and a pinning package such as nvidia-driver-pinning-580 installed first. On RHEL 10, Red Hat-signed kmod-nvidia-open packages exist; RHEL 8 and 9 use DKMS streams with a key of your own under Secure Boot. Our guide to the NVIDIA driver under Secure Boot covers each route, and our guide to driver branches explains why a production server runs R580 or R595.

After the reboot, three checks confirm the result. nvidia-smi -L lists four GPUs. /proc/driver/nvidia/version begins "NVRM version: NVIDIA UNIX Open Kernel Module", the line NVIDIA's README gives for the open flavour. And the CUDA version nvidia-smi prints is the newest CUDA the driver supports, which is what container images are later compared with.

We install the operating system, the driver branch, CUDA and a container runtime on request before delivery. Send us the distribution and whether Secure Boot stays on, and the server arrives at step 7 of this list.

Persistence and power limits

Without a client, the driver deinitialises an idle GPU, which NVIDIA says leads to CUDA load times "on the order of seconds" and clears settings such as a software power cap. nvidia-persistenced comes with the driver, but NVIDIA does not promise that it starts by default, so enable it and check after a reboot that it runs.

Then read the power data with nvidia-smi -q -d POWER. Each Server Edition card should report the maximum limit its cable allows, 600 W in the configuration above. If you run the cards capped, for example at 450 W to fit the feeds, set the cap after the persistence daemon is up and again after every driver reload, because a cap set with nvidia-smi does not survive one.

Docker, the Container Toolkit and CDI

The models run in containers, so the next layer is the NVIDIA Container Toolkit. NVIDIA's installation guide, updated 18 September 2026, installs version 1.20.1 from its repository at nvidia.github.io and configures Docker with nvidia-ctk runtime configure with --runtime=docker, followed by a restart of Docker; for containerd and CRI-O the same command writes a drop-in file.

Podman uses CDI, the Container Device Interface, in NVIDIA's words its recommended way to reach the GPUs. Since toolkit 1.18.0 a systemd service, nvidia-cdi-refresh, generates the CDI specification and updates it after driver or toolkit upgrades and at boot. nvidia-ctk cdi list shows the device names, nvidia.com/gpu=0 to nvidia.com/gpu=3 on this server plus nvidia.com/gpu=all. NVIDIA's own first test runs nvidia-smi -L inside a plain Ubuntu image, with --device nvidia.com/gpu=all on Podman or the nvidia runtime on Docker. If all four cards appear there, the runtime is done. How to give each container its own card, by index, UUID or MIG device, is in our guide to GPU assignment for Docker without Kubernetes.

Health baseline with DCGM

DCGM 4.6 is packaged per CUDA major version, as datacenter-gpu-manager-4-cuda13 for the R580 and R595 drivers, and runs as the service nvidia-dcgm, started with systemctl --now enable nvidia-dcgm. dcgmi discovery -l should list all four cards. Save three outputs as the server's baseline: nvidia-smi -q -x, the GPU placement from nvidia-smi topo -m and the JSON of dcgmi diag -r 3 -j.

The diagnostics at level 3 take in the order of 15 to 30 minutes and should pass on every card; our acceptance testing guide explains what each test covers on RTX PRO cards and how to extend the run under sustained power. The same counters, ECC errors and XID messages, are what monitoring watches from now on.

MIG or whole cards

Decide the layout before users arrive, because changing it later means stopping the workloads on that card. Whole cards suit models split across GPUs with tensor parallelism, since NCCL is not supported with MIG. MIG splits an RTX PRO 6000 Server Edition into up to four 1g.24gb, two 2g.48gb or one 4g.96gb instances, each with its own memory, which suits several small models or teams on one server. Our MIG runbook for RTX PRO Blackwell has the commands and the prerequisites, among them a driver from 575.51.03 on.

Two points carry over to the steps around it. MIG instances do not survive a reboot on Hopper and later GPUs, so a start-up service recreates them. And the CDI refresh does not cover MIG changes: restart nvidia-cdi-refresh.service after each one, so containers see the new devices.

The first model in vLLM

vLLM's documentation gives one command for its official image: docker run with --runtime nvidia --gpus all, the Hugging Face cache mounted from the host, port 8000 published, --ipc=host for shared memory, the image vllm/vllm-openai and the model as --model. For a server, pin the image to a release, vllm/vllm-openai:v0.32.0 on 25 October 2026, instead of latest. To keep the first model on one card, replace --gpus all with the CDI name, NVIDIA_VISIBLE_DEVICES=nvidia.com/gpu=0 as an environment variable with the nvidia runtime.

A good first model is gpt-oss-120b. Its Hugging Face card gives 117 billion parameters with 5.1 billion active and the Apache 2.0 licence, and the 15 safetensors files on its Files page add up to about 65 GB, or 60.8 GiB. By our users-per-card rule (0.9 × 95.6 GiB driver-visible memory − 3 GiB overhead − 60.8 GiB weights) one card keeps 22.2 GiB for the KV cache, about 19 conversations of 32K tokens with a 16-bit cache (1.125 GiB each), the figure our four-card server guide uses. The other three cards stay free for a second copy, a tensor-parallel pair or other services.

When the container logs that the server is up, curl localhost:8000/v1/models should list the model. Before anyone else uses the endpoint, two facts from vLLM's security documentation matter: the server listens on all interfaces, and --api-key protects only the paths under /v1, /v2, /inference and /cohere, so /metrics, /health and other routes answer without the key. Put the endpoint behind a gateway or a firewall rule. Our guide to vLLM KV cache size and concurrency helps with the settings after the first test.

After the first model

Two tasks turn a working server into a service. Monitoring comes first: DCGM metrics, XID messages and the BMC's sensors feed the system you already run, as our guide to GPU monitoring with Zabbix and Checkmk shows. Then write down what you did, because the second server should take a fraction of the time: the BIOS settings, the driver branch, the toolkit version and the MIG layout are exactly the variables an Ansible inventory holds.

What we supply

We build AI servers to order: an entry server with two RTX PRO 6000 Server Edition cards, the inference node with four used in this guide, and a training node with eight H200 NVL, each with two AMD EPYC processors, assembled and tested before delivery and delivered across the EU on one contract and invoice. Commissioning on site is available on request. Describe the models and users you plan for, and our team replies with a configuration within one business day.

FAQ