Automate GPU server setup with Ansible: NVIDIA's driver role, Container Toolkit, DCGM, MIG

What Ansible should own on a GPU server

To automate a GPU server with Ansible, use NVIDIA's role nvidia.nvidia_driver for the driver (with your own open-module package list on Blackwell cards), a few documented tasks for the Container Toolkit and DCGM, an nvidia-mig-parted file for MIG layouts, and roll driver updates out in batches with serial. Those are the five layers a GPU server adds to a normal Linux host: the driver, the Container Toolkit with its CDI device list, DCGM for health and diagnostics, a MIG layout on cards that use one, and settings such as persistence mode.

Each layer has a version and an order, and each breaks quietly when one host differs from the others. Written as Ansible roles and an inventory, they give an infrastructure team the same result on the fifth server as on the first, and a record of which branch runs where.

Firmware and BIOS settings are a different matter. Ansible can read the vBIOS version on every card and fail when one differs, but the update itself goes through the server maker's tools and the BMC, as our guide to firmware updates for vBIOS, BMC and BIOS describes. On Kubernetes nodes, the GPU Operator installs the driver and toolkit itself; Ansible then prepares the operating system and stops there.

NVIDIA ansible-role-nvidia-driver README and defaults, Container Toolkit install guide and CDI page (18 September 2026), DCGM 4.6 user guide (17 June 2026) and nvidia-mig-parted README, read on 25 October 2026.

The Ansible role for the NVIDIA driver: what it does and where it runs

NVIDIA publishes an Ansible role for the driver on GitHub under the BSD-3-Clause licence. It is installed with ansible-galaxy install nvidia.nvidia_driver and needs the community.general collection, which has to be installed separately; a source checkout lists its dependencies in requirements.yml. The README's example playbook applies the role to a host group called gpu_nodes.

The role reboots nodes during installation, and the README is direct about the consequence: run ansible-playbook from a machine other than the GPU node. With a local connection the role refuses to run, and over SSH to localhost it can reboot the machine running the playbook. The variable nvidia_driver_skip_reboot turns the reboot off, which suits a maintenance process that reboots on its own schedule.

NVIDIA ansible-role-nvidia-driver README on GitHub, read on 25 October 2026.

Ubuntu 26.04, RHEL 10 and Rocky Linux 10 are missing from that list, although CUDA and the GPU Operator support them, as our comparison of Linux for GPU servers shows. A fleet on those releases either tests the role there first or writes its own driver tasks from NVIDIA's installation guide.

Variables to set per inventory group

The defaults file shows how the role decides what to install. nvidia_driver_branch is "580", the long term support branch that NVIDIA supports to June 2028, as our guide to NVIDIA driver branches and CUDA versions explains. nvidia_driver_package_version is empty, so the newest package of the branch is installed, and nvidia_driver_package_state is present. Persistence mode is on by default (nvidia_driver_persistence_mode_on: yes), and module options go to /etc/modprobe.d/nvidia.conf.

On Ubuntu the role installs from Canonical's repositories unless nvidia_driver_ubuntu_install_from_cuda_repo is set. Its package list combines the branch with the suffix "-server": nvidia-headless, nvidia-utils, nvidia-headless-no-dkms and nvidia-kernel-source, each followed by the branch and suffix. From the CUDA repository it installs cuda-drivers- plus the branch instead. The README adds one trap for pinning: on Ubuntu the branch is part of the package name, so a version pinned from an older branch needs the matching branch variable as well, or the role combines the default name with an incompatible version.

On RHEL and Rocky 8 and later, the install task enables a DNF module stream of the nvidia-driver module, by default the branch followed by "-dkms". It also installs the kernel headers and kernel-devel for the running kernel, DKMS from EPEL, and blacklists nouveau.

Open kernel modules for RTX PRO Blackwell

None of the default package lists names an open kernel module package. For Blackwell this matters: NVIDIA states that the proprietary drivers are unsupported on Blackwell and that the open modules must be used, and every RTX PRO Blackwell card, the RTX PRO 6000 Server Edition included, is a Blackwell GPU. Canonical publishes the open flavour with the suffix "-server-open", for example nvidia-driver-580-server-open and the precompiled modules linux-modules-nvidia-580-server-open-generic, which our guide to the NVIDIA driver under Secure Boot lists from Canonical's archive. On RHEL 8 and 9, NVIDIA's guide names the open DKMS streams open-dkms or 580-open.

For the H200 NVL, L40S and L4 the proprietary flavour still works, but the README of NVIDIA's 595.84 driver recommends the open modules for every GPU that supports them. Our example layout therefore overrides nvidia_driver_ubuntu_packages in group_vars with the open packages for the branch, checked with apt-cache policy on one host first. After the first run, the module flavour is visible on the host: with the open modules, /proc/driver/nvidia/version begins "NVRM version: NVIDIA UNIX Open Kernel Module" and modinfo nvidia reports the licence "Dual MIT/GPL"; the proprietary flavour reports "NVIDIA". Keep one inventory group per card type, so that a MIG layout or a branch test for one type never reaches the others by accident.

Our example layout. Branch support dates from NVIDIA's driver lifecycle table (9 September 2026); package names from Canonical's archive as listed in our Secure Boot guide (24 October 2026); module flavours from the NVIDIA 595.84 driver README, read on 25 October 2026.

Servers we build to order arrive with the operating system, the driver branch and the container runtime installed on request, so the first Ansible run finds a known state. Tell us your distribution and the branch your playbooks pin, and we install that before delivery.

Container Toolkit and CDI as Ansible tasks

NVIDIA's installation guide for the Container Toolkit, last updated 18 September 2026, sets NVIDIA_CONTAINER_TOOLKIT_VERSION to 1.20.1-1 and installs four packages pinned to that version: nvidia-container-toolkit, nvidia-container-toolkit-base, libnvidia-container-tools and libnvidia-container1. The repository comes from nvidia.github.io: a GPG key dearmored into a keyring and a list file for apt, or a .repo file for dnf. Each step maps to one Ansible task, with the version as a variable in group_vars.

Configuring the runtime is a command that writes a file. nvidia-ctk runtime configure with --runtime=docker updates /etc/docker/daemon.json; for containerd and CRI-O it creates the drop-in files 99-nvidia.toml under /etc/containerd/conf.d and /etc/crio/conf.d. In a playbook the command needs a creates argument or a check of the file, and a handler that restarts the engine only when the file changed. The driver must be in place first; NVIDIA's guide lists it as a prerequisite.

CDI, the Container Device Interface, describes the GPUs as named devices for Podman and other engines. Since toolkit 1.18.0 the specification is, in NVIDIA's words, "automatically generated and updated by a systemd service called nvidia-cdi-refresh": a path unit triggers it after a toolkit or driver install or upgrade and at boot. Two events are not covered, driver removal and MIG reconfiguration. A playbook that changes a MIG layout should therefore restart nvidia-cdi-refresh.service afterwards and check the result with nvidia-ctk cdi list, which shows names such as nvidia.com/gpu=0 or nvidia.com/gpu=1:0 for the first MIG device on GPU 1. Our guide to GPU assignment for Docker without Kubernetes shows how containers use those names.

DCGM: install, start, verify

DCGM 4.6, in NVIDIA's user guide last updated 17 June 2026, is packaged per CUDA major version: datacenter-gpu-manager-4-cuda followed by the CUDA major version that nvidia-smi reports for the driver, 13 on R580 and R595 hosts. The guide removes older datacenter-gpu-manager packages first, installs with recommended packages (--install-recommends on apt, weak dependencies on dnf) and starts the service with systemctl --now enable nvidia-dcgm. Its check is dcgmi discovery -l, which lists the GPUs DCGM sees.

In Ansible the CUDA major version is a fact you read once from nvidia-smi on the host, and dcgmi discovery -l becomes a task whose output is registered and compared with the card count expected for that card type. A host with three of four cards visible then fails the play instead of joining production. DCGM also runs the diagnostics our acceptance testing guide uses, and the metrics our Zabbix and Checkmk monitoring guide collects.

MIG layouts as code with nvidia-mig-parted

NVIDIA's MIG Partition Editor, nvidia-mig-parted, is released under Apache 2.0 as deb and rpm packages. Its configuration file is declarative: version: v1, then a section mig-configs whose entries have names you choose, each listing the target devices, whether MIG is enabled and the MIG devices to create as profile and count. For an RTX PRO 6000 Server Edition, an entry with four 1g.24gb devices splits the card into quarters, the profiles our MIG profile guide covers.

Three commands make it fit Ansible. nvidia-mig-parted assert -f with the file and -c with the entry name exits 0 when the layout is applied and 1 when it is not; apply with the same arguments creates it; export prints the current state. A task that runs assert, registers the return code and runs apply only on failure is idempotent by construction. The README notes that the tool does not make sure the node is in a state where MIG mode changes apply cleanly, and lists one known issue: a loaded nvidia_drm module must be unloaded before apply and reloaded after. Stop GPU workloads on the host before the play, and restart nvidia-cdi-refresh.service after it.

MIG instances do not survive a reboot on Hopper and later GPUs. NVIDIA suggests a systemd service around nvidia-mig-parted for start-up, which the README points to in its deployments/systemd directory; our MIG runbook for RTX PRO Blackwell covers the driver and vBIOS prerequisites.

Rolling driver updates without losing the service

A driver update on a production fleet needs batches. Ansible's serial keyword takes a number, a percentage or a list: serial: [1, 5, 10] updates one host first, then five, then ten at a time. With serial, failures count per batch, and max_fail_percentage applies to each batch; Ansible's documentation adds that the percentage must be exceeded, not equalled, before the rest of the play is aborted. Tasks marked run_once run on one host in each batch.

A play for one batch runs in this order:

1. Drain the host: move its containers or cordon it in the scheduler.
2. Read the vBIOS of every card with nvidia-smi --query-gpu=vbios_version with --format=csv and fail if identical cards differ.
3. Apply the driver role with the new package version and let it reboot, or reboot in a separate task.
4. Check nvidia-smi -L against the expected card count and dcgmi discovery -l.
5. Assert the MIG layout, restart nvidia-cdi-refresh.service, and check nvidia-ctk cdi list.
6. Return the host to service, and let the next batch start.

Before step 3, read the release notes and NVIDIA's latest security bulletin, as the checklist in our driver branch guide describes. A software power cap set with nvidia-smi is cleared when the driver reloads, so a play that caps the RTX PRO 6000 Server Edition at 450 W sets the cap again after the reboot.

When the GPU Operator is the better tool

On a Kubernetes cluster, the NVIDIA GPU Operator manages the driver, the Container Toolkit, DCGM and MIG through its own resources, and Ansible installing the same layers on the host would compete with it. There, Ansible prepares the operating system, kernel and container runtime, and the Operator takes over, as our guide to GPU sharing in Kubernetes describes. For servers without internet access, the repositories in these tasks point to a local mirror; our guide to an air-gapped LLM server shows the staging steps.

What we supply

We build AI servers to order for fleets managed with Ansible: an entry server with two RTX PRO 6000 Server Edition cards, an inference node with four, and a training node with eight H200 NVL, each with two AMD EPYC processors, assembled and tested before delivery and delivered across the EU on one contract and invoice. Our new GPU server setup guide gives the manual order of the same steps. Send us your inventory layout and card types, and our team replies with a configuration that fits them.

FAQ