Docker GPU assignment without Kubernetes: --gpus, Compose, CDI and MIG on one server

Assigning GPUs to Docker containers at a glance

To give a Docker container one specific GPU, start it with --gpus device=1 or with the card's UUID, --gpus device=GPU-…, which nvidia-smi -L lists; in Docker Compose, the same choice goes into device_ids under deploy.resources.reservations.devices. With CDI, the Container Device Interface that NVIDIA's Container Toolkit generates, the card is nvidia.com/gpu=1 and a MIG slice nvidia.com/gpu=1:0, passed with --device in Docker or Podman. On a server with four or eight GPUs, pin every service by UUID or CDI name and set CUDA_DEVICE_ORDER=PCI_BUS_ID, so that GPU 1 in your files is GPU 1 on the hardware.

Most GPU servers with two to eight cards run their services this way: vLLM for the language model, an embedding and a reranking service, Open WebUI and a few tools, all in Docker Compose, without a Kubernetes cluster. Kubernetes does the assignment for you through the device plugin or DRA, as our guides to Kubernetes GPU sharing and Kubernetes DRA for NVIDIA GPUs describe. Without it, you set each container's cards yourself, and the mistakes are quiet ones: two services on the same card while another card idles, or a container that sees every GPU.

The commands below come from Docker's documentation and NVIDIA's Container Toolkit documentation (last updated 18 September 2026), read on 25 October 2026. Which department gets which cards on a large server is a separate question, covered in our article on GPU allocation on an eight-card server.

Three ways to select a GPU

Docker has three mechanisms, and all three end with the NVIDIA Container Toolkit injecting the device nodes and driver libraries into the container.

Docker documentation "GPU access" (19 March 2026) and dockerd reference; NVIDIA Container Toolkit pages "Specialized configurations with Docker" and "Support for Container Device Interface", updated 18 September 2026; read 25 October 2026.

The --gpus option is the one most guides show. Docker's page gives --gpus all, --gpus device=0, a UUID in the form device=GPU-3a23c669-…, and for several cards --gpus '"device=0,2"', which "must be quoted because it contains a comma". It also takes capabilities, as in --gpus 'all,capabilities=utility'.

CDI is the newer path. NVIDIA calls CDI an open specification that abstracts what access to a device means; in practice it is a file, read from /etc/cdi or /var/run/cdi, that lists per device the device nodes, driver libraries and hooks the runtime injects. Docker's dockerd reference states that CDI "is enabled by default since Docker Engine 28.3.0" for Linux containers, and that docker info shows the configured spec directories. On older engines it was an experimental feature switched on with features.cdi in daemon.json.

The environment variable is the oldest mechanism. With --runtime=nvidia, NVIDIA_VISIBLE_DEVICES takes a list of indices or UUIDs, all, none (no GPU, but driver capabilities) or void (behaves like runc). NVIDIA notes that "all" is the default value in its base CUDA images, which matters for the pitfalls further down.

Index, UUID and the device order inside the container (CUDA_VISIBLE_DEVICES)

A GPU index is a position in an enumeration; a UUID belongs to the card. nvidia-smi -L lists both, and NVIDIA's example for Docker queries the UUID of card 3 with nvidia-smi -i 3 plus --query-gpu=uuid --format=csv before starting a container with it. For files that should mean the same thing next year, UUIDs are the safer choice. The trade-off is replacement: a new card has a new UUID, so the Compose file changes with the hardware.

Inside the container, the numbering starts again. CUDA's environment-variable reference states that cudaGetDeviceCount() "includes only the visible devices", so a container given host GPU 3 sees it as device 0. A vLLM container started with two host cards sees devices 0 and 1, whatever their host numbers are, and its --tensor-parallel-size 2 uses both.

The order itself has a catch. CUDA_DEVICE_ORDER defaults to FASTEST_FIRST, which enumerates "from fastest to slowest using a simple heuristic"; PCI_BUS_ID enumerates "by PCI bus ID in ascending order". The difference shows most in a workstation or server with mixed cards, where CUDA's device 0 can be another card than nvidia-smi's GPU 0. Set CUDA_DEVICE_ORDER=PCI_BUS_ID in the environment of every GPU service; NVIDIA's Isaac Lab documentation gives the same setting as the fix when CUDA and the renderer count GPUs differently, as our robotics server guide notes.

CUDA_VISIBLE_DEVICES inside a container then works on the container's own numbering. Its accepted values, per the CUDA Programming Guide 13.4.2 of 10 September 2026, are indices, UUIDs as nvidia-smi -L prints them, unique UUID prefixes, and MIG devices as MIG-<GPU-UUID>/<GPU instance ID>/<compute instance ID>, with the warning that "Only single MIG instance enumeration is supported." An invalid index hides every device after it.

Our AI servers come with the operating system, NVIDIA driver, CUDA and a container runtime installed on request. Send us the services you plan to run and how many cards each needs, and our team replies with a configuration and a card layout to match.

Docker Compose on a four-GPU server

Docker's Compose page documents GPU access as a device reservation. A service lists, under deploy.resources.reservations.devices, one entry with driver: nvidia, capabilities: [gpu] and either count or device_ids. The rules are strict: capabilities must be set, count and device_ids "are mutually exclusive", an invalid device ID or a count above the number of GPUs is an error, and if count is all or not set, or device_ids is not set, the service gets all GPUs of the host.

That last rule is the common mistake. A reranker defined with driver: nvidia and capabilities: [gpu] but no device_ids lands on every card, beside the vLLM process that expects to own them. For a four-card RTX PRO 6000 Server Edition server we would lay out a typical stack like this:

Compose syntax from Docker's "GPU support in Docker Compose" page, read 25 October 2026; the layout is our example.

Compose also offers a shorter gpus: attribute on a service, which its reference calls "equivalent to a device request with an implicit gpu capability", with gpus: all or a list of entries with driver and count. For CDI, the devices: attribute accepts a CDI name directly, so a service can list devices: ["nvidia.com/gpu=2"]. The reference shows this form with a placeholder vendor; check it with your Compose version before rolling it out.

How the tensor-parallel pair performs on PCIe cards without NVLink is the subject of our article on RTX PRO 6000 without NVLink; which models fit on one or two cards is covered by our page on the four-card RTX PRO 6000 Server Edition server.

NVIDIA Container Toolkit and CDI in Docker

NVIDIA's Container Toolkit generates the CDI spec for the GPUs it finds. The manual command is sudo nvidia-ctk cdi generate with --output=/var/run/cdi/nvidia.yaml, and nvidia-ctk cdi list shows the device names the spec contains; nvidia-ctk --debug cdi list explains errors when a spec does not load. NVIDIA documents the names nvidia.com/gpu=all, nvidia.com/gpu=0 for the first GPU and nvidia.com/gpu=1:0 for the first MIG device on GPU 1. Read the list on your own server before writing names into files.

Toolkit 1.18.0 changed two defaults. The NVIDIA Container Runtime now uses a just-in-time CDI spec instead of its legacy mode, and a systemd service, nvidia-cdi-refresh, regenerates /var/run/cdi/nvidia.yaml when the toolkit is installed or upgraded, when the driver is installed or upgraded, and at reboot. NVIDIA lists two things it does not handle: removal of the driver and reconfiguration of MIG devices. After either, regenerate the spec, preferably by restarting the unit nvidia-cdi-refresh.service with systemctl.

Later releases fixed the refresh service. In 1.20.1, the newest in NVIDIA's release notes, it creates the NVIDIA control device nodes before generating the spec, and the service is now ordered before Docker, containerd and CRI-O, so the spec is in place when those engines start their containers at boot. NVIDIA's install guide pins 1.20.1 in its commands. The Toolkit's security history matters too: our guide to securing a GPU server lists the CVEs fixed in earlier releases.

For Docker, nvidia-ctk runtime configure with --runtime=docker, run with sudo and followed by a restart of Docker, writes the runtime into daemon.json. In rootless mode NVIDIA points --config at $HOME/.config/docker/daemon.json, adds a user-level restart and the setting nvidia-container-cli.no-cgroups.

Podman and rootless containers

For Podman, NVIDIA recommends CDI. Podman supports CDI devices in --device from release 4.1.0, and NVIDIA's example runs podman run --rm with --device nvidia.com/gpu=all and --security-opt=label=disable on an Ubuntu image with nvidia-smi -L. NVIDIA uses the label option in all of its Podman examples without further comment, so keep it as shown. For one card, replace all with the index or the MIG name from nvidia-ctk cdi list.

NVIDIA's install guide gives Podman no runtime configuration step, only the recommendation to use CDI. That makes Podman the simpler choice for CI jobs on a GPU server, where a runner starts short-lived containers; our article on GPU CI runners for GitLab and GitHub Actions covers that case.

MIG slices for containers

Once a MIG slice exists, a container gets it like a whole card. On an RTX PRO 6000 a card splits into up to four 1g.24gb instances, on an H200 NVL into up to seven 1g.18gb, as our MIG step-by-step guide shows. With CDI the slice is nvidia.com/gpu=<card>:<slice>; with the nvidia runtime, NVIDIA_VISIBLE_DEVICES takes the MIG UUID or the index form, such as 0:0.

The order of operations decides whether it works after a reboot. MIG mode on Hopper and later GPUs lasts only as long as the driver is loaded, so the layout has to be recreated at start-up, for example with nvidia-mig-parted in a systemd unit. The CDI spec must then be regenerated, because nvidia-cdi-refresh does not react to MIG changes. Only after both steps should the containers start; a Compose stack that starts earlier does not find the slices it names.

MIG suits the small services in a stack: an embedding model, a reranker, a speech model. NCCL is not supported with MIG, so a model split across several GPUs needs whole cards. Toolkit 1.20.1 also injects the MIG management devices when NVIDIA_MIG_CONFIG_DEVICES or NVIDIA_MIG_MONITOR_DEVICES is all in a container with CAP_SYS_ADMIN; NVIDIA notes that this cannot be combined with per-instance visibility in the same container.

Pitfalls on shared GPU servers

Four mistakes are easy to make on a server that several services share.

1. A service without device_ids. In Compose it receives all GPUs. Give every GPU service an explicit list.
2. The nvidia runtime as default. If daemon.json makes nvidia Docker's default runtime, a container from a CUDA base image starts with NVIDIA_VISIBLE_DEVICES=all from the image, by our reading of NVIDIA's table, and sees every card even without --gpus. Keep runc as the default and request GPUs explicitly.
3. Driver capabilities. Without NVIDIA_DRIVER_CAPABILITIES the default is utility,compute; a container that encodes or decodes video needs video as well, and the variable "replaces the default capabilities rather than adding to them", so list compute again.
4. A stale CDI spec. After a driver upgrade on Toolkit 1.18 or later the refresh service handles it; after a MIG change or a driver removal it does not.

Toolkit 1.18.2 also fixed a bug in which NVIDIA_VISIBLE_DEVICES=none exposed all GPUs, one more reason to run a current release. When a card has to be identified after an error, the PCI bus ID links the kernel log to the UUID in your Compose file, as our guide to identifying a faulty GPU explains.

What we supply

We build AI servers to order for container stacks like this: an entry server with two RTX PRO 6000 Server Edition cards, an inference node with four and a training node with eight H200 NVL, each with two AMD EPYC processors, assembled and tested before delivery and delivered across the EU on one contract and invoice. On request they arrive with the operating system, driver, CUDA and container runtime installed, and with a MIG layout if you plan one. Describe your container stack, with the models per service, and our team replies within one business day.

FAQ