Find the faulty GPU in a multi-GPU server: index, PCI bus ID, physical slot, serial

From an alert on "GPU 3" to the card in the chassis

To find a faulty GPU in a server with four or eight cards, start from the PCI bus ID, not from the GPU number. Take the bus ID from the XID line or the monitoring alert, look up the serial number and UUID for that bus ID with nvidia-smi, find the physical slot for it with lspci or the BMC's Redfish interface, and confirm the slot in the server maker's slot map. Compare the serial on the card's label with the one nvidia-smi reports before you pull it.

The GPU number is the weak link because two numbering schemes exist on the same host. nvidia-smi numbers the GPUs in "the natural enumeration returned by the driver", whose order, NVIDIA warns, can differ from one boot to the next. CUDA numbers them by its own rule. A dashboard, a job log and an nvidia-smi listing can therefore mean three different cards by "GPU 3". The identifiers below do not have that problem.

NVIDIA nvidia-smi documentation and CUDA Programming Guide 13.4.2 (environment variables, updated 10 September 2026), read 24 October 2026.

The bus ID follows the slot, the UUID and the serial number follow the card. When a card moves to another slot, its bus ID changes and its serial stays, which is exactly what the slot swap test in our article on XID 79 relies on.

Why "GPU 0" in CUDA can differ from nvidia-smi

NVIDIA's CUDA Programming Guide defines CUDA_DEVICE_ORDER with two values. FASTEST_FIRST, the default, enumerates the devices "from fastest to slowest using a simple heuristic". PCI_BUS_ID enumerates them "by PCI bus ID in ascending order". On a server with identical cards the heuristic has nothing to rank them by, and the guide does not say how it breaks ties, so do not assume that CUDA device 3 and nvidia-smi index 3 are the same card.

Three settings remove the doubt:

• set CUDA_DEVICE_ORDER=PCI_BUS_ID in the environment of every job and service, so CUDA counts in bus order;
• pass UUIDs instead of numbers in CUDA_VISIBLE_DEVICES, which the guide allows in the format nvidia-smi -L prints;
• log the UUID or bus ID of each GPU at job start, so an error in a job log can be traced to a card later.

On a GPU split with MIG, CUDA addresses an instance as MIG-<GPU-UUID>/<gi>/<ci>, the GPU UUID followed by the GPU instance ID and the compute instance ID. The faulty hardware is then the parent GPU, so map the GPU UUID part, not the instance. Inside a virtual machine, the PCI addresses are those the hypervisor presents to the guest, so do the mapping to a slot on the host. Our H200 NVL NVLink bridge guide shows why CUDA_DEVICE_ORDER also matters when you pick the cards of one NVLink set.

Step 1: take the PCI bus ID from the alert

NVIDIA's XID documentation shows the format in two lines. The first XID in a log is preceded by a line that names the PCI address and the UUID, in NVIDIA's example NVRM: GPU at 0000:03:00: followed by GPU-b850f46d-d5ea-c752-ddf3-c4453e44d3f7. The XID line then repeats the address: NVRM: Xid (0000:03:00): 14 and the error data.

NVIDIA calls "0000:03:00" the PCI Express bus, device and function ID and the GPU- string "a globally unique, immutable identifier for each GPU". Search the kernel log with journalctl -k | grep 'NVRM: Xid', as NVIDIA's documentation suggests grepping for "NVRM: Xid". nvidia-smi writes the address as domain:bus:device.function in hex, with the function number added at the end, so compare the bus and device digits.

If the alert came from monitoring, use the bus ID or UUID the alert carries, where your exporter labels its metrics with them, rather than the GPU number. Our guide to GPU server monitoring with DCGM covers the XID fields and alert rules, and the NVIDIA XID error codes reference says what each code asks you to do before the card comes out.

Step 2: map the bus ID to UUID and serial number

nvidia-smi accepts the bus ID directly. The manual says -i takes the 0-based index, "the GPU's board serial number, the GPU's UUID, or the GPU's PCI bus ID (as domain:bus:device.function in hex)". So nvidia-smi -q -i 0000:03:00.0 prints the full record of exactly the card in the XID line. The fields that matter here:

• Serial Number: "This number matches the serial number physically printed on each board"; "a globally unique immutable alphanumeric value".
• GPU UUID: "The globally unique immutable alphanumeric identifier of the GPU."
• Bus Id: "PCI bus id as "domain:bus:device.function", in hex".
• Board Part Number and GPU Part Number, which the warranty provider may ask for.

For all cards at once, nvidia-smi -L lists every GPU with its UUID, and --query-gpu with --format=csv prints chosen fields per card; nvidia-smi --help-query-gpu lists the field names your driver supports, and CUDA's programming guide uses nvidia-smi --query-gpu=name,pci.bus_id to list the bus IDs. One caveat from the manual: the "board serial number might be shared between multiple GPUs on the same board". The single-GPU PCIe cards in our servers carry one GPU per board, so the serial identifies one card.

Layout of the inventory table we suggest; the bus IDs are placeholders, not values from a real server.

Step 3: find the physical slot for the bus ID

Three sources tie a bus address to a slot on the host side. Use at least two and compare.

lspci. lspci -vmm -s 0000:3d:00.0 prints a record for that device in the machine-readable format. Its tags include PhySlot, "The physical slot where the device resides (optional, Linux only)", and NUMANode, the NUMA node the device is connected to. lspci -t draws the tree of buses and bridges, which shows which PCIe switch the card sits behind. PhySlot is optional because the server firmware supplies it, and its value is the firmware's slot name, which need not match the label on the riser.

sysfs. The kernel's ABI documentation describes a label attribute per PCI device that gives "the firmware given name (SMBIOS type 41 string or ACPI _DSM string) of the PCI device". Where the maker fills it, it is the most readable name the host offers, found under /sys/bus/pci/devices/ and the card's address.

Redfish through the BMC. DMTF's PCIeDevice schema (v1_23_0) gives each device a SerialNumber, a PartNumber and a Slot object whose Location carries "part location information, including a ServiceLabel property". The older PCIeSlots resource is deprecated since v1_6_0 "in favor of the PCIeDevice schema". A BMC that fills these fields lets you match the serial from step 2 to a slot label without opening the server; what each maker's BMC shows for GPUs is in our article on out-of-band GPU monitoring through the BMC and Redfish.

nvidia-smi also has a "Slot Number" field, but the manual places it under Platform Info, "compute tray platform specific information", with chassis serial number and tray index. It describes rack-scale tray systems, not the PCIe slots of a standard GPU server.

We build AI servers with 2, 4 or 8 RTX PRO 6000 Server Edition or H200 NVL cards; send us the server model and the number of cards you plan, and our team replies with a configuration within one business day.

Step 4: check the server maker's slot map

The last link is the maker's documentation, because only it says which slot number on the chassis belongs to which processor and switch. Dell's technical guide for the PowerEdge XE7740 (Rev. A04, April 2026) is an example. It numbers the GPU slots 21 to 36, connects slots 21 to 28 to processor 0 and 29 to 36 to processor 1, and maps each slot to one of four PCIe switches. For double-width cards such as the H200 NVL and the RTX PRO 6000 it fills the slots in the order 25, 27, 29, 31, 21, 23, 33, 35.

Dell PowerEdge XE7740 Technical Guide, Rev. A04 (April 2026), tables 19, 20 and 21 (expansion slots, slot priority for double-width GPUs, GPU slot mapping), read 24 October 2026.

Two cross-checks follow from such a table. The NUMA node that lspci reports must belong to the processor of the slot you found; with one NUMA node per socket, a card on node 1 cannot sit in slot 25. And the cards behind one switch in lspci -t must sit in slots of the same switch in the table. In this server "GPU 3" in bus order cannot sit in a slot numbered 3, because no GPU slot has that number; the slot comes from the map, not from the index.

Step 5: confirm before you pull the card

Before you power down, write the expected slot and serial on the work order. Then follow the maker's service procedure for that chassis, with the power cords disconnected where the manual says so, and read the serial label on the card before it leaves the slot. If label and nvidia-smi disagree, stop: either the mapping is wrong or the server was changed since the inventory was taken.

On servers with NVLink bridges, the bridge comes off first with the tools the maker names; our article on XID 74 and H200 NVL bridges quotes Dell's rule of two removal tools per bridge. After the replacement, run nvidia-smi -q again: the new card shows a new serial and UUID at the old bus ID, and the inventory table needs that update.

What to record for the warranty case

NVIDIA's GPU Debug Guidelines list what a GPU issue report should contain and end with "Submit a ticket to your system vendor". The identification steps above add the fields that tie the report to one physical card.

NVIDIA GPU Debug Guidelines, GPU node triage, "Reporting a GPU Issue", updated 4 October 2026, read 24 October 2026; the serial, slot and part-number rows are our recommendation.

NVIDIA adds that its field diagnostic "is usually required before an RMA can be started" and that the system vendor says when and how to run it. Which code justifies a case is covered per code: XID 119 and 120 GSP timeouts, ECC errors and row remapping and GPU reset or reboot after XID 154 and 62.

Keep the map from the first day

The mapping is quickest when it already exists. At acceptance, save nvidia-smi -q and an lspci listing per server and fill in the inventory table from step 2 with the slot column from the maker's map; our acceptance testing guide records the serials against the board labels at the same point. Set CUDA_DEVICE_ORDER=PCI_BUS_ID in the base image or the service units, label monitoring metrics with UUID and bus ID, and update the table after every card swap. Then the next alert on "GPU 3" can be traced from the table. If you are planning new servers, our guide to how many GPUs fit in one server covers slot layouts per chassis; describe your fleet and its monitoring and we suggest a configuration.

FAQ