XID 13, 31, 32, 43 and 45: application faults or a faulty GPU, and how to tell them apart

What XID 13, 31, 32, 43 and 45 have in common

XID 13, 31, 32, 43 and 45 are the NVIDIA driver's reports that a GPU job failed, and in most cases the cause lies in the job itself. NVIDIA's XID Catalog gives a restart of the application as the immediate action for 13, 31 and 32, marks 43 as IGNORE and handles 45 through a short workflow. What points to the hardware is a pattern: the same code on the same GPU with different jobs, or a code that a known good application also triggers.

An XID is, in NVIDIA's words, "an error report from the NVIDIA driver that is printed to the operating system's kernel log or event log", and the messages "can be indicative of a hardware problem, an NVIDIA software problem, or a user application problem". Our NVIDIA XID error codes reference lists every common code with its actions. This article covers only the five that a running application usually causes and shows how to decide whether the card needs attention.

NVIDIA XID Catalog, "Analyzing Xid Errors with the Xid Catalog", last update 9 September 2026, and its PDF edition (release 615), read on 24 October 2026. The catalogue marks all five codes as applicable to A100, H100, B100 and GB200.

The PDF edition defines RESTART_APP as "The application should be restarted" and CHECK_APP/CUDA as an issue "likely caused by an application passing bad data or utilizing incorrect methods in communications with GPU", adding: "If determined to be a driver issue then REPORT_ISSUE".

Reading the log line: PCI address, PID and process name

Every XID line starts with NVRM: Xid, followed by the PCI address of the GPU and the code. NVIDIA's catalogue adds that "Some errors will contain PID that can be used to identify source of the problem", and for 13 and 31 the driver prints more detail after the code. An unconfirmed user report on NVIDIA's developer forum from March 2026, on an RTX PRO 6000 Blackwell Workstation Edition with driver 590.48.01, shows what such lines contain.

Log lines quoted in a user report on NVIDIA's developer forum, 21 March 2026 (two RTX PRO 6000 Blackwell Workstation Edition cards, driver 590.48.01), read on 24 October 2026. A report, not an NVIDIA statement; the field layout can change between driver branches.

Three steps tie a line to a workload. First map the PCI address to a card: nvidia-smi -q -i accepts the PCI bus ID and prints the serial number, so that every later note names the card and not only the slot. Our guide to finding the faulty GPU by bus ID, slot and serial shows the steps. Then match the PID and process name to the job that ran at that time: by the time you read the log, the process has usually exited, so the scheduler's job log or the container runtime's log is the place to look. nvidia-smi -q -d PIDS lists only the processes running on the GPU now. Finally record the GPC, TPC and SM numbers of an XID 13, because the catalogue's workflow asks whether repeats hit the same unit.

In Kubernetes, NVIDIA's device plugin watches XIDs and marks a GPU unhealthy on a critical one. Its health check, in the main branch read on 24 October 2026, skips a hard-coded list it labels "Application errors: the GPU should still be healthy": 13, 31, 43, 45, 68 and 109. A pod that dies with an XID 31 therefore leaves its GPU schedulable, and the next pod lands on it. Other codes, such as 48 or 62, mark the device unhealthy when NVML reports them as critical XID events, and the file has no code path that marks a device healthy again. The variable DP_DISABLE_HEALTHCHECKS can switch the check off, set to "all" or "xids" or given a list of codes.

We build GPU servers with RTX PRO 6000 Server Edition, L40S, L4 and H200 NVL cards and install the drivers, CUDA and a container runtime on request; send us your workload and the XID lines you see, and our team replies with what we would check first.

XID 13: Graphics Engine Exception

The catalogue says XID 13 "is logged for general user application faults" and that "Typically this is an out-of-bounds error where the user has walked past the end of an array". The immediate action is a restart of the application. NVIDIA's GPU Debug Guidelines, updated on 4 October 2026, add a step: "Run DCGM and Field diagnostics to confirm if the issue is related to hardware", and if the hardware is fine, debug the application.

The investigatory workflow in the PDF edition sorts XID 13 by pattern:

1. Repeats on the same TPC and GPC, on different SMs or on one SM: run DCGM diagnostics as a "possible HW issue", and if they pass, NVIDIA's field diagnostic.
2. A single event with no burst, or repeats that do not hit the same TPC and GPC: CHECK_APP/CUDA, an application issue.
3. In a non-production environment: CHECK_APP/CUDA.
4. A known good application and a single event: REPORT_ISSUE.

NVIDIA does not define "solo" and "burst" in the catalogue. We read "solo" as an XID 13 with no other XID around it and "burst" as several in quick succession.

For the application side, NVIDIA's Compute Sanitizer (version 2026.3.0) has a memcheck tool that "can precisely detect and report out of bounds and misaligned memory accesses to global, local and shared memory", enabled by default and started as compute-sanitizer --tool memcheck in front of the program. Run it on a test instance with the input that failed, not on the production service, since instrumentation slows the program down.

One known driver issue can produce the same symptom on Blackwell. The release notes of driver 595.112.07 (22 October 2026) state that applications using cuTensorMapEncodeTiled() or cuTensorMapEncodeIm2col*() on Blackwell GPUs "may encounter sporadic illegal memory accesses (IMA) or MMU faults (like XID 13)" when the tensor's backing allocation is smaller than 128 KB and the tensor is not dense and non-overlapping. The notes give a workaround in the application code and name no driver release with the fix. If XID 13 or 31 appears on RTX PRO Blackwell cards with kernels that use such tensor maps, check this entry before you suspect the card.

XID 31: GPU memory page fault

XID 31 "is logged when a fault is reported by the MMU", the GPU's memory management unit, when a process touches an address it has no mapping for. The catalogue states: "Typically these are application-level bugs, but can also be driver bugs or hardware bugs." Its workflow starts with "Multiple runs needed to establish pattern":

1. Repeated MMU faults on the same GPU, identified by its PCI ID: DCGM diagnostics as a possible hardware issue, then the field diagnostic if they pass.
2. Repeated MMU faults on different GPUs: CHECK_APP/CUDA.
3. A single fault with no burst: CHECK_APP/CUDA.
4. A fault from a known good application: REPORT_ISSUE.

NVIDIA's two documents weigh XID 31 differently. The catalogue calls it typically an application bug, while the triage table of the GPU Debug Guidelines lists it as "Suspected Hardware Problems" with the step "Contact the hardware vendor". The catalogue's pattern rule is the more useful guide in practice: one fault, or faults that move between GPUs with one application, belong to the application owner; faults that stay with one card across jobs go to the vendor.

The forum report quoted above follows that second pattern. The poster logged 47 XID events on one of two identical cards, saw the fault follow the card when it moved to another PCIe slot, and saw none on the second card under the same vLLM workload; the poster also reported that NVIDIA's diagnostic tool nvvs triggered an XID 31 on the faulty card. As read on 24 October 2026, the thread records no confirmed cause and no fix, so treat it as one user's observation. The slot swap itself is described step by step in our article on XID 79, the GPU that has fallen off the bus.

XID 32: corrupted push buffer, a PCIe signal

XID 32 has the same immediate action as 13 and 31, a restart of the application, but a different cause. The catalogue says that "These failures primarily involve quality issues on PCI, and are generally not caused by user application actions." One XID 32 after a restart of the job needs no more than a note. Repeats belong with the PCIe checks: link generation and width under load, "Replays Since Reset" in nvidia-smi -q, the riser and its cable, and the AER counters, in the order our XID 79 article gives.

XID 43 and 45: informational codes

XID 43 "is logged when a user application hits a software induced fault and must terminate", and NVIDIA adds: "The GPU remains in a healthy state." The immediate action is IGNORE. CONTACT_SUPPORT is the investigatory action for the case where it recurs without an explanation in the application.

XID 45 "is logged when the user application aborts and the kernel driver tears down the GPU application". The catalogue adds that it is "Most likely to see when running multiple cuda applications and hitting a DBE", a double-bit ECC error, and that "In many cases, this is not indicative of a bug but rather a user or system action." The GPU Debug Guidelines call it "informative only". The workflow reads "Solo: RESTART_FM Not Solo: IGNORE (follow guidance in other Xid)". By our reading, RESTART_FM refers to NVIDIA's Fabric Manager, which the debug guidelines discuss for systems with NVSwitch; they describe a solo 45 on the A100 after an unexpected Fabric Manager shutdown.

The second half of that rule matters more on PCIe servers. When 45 appears together with 48, 94 or 95, the catalogue's entries for 94 and 95 say "Xid 45 will be seen in relation to this error", and the memory error decides the next step, as our guide to XID 94, 95, 48 and 63, ECC errors and row remapping describes. A 45 next to a 62 or a 154 belongs to our article on XID 154 and 62, GPU reset or server reboot.

When an application fault points to the card

The table condenses NVIDIA's workflows into the patterns an operator sees in a week of logs.

NVIDIA XID Catalog workflows WORKFLOW_XID_13, WORKFLOW_XID_31 and WORKFLOW_XID_45 (release 615, 9 September 2026) and NVIDIA GPU Debug Guidelines (4 October 2026); the grouping into log patterns is ours.

Take a server with four RTX PRO 6000 Server Edition cards running four copies of a model with vLLM, one per card. GPU 2 logs XID 31 three times in a week, each time under a different model version, and the other three cards stay clean. The fault stays with one PCI address across jobs, so the catalogue's first branch applies: drain that card's copy, run dcgmi diag -r 3 on it, and if it passes, ask the vendor about the field diagnostic. If instead all four cards log XID 31 within an hour of rolling out a new container image, the fault moves between GPUs with one application, and the image is the suspect.

DCGM does not run every diagnostic level on every card. Our acceptance testing guide explains which levels NVIDIA's feature table gives to data-centre cards, why it names no RTX PRO Blackwell card, and what DCGM 4.6.0 changed. Alerting on these codes is covered in our guide to GPU server monitoring with DCGM.

What to collect before you open a case

NVIDIA's GPU Debug Guidelines list what a report to the vendor should contain, under "Reporting a GPU Issue":

1. Basic system configuration, such as the operating system and driver version.
2. A clear description of the issue with the key log messages.
3. The debug steps already taken.
4. A full listing of the log that contains those messages.
5. The output of nvidia-bug-report.sh.
6. DCGM diagnostics logs.
7. Application details, if the error is tied to one.

For application faults, add what the workflows ask about: every XID line with its time, PCI address, PID and process name, the serial number of each card, the container image and framework versions, and whether a second card ran the same job without errors. The guidelines end with "Submit a ticket to your system vendor". NVIDIA calls its field diagnostic "the authoritative and comprehensive NVIDIA tool for determining the health of GPUs" and says it "is usually required before an RMA can be started"; the system vendor says when and how to run it.

For GPUs bought from us, the manufacturer warranty is handled through us, with support from our engineering partner's service team. Write to us with the XID lines, the serial numbers and the diagnostic output, and we reply within one business day with the next step for the card.

What we supply

We build AI servers to order with RTX PRO 6000 Server Edition, L40S, L4 and H200 NVL cards, assembled and tested before delivery, with manufacturer warranty on every component and DOA units replaced, on one EU contract and invoice. The operating system, drivers, CUDA and a container runtime are installed on request. For memory errors, GSP timeouts and resets, see our articles on XID 119 and 120, XID 94 and 95, and XID 154 and 62.

FAQ