XID 154 and XID 62 on NVIDIA GPUs: GPU reset, drain or a full server reboot

What XID 154 and XID 62 mean

XID 154 tells you what the GPU needs to recover from another error: nothing, a GPU reset, a reboot of the node, a drain of peer-to-peer traffic, or a drain followed by a reset. NVIDIA's XID Catalog calls it "GPU Recovery Action Changed" and says it "will be seen in conjunction with other Xids and summarizes the recovery action required for other Xids". XID 62 is one of those other errors, an internal micro-controller halt on the GPU, for which the catalogue gives a GPU reset as the immediate action and a support case as the next step.

The catalogue, last updated on 9 September 2026 (release 615), gives a sample line: "Xid 154 GPU recovery action changed from 0x0 (None) to 0x2 (Node Reboot Required)". The bucket for 154 is XID_154, defined as "Follow XID 154 reported guidance", and its investigatory action reads "N/A Informational only regarding another Xid". The catalogue's XID 154 column marks the codes that trigger it from "CUDA 12.7; GPU driver R565" on, among them 48, 62, 63, 64, 74, 94, 95 and 120.

Codes 13, 31, 43 and 45, the application faults, are covered in our article on XID 13, 31, 43 and 45, memory errors in our guide to XID 94, 95, 48 and 63, and the full list in our NVIDIA XID error codes reference. This article is about the decision after the error: reset the GPU, drain the node, or reboot it.

The five recovery actions and what each asks of you

NVIDIA defines the actions in its GPU memory error management document and lists the same five values for the GPU Recovery Action field in the nvidia-smi manual.

NVIDIA nvidia-smi manual, NVIDIA XID Catalog (value names for XID 154) and NVIDIA GPU memory error management, "Error Recovery and Response Flags", updated 9 September 2026, all read on 24 October 2026; the operator step is our summary of NVIDIA's descriptions. Our other XID articles use the nvidia-smi names in the first column.

Three details change what you do. Reset means, in NVIDIA's words, "Terminate all GPU processes, reset the GPU using “nvidia-smi -r”", then start new processes. Reboot is for a fault that may have left the operating system in an inconsistent state; the nvidia-smi manual adds that a warm reboot of the operating system is enough and no AC or DC power cycle is needed. Drain P2P asks you to end processes that use peer-to-peer traffic and to disable UVM persistence mode; after that, a new query returns one of the other actions, and the nvidia-smi manual says to reset the GPU if it still reads Drain P2P. Drain and Reset marks a GPU at reduced capacity, for example with part of its memory offlined or some MIG partitions down, where existing work "that didn't get affected" may finish before the reset.

NVIDIA's catalogue shows 0x0 for None and 0x2 for Node Reboot Required. The XID 154 lines in the forum reports below read "from 0x0 (None) to 0x1 (GPU Reset Required)". The pages read for this article give no hexadecimal values for the other two actions, so read the text in brackets, not the number.

The nvidia-smi manual adds that the field is "not intended for determining which fault triggered recovery action". Read the XID lines in the kernel log for the cause and the recovery action for the step. Monitoring code reads the value through NVML as NVML_FI_DEV_GET_GPU_RECOVERY_ACTION. Since the driver logs a 154 whenever the action changes, an alert on XID 154 with the text in brackets tells the on-call person what kind of maintenance window is needed before anyone logs in. The alerting side is covered in our guide to GPU server monitoring with DCGM.

XID 62: a halted micro-controller

The catalogue names XID 62 "Internal micro-controller halt (newer drivers)", with RESET_GPU as the immediate action and CONTACT_SUPPORT as the investigatory one, and gives no trigger text. NVIDIA's GPU Debug Guidelines, updated on 4 October 2026, treat it together with XID 61: "Report a GPU Issue and Reset GPU(s) reporting the XID." The catalogue lists a second halt code, XID 169, also described as "Internal micro-controller halt", with no actions given.

Two unconfirmed user reports on NVIDIA's developer forum, opened in March and July 2026, describe XID 62 on an RTX PRO 6000 Blackwell Workstation Edition in a two-card workstation with driver 595.58.03, while vLLM 0.17.1 served an FP8 mixture-of-experts model over two cards. The posted log shows the XID 62 line, a GSP message that the "PMU has halted" and an XID 154 changing the action to "GPU Reset Required"; an earlier crash on the same card logged XID 120. The poster wrote that the GPU did not recover without a full power cycle. Another user reported that an undersized power supply in their own workstation caused similar crashes and that capping the card's power stopped them. As read on 24 October 2026, neither thread has a reply from NVIDIA, a confirmed cause or a fixed driver version.

These are reports from workstations, not NVIDIA statements, and they do not show that XID 62 is typical of the card. They show the order to work in: follow the recovery action the driver reports, collect evidence, check the power delivery against the system maker's rules, and update driver and firmware before opening a case. The power cap commands are in our guide to the RTX PRO 6000 power limit, GSP errors such as XID 120 in our article on XID 119 and 120 GSP timeouts.

GPU reset with nvidia-smi: requirements and limits

nvidia-smi -r, long form --gpu-reset, "can be used to clear GPU HW and SW state in situations that would otherwise require a machine reboot", in the manual's words. Its conditions are strict:

• It requires root.
• No application may use the GPU, which the manual spells out as CUDA applications, graphics applications and monitoring applications "like other instance of nvidia-smi".
• -i selects one or more GPUs; "Without this option, all GPUs are reset."
• Where an individual reset is not feasible, no compute application may run on any other GPU in the system either.
• From Ampere on, GPUs with NVLink connections can be reset individually; without Fabric Manager or before Ampere, linked GPUs must be reset in the same command.

NVIDIA also writes that a reset does not succeed in every case and that "It is not recommended for production environments at this time", and adds: "If any GPU is not healthy a complete reset should be instigated by power cycling the node." The GPU Debug Guidelines add the virtual machine case: "GPU reset not supported in VMs. Restart the VM." For a GPU passed through to a VM, the VM restart is the reset.

A reset of one GPU on a running server, in the order we would follow it:

1. Run nvidia-bug-report.sh as root while the error state is still visible; NVIDIA's guidelines list its output among what a report needs.
2. Drain the node or remove that GPU's workload from the scheduler and the load balancer, and wait for running work to end.
3. Check with nvidia-smi -q -d PIDS that no process uses the GPU, and stop monitoring agents that hold it open.
4. Reset the GPU with nvidia-smi -r -i 2, where 2 is its index.
5. Read nvidia-smi -q: the GPU Recovery Action should be None and, for memory errors, the row remapper should show Pending: No.
6. Run dcgmi diag -r 3 on the GPU before it returns to service; our acceptance testing guide explains which levels apply to RTX PRO cards.

If the reset fails or the action reads Reboot, reboot the node; if the card is still unhealthy, power the server off fully, as the manual says.

We supply the RTX PRO 6000 Server Edition and build servers with two, four or eight of them; tell us your scheduler and how many nodes you run, and our team replies with a layout in which one node can be drained without stopping the service.

Draining the node: Slurm and Kubernetes

A reset or reboot only goes smoothly when no new work arrives while you wait for the running jobs. Both common schedulers have a documented way to do that.

Slurm scontrol manual, version 26.05, and Kubernetes kubectl drain reference, read on 24 October 2026; n01 is an example node name.

In Slurm, scontrol update NodeName=n01 State=DRAIN with a Reason such as "xid 154 reset GPU 2" means, in the manual's words, that "no new jobs may be started on this node. Existing jobs are allowed to run to completion, leaving the node in a DRAINED state once all the jobs have completed." For a reboot, scontrol reboot with ASAP adds the DRAIN flag and reboots once the node is idle; with nextstate=DOWN the node "will remain in a down state after rebooting", so it returns only after you have checked it.

In Kubernetes, kubectl drain marks the node unschedulable and evicts its pods. It "will not delete any daemon set-managed pods", and it refuses to proceed while such pods exist unless --ignore-daemonsets is given. Monitoring agents that run as DaemonSets, such as a DCGM exporter, therefore stay on the drained node; stop the one on that GPU before nvidia-smi -r if it holds the device. The health check of NVIDIA's device plugin, in the main branch read on 24 October 2026, marks a GPU unhealthy on critical XIDs, and its code contains no path that marks it healthy again, so after the repair check in the node's allocatable resources that the GPU is back.

In a Slurm cluster of eight-card RTX PRO 6000 Server Edition nodes, GPU 5 of node n07 logs XID 62, then XID 154 to GPU Reset Required. The operator drains n07 with a reason, collects the bug report and lets the running jobs finish. After the reset, the action reads None and the level 3 diagnostic passes, so n07 returns with State=RESUME. Had the 154 read Node Reboot Required, the operator would have used scontrol reboot ASAP nextstate=DOWN n07 and resumed the node only after the diagnostic.

When repeats justify a support case

XID 62's investigatory action is CONTACT_SUPPORT, so a repeat on the same GPU, or a reset that does not bring the action back to None, is a case for the vendor. NVIDIA's guidelines list what the report needs: OS and driver details, the key log messages, the debug steps taken, nvidia-bug-report.sh output and DCGM diagnostics logs, submitted "to your system vendor". Add the serial number, the slot, the power cap setting and the workload at the time.

Before opening the case, rule out the parts you can change. Compare the driver with the current release of your branch, as our guide to NVIDIA driver branches and CUDA versions describes, and the vBIOS, BIOS and BMC with the system maker's current releases, as in our guide to GPU server firmware updates. NVIDIA's field diagnostic "is usually required before an RMA can be started", and the system vendor says when and how to run it. Errors that follow the card across slots point to the card, as the swap test in our article on XID 79 shows.

For cards bought from us, the manufacturer warranty is handled through us and support by our engineering partner's service team. Send us the XID lines, the recovery action and the serial number, and we reply within one business day with the next step.

What we supply

We supply the RTX PRO 6000 Server Edition, the Max-Q and the Workstation Edition, and build AI servers to order with two to eight Server Edition cards, assembled and tested before delivery, with manufacturer warranty on every component and DOA units replaced, on one EU contract and invoice. The operating system, drivers, CUDA and a container runtime are installed on request.

FAQ