XID 119 GSP RPC timeout and XID 120 GSP error on NVIDIA GPUs: causes and supported fixes

What XID 119 and XID 120 mean

XID 119 is a GSP RPC timeout and XID 120 a GSP error: the NVIDIA driver logs them when code running on the GPU System Processor, a small processor on the GPU, fails or does not answer a request in time. NVIDIA's XID Catalog gives both codes the same actions: reset the GPU now, then investigate the software, which in practice means the driver. If the error persists, a GPU reset or a node power cycle may be needed; if it comes back after a power cycle, NVIDIA points to its GPU Debug Guidelines.

The catalogue, last updated on 9 September 2026, uses one trigger text for both codes. They "may be logged when an error occurs in code running on the GSP core of the GPU and/or a timeout occurs while waiting for the GSP core of the GPU to respond to an RPC message". An RPC here is a request from the driver on the host to the firmware on the GPU. So 119 means the driver waited and gave up, while 120 means the firmware reported an error of its own. Both stop the GPU for the work in flight.

NVIDIA XID Catalog, "Analyzing Xid Errors with the Xid Catalog", updated 9 September 2026, read 24 October 2026. Its applicability columns mark both codes for the A100, H100, B100 and GB200.

The last column matters for automation. XID 154, "GPU Recovery Action Changed", summarises the recovery action other codes require. The catalogue lists driver R565 in that column for 120 and nothing for 119, so by our reading a 120 on a current driver can be followed by a 154 line that names the recovery action, while a 119 is not. Our guide to XID 154 and 62, GPU reset or reboot covers how to act on that line, and the NVIDIA XID error codes reference lists the other codes. This article covers 119 and 120 alone.

What the GSP does and where its firmware comes from

NVIDIA's Linux driver README describes the GSP in one sentence: "Some GPUs include a GPU System Processor (GSP) which can be used to offload GPU initialization and management tasks." The driver uses the GSP firmware "by default for all Turing and later GPUs". Every card we supply falls in that range: the L4 and L40S (Ada), the H200 NVL (Hopper) and the RTX PRO Blackwell cards, including the RTX PRO 6000 Server Edition.

The firmware is part of the driver, not of the card. The README of driver 575.64 installs the files in /lib/firmware/nvidia/575.64/, named gsp_*.bin after the GPU architectures they serve. Its example output shows the GSP firmware version equal to the driver version. A driver update therefore replaces the GSP firmware too, while the VBIOS on the card stays as it was. Two places show which firmware runs:

• nvidia-smi -q -d GSP_FIRMWARE_VERSION prints the field "GSP Firmware Version" per GPU; a version number means GSP is in use, "N/A" means it is disabled.
• /proc/driver/nvidia/gpus/ followed by the GPU's PCI bus ID holds an information file with a "GSP Firmware" line.

On a server with mixed driver installs, for example after a failed upgrade, this is the quickest check that every GPU runs the firmware of the driver you think is loaded.

Turning GSP off is not an option on Blackwell

The same README documents a switch: "The driver can be forced to disable use of GSP firmware by setting the kernel module parameter NVreg_EnableGpuFirmware=0." On Blackwell that parameter has no supported use. NVIDIA's open GPU kernel modules README (version 615.71.09) states that the modules "must be used with GSP firmware and user-space NVIDIA GPU driver components". NVIDIA's blog of 17 July 2024 on the move to open modules says: "For cutting-edge platforms such as NVIDIA Grace Hopper or NVIDIA Blackwell, you must use the open-source GPU kernel modules. The proprietary drivers are unsupported on these platforms."

Taken together, the RTX PRO 6000 in all three editions, the RTX PRO 4500 Server Edition and the other RTX PRO Blackwell cards always run with GSP. On the H200 NVL, L40S and L4, the proprietary flavour still exists in the R580 and R595 branches, but from R615 NVIDIA's packages provide only the open modules, as our guide to NVIDIA driver branches and CUDA versions explains. Turning GSP off on those cards means moving to a kernel module flavour that is on its way out. We would use it only when the system vendor's or NVIDIA's support asks for it as a test during a case, and never as the fix for a production fleet. Servers we build to order ship with the operating system and the NVIDIA driver installed on request; name the branch and module flavour you need and the server arrives set up that way.

NVIDIA open-gpu-kernel-modules README (615.71.09) and NVIDIA technical blog of 17 July 2024, read 24 October 2026; R615 packaging from the R615 release notes as cited in our driver guide. The last column is our reading of these statements.

First response to an XID 119 or 120

The catalogue's immediate action is a GPU reset, and NVIDIA's nvidia-smi manual sets its limits: nvidia-smi -r "Requires root", there "can't be any applications using these devices", a reset does not succeed in every case, and "It is not recommended for production environments at this time". So treat the reset as a maintenance step on a drained GPU.

1. Record the XID line, the time and the PCI address in it. The address belongs to the slot; our guide to finding the faulty GPU by bus ID, slot and serial turns it into a card and a serial number.
2. Check the lines around it. A 154 line names the recovery action, a 62 points at an internal microcontroller halt, and a 79 means the GPU has fallen off the bus, which has its own article on XID 79.
3. Run nvidia-bug-report.sh before any reset, so the evidence survives it.
4. Drain the GPU: take it out of the scheduler or the load balancer and stop the processes on it.
5. Reset the GPU with nvidia-smi -r -i and its bus ID. If the GPU is not healthy afterwards, the manual says "a complete reset should be instigated by power cycling the node".
6. If 119 or 120 returns after the power cycle, follow NVIDIA's GPU Debug Guidelines and open a case with the system vendor.

On a GPU connected over NVLink, as in an H200 NVL server with bridges, the manual states that from the Ampere architecture on such GPUs "can be individually reset". Inside a virtual machine, NVIDIA's debug guidelines say "GPU reset not supported in VMs. Restart the VM."

The driver side: branch, release notes and a fixed crash

INVESTIGATE_SW sends you to the driver first, because the GSP firmware and the code that talks to it both come from the driver package. Check three things.

• The branch. For production, R580 (Long Term Support Branch) or R595 (Production Branch), as listed in NVIDIA's driver table of 9 September 2026.
• The release. Driver 580.196.07 of 21 October 2026 lists a fixed issue: "Fixed a race condition between GPU event notification and subdevice teardown that could cause a system crash. On Linux, the issue was observed following a GSP RPC timeout." The fix removes a crash that followed a 119, not the timeout itself. If 119 on an R580 host took the whole node down, this release is the first step.
• The VBIOS before the next upgrade. The release notes of 595.112.07 of 22 October 2026 carry a known issue for Hopper: the driver "will fail to initialize on systems with Hopper GPUs subrevision = 3 and VBIOS versions older than 96.00.68.00.xx", and NVIDIA asks for VBIOS 96.00.68.00.xx or newer before the upgrade. Read the VBIOS of each H200 NVL with nvidia-smi before moving such a host to R595.

NVIDIA data-centre driver release notes 580.196.07 and 595.112.07, read 24 October 2026; NVIDIA knowledge base article on useful nvidia-smi queries (fields gpu_name, gpu_bus_id, vbios_version); pinning as described in our driver guide.

Neither of the two release notes lists a known issue for 119 or 120. When the code repeats on a current release of a supported branch, the cause is outside the driver's known issues, and the case to the system vendor is the next step. We build servers with RTX PRO 6000 Server Edition and H200 NVL cards and install the driver branch your fleet runs; tell us which branch and cards you operate and our team replies with a configuration on that branch.

vBIOS and server firmware come through the server maker

The VBIOS lives on the card, BIOS and BMC firmware on the server. NVIDIA's Hopper note above shows that the two have to fit the driver. For cards in a rack server, take VBIOS updates from the server maker's support pages for that model, because the maker qualifies the card firmware with its BIOS and slot layout; our guide to GPU server firmware updates for vBIOS, BMC and BIOS describes the order.

Our acceptance testing guide records the VBIOS and GSP firmware versions of every card at delivery. That record pays off here: if one card of eight logs 119 and its VBIOS differs from the other seven, the version is the first lead. Identical cards bought together should report identical versions.

Persistence mode and monitoring for GSP errors

The nvidia-smi manual says that with persistence mode on, "the NVIDIA driver remains loaded even when no active clients, such as X11 or nvidia-smi, exist", and that after each reboot the mode set with nvidia-smi -pm "will default to "Disabled"". On a GPU server the driver should stay loaded, so check after a reboot that persistence mode is enabled again. NVIDIA does not name persistence mode as a cause or a fix for 119 or 120; it belongs on the checklist because a reboot during the incident sets it back to Disabled.

For alerts, dcgm-exporter's default field file exposes DCGM_FI_DEV_XID_ERRORS, the "Value of the last XID error encountered", and leaves the counter DCGM_EXP_XID_ERRORS_TOTAL commented out. One 119 between two other codes can vanish from the gauge, so enable the counter or alert from the kernel log. Our guide to GPU server monitoring with DCGM lists the fields.

When a GSP error points at the card

The catalogue files 119 and 120 under software, so a single occurrence that a reset clears is a driver matter. Three patterns, by our reading of NVIDIA's actions, shift the suspicion to hardware:

• the code returns after a full power cycle on a current driver of a supported branch;
• it follows one card and not the slot after a slot swap, as described in our article on XID 79;
• it comes with memory, PCIe or NVLink codes on the same GPU, whose own actions then apply.

In each case, run dcgmi diag -r 3 on the affected GPU before the case is opened, and compare with the acceptance baseline.

What to send with a support case

NVIDIA's GPU Debug Guidelines (updated 4 October 2026) list what a report should contain and end with "Submit a ticket to your system vendor". For GSP errors, add the versions that decide the software side.

NVIDIA GPU Debug Guidelines, GPU node triage, "Reporting a GPU Issue", read 24 October 2026; rows marked "our recommendation" are ours.

The guidelines call NVIDIA's field diagnostic "the authoritative and comprehensive NVIDIA tool for determining the health of GPUs" and say it "is usually required before an RMA can be started"; the system vendor says when and how to run it. For GPUs we supply, the manufacturer warranty is handled through us; send us the XID lines and the versions above and we reply within one business day.

FAQ