XID 94, 95, 48 and 63: GPU memory errors, row remapping and the warranty case

What XID 94, 95, 48 and 63 tell you

XID 48 means the GPU found an uncorrectable ECC error in its memory. On a GPU with error containment, such as the H200 NVL, XID 94 follows when the error stayed with one application, which must restart, and XID 95 when it reached several, which can restart only after a GPU reset; XID 63 then records a row remap that takes effect at the next reset. One such sequence is normal handling of a memory error; NVIDIA treats a card as an RMA case when the row remapper's failure flag is set and its field diagnostic confirms it.

XID 64 means the remap could not be recorded. The codes have their place in our NVIDIA XID error codes reference, and alert rules for them are in our guide to GPU server monitoring with DCGM. This article follows one memory error from the log line to the reset and, where it comes to that, the warranty case.

NVIDIA XID Catalog, last update 9 September 2026 (release 615), and NVIDIA's GPU memory error management document, "User Visible Statistics", updated 9 September 2026, both read on 24 October 2026. The catalogue marks all six as applicable to A100, H100, B100 and GB200.

Two newer codes add detail. NVIDIA's memory error document says "XID 171 indicates if it is DRAM and XID 172 indicates if it is SRAM", and the catalogue lists both as additions to XID 48. It gives no actions for 171 and 172, so the action stays the one for 48.

The sequence on the card: 48, then 94 or 95, then 63

NVIDIA's memory error document, last updated on 9 September 2026, describes the handling that the codes report. Error containment, introduced with Ampere, is meant to "limit the impact of uncorrectable ECC errors on GPU applications". When it works, "the affected application is terminated", "Unaffected applications will continue to run", and dynamic page offlining "marks the page containing the faulty memory as unusable", so new work can start on the GPU "without requiring a GPU reset". In rare cases, the document adds, uncorrectable errors "are still uncontained and might impact all the workloads being processed in the GPU".

NVIDIA's examples of the two log lines show the difference. The contained one, on a GPU with MIG, reads 94, pid=7194, Contained: CE User Channel and ends with "RST: No, D-RST: No"; the uncontained one reads 95, pid=7062, Uncontained: LTC TAG and ends with "RST: Yes". The RST field matches the catalogue: no reset after 94, a reset after 95. The documents read for this article do not define D-RST. Both entries in the catalogue add: "Xid 45 will be seen in relation to this error", so the application's teardown appears as an XID 45 next to them.

Then the row remapper acts. Every DRAM bank carries spare rows, and the remapper "replaces degrading memory cells with spare ones to avoid offlining regions of memory in software". The document's statistics page describes XID 63 as "successful recording of a row-remapping entry to the InfoROM" and XID 64 as "a failure in recording a row-remapping entry to the InfoROM". Row remapping supports up to 512 remappings, against 64 page retirements on older GPUs, and it "requires a GPU reset to take effect".

The catalogue's workflow for 48 joins the steps. Its immediate action reads "Solo: RESET_GPU w/ 63 or 64: DRAIN_AND_RESET", and its investigatory action "Solo: RUN_FIELDDIAG", with the note that for frame-buffer errors you "follow Xid63/64 guidance" and for all other SRAM "check SRAM Error Threshold flag". NVIDIA's GPU Debug Guidelines, updated on 4 October 2026, say the same in prose: if 63 or 64 follows a 48, "Drain/cordon the node, wait for all work to complete, and reset GPU(s) reporting the XID", and if neither follows, run the field diagnostic.

Which cards support what: H200 NVL, L40S, L4 and RTX PRO

NVIDIA lists the memory features per chip, and the chip decides which codes you can see.

NVIDIA GPU memory error management, "Supported GPUs" (GA100, GA10x, Ada AD10x, GH100, GB10x), updated 9 September 2026, read on 24 October 2026; memory sizes from NVIDIA's product pages. GB10x, the data-centre Blackwell chip, also has RAS repair of GPU memory, which NVIDIA does not promise for every Blackwell GPU.

On an H200 NVL all the codes above can appear, and a contained error costs one job. The L40S and L4 have no error containment in NVIDIA's table, so XID 94 and 95 are not expected on them; an uncorrectable error there is handled by the workflow for 48, with a reset or reboot to clear it.

The RTX PRO Blackwell cards are missing from the table. The nvidia-smi manual describes the row remapper as "available on Ampere+", and NVIDIA's vPC sizing guide states that GDDR7 ECC on RTX PRO Blackwell cards is always enabled. The catalogue's applicability columns name the B100 and GB200, which are data-centre Blackwell parts, not the GB202 of the RTX PRO 6000. So run nvidia-smi -q -d ROW_REMAPPER on the delivered card and record what it reports: a card that shows the row remapper section can log 63 and 64, and its counts belong in your acceptance record. Our acceptance testing guide puts that reading into the delivery checks.

We build servers with H200 NVL, L40S, L4 and RTX PRO 6000 Server Edition cards and size the layout so that one card can leave service without stopping the rest; send us your models and the redundancy you need, and our team replies with a configuration.

Reading the row remapper in nvidia-smi

nvidia-smi -q -d ROW_REMAPPER prints five values per GPU, and nvidia-smi -q -d ECC the error counts behind them.

NVIDIA nvidia-smi manual, read on 24 October 2026; the right column is our reading of NVIDIA's memory error document.

Two details matter when you read the numbers. The remap counts are, in NVIDIA's words, "the number of entries recorded in the InfoROM, not the ones remapped in hardware", so a pending remap is already counted. For ECC, "Volatile error counters track the number of errors detected since the last driver load", while "Aggregate error counts persist indefinitely and thus act as a lifetime counter", split into SRAM and DRAM, correctable and uncorrectable. For scripts, nvidia-smi offers --query-remapped-rows; --help-query-remapped-rows lists its fields. The same values reach monitoring through DCGM, as our monitoring guide shows.

Applying the remap: drain, reset and check

The pending flag "indicates that row-remapping will happen at the next GPU reset". Until then, NVIDIA writes, unaffected applications keep running and new workloads can launch, and it recommends a reset before a live VM migration onto that GPU. When the driver reports the recovery action Drain and Reset, which the nvidia-smi manual says is "Asserted only for UCE row remaps", the GPU runs at reduced capacity, for example with part of its memory offlined: schedule no new work, let existing work finish, then "reset the GPU to regain its full capacity".

The reset itself needs root and no application on the GPU, and inside a VM the GPU Debug Guidelines say "GPU reset not supported in VMs. Restart the VM." The steps, the limits of nvidia-smi -r and draining a Slurm or Kubernetes node are in our article on XID 154 and 62, GPU reset or server reboot.

For XID 95 the guidelines distinguish two cases, in a row they label for the A100. With MIG enabled, "drain any work on the other GPU instances, wait for all work to complete" and reset the GPU. With MIG disabled, "the node should be rebooted immediately", because the error is uncorrectable and uncontained. For XID 64 the catalogue gives a GPU reset and a support case; the guideline row for the A100 says "The node should be rebooted immediately since there is a recording failure."

A worked example: a server with four H200 NVL cards serves two models, one TP=2 pair each. At 14:10, GPU 3 logs XID 48, then 94 with the PID of one vLLM worker, a 45 and a 63. The pair on GPUs 2 and 3 loses its worker process and restarts it; the pair on GPUs 0 and 1 is not affected. nvidia-smi -q -d ROW_REMAPPER on GPU 3 shows Pending: Yes. The operator removes the pair on GPUs 2 and 3 from the load balancer in the evening, waits for open requests to finish, resets GPU 3, and confirms Pending: No and Remapping Failure Occurred: No before returning them. Had the line read 95, the whole GPU would have needed the reset before any restart.

When a memory error makes the card an RMA case

NVIDIA's RMA policy for row remapping reads: "the RMA criteria is met when the row-remapping failure flag is set and validated by the field diagnostic". The flag is set in three cases:

1. A remap for an uncorrectable error is needed on a bank that already has eight remapped rows.
2. A remap for an uncorrectable error is attempted on a row that was already remapped, even if its bank has fewer than eight.
3. The GPU reaches 512 remaps for uncorrectable errors.

For Blackwell GPUs, the policy adds that a third remap attempt for an uncorrectable error on a bank invokes HBM channel repair if a spare channel is available; if the repair succeeds, the GPU is ready for use again, otherwise remapping continues until the failure flag is set. SRAM errors have their own threshold: NVIDIA sets an "SRAM Threshold Exceeded" flag after more than four unique uncorrectable events in one address bank of a parity-protected SRAM, or more than two in a SECDED-protected one, and its field diagnostic then "will determine whether a GPU qualifies for RMA". For XID 92, a high rate of corrected single-bit errors, the debug guidelines also send you to the field diagnostic. The release notes of driver 595.112.07 (22 October 2026) list a reduced XID 92 threshold for DRAM correctable errors, which lets the driver detect a storm of such errors and disable the DRAM CE interrupt until the next driver reload, to prevent the loss of performance they cause.

NVIDIA XID Catalog (9 September 2026), GPU Debug Guidelines (4 October 2026) and GPU memory error management, "RMA Policy Thresholds for Row-Remapping" (9 September 2026); the grouping is ours.

A warranty case needs what NVIDIA's guidelines list under "Reporting a GPU Issue": OS and driver version, the key log lines, the debug steps taken, nvidia-bug-report.sh output and the DCGM diagnostics log. For memory errors add the ROW_REMAPPER and ECC output before and after the reset, the serial number, and the times of the 48, 94 or 95 and 63 or 64 lines. The field diagnostic is "usually required before an RMA can be started", and the system vendor says when and how to run it.

For cards bought from us, the manufacturer warranty is handled through us. Write to us with the nvidia-smi output and the XID lines, and we reply within one business day with how the case proceeds for that card.

What we supply

We build AI servers to order with H200 NVL, L40S, L4 and RTX PRO 6000 Server Edition cards, assembled and tested before delivery, with manufacturer warranty on every component, DOA units replaced and the assembly covered by us, on one EU contract and invoice. Application faults such as XID 13 and 31 are covered in our article on XID 13, 31, 43 and 45, and MIG layouts that let one instance fail alone in our H200 NVL MIG profiles.

FAQ