Monitoring GPU servers in production: DCGM metrics, clock event reasons, memory errors and XIDs

What DCGM covers, and on which cards

NVIDIA’s Data Center GPU Manager, DCGM, is “a suite of tools for managing and monitoring NVIDIA datacenter GPUs in cluster environments”, and dcgm-exporter turns its fields into Prometheus metrics. DCGM supports data-centre GPUs from Kepler on and other GPUs from Maxwell on, the latter with “limited DCGM functionality”. Its feature table still uses NVIDIA’s old brand names: Tesla, the data-centre line, gets everything; Quadro, Titan and GeForce cards get metrics, configuration and health checks, but no policy notifications and only the first diagnostic level. No RTX PRO Blackwell card is named. The release notes add them one at a time instead, the RTX PRO 6000 Max-Q in DCGM 4.5.3 and an RTX PRO 5000 variant in 4.6.0; 4.7.0 was the newest listed when we read them in October 2026, and the release notes of the R580 driver 580.196.07 state that it works only with DCGM 4.3 or newer. Keep DCGM current and test what it reports on workstation editions; the driver side is in our driver and CUDA guide.

NVIDIA’s quick start runs dcgm-exporter as a container with --cap-add SYS_ADMIN, and DCGM notes that profiling counters need administrator privileges. The metrics appear on port 9400, where curl localhost:9400/metrics shows them. The field list comes from /etc/dcgm-exporter/default-counters.csv, and -f points to your own. Keep port 9400 on the management network and the exporter current. NVIDIA rates CVE-2026-47483, published on 28 July 2026, at 8.2 (high); it concerns the /debug/pprof endpoints of dcgm-exporter up to version 4.8.2, and the CVE record also lists DCGM itself up to 4.5.2 as affected. Our guide to securing a GPU server covers the BMC network, driver fixes and Container Toolkit CVEs. Our AI servers, among them a four-card RTX PRO 6000 Server Edition inference node, come with the operating system, NVIDIA driver and container runtime installed on request. Tell us which cards you monitor and with what, and our team will help.

Why GPU utilisation misleads

nvidia-smi defines GPU utilisation as the “percent of time over the past sample period during which one or more kernels was executing on the GPU”, with a sample period of 1 to 1/6 second depending on the product. It ignores how much of the GPU the kernels use. In NVIDIA’s DCGM example, on a GPU with N streaming multiprocessors (SMs), a kernel with N/5 blocks that runs for the whole interval gives an SM activity of 0.2, while by the definition above it shows 100 per cent utilisation. On an RTX PRO 6000, with 188 multiprocessors, one busy multiprocessor reads about 0.005 by our arithmetic. For SM activity, NVIDIA states: “A value of 0.8 or greater is necessary, but not sufficient, for effective use of the GPU. A value less than 0.5 likely indicates ineffective GPU usage.” On drivers before 580.196.07 and 595.112.07, a GPU could also report 100 per cent utilisation after a CUDA job was stopped with Ctrl-C; NVIDIA lists “GPU utilization reported 100% following SIGINT-terminated CUDA workloads” as fixed in these releases of 21 and 22 October 2026.

NVIDIA DCGM feature overview and 4.6.0 release notes, which keep the classic names as deprecated aliases; GPU utilisation as defined in the nvidia-smi documentation.

For language models, read the last two rows together: token generation at low batch sizes is bound by memory bandwidth and prompt processing by compute, so expect a server that generates for a few users at a time to show high memory activity and modest tensor activity. The exporter’s default file enables graphics engine, tensor and memory activity, leaves SM activity and occupancy commented out, and still uses the classic names. When you move to the new names, check the unit as well: since DCGM 4.7.0, DCGM_FI_DEV_GPU_UTIL_RATIO is a field of its own, ID 1613, that reads 0.0 to 1.0, while the deprecated DCGM_FI_DEV_GPU_UTIL, ID 203, stays an integer percentage, so an alert threshold of 90 becomes 0.9.

Power, temperature and clock event reasons

Power and temperature come from DCGM_FI_DEV_POWER_USAGE and DCGM_FI_DEV_GPU_TEMP, renamed in 4.6 to DCGM_FI_DEV_BOARD_POWER_WATTS and DCGM_FI_DEV_GPU_TEMP_CELSIUS. Take thresholds from the card: nvidia-smi reports each GPU’s Slowdown Temp, “the temperature at which a GPU HW will begin optimizing clocks due to thermal conditions”, its Max Operating Temp and its Shutdown Temp, and DCGM has fields for all three.

The Clocks Event Reasons block of nvidia-smi -q, the current name for what older tools called throttle reasons, says why clocks drop; DCGM has deprecated DCGM_FI_DEV_CLOCK_THROTTLE_REASONS for DCGM_FI_DEV_CLOCKS_EVENT_REASONS. SW Power Cap holds clocks down “because the GPU is consuming too much power”. Under full load that means the card is at its power limit, so read the Enforced Power Limit beside it. If an RTX PRO 6000 Server Edition enforces 450 W instead of 600 W and was not ordered as a 450 W power-capped option, the usual cause is the power cable, as our rack power guide explains. HW Slowdown cuts “the core clocks by a factor of 2 or more” whenever HW Thermal Slowdown or HW Power Brake is active, the brake being an external signal, “e.g. by the system power supply”. SW Thermal Slowdown means the GPU is above its Max Operating Temp. Passive cards get there when chassis airflow falls short, the limit our guide to GPUs per server works through. For trends, use the time counters: nvidia-smi’s Clock Event Reasons Counters, and DCGM’s DCGM_FI_DEV_POWER_VIOLATION and DCGM_FI_DEV_THERMAL_VIOLATION, which the exporter’s default field list leaves commented out.

ECC and row remapping

nvidia-smi splits ECC errors into correctable and uncorrectable, SRAM and DRAM, and volatile counts since the last driver load against aggregate counts that “persist indefinitely”. The matching exporter fields, DCGM_FI_DEV_ECC_DBE_VOL_TOTAL and its siblings, are commented out by default.

Row remapping swaps a failing memory row for one of the spare rows each DRAM bank carries, up to 512 per GPU; the remap “requires a GPU reset to take effect” and then lasts for the life of the GPU. nvidia-smi -q -d ROW_REMAPPER shows correctable and uncorrectable remaps, Pending (a reset is needed), Remapping Failure Occurred, and a histogram of the spare rows each bank has left. NVIDIA’s RMA policy flags a row remapping failure in three cases: a remap for an uncorrectable error hits a bank that already has eight uncorrectable remaps, hits a row that was already remapped, or comes after 512 uncorrectable remaps. The GPU qualifies for RMA when that flag is set and NVIDIA’s field diagnostic validates it.

NVIDIA’s memory-error document lists row remapping for GA100, GA10x, Ada AD10x, GH100 and GB10x chips, and error containment and dynamic page offlining for GA100, GH100 and GB10x only. Among the cards we supply, the H200 NVL, a GH100, gets all three; the L40S and L4, both Ada, get row remapping. The document names chips rather than memory types; for these cards that means HBM3e on the H200 NVL and GDDR6 on the L40S and L4. The table, updated 9 September 2026, has no entry for the RTX PRO Blackwell chips, although the nvidia-smi manual describes the row remapper as “available on Ampere+”, so check what nvidia-smi -q -d ECC,ROW_REMAPPER reports on the delivered card before alerting on it. The exporter carries the remapped-row counts and the failure flag by default; DCGM_FI_DEV_ROW_REMAP_PENDING has to be added. What XID 94, 95, 48 and 63 mean for each card is in our article on ECC errors and row remapping.

NVLink on a bridged H200 NVL

Hopper uses fourth-generation NVLink with up to 18 links per GPU, and the H200 NVL brief lists 18. nvidia-smi nvlink -s shows the state of each link and nvidia-smi nvlink -e its error counters, on NVLink 4 replay, recovery and CRC errors. The exporter’s default file marks its flit CRC, data CRC, replay and recovery counters as fields for GPUs before Hopper and lists separate Hopper counters, DCGM_FI_DEV_NVLINK_CRC_ERROR_TOTAL, DCGM_FI_DEV_NVLINK_REPLAY_TOTAL and DCGM_FI_DEV_NVLINK_RECOVERY_TOTAL. All of them are commented out, so enable the Hopper set on hosts with bridged H200 NVL cards and record the link state after installation. A failing link also logs Xid 74. The RTX PRO, L40S and L4 cards have no NVLink.

Common XID errors and NVIDIA’s recommended actions

An Xid is “an error report from the NVIDIA driver that is printed to the operating system’s kernel log”, on Linux a line containing NVRM: Xid. The causes range from hardware and driver bugs to “a user application problem”. NVIDIA’s catalog covers Ampere and newer GPUs, PCIe cards included, and gives each Xid an immediate and an investigatory action. The table below lists common codes; more codes, both actions for each and how to find them in the kernel log are in our NVIDIA XID error codes reference.

NVIDIA Xid catalog (immediate actions, trigger conditions), updated 4 November 2026, and GPU Debug Guidelines, updated 4 October 2026. For Xid 94, the contained counterpart of 95, the affected application must be restarted, other applications keep running, and NVIDIA recommends a GPU reset when convenient.

The exporter’s DCGM_FI_DEV_XID_ERRORS holds only the “Value of the last XID error encountered”; enable the commented-out counter DCGM_EXP_XID_ERRORS_TOTAL or collect the kernel log too. The drivers 580.196.07 and 595.112.07 of 21 and 22 October 2026 fix rare “GPU fallen off bus errors” caused by NVML calls in paths such as nvidia-smi -q, a driver-side cause of Xid 79, so update the driver before you report a card as faulty, and on older drivers do not poll nvidia-smi -q at a high rate. Before opening a case, run nvidia-bug-report.sh as root and dcgmi diag -r 3, which the triage guide lists. DCGM’s feature table gives cards outside the data-centre line only level 1, and since DCGM 4.6.0 levels 3 and 4 can run as stability checks on GPUs without SKU-specific calibration, with -p "generic_mode=True".

For GPUs bought from us, the manufacturer warranty is handled through us, with support from our engineering partner’s service team. Write to us with the card and the XID you see, and we reply within one business day.

nvidia-smi dmon, pmon and persistence mode

nvidia-smi dmon watches up to 16 GPUs, by default power, temperature, clocks and utilisation, and -s selects other sets, among them v for power and thermal violations and e for ECC and PCIe replay errors. nvidia-smi pmon shows utilisation per process. Both suit a look at one node; trends belong in DCGM.

Persistence mode affects what the counters show. Without a client the driver deinitialises the GPU, which NVIDIA says “leads to long load times for each CUDA job, on the order of seconds”, and clears the compute mode, application clocks, a software power cap and the volatile ECC counts. NVIDIA calls the legacy persistence mode “near end-of-life”; nvidia-persistenced comes with the driver, but NVIDIA “cannot guarantee” that it runs by default, so check that it starts at boot.

What to alert on: examples

These rules are our examples, not NVIDIA’s; the right column names the NVIDIA statement each one rests on. Xid codes reach them only in-band, through the kernel log or DCGM. What the server’s BMC reports out-of-band, and which readings it keeps when the host hangs, is in our article on out-of-band GPU monitoring through the BMC and Redfish.

The one-minute duration, the 5 °C margin and the 0.5 trigger are our example values; test them against your own monitoring data first, starting from the baseline your acceptance test and burn-in recorded.

What we supply

We supply RTX PRO Blackwell workstation cards from the RTX PRO 2000 to the RTX PRO 6000, the RTX PRO 6000 Server Edition for rack servers, the L4, the L40S and the H200 NVL. We deliver them across the EU with manufacturer warranty. AI servers we build to order are assembled and tested before delivery and arrive with the operating system, drivers and container runtime installed on request.

FAQ