Monitoring GPU servers in production: DCGM metrics, clock event reasons, memory errors and XIDs

What DCGM covers, and on which cards

NVIDIA’s Data Center GPU Manager, DCGM, is “a suite of tools for managing and monitoring NVIDIA datacenter GPUs in cluster environments”, and dcgm-exporter turns its fields into Prometheus metrics. DCGM supports data-centre GPUs from Kepler on and other GPUs from Maxwell on, the latter with “limited DCGM functionality”. Its feature table still uses NVIDIA’s old brand names: Tesla, the data-centre line, gets everything; Quadro, Titan and GeForce cards get metrics, configuration and health checks, but no policy notifications and only the first diagnostic level. No RTX PRO Blackwell card is named. The release notes add them one at a time instead, the RTX PRO 6000 Max-Q in DCGM 4.5.3 and an RTX PRO 5000 variant in 4.6.0; 4.7.0 is the newest listed. Keep DCGM current, test what it reports on workstation editions, and see our driver and CUDA guide for the driver side.

NVIDIA’s quick start runs dcgm-exporter as a container with --cap-add SYS_ADMIN, and DCGM notes that profiling counters need administrator privileges. The metrics appear on port 9400, where curl localhost:9400/metrics shows them. The field list comes from /etc/dcgm-exporter/default-counters.csv, and -f points to your own.

Why GPU utilisation misleads

nvidia-smi defines GPU utilisation as the “percent of time over the past sample period during which one or more kernels was executing on the GPU”, with a sample period of 1 to 1/6 second depending on the product. It ignores how much of the GPU the kernels use. NVIDIA’s own DCGM example: on a GPU with N multiprocessors, a kernel with N/5 blocks that runs for the whole interval gives an SM activity of 0.2, while by the definition above it shows 100 per cent utilisation. On an RTX PRO 6000, with 188 multiprocessors, one busy multiprocessor reads about 0.005 by our arithmetic. NVIDIA’s guidance: “A value of 0.8 or greater is necessary, but not sufficient, for effective use of the GPU. A value less than 0.5 likely indicates ineffective GPU usage.”

NVIDIA DCGM feature overview and 4.6.0 release notes, which keep the classic names as deprecated aliases; GPU utilisation as defined in the nvidia-smi documentation.

For language models, read the last two rows together: token generation at low batch sizes is bound by memory bandwidth and prompt processing by compute, so expect a server that generates for a few users at a time to show high memory activity and modest tensor activity. The exporter’s default file enables graphics engine, tensor and memory activity, leaves SM activity and occupancy commented out, and still uses the classic names.

Power, temperature and clock event reasons

Power and temperature come from DCGM_FI_DEV_POWER_USAGE and DCGM_FI_DEV_GPU_TEMP, renamed in 4.6 to DCGM_FI_DEV_BOARD_POWER_WATTS and DCGM_FI_DEV_GPU_TEMP_CELSIUS. Take thresholds from the card: nvidia-smi reports each GPU’s Slowdown Temp, “the temperature at which a GPU HW will begin optimizing clocks due to thermal conditions”, its Max Operating Temp and its Shutdown Temp, and DCGM has fields for all three.

The Clocks Event Reasons block of nvidia-smi -q, the current name for what older tools called throttle reasons, says why clocks drop; DCGM has deprecated DCGM_FI_DEV_CLOCK_THROTTLE_REASONS for DCGM_FI_DEV_CLOCKS_EVENT_REASONS. SW Power Cap holds clocks down “because the GPU is consuming too much power”: under full load that is a card at its limit, so read the Enforced Power Limit beside it, and an RTX PRO 6000 that enforces 450 W instead of 600 W usually points to the power cable, as our rack power guide explains. HW Slowdown cuts “the core clocks by a factor of 2 or more” whenever HW Thermal Slowdown or HW Power Brake is active, the brake being an external signal, “e.g. by the system power supply”. SW Thermal Slowdown means the GPU is above its Max Operating Temp. Passive cards get there when chassis airflow falls short, the limit our guide to GPUs per server works through. For trends, use the time counters: nvidia-smi’s Clock Event Reasons Counters, and DCGM’s DCGM_FI_DEV_POWER_VIOLATION and DCGM_FI_DEV_THERMAL_VIOLATION, which the exporter’s default field list leaves commented out.

ECC and row remapping

nvidia-smi splits ECC errors into correctable and uncorrectable, SRAM and DRAM, and volatile counts since the last driver load against aggregate counts that “persist indefinitely”. The matching exporter fields, DCGM_FI_DEV_ECC_DBE_VOL_TOTAL and its siblings, are commented out by default.

Row remapping swaps a failing memory row for one of the spare rows each DRAM bank carries, up to 512 per GPU; the remap “requires a GPU reset to take effect” and then lasts for the life of the GPU. nvidia-smi -q -d ROW_REMAPPER shows correctable and uncorrectable remaps, Pending (a reset is needed), Remapping Failure Occurred, and a histogram of the spare rows each bank has left. NVIDIA’s RMA policy flags a row remapping failure when a remap for an uncorrectable error hits a bank that already has eight uncorrectable remaps, or a row that was already remapped, or comes after 512 uncorrectable remaps; the GPU qualifies for RMA when that flag is set and NVIDIA’s field diagnostic validates it.

NVIDIA’s memory-error document lists row remapping for GA100, GA10x, Ada AD10x, GH100 and GB10x chips, and error containment and dynamic page offlining for GA100, GH100 and GB10x only. Among our cards, the H200 NVL, a GH100, gets all three; the L40S and L4, both Ada, get row remapping. The document names chips rather than memory types; for these cards that means HBM3e on the H200 NVL and GDDR6 on the L40S and L4. The RTX PRO Blackwell chips, GB202 and GB203, are not listed, although the nvidia-smi manual describes the row remapper as “available on Ampere+”, so check what nvidia-smi -q -d ECC,ROW_REMAPPER reports on the delivered card before alerting on it. The exporter carries the remapped-row counts and the failure flag by default; DCGM_FI_DEV_ROW_REMAP_PENDING has to be added.

NVLink on a bridged H200 NVL

Hopper uses fourth-generation NVLink with up to 18 links per GPU, and the H200 NVL brief lists 18. nvidia-smi nvlink -s shows the state of each link and nvidia-smi nvlink -e its error counters, on NVLink 4 replay, recovery and CRC errors. DCGM has counters for flit CRC, data CRC, replay and recovery errors, all four commented out in the exporter, so enable them on bridged hosts and record the link state after installation. A failing link also logs Xid 74. The RTX PRO, L40S and L4 cards have no NVLink.

The XID errors you will actually meet

An Xid is “an error report from the NVIDIA driver that is printed to the operating system’s kernel log”, on Linux a line containing NVRM: Xid. The causes range from hardware and driver bugs to “a user application problem”. NVIDIA’s catalog covers Ampere and newer GPUs, PCIe cards included, and gives each Xid an immediate and an investigatory action.

NVIDIA Xid catalog (immediate actions, trigger conditions) and GPU debug guidelines, September 2026. For Xid 94, the contained counterpart of 95, the affected application must be restarted, other applications keep running, and NVIDIA recommends a GPU reset when convenient.

The exporter’s DCGM_FI_DEV_XID_ERRORS holds only the “Value of the last XID error encountered”; enable the commented-out counter DCGM_EXP_XID_ERRORS_TOTAL or collect the kernel log too. Before opening a case, run nvidia-bug-report.sh as root and dcgmi diag -r 3, which the triage guide lists; outside the data-centre line DCGM runs only level 1.

nvidia-smi on the node, and persistence

nvidia-smi dmon watches up to 16 GPUs, by default power, temperature, clocks and utilisation, and -s selects other sets, among them v for power and thermal violations and e for ECC and PCIe replay errors. nvidia-smi pmon shows utilisation per process. Both suit a look at one node; trends belong in DCGM.

Persistence decides what the counters mean. Without a client the driver deinitialises the GPU, which NVIDIA says “leads to long load times for each CUDA job, on the order of seconds”, and clears the compute mode, application clocks, a software power cap and the volatile ECC counts. NVIDIA calls the legacy persistence mode “near end-of-life”; nvidia-persistenced comes with the driver, but NVIDIA “cannot guarantee” that it runs by default, so check that it starts at boot.

What to alert on: examples

These rules are our examples, not NVIDIA’s; the right column names the NVIDIA statement each one rests on.

The one-minute duration, the 5 °C margin and the 0.5 trigger are our example values; test them on your own data first.

What we supply

Eurokommerz supplies RTX PRO Blackwell workstation cards from the RTX PRO 2000 to the RTX PRO 6000, the L4, the L40S and the H200 NVL EU-wide with manufacturer warranty, and AI servers built to order, load-tested before shipment and delivered with the operating system, drivers and container runtime installed on request. Tell us the cards and your current monitoring, and we will say which fields and alerts fit them.

FAQ