Monitor NVIDIA GPUs with Zabbix and Checkmk: templates, metrics, XID alerts and the gaps

NVIDIA GPU monitoring in Zabbix and Checkmk at a glance

To monitor NVIDIA GPUs with Zabbix, install the NVIDIA plugin for Zabbix agent 2 on the GPU server and link the official template "Nvidia by Zabbix agent 2" of Zabbix 7.4, which finds every GPU through NVIDIA's management library NVML and tracks utilisation, temperature, power, memory, clocks, ECC errors and PCIe traffic, with triggers. Checkmk monitors NVIDIA GPUs with its built-in nvidia_smi checks, available since Checkmk 2.2.0, which read the same basics through the nvidia-smi tool. Both leave out XID errors and the reasons a GPU slows down, which you add from the kernel log and, where needed, from NVIDIA's DCGM.

This guide is for an IT team that already runs Zabbix or Checkmk and wants a new GPU server in the same monitoring, without building a separate Prometheus and Grafana stack. What to measure and which values mean trouble is covered in our guide to GPU server monitoring with DCGM and XID errors; readings that survive a hung host come from the BMC, as our guide to out-of-band GPU monitoring with Redfish explains. Here the subject is the two tools: what their NVIDIA templates do, what to change, and what to add.

Versions are as of October 2026. Zabbix's documentation lists 7.4 as the current release and 8.0 as in development; the Zabbix figures below come from its integration page for NVIDIA and its package repository, the Checkmk figures from Checkmk's check manual pages and werk 14723, all read on 25 October 2026.

Zabbix: the NVIDIA plugin and the agent 2 template

Zabbix publishes two official templates for NVIDIA GPUs: "Nvidia by Zabbix agent 2" for passive checks and "Nvidia by Zabbix agent 2 active" for active ones. Both require Zabbix 7.4 or later; for 7.2 there is a passive version. The template's setup has two steps: set up "Zabbix agent 2 compiled with the Nvidia monitoring plugin", then create a host and link the template, with a Zabbix agent interface for the passive variant. Zabbix's repository carries the plugin as its own package, zabbix-agent2-plugin-nvidia-gpu; the newest build for 7.4 in the Debian pool is 7.4.16, uploaded on 20 October 2026.

Zabbix's own test of the setup is a single command from the server or proxy: zabbix_get with -s nvidia-host and -k nvml.system.driver.version, which returns the driver version if the agent and the plugin can reach NVML. If it fails, the plugin is missing or the NVIDIA driver is not loaded on the host.

The template has four items at host level: the driver version, the NVML library version, the number of devices and a device list. A discovery rule, GPU Discovery, creates the per-GPU items from that list. Its filter macros {$NVIDIA.NAME.MATCHES} and {$NVIDIA.UUID.MATCHES} default to .*, so every GPU is discovered; the NOT_MATCHES macros exclude cards by name or UUID.

Zabbix integration page "NVIDIA" (template for Zabbix 7.4), item and item prototype lists, read 25 October 2026.

Zabbix notes that the template was tested on an "Nvidia GTX 1650s" and an "Nvidia RTX 2070Ti", so it has not been described on data-centre cards. On a passive card such as the RTX PRO 6000 Blackwell Server Edition or the H200 NVL, which has no fan of its own, expect the fan speed item to return nothing useful; the chassis fans are a BMC reading. Check every item on the delivered server once before you trust the dashboards.

Zabbix triggers and macros to change for GPU servers

The template's triggers fire when a value stays above a threshold for three minutes, with a trigger of severity Warning at 80 and one of severity Average at 90 for GPU, memory, encoder and decoder utilisation, fan speed, temperature and power. Most of them need changing on a server that runs AI workloads.

Defaults from the Zabbix NVIDIA template for 7.4 (macros {$NVIDIA.*.WARN} and {$NVIDIA.*.CRIT}), read 25 October 2026; our recommendations in the third column.

High utilisation is the normal state of a GPU that serves a model, and NVIDIA's definition of it only counts the time any kernel runs, so the 90 % trigger tells you nothing about trouble. The framebuffer items show a card that serves a model almost full on purpose: vLLM reserves 92 % of the GPU memory at start-up by default since version 0.20.0, so an alert on used framebuffer memory would fire all day.

Temperature is the trigger to get right. The template describes {$NVIDIA.TEMPERATURE.WARN} and .CRIT as "in %", while the item they are compared with is in degrees C, so the defaults mean 80 °C and 90 °C. Take the thresholds from the card instead: nvidia-smi -q reports each GPU's Slowdown Temp, Max Operating Temp and Shutdown Temp, as our DCGM guide describes, and the warning should sit a few degrees below the slowdown point. Set them as host macros on the GPU server, so other hosts keep the defaults.

Power usage above 90 % of the limit is what a well-used card does. The useful power trigger is "Power limit changed": if an RTX PRO 6000 Server Edition that was not ordered with a 450 W cap reports 450 W instead of 600 W, the usual cause is the power cable, as our DCGM guide explains; the power levels of this card are set out in our page on RTX PRO 6000 Server Edition power at 600, 450 and 300 W. Raise "Number of devices has changed" to severity High, because a card that disappears from NVML is a hardware event, and treat a rising count of uncorrected memory ECC errors as an incident, following our page on XID 94 and 95 and row remapping.

On request our AI servers arrive with the operating system, the NVIDIA driver, CUDA and a container runtime installed, so your monitoring agent goes on as on any other Linux host; tell us which monitoring system you run, and our team replies with a configuration.

Checkmk: the nvidia_smi checks

Checkmk added its NVIDIA support with werk 14723, in version 2.2.0b1 of 14 December 2022: a Windows agent plugin called nvidia_smi, with a configurable path to nvidia-smi.exe, and five check plugins. The check manual pages list Linux and Windows as supported agents for the checks and say each one "only works if that tool is installed", the tool being nvidia-smi, which comes with the NVIDIA driver.

Checkmk werk 14723 and the Checkmk check manual pages for nvidia_smi_temperature, nvidia_smi_gpu_util, nvidia_smi_power and nvidia_smi_memory_util, read 25 October 2026.

Each check creates one service per card, and the item is the card's PCI bus ID, not its index. That is an advantage on four- and eight-card servers: the bus ID stays with the slot when the index order changes, and it is the same address the kernel writes into an XID line. Our guide to identifying a faulty GPU by bus ID and serial shows how to go from that address to the serial number for a replacement.

The manual pages give no default levels, so set the temperature rule per host from the Slowdown Temp as above. The memory check monitors FB and BAR1 memory separately; Checkmk's page explains BAR1 as the window through which the CPU or other devices reach the card's memory. A full framebuffer is again normal on an inference server.

On the Checkmk Exchange, the package NVIDIA_GPU, version 2.0 of 9 December 2021 for Checkmk 2.0.0p3, predates the built-in checks. For Checkmk 2.2 and later we would use the built-in plugins and avoid two sources for the same services.

What the NVIDIA templates do not see

Both tools read what NVML and nvidia-smi report by default. The values that explain a failed job or a slow server sit elsewhere.

Item lists of the Zabbix template and Checkmk check manuals (25 October 2026); sources of each signal as described in our DCGM, XID and BMC guides.

Clock event reasons tell you why clocks drop: SW Power Cap at the power limit, HW Slowdown from a thermal or power brake signal, SW Thermal Slowdown above the Max Operating Temp. On a server whose cards run slower than expected, these explain more than any temperature graph. Row remapping and NVLink errors matter for the replacement decision and, on H200 NVL servers with bridges, for whether a pair of cards still runs at full link speed; our page on XID 74 and NVLink errors covers the second.

XID errors into Zabbix or Checkmk

An XID is an error report the NVIDIA driver writes to the kernel log, a line with NVRM: Xid, the GPU's PCI address and a code, as our XID error codes reference explains. NVIDIA writes that the messages are logged in the kernel log buffer, usually reaching the journal and from there /var/log/messages or /var/log/syslog, and tells you to search for "NVRM: Xid". That makes XID alerts a log-monitoring job in either tool.

In Zabbix, log file monitoring is a feature of the Zabbix agent item types, documented in its manual under that name. A log item on the system log of the GPU server, filtered on NVRM: Xid, gives you one event per XID line, and a trigger can then match the codes that need action. In Checkmk, the same lines can come in through its log monitoring or as syslog messages forwarded from the server. We give no item syntax here: follow the log monitoring page of your release, and test with a harmless line written to the log by hand.

Match codes in three classes, as NVIDIA's XID catalogue does. Application faults such as 13, 31 and 43 point at a job and can stay at warning. Codes whose immediate action is a GPU reset, such as 95, 119 and 120, need a person the same day. A 79, a GPU that has fallen off the bus, needs a drained node and a call to the system vendor. Include the PCI address in the alert text, so it can be matched to the Checkmk service of that card or to the UUID in Zabbix.

When to add dcgm-exporter anyway

The NVML template and the nvidia_smi checks cover a two- or four-card server that runs a few models for one team. Three situations call for NVIDIA's Data Center GPU Manager as well: H200 NVL servers with NVLink bridges, where the link counters are only in DCGM; servers where you need SM activity to tell a busy card from a card that is only nominally busy; and fleets where profiling and health checks should look the same on every host.

You do not need a second monitoring system for that. dcgm-exporter serves its metrics on port 9400 in the Prometheus format, as our DCGM guide describes, and Zabbix lists an official Prometheus integration, "Prometheus checks", since Zabbix 4.2, described as native support of PromQL. Keep port 9400 on the management network. Our DCGM guide also names the security fix for dcgm-exporter that should be installed before you expose it.

For the BMC, the chassis side of the same server, use the paths our Redfish guide lists for each server maker. The BMC reports temperatures, power and fans with the operating system down, which neither agent can.

What we supply

We build AI servers to order: an entry server with two RTX PRO 6000 Server Edition cards, an inference node with four and a training node with eight H200 NVL, each with two AMD EPYC processors, assembled and tested before delivery and delivered across the EU on one contract and invoice. On request they arrive with the operating system, driver and container runtime installed; the Zabbix agent or Checkmk agent then goes on like on any other Linux host. Send us your monitoring setup and the alerts you need, and our team replies within one business day. Before the server goes into production, our guide to GPU server acceptance testing helps you check that every card reports what it should.

FAQ