NVIDIA GPU Monitoring with DCGM Exporter, Prometheus, and Grafana

Your GPU box reports 100 percent utilisation and the job is still slow. nvidia-smi will not tell you why, because utilisation only counts whether a kernel was resident, not whether the card was allowed to run at speed. NVIDIA DCGM will tell you. It reads the same driver telemetry, adds health watches and diagnostics on top, and hands the whole lot to Prometheus through an exporter.

Original content from computingforgeeks.com - post 171116

This NVIDIA GPU monitoring guide sets up DCGM, dcgm-exporter, Prometheus, Grafana, and Alertmanager on a real GPU host, then builds alert rules for the four signals that matter on a GPU fleet: utilisation, memory, power, and thermal health. Every command, error, and number below came off a live NVIDIA A10G, with a second pass on an A100 to catch what changes inside a container.

Ran this end to end on Ubuntu 24.04 with an A10G in August 2026; it works start to finish.

Why NVIDIA GPU monitoring needs more than nvidia-smi

DCGM is a daemon plus a CLI plus a library. The daemon, nv-hostengine, holds the watches on GPU fields so several clients can read telemetry without each opening its own NVML session. That daemon model is what makes fleet monitoring practical.

Capability	nvidia-smi	DCGM
Utilisation, memory, power, temperature	Yes	Yes
Per-engine activity (tensor, DRAM, graphics)	No	Yes, the profiling fields
Background health watches with pass/warn/fail	No	Yes
Active hardware diagnostics that run a workload	No	Yes, four levels
Decoded clock throttle reasons as separate signals	Text only	Yes, as labelled metrics
Prometheus output	No	Yes, via dcgm-exporter

DCGM runs on non-datacenter cards too, but not all of it. Field value watches and the active health checks work on GeForce and Quadro as well as Tesla. What is restricted is policy notification and the deeper diagnostic levels, which stay Tesla-only, and the profiling fields, which need a datacenter card from Volta onwards. So the health watches below work almost anywhere; the DCGM_FI_PROF_* metrics in this guide assume a Tesla, A-series, L-series, or H-series card.

Prerequisites

You need a host with an NVIDIA datacenter GPU, a working driver, and root. If the driver is not installed yet, work through installing the NVIDIA drivers and CUDA toolkit first, then come back. Confirm the card is visible:

nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv

The card, driver, and framebuffer size print on one line:

name, driver_version, memory.total [MiB]
NVIDIA A10G, 595.91.07, 23028 MiB

Note the driver version. The next step depends on it.

1. Install DCGM on Ubuntu and RHEL

DCGM 4 splits its binaries by CUDA major version, and the package you want is keyed to the CUDA version your driver reports, not to any CUDA toolkit you may have installed separately. Read it off the driver:

nvidia-smi -q | grep "CUDA Version"

A current driver reports CUDA 13, which means the cuda13 package:

CUDA Version                              : 13.2

Add the NVIDIA CUDA repository and install. On Ubuntu 24.04:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get install -y datacenter-gpu-manager-4-cuda13

The metapackage pulls three more, and it is not small. Budget the disk before you run it on a thin root volume:

The following NEW packages will be installed:
  datacenter-gpu-manager-4-core datacenter-gpu-manager-4-cuda13
  datacenter-gpu-manager-4-proprietary
  datacenter-gpu-manager-4-proprietary-cuda13
0 upgraded, 4 newly installed, 0 to remove and 36 not upgraded.
Need to get 587 MB of archives.
After this operation, 1965 MB of additional disk space will be used.

If the driver reports CUDA 12, swap the suffix and install datacenter-gpu-manager-4-cuda12 instead. Picking the wrong one installs binaries that will not load against your driver.

On RHEL, Rocky Linux, and AlmaLinux the repository changes but the package names do not:

sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo
sudo dnf install -y datacenter-gpu-manager-4-cuda13 datacenter-gpu-manager-exporter

Two things change on EL10. The repository path becomes rhel10, and dnf5 replaced the old subcommand, so the middle line is sudo dnf config-manager addrepo --from-repofile=<url> rather than --add-repo. You can drop the dnf-plugins-core line entirely there, because dnf5 has config-manager built in. Everything after the install step, including the unit names, is identical to the Ubuntu path below.

One message during setup is harmless and appears on any host that has had DCGM before: useradd: user 'nvidia-dcgm' already exists.

2. Start the host engine and confirm the GPU is discovered

The package ships a systemd unit that runs the daemon as root and drops privileges to a dedicated service account:

sudo systemctl enable --now nvidia-dcgm
systemctl cat nvidia-dcgm.service | grep -E "ExecStart|User"

That is the whole service definition worth knowing:

User=root
# ExecStart=%Y/../../../bin/nv-hostengine -n --service-account nvidia-dcgm
ExecStart=/usr/bin/nv-hostengine -n --service-account nvidia-dcgm

Ask DCGM what hardware it found:

dcgmi discovery -l

Each GPU gets an ID that every later command uses, alongside counts for NVSwitch, ConnectX, and CPU entities:

1 GPU found (Active).
+--------+----------------------------------------------------------------------+
| GPU ID | Device Information                                                   |
+--------+----------------------------------------------------------------------+
| 0      | Name: NVIDIA A10G                                                    |
|        | PCI Bus ID: 00000000:00:1E.0                                         |
|        | Device UUID: GPU-d3ed7d5d-47b5-609f-56ed-d76ebb18890e                |
+--------+----------------------------------------------------------------------+
0 NvSwitches found.
0 ConnectX found.
0 CPUs found.

The attribute dump is where you find the numbers your alert thresholds should be built from. Note the flag is -i, which is the short form of --info, so passing both is an error:

dcgmi discovery -i aptc --gpuid 0

Power limits and the two thermal limits are the useful part:

| Device Name              | NVIDIA A10G                                     |
| Serial Number            | 1320522015573                                   |
| InfoROM Version          | G133.0210.00.04                                 |
| VBIOS                    | 94.02.75.00.01                                  |
| Current Power Limit (W)  | 300                                             |
| Default Power Limit (W)  | 300                                             |
| Max Power Limit (W)      | 300                                             |
| Min Power Limit (W)      | 100                                             |
| Enforced Power Limit (W) | 300                                             |
| Shutdown Temperature (C) | 98                                              |
| Slowdown Temperature (C) | 95                                              |

Slowdown at 95 C and shutdown at 98 C are the hardware’s own limits. Any temperature alert you write should fire well below 95 C, because by the time the card is cutting clocks to protect itself you have already lost throughput.

3. Watch live GPU metrics with dcgmi dmon

dcgmi dmon streams fields by numeric ID. These are the ones worth memorising:

Field ID	Short tag	What it is
150	TMPTR	GPU core temperature, C
140	MMTMP	Memory temperature, C
155	POWER	Power draw, W
203	GPUTL	GPU utilisation, percent
252	FBUSD	Framebuffer used, MiB
100 / 113	SMCLK / SMMAX	Current and maximum SM clock, MHz
112	DVCCTR	Current clock event reasons, bitmask
230	XIDER	XID error count
1001 / 1004 / 1005	GRACT / TENSO / DRAMA	Graphics engine, tensor pipe, DRAM activity

Watch the four fleet signals at once:

dcgmi dmon -e 203,252,155,150 -c 5

Under a sustained fp16 matmul the A10G pinned itself against its power cap:

#Entity         GPUTL             FBUSD             POWER             TMPTR
ID                                                   W                 C
GPU 0           100               21288             296.935           73
GPU 0           100               21288             301.198           73
GPU 0           100               21288             297.752           73
GPU 0           100               21288             302.069           73
GPU 0           100               21288             298.164           73

Compare that against the same card idle, and you have the ranges your thresholds live between:

Signal	Idle	Under load
Utilisation	0 percent	100 percent
Framebuffer used	0 MiB	21288 MiB of 23028
Power draw	60 W	297 to 302 W against a 300 W cap
Core temperature	31 C	73 C

Readings above the cap are normal, not a measurement bug. The enforced limit is an average the board management holds you to, so instantaneous samples ride a couple of watts either side of it. Alert on sustained draw, never on a single sample crossing the number.

The gotcha here is the thing utilisation hides. On the A100 the same test reported 100 percent utilisation while the SM clock sat at 1050 MHz against a 1410 MHz maximum, because the card was clamped by its power cap. Utilisation was perfect and the card was running a quarter slower than it could. That is why the power and clock fields belong in the same dashboard as utilisation, and it is the reason for field 112.

4. Run GPU health checks

Health watches are a background system: you enable a set of watch groups, DCGM samples them, and you ask for a verdict. Enable and check:

dcgmi health -g 0 -s a
dcgmi health -g 0 -c

The selector letters are p for PCIe, m for memory, i for InfoROM, t for thermal and power, n for NVLink, d for driver, x for ConnectX, and a for all of them. With the card sitting on its power limit, the verdict is a warning rather than a failure:

+---------------------------+----------------------------------------------------------+
| Health Monitor Report                                                                |
+===========================+==========================================================+
| Overall Health            | Warning                                                  |
| GPU                       |                                                          |
| -> 0                      | Warning                                                  |
|    -> Errors              |                                                          |
|       -> Power system     | Warning                                                  |
|                           | Detected clocks event due to power violation in GPU 0.   |
|                           | Monitor the power conditions. This GPU can still         |
|                           | perform workload.                                        |
+---------------------------+----------------------------------------------------------+

Two things trip people up here. First, watches need time to accumulate. Checking immediately after enabling them returns Healthy because nothing has been sampled yet, which is easy to mistake for a clean bill of health. DCGM’s own help text puts the figure at 60 seconds before the first meaningful query, so give it a full minute.

Second, -s a produces a false failure on PCIe cards that carry NVLink hardware with no peer attached. On an A100 PCIe the NVLink watch reports all 12 links down and drags the verdict to Overall Health: Failure on a perfectly healthy card. Drop the NVLink watch on those hosts:

dcgmi health -g 0 -s pmit

Cards with no NVLink hardware at all, like the A10G, are not affected and can safely use -s a. Check which case you are in before you decide:

dcgmi nvlink -s

Links rendered as underscores mean no NVLink hardware is present, so the n watch has nothing to complain about. Links reported as Down are the false-failure case.

5. Run GPU diagnostics

Diagnostics actively run work on the card, unlike health watches which only observe. There are four levels, and they cost progressively more time:

dcgmi diag -r 1

Level 1 runs the software plugin only, and reports the versions it detected alongside the result:

+---------------------------+------------------------------------------------+
| Diagnostic                | Result                                         |
+===========================+================================================+
|-----  Metadata  ----------+------------------------------------------------|
| DCGM Version              | 4.6.1                                          |
| Driver Version Detected   | 595.91.07                                      |
| GPU Device IDs Detected   | 2237                                           |
|-----  Deployment  --------+------------------------------------------------|
| software                  | Pass                                           |
|                           | GPU0: Pass                                     |
+---------------------------+------------------------------------------------+

Level 2 adds the memory and PCIe plugins:

dcgmi diag -r 2

NVIDIA documents level 2 as taking between 2.5 and 10.5 minutes, but that range assumes a 4-GPU or 8-GPU system. Measured on single-GPU hosts it is far quicker:

Level	Plugins	A10G, 1 GPU	A100, 1 GPU
1	software	1 s	2 s
2	software, memory, pcie	9 s	10 s
3	adds diagnostic, nvbandwidth, memory_bandwidth, targeted_stress, targeted_power	348 s	not measured

Level 3 is a different kind of commitment. It saturates the card for the best part of six minutes, and on a single-GPU host the nvbandwidth plugin reports Skip because there is no peer to measure against:

| software                  | Pass                                           |
| memory                    | Pass                                           |
| diagnostic                | Pass                                           |
| nvbandwidth               | Skip                                           |
| pcie                      | Pass                                           |
+-----  Stress  ------------+------------------------------------------------+
| memory_bandwidth          | Pass                                           |
| targeted_stress           | Pass                                           |
| targeted_power            | Pass                                           |

Run level 3 or 4 on a card that is draining or freshly racked, never on one serving traffic, and keep them out of any cron job that can overlap a real workload. Level 1 is cheap enough to run on every boot.

6. Install dcgm-exporter and expose metrics to Prometheus

The exporter ships in the same repository you already added:

sudo apt-get install -y datacenter-gpu-manager-exporter
dcgm-exporter --version

The package version and the exporter’s own version string differ, which matters when you are matching against upstream release notes:

DCGM Exporter version 4.6.0-4.8.3

Start it and check what it is serving:

sudo systemctl enable --now nvidia-dcgm-exporter
curl -s localhost:9400/metrics | grep -c "^DCGM"

A healthy single-GPU host serves a couple of dozen series. If that count comes back 0, skip ahead to the troubleshooting section, because the exporter fails all-or-nothing and a zero here has two completely different causes.

One metric line shows the label set every rule and dashboard will key on:

DCGM_FI_DEV_GPU_TEMP{gpu="0",UUID="GPU-d3ed7d5d-47b5-609f-56ed-d76ebb18890e",
pci_bus_id="00000000:00:1E.0",device="nvidia0",modelName="NVIDIA A10G",
hostname="ip-10-42-1-55",DCGM_FI_DRIVER_VERSION="595.91.07"} 73

That hostname label is lowercase. The rename landed in 4.8.3, one release after 4.8.2 still emitted Hostname with a capital H, so this breaks on upgrade between two consecutive versions. Any dashboard or alert annotation copied from an older source that references {{ $labels.Hostname }} renders empty now. The two popular community dashboards are unaffected because they key on instance and gpu instead.

7. Enable the clock event, XID, and health metrics

The stock counters file leaves the most useful alerting metrics commented out. Look at what is sitting behind a #:

grep "DCGM_EXP" /etc/dcgm-exporter/default-counters.csv

Six exporter-computed metrics ship disabled:

# DCGM_EXP_CLOCK_EVENTS_COUNT, gauge,   Clock events observed during the configured time window.
# DCGM_EXP_CLOCK_EVENTS_TOTAL, counter, Total clock events observed since exporter start (edge-counted).
# DCGM_EXP_XID_ERRORS_COUNT,   gauge,   XID errors observed during the configured time window.
# DCGM_EXP_XID_ERRORS_TOTAL,   counter, Total XID errors observed since exporter start.
# DCGM_EXP_GPU_HEALTH_STATUS,  gauge,   Current DCGM-reported GPU health status.
# DCGM_EXP_P2P_STATUS,         gauge,   Current NVLink P2P status per peer link.

These matter because of a limitation you cannot work around in PromQL. Field 112 carries the clock event reasons as a raw bitmask. It is not in the stock counters file in any form, commented or otherwise, so you would have to add the line yourself, and even then PromQL has no bitwise operators, which makes testing a single bit cleanly impossible. DCGM_EXP_CLOCK_EVENTS_COUNT decodes the bitmask for you and emits one series per reason.

Copy the stock file and uncomment the five you want:

sudo cp /etc/dcgm-exporter/default-counters.csv /etc/dcgm-exporter/gpu-counters.csv
sudo sed -i 's/^# \(DCGM_EXP_CLOCK_EVENTS\)/\1/; s/^# \(DCGM_EXP_XID_ERRORS\)/\1/; s/^# \(DCGM_EXP_GPU_HEALTH_STATUS\)/\1/' /etc/dcgm-exporter/gpu-counters.csv

Point the unit at your file with a systemd override. Blanking ExecStart before setting it again is required, because systemd appends rather than replaces otherwise:

sudo mkdir -p /etc/systemd/system/nvidia-dcgm-exporter.service.d
sudo vim /etc/systemd/system/nvidia-dcgm-exporter.service.d/override.conf

Add the two lines:

[Service]
ExecStart=
ExecStart=/usr/bin/dcgm-exporter -f /etc/dcgm-exporter/gpu-counters.csv

Reload and restart both services. Restarting the host engine first matters, for reasons covered in troubleshooting:

sudo systemctl daemon-reload
sudo systemctl restart nvidia-dcgm
sudo systemctl restart nvidia-dcgm-exporter

The clock event metric now carries a readable reason:

DCGM_EXP_CLOCK_EVENTS_COUNT{gpu="0",modelName="NVIDIA A10G",hostname="ip-10-42-1-55",
clock_event="power_cap",window_size_in_ms="300000"} 8

The clock_event label takes one of nine values, each mapping to a bit in field 112:

Bit	clock_event label	Meaning
0x001	gpu_idle	Card is idle, clocks dropped on purpose
0x002	clocks_setting	Application clocks were set manually
0x004	power_cap	Software power cap is clamping clocks
0x008	hw_slowdown	Hardware slowdown engaged
0x010	sync_boost	Held back to keep a sync boost group aligned
0x020	sw_thermal	Software thermal slowdown
0x040	hw_thermal	Hardware thermal slowdown
0x080	hw_power_brake	External power brake asserted
0x100	display_clocks	Limited by display clock setting

The health metric is the other prize. It puts the same verdicts dcgmi health prints into Prometheus, one series per watch, with the DCGM error code as a label:

DCGM_EXP_GPU_HEALTH_STATUS{gpu="0",hostname="ip-10-42-1-55",
health_error_category="HARDWARE_POWER",health_error_code="DCGM_FR_CLOCK_THROTTLE_POWER",
health_error_severity="MONITOR",health_watch="POWER"} 10

Watches reported are SM, MEM, PCIE, POWER, THERMAL, NVLINK, DRIVER, INFOROM, MCU, PMU, and ALL. A value of 0 is healthy, 10 is a warning, and 20 is a failure. Worth knowing: the exporter sets its own watches on its own group, so this metric populates even when dcgmi health -g 0 -c still insists that watches are not enabled.

8. Scrape the exporter with Prometheus

If Prometheus is not on this host yet, the Prometheus installation guide covers the binary, user, and unit. Create the rules directory first, because the config below references a rules file and promtool refuses to validate a config that points at a path which does not exist yet:

sudo mkdir -p /etc/prometheus/rules
sudo touch /etc/prometheus/rules/gpu-alerts.yml
sudo chown -R prometheus:prometheus /etc/prometheus/rules

Section 10 fills that file. Now point Prometheus at port 9400:

sudo vim /etc/prometheus/prometheus.yml

The scrape job and the alerting wiring both go in:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/gpu-alerts.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["localhost:9093"]

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: dcgm-exporter
    static_configs:
      - targets: ["localhost:9400"]

On a fleet, replace that static target list with your service discovery of choice and keep the job name, because the alert rules below match on it. Validate before restarting:

promtool check config /etc/prometheus/prometheus.yml
sudo systemctl restart prometheus

Prometheus and promtool disagree about a missing rules file, which is worth knowing before it costs you an afternoon. Prometheus globs rule_files and starts happily with zero rules loaded, while promtool check config hard-errors with does not point to an existing file and exits non-zero. The dangerous half is Prometheus: the service looks healthy and no rules are evaluated.

Check the target went green in the Prometheus UI under Status then Targets.

With the target up, GPU telemetry is queryable. Graphing DCGM_FI_DEV_POWER_USAGE across a load test shows the card jumping from its idle draw to the power ceiling and staying there.

The expression browser is fine for confirming a metric exists. It is not where you want to be at three in the morning, which is what the next section fixes.

9. Build the GPU dashboard in Grafana

Install Grafana if it is not already running, following the Grafana setup guide, then add Prometheus as a datasource. Provisioning it from a file survives reinstalls better than clicking through the UI:

sudo vim /etc/grafana/provisioning/datasources/prometheus.yml

Four lines of configuration are enough:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://localhost:9090
    isDefault: true

Restart Grafana to pick it up. If you need to reset the admin password on a current release, the old grafana-cli wrapper now fails with Grafana-server Init Failed: Could not find config defaults unless you pass a homepath, and it prints a deprecation notice pointing at the new subcommand:

sudo grafana cli --homepath /usr/share/grafana admin reset-admin-password 'YourStrongPassword'

Build panels around the four fleet signals, then add clocks and clock events beside them so a throttled card is visible at a glance rather than hidden behind a green utilisation number.

The queries behind those panels are short. Framebuffer percentage has to be computed because DCGM exports used and free separately rather than a ratio:

DCGM_FI_DEV_GPU_UTIL
DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE) * 100
DCGM_FI_DEV_POWER_USAGE
DCGM_FI_DEV_GPU_TEMP
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE * 100
DCGM_EXP_CLOCK_EVENTS_COUNT

Put a threshold line on the power panel at the card’s enforced limit and on the temperature panel at your warning level. A flat line riding the ceiling is the visual signal that the card is capped.

Below those, the clock event panel and a table of DCGM health watches turn two of the harder-to-read signals into something you can scan.

If you would rather start from something prebuilt, dashboards 12239 and 15117 on grafana.com both target dcgm-exporter and work unmodified against current releases.

Import both and you will find only one. They ship the same dashboard UID, so importing the second overwrites the first. Give one of them a fresh UID during import if you want to keep both.

10. Alert on GPU utilisation, memory, power, and thermal health

Fill in the rules file created back in section 8:

sudo vim /etc/prometheus/rules/gpu-alerts.yml

These rules cover the four signals plus the hardware faults that end a GPU’s life:

groups:
  - name: gpu-availability
    rules:
      - alert: GPUExporterDown
        expr: up{job="dcgm-exporter"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "DCGM exporter on {{ $labels.instance }} is not responding"

  - name: gpu-thermal
    rules:
      - alert: GPUTemperatureHigh
        expr: DCGM_FI_DEV_GPU_TEMP > 80
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "GPU {{ $labels.gpu }} on {{ $labels.hostname }} at {{ $value }}C"

      - alert: GPUTemperatureCritical
        expr: DCGM_FI_DEV_GPU_TEMP > 90
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "GPU {{ $labels.gpu }} on {{ $labels.hostname }} at {{ $value }}C"
          description: "Thermal slowdown is imminent. Check airflow and inlet temperature."

      - alert: GPUThermalSlowdown
        expr: DCGM_EXP_CLOCK_EVENTS_COUNT{clock_event=~"sw_thermal|hw_thermal"} > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "GPU {{ $labels.gpu }} is in thermal slowdown ({{ $labels.clock_event }})"

  - name: gpu-power-and-clocks
    rules:
      - alert: GPUPowerCapThrottling
        expr: DCGM_EXP_CLOCK_EVENTS_COUNT{clock_event="power_cap"} > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "GPU {{ $labels.gpu }} on {{ $labels.hostname }} is clamped by its power cap"
          description: "The card is drawing its full budget and running below boost clock."

      - alert: GPUHardwareSlowdown
        expr: DCGM_EXP_CLOCK_EVENTS_COUNT{clock_event=~"hw_slowdown|hw_power_brake"} > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "GPU {{ $labels.gpu }} in hardware slowdown ({{ $labels.clock_event }})"

  - name: gpu-memory
    rules:
      - alert: GPUMemoryNearlyFull
        expr: >
          DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE) * 100 > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "GPU {{ $labels.gpu }} framebuffer {{ $value | printf \"%.1f\" }}% full"
          description: "The next allocation is likely to fail with an out of memory error."

  - name: gpu-hardware-health
    rules:
      - alert: GPUHealthDegraded
        expr: DCGM_EXP_GPU_HEALTH_STATUS{health_error_severity!="MONITOR"} > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "DCGM health watch {{ $labels.health_watch }} degraded on GPU {{ $labels.gpu }}"
          description: "Error {{ $labels.health_error_code }} on {{ $labels.hostname }}."

      - alert: GPUXidError
        expr: DCGM_EXP_XID_ERRORS_COUNT > 0
        labels:
          severity: critical
        annotations:
          summary: "XID error on GPU {{ $labels.gpu }} of {{ $labels.hostname }}"
          description: "Cross-reference the code in dmesg before recycling the node."

      - alert: GPURowRemapFailure
        expr: DCGM_FI_DEV_ROW_REMAP_FAILURE > 0
        labels:
          severity: critical
        annotations:
          summary: "Row remap failure on GPU {{ $labels.gpu }}"
          description: "The card is out of spare memory rows. Drain the node and replace the GPU."

      - alert: GPUUncorrectableRemappedRows
        expr: increase(DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS[30m]) > 0
        labels:
          severity: warning
        annotations:
          summary: "Uncorrectable memory row remapped on GPU {{ $labels.gpu }}"

  - name: gpu-utilisation
    rules:
      - alert: GPUSaturated
        expr: avg_over_time(DCGM_FI_DEV_GPU_UTIL[10m]) > 90
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "GPU {{ $labels.gpu }} saturated at {{ $value | printf \"%.0f\" }}%"

      - alert: GPUIdleButAllocated
        expr: avg_over_time(DCGM_FI_PROF_GR_ENGINE_ACTIVE[1h]) < 0.05
        for: 1h
        labels:
          severity: info
        annotations:
          summary: "GPU {{ $labels.gpu }} on {{ $labels.hostname }} nearly idle for an hour"
          description: "This card is being paid for and not used."

Validate and reload without a restart:

promtool check rules /etc/prometheus/rules/gpu-alerts.yml
sudo systemctl kill -s HUP prometheus

Use the signal, not the HTTP endpoint. Plenty of guides reach for curl -X POST http://localhost:9090/-/reload, but the management API is disabled unless Prometheus was started with --web.enable-lifecycle. Without the flag it answers 403 Lifecycle API is not enabled. and curl still exits 0, so you get a clean-looking terminal and rules that never loaded. Add the flag to the unit if you want that endpoint; SIGHUP works on a stock install either way. Prefer the unit-scoped form above over kill -HUP $(pidof prometheus): if the service is down, pidof returns nothing and the command collapses into a bare kill -HUP that prints usage text, which is a confusing thing to meet while debugging a dead service.

Three of those rules deserve an explanation, because they came out of watching the rules misbehave on a real card.

GPUHealthDegraded filters out MONITOR severity deliberately. A GPU running flat out at its power limit sets the POWER watch to 10 with severity MONITOR, so an unfiltered > 0 rule fires on every healthy busy GPU in the fleet. Filter it out and you still catch the watches that mean something. The same rule needs a second exclusion on PCIe fleets: the exporter always watches every category including NVLINK, and the down-link error carries triage severity rather than MONITOR, so the false failure described in section 4 fires here permanently too. Add health_watch!="NVLINK" to the selector on hosts with unconnected NVLink hardware.

The framebuffer percentage reads very slightly high. There is no total-memory field, so the expression divides used by used plus free, and DCGM reports a third bucket, DCGM_FI_DEV_FB_RESERVED, that is left out of the denominator. On a 24 GB card the reserved slice is a few hundred MiB, enough to reach a > 90 threshold marginally sooner than the card really does.

Two rules also take twice as long to fire as they look. GPUSaturated averages over 10 minutes and then waits for: 10m, so first notification is around 20 minutes in; GPUIdleButAllocated does the same over an hour and needs two hours of history behind it. Neither is broken when it stays quiet on a fresh Prometheus.

GPUIdleButAllocated is a cost rule rather than a fault rule. On rented GPUs, an idle card with the graphics engine below 5 percent for an hour is money leaving the building, and nothing else in this rule set would tell you.

The throttle rules deliberately do not use the violation counters. DCGM exposes power_violation (field 240) and per-reason nanosecond counters like clocks_event_power_cap_ns (field 1420), and they look like the obvious thing to put a rate() on. On an A100 running 40 seconds of continuous, confirmed power capping, both stayed frozen at exactly the same value before and after. So did nvidia-smi’s own SW Power Capping counter, which means the underlying NVML accumulator was not advancing rather than DCGM misreporting it. A rate() alert on those fields would never have fired. The exporter’s clock event metrics are edge-counted independently, which is why they work.

With a load running, the rules move through pending into firing.

One threshold will not fire on every card, and that is correct. The A10G peaked at 73 C under sustained full load, so GPUTemperatureHigh at 80 C stayed quiet. Those thresholds are sized for datacenter cards in a warm aisle, where 80 C is a genuine early warning ahead of the hardware’s own 95 C slowdown point.

11. Route GPU alerts through Alertmanager

Alertmanager handles grouping and delivery. Group by GPU and host so a bad node produces one notification rather than one per rule:

sudo vim /etc/alertmanager/alertmanager.yml

The grouping keys are the important part:

route:
  receiver: gpu-oncall
  group_by: ["alertname", "gpu", "hostname"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h

receivers:
  - name: gpu-oncall
    webhook_configs:
      - url: http://127.0.0.1:5001/alerts

Swap that webhook receiver for Slack, email, or PagerDuty using the receiver configuration in the Alertmanager setup guide. Firing alerts arrive grouped by the labels above:

Notice the grouping labels use lowercase hostname. A group_by naming Hostname would silently group everything into one bucket, because the label would not exist.

12. Run dcgm-exporter on Kubernetes

On a cluster, the exporter runs as a DaemonSet. The NVIDIA GPU Operator installs it alongside the driver and device plugin, or you can deploy the chart on its own:

helm repo add gpu-helm-charts https://nvidia.github.io/dcgm-exporter/helm-charts
helm repo update
helm install dcgm-exporter gpu-helm-charts/dcgm-exporter --namespace gpu-monitoring --create-namespace

The chart’s default security context is the part to read before you tighten anything:

securityContext:
  runAsNonRoot: false
  runAsUser: 0
  capabilities:
    add: ["SYS_ADMIN"]  # Required for profiling metrics (DCGM_FI_PROF_*)
    drop: ["ALL"]
  allowPrivilegeEscalation: false

That SYS_ADMIN capability is load-bearing. Strip it to satisfy a pod security policy and the profiling fields stop working, which takes the whole exporter down rather than degrading it. The chart’s comment covers half of it, telling you to set runAsNonRoot: true, runAsUser: 1000 and drop SYS_ADMIN for a non-root install without profiling metrics. It does not mention the other half, which the failure above makes clear: you must also remove the DCGM_FI_PROF_* lines from the counters file, or the exporter serves nothing at all.

The chart enables a ServiceMonitor by default, so a kube-prometheus-stack install picks the target up with no extra work:

serviceMonitor:
  enabled: true
  interval: 30s
  scrapeTimeout: 25s

Section 7 has a Kubernetes counterpart, and skipping it quietly disarms five of the rules. The chart mounts its own ConfigMap over /etc/dcgm-exporter/default-counters.csv, and in that ConfigMap every DCGM_EXP_* line is commented out exactly as it is in the deb. So on a default chart install GPUThermalSlowdown, GPUPowerCapThrottling, GPUHardwareSlowdown, GPUHealthDegraded, and GPUXidError reference metrics that do not exist. Supply your own counters file instead, remembering that it replaces the list rather than adding to it:

helm upgrade --install dcgm-exporter gpu-helm-charts/dcgm-exporter \
  --namespace gpu-monitoring --create-namespace \
  --set-file customMetrics=/etc/dcgm-exporter/gpu-counters.csv

That is the edited file from section 7, and it has to be readable from wherever you run helm, not from the GPU node. Copy it to your workstation first if the two are different machines. With the right file in place the same rule set works on a cluster. Two other defaults catch people out: the chart only tolerates the control-plane taint, so GPU nodes tainted nvidia.com/gpu=present:NoSchedule need that toleration added, and installing this chart on a cluster already running the GPU Operator gives you two exporters, because the Operator ships its own.

With the counters file sorted, the alert rules from section 10 carry over to a Prometheus and Grafana Helm stack as a PrometheusRule. They are also the missing piece for anyone running vLLM on Kubernetes with NVIDIA GPUs, where knowing whether a card is thermally capped explains a lot of unexplained latency.

Troubleshooting the errors you will actually hit

Error: “Failed to watch DCGM fields … Host engine is running as non-root”

The exporter starts, logs this, and exits. curl localhost:9400/metrics returns zero DCGM metrics, not a partial set:

level=ERROR msg="Failed to watch DCGM fields" entity_type=GPU
  error="error watching fields: Host engine is running as non-root"
level=ERROR msg="DCGM collector for entity type 'GPU' cannot be initialized"

The message points at the wrong thing. On a container where this reproduces, the host engine was already running as root. The real blocker is the missing CAP_SYS_ADMIN capability, which the profiling fields require. The 4.x images are distroless, so there is no shell to run capsh in. Check from the host instead with docker inspect --format '{{.HostConfig.CapAdd}}' <container>, which prints [] when nothing was granted. Note that this reports what was requested at run time rather than the effective capability set the process ended up with, so it answers this particular question and is not a drop-in for capsh. Then add the capability:

docker run --gpus all --cap-add SYS_ADMIN -p 9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:4.6.0-4.8.3-distroless

The reason a single unwatchable field kills everything is that the exporter puts all 26 default fields into one field group and watches them in one call. Five of those are profiling fields. If any one field in the group cannot be watched, the group fails and the collector never initialises, so you get zero metrics instead of the 21 that would have worked. Proving it takes one command: strip the DCGM_FI_PROF_* lines from a copy of the counters file and the same exporter serves metrics happily.

Error: “The third-party Profiling module returned an unrecoverable error”

Same zero-metrics symptom, completely different cause, and this one bites on bare metal with full root:

level=ERROR msg="Failed to watch DCGM fields" entity_type=GPU
  error="error watching fields: The third-party Profiling module returned an unrecoverable error"
systemd[1]: nvidia-dcgm-exporter.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: nvidia-dcgm-exporter.service: Scheduled restart job, restart counter is at 4

This trips people up because the natural order of work causes it. You prove the profiling fields work with dcgmi dmon, then start the exporter, and the exporter fails. Watching the profiling fields with dcgmi leaves the profiling module in a state the next client cannot acquire. Three runs, restarting the host engine before each, isolate it:

Sequence after restarting nvidia-dcgm	Exporter result
Start the exporter immediately	Works
Run dcgmi dmon -e 1001,1004,1005 first	Fails, unrecoverable profiling error
Run dcgmi dmon -e 150,155,203 first	Works

Only the profiling fields, the ones numbered 1001 and above, cause it. Ordinary fields do not. The fix is one restart, and the ordering rule is to always bounce the host engine before starting the exporter after any manual dcgmi dmon profiling session:

sudo systemctl restart nvidia-dcgm
sudo systemctl restart nvidia-dcgm-exporter

Restart the host engine first and the exporter second, because the exporter has to acquire the profiling module from a freshly started engine.

Error: “Health watches not enabled. Please enable watches.”

dcgmi health -g 0 -c returns this even while the exporter is happily publishing DCGM_EXP_GPU_HEALTH_STATUS. Both are true at once. The exporter creates its own group and sets its own watches, and group 0 that the CLI is asking about is a different group. Enable watches for the CLI separately with dcgmi health -g 0 -s pmit.

PARSE ERROR: Argument: -v Couldn’t find match for argument

Guides written against DCGM 2.x and 3.x use -v for verbose health output. The flag is gone from dcgmi health specifically; it still exists on dcgmi diag and dcgmi discovery, which is why the error looks inconsistent. Use -j for JSON output instead, which is more useful for scripting anyway.

A memory temperature alert that never fires

Field 140 is not populated on every card. The A10G reports 0 for DCGM_FI_DEV_MEMORY_TEMP, and nvidia-smi -q -d TEMPERATURE confirms Memory Current Temp : N/A. The A100 reports it properly at 58 C under load. Before you rely on a memory temperature rule, check the metric is non-zero on the cards you actually run, or the rule is decoration.

Everything above stands on the exporter staying alive, so the one rule to keep even if you drop the rest is GPUExporterDown. A GPU fleet with a dead exporter looks exactly like a GPU fleet with no problems.