Skip to content

fix(infiniband): skip sysfs reads for excluded devices - #3824

Open
pujitha24 wants to merge 1 commit into
prometheus:masterfrom
pujitha24:auto/issue-3823
Open

pujitha24 wants to merge 1 commit into
prometheus:masterfrom
pujitha24:auto/issue-3823

Conversation

@pujitha24

Copy link
Copy Markdown

Motivation:
--collector.infiniband.device-exclude (and device-include) correctly
filtered excluded devices out of emitted metrics, but the collector's
Update() called sysfs.FS.InfiniBandClass(), which eagerly parses every
device under /sys/class/infiniband -- including reading each port's
counter files -- before the device filter is ever applied. On hosts
with firmware-managed restricted ports, reading those counter files
can trigger a firmware command that fails and floods the kernel log,
even though the offending device was explicitly excluded.

Approach:
Enumerate device names first via sysfs.FS.InfiniBandClassDevices(),
which only lists directory entries and performs no attribute or
counter reads. Apply the existing device filter against the name,
and only call sysfs.FS.InfiniBandDevice(name) -- which does the
actual sysfs reads -- for devices that pass the filter. This does
not change emitted metrics for included devices; it only avoids
touching excluded devices' sysfs files at all.

Validation:
Added TestInfiniBandCollectorSkipsSysfsReadsForExcludedDevices, which
builds a synthetic sysfs tree with a healthy device and a device
missing its required fw_ver file (simulating a device whose
attributes/counters cannot be read), then excludes the broken device
via device-exclude and asserts Update() succeeds and only the
included device's metrics are emitted.

Report: #3823
Signed-off-by: Pujitha Paladugu 10557236+pujitha24@users.noreply.github.com
Assisted-by: claude-sonnet-5 (via Claude Code)

Motivation:
--collector.infiniband.device-exclude (and device-include) correctly
filtered excluded devices out of emitted metrics, but the collector's
Update() called sysfs.FS.InfiniBandClass(), which eagerly parses every
device under /sys/class/infiniband -- including reading each port's
counter files -- before the device filter is ever applied. On hosts
with firmware-managed restricted ports, reading those counter files
can trigger a firmware command that fails and floods the kernel log,
even though the offending device was explicitly excluded.

Approach:
Enumerate device names first via sysfs.FS.InfiniBandClassDevices(),
which only lists directory entries and performs no attribute or
counter reads. Apply the existing device filter against the name,
and only call sysfs.FS.InfiniBandDevice(name) -- which does the
actual sysfs reads -- for devices that pass the filter. This does
not change emitted metrics for included devices; it only avoids
touching excluded devices' sysfs files at all.

Validation:
Added TestInfiniBandCollectorSkipsSysfsReadsForExcludedDevices, which
builds a synthetic sysfs tree with a healthy device and a device
missing its required fw_ver file (simulating a device whose
attributes/counters cannot be read), then excludes the broken device
via device-exclude and asserts Update() succeeds and only the
included device's metrics are emitted.

Report: prometheus#3823
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
Assisted-by: claude-sonnet-5 (via Claude Code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant