Conversation
Motivation: --collector.infiniband.device-exclude (and device-include) correctly filtered excluded devices out of emitted metrics, but the collector's Update() called sysfs.FS.InfiniBandClass(), which eagerly parses every device under /sys/class/infiniband -- including reading each port's counter files -- before the device filter is ever applied. On hosts with firmware-managed restricted ports, reading those counter files can trigger a firmware command that fails and floods the kernel log, even though the offending device was explicitly excluded. Approach: Enumerate device names first via sysfs.FS.InfiniBandClassDevices(), which only lists directory entries and performs no attribute or counter reads. Apply the existing device filter against the name, and only call sysfs.FS.InfiniBandDevice(name) -- which does the actual sysfs reads -- for devices that pass the filter. This does not change emitted metrics for included devices; it only avoids touching excluded devices' sysfs files at all. Validation: Added TestInfiniBandCollectorSkipsSysfsReadsForExcludedDevices, which builds a synthetic sysfs tree with a healthy device and a device missing its required fw_ver file (simulating a device whose attributes/counters cannot be read), then excludes the broken device via device-exclude and asserts Update() succeeds and only the included device's metrics are emitted. Report: prometheus#3823 Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com> Assisted-by: claude-sonnet-5 (via Claude Code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation:
--collector.infiniband.device-exclude (and device-include) correctly
filtered excluded devices out of emitted metrics, but the collector's
Update() called sysfs.FS.InfiniBandClass(), which eagerly parses every
device under /sys/class/infiniband -- including reading each port's
counter files -- before the device filter is ever applied. On hosts
with firmware-managed restricted ports, reading those counter files
can trigger a firmware command that fails and floods the kernel log,
even though the offending device was explicitly excluded.
Approach:
Enumerate device names first via sysfs.FS.InfiniBandClassDevices(),
which only lists directory entries and performs no attribute or
counter reads. Apply the existing device filter against the name,
and only call sysfs.FS.InfiniBandDevice(name) -- which does the
actual sysfs reads -- for devices that pass the filter. This does
not change emitted metrics for included devices; it only avoids
touching excluded devices' sysfs files at all.
Validation:
Added TestInfiniBandCollectorSkipsSysfsReadsForExcludedDevices, which
builds a synthetic sysfs tree with a healthy device and a device
missing its required fw_ver file (simulating a device whose
attributes/counters cannot be read), then excludes the broken device
via device-exclude and asserts Update() succeeds and only the
included device's metrics are emitted.
Report: #3823
Signed-off-by: Pujitha Paladugu 10557236+pujitha24@users.noreply.github.com
Assisted-by: claude-sonnet-5 (via Claude Code)