infiniband collector fails on MANA RDMA devices before device exclusion is applied

[chmill-zz]

Host operating system: output of uname -a

Observed on an Azure Ubuntu 22.04 VM using a Microsoft Azure Network Adapter (MANA). The public MANA documentation notes that Linux kernel 6.2 adds InfiniBand/RDMA support:

https://learn.microsoft.com/azure/virtual-network/accelerated-networking-mana-linux

The upstream driver is also under drivers/infiniband/hw/mana:

https://github.com/torvalds/linux/tree/master/drivers/infiniband/hw/mana

node_exporter version: output of node_exporter --version

node_exporter, version v1.12.1

This is the Microsoft Dalec-built node-exporter-kubernetes package from the upstream v1.12.1 source.

node_exporter command line flags

The collector is enabled by default. We also tried the new v1.12 device exclusion:

--collector.infiniband.device-exclude=^mana_.*

node_exporter log output

level=ERROR source=collector.go:168 msg="collector failed" name=infiniband err="error obtaining InfiniBand class info: read /sys/class/infiniband/mana_0/ports/1/rate: invalid argument"

Are you running node_exporter in Docker?

No. It runs directly as a systemd service on the VM.

What did you do that produced an error?

Scraped /metrics on a MANA-enabled Azure VM. MANA registers mana_0 under /sys/class/infiniband, but reading /sys/class/infiniband/mana_0/ports/1/rate returns EINVAL.

Adding --collector.infiniband.device-exclude=^mana_.* does not avoid the error. In v1.12.1, infinibandCollector.Update first calls c.fs.InfiniBandClass(), which fully parses every device, and applies c.deviceFilter.ignored(device.Name) only afterward. The failure therefore happens before the exclusion is evaluated.

Relevant ordering:

https://github.com/prometheus/node_exporter/blob/v1.12.1/collector/infiniband_linux.go

The underlying procfs parser treats rate as required and propagates its read error:

https://github.com/prometheus/procfs/blob/v0.21.1/sysfs/class_infiniband.go

What did you expect to see?

At minimum, the device exclusion should prevent node_exporter from parsing mana_*, allowing operators to avoid unsupported device attributes while retaining metrics for other HCAs on mixed-device nodes.

Ideally, procfs/node_exporter could also support MANA directly or tolerate an unavailable rate attribute if the remaining MANA RDMA data is useful.

What did you see instead?

The whole InfiniBand collector fails and reports:

node_scrape_collector_success{collector="infiniband"} 0

On mixed MANA and Mellanox nodes, the late filter means there is currently no configuration-only way to ignore MANA and retain collection from the other HCAs. The only working operator mitigation is --no-collector.infiniband, which disables all InfiniBand/RDMA metrics.

This resembles #3265 / prometheus/procfs#704, but that fix skips EINVAL for individual counter files. This failure is on the required port rate file and occurs before the v1.12 device filter is applied.

[chmill-zz]

Additional context: direct MANA support would be useful, rather than treating MANA as a device that should always be ignored.

The public Linux MANA RDMA driver exports a substantial set of hardware statistics through the standard RDMA stats interface, including requester/responder errors, retry exhaustion, CNP/ECN and congestion events, current rate, TX/RX bytes and packets, and RDMA read/write request counts:

https://github.com/torvalds/linux/blob/master/drivers/infiniband/hw/mana/counters.c

MANA also implements query_port, derives active speed/width through ib_get_eth_speed, and reports an Ethernet link layer:

https://github.com/torvalds/linux/blob/master/drivers/infiniband/hw/mana/main.c

So the preferred outcome seems to be:

1. Support MANA attributes and hardware counters in procfs/node_exporter where they map cleanly.
2. Independently move include/exclude filtering before full device parsing, so one unsupported or broken RDMA device cannot fail collection for every HCA.

The second item remains valuable even with direct MANA support because mixed-device failures have happened with other RDMA drivers as well. Until one of those is available, operators can only disable the entire collector on MANA nodes.