Metrics stuck / deadlock on some kernel versions

[nevgin]

Host operating system: output of uname -a

Linux host 5.4.211-1.el8.elrepo.x86_64 #1 SMP Wed Aug 24 08:50:28 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.4.0 (branch: HEAD, revision: 7da1321)
build user: root@83d90983e89c
build date: 20220926-12:32:56
go version: go1.19.1
platform: linux/amd64

node_exporter command line flags

--no-collector.cpufreq --no-collector.systemd --collector.sockstat --no-collector.logind --collector.textfile.directory=/var/lib/prometheus

Are you running node_exporter in Docker?

No

What did you do that produced an error?

after 2-24 hour, randomized

What did you expect to see?

Not stuck node_exporter

What did you see instead?

Stuck node exporter after random time.
After run strace -f -p pgrep node_exporter
node exporter unstuck
Out pprof:
debug1.txt
debug2.txt

Responce > 1 minute

[discordianfish]

So it stops responding to requests?
Can you please attach the log?

[nevgin]

So.
node_exporter.tar.gz

[discordianfish]

Have you tried using e.g curl to retrieve the metrics? Might be an issue on the networking/prometheus server side.
There is also an unexpected number of irq/idle counter jumps..

[stephrdev]

I can confirm the same issue, even a curl running against localhost:9100 is hanging until the exporter is restarted. After some undefined time the exporter responds again.

I noticed this first after a recent kernel update but I'm not able to specify the exact kernel version when the issues came up first. I can also confirm that this happend with a quite old node exporter version and also with the latest 1.4.0.

After trying to find out more about the issue using log.level=debug I can say that the log didn't showed any helpful information.

[nevgin]

Yes of course

[nevgin]

Issue same on all system with kernel version 5.4.211, 5.4.212

[ventifus]

Is it intentional that the system is getting scraped 24 times per second? 60 seconds to scrape /metrics is unusually long probably caused by kernel-level rate limits. It would mean you have 1440 concurrent scrapes at any given time. I usually see scrape intervals of 10 or 15 seconds.

egrep 'ts=2022-10-05T13:47:59.*name=uname' ~/Downloads/node_exporter/tmp/node_exporter Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.349Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=2.9704e-05
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.355Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=0.004351489
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.357Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=0.000320838
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.360Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=0.00315381
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.364Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=0.009627007
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.354Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=0.002329661
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.367Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=0.00543363
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.370Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=0.008927494
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.372Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=0.011827477
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.376Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=0.014796259
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.380Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=6.984e-06
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.381Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=4.1388e-05
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.381Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=4.2767e-05
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.383Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=0.028515173
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.389Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=0.000264836
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.389Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=1.9946e-05
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.373Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=0.000142291
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.386Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=0.000344837
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.397Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=0.017266906
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.398Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=1.0905e-05
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.401Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=0.000125761
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.404Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=0.000226375
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.410Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=7.172e-06
Oct 05 16:47:59 host.net node_exporter[2947760]: ts=2022-10-05T13:47:59.415Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=0.019929624

[nevgin]

Metrics are collected by the exporter_exporter. He stupidly scrapes metrics, regardless of whether he gave the metrics or not. Here is the moment at which the node_exporter began to respond and fulfilled all requests.

[nevgin]

Scrapping every 5seconds.
Log:
node_exporter.log

[discordianfish]

Can you attach the binary pprof profiles? e.g:

for p in goroutine block mutex profile; do
  curl -O localhost:9100/debug/pprof/$p
done

[nevgin]

Yes of course.
out.tar.gz

[nadavgolden]

Any news on this?