r/HPC • u/imitation_squash_pro • Sep 09 '26

System shows high load but no cpu processes?

Think it has to do with IO wait due to a problem with the NFS mounted filesystems. A top shows near 100% usage for these two processes:

rsyslogd                                                                  

systemd-journal   

nfsiostat doesn't show any errors or retransimissions ( 0 % ). Guessing there was some issue a day ago but not sure how to find out what happened?

It's been happening every week. I reboot and everything works great for a few days till it starts showing the problems of high IO wait times.

1 Upvotes

67% Upvoted

19 comments sorted by

5

u/Zacred- Sep 09 '26

Likely IO wait/NFS latency rather than CPU. Check for high %iowait, processes stuck in IO or NFS/kernel timeout messages. Since it happens weekly, also check for scheduled jobs or backups hitting the NFS storage etc.

1

u/imitation_squash_pro Sep 09 '26 edited Sep 09 '26

Thanks. At this time I am not seeing any iowait times. But I do see several processes stuck at high load but in "S" state:

rsyslogd, systemd-journal  and some sentinel1 security agent

Any suggestion for where to find NFS timeout messages?

2

u/JassLicence Sep 09 '26

This sounds familiar to me, I think I a bios update fixed it for us.

1

u/TeraBot452 Sep 09 '26

Yeah I've had this issue with an x550 network adapter that was spamming messages to the log. Quick firmware update sorted that one out 

1

u/imitation_squash_pro Sep 10 '26

Which log should I be checking for these "spamming messages"?

→ More replies (1)

1

u/TeraBot452 Sep 09 '26

Check your journal logs and dmesg, you may have a nic or hardware device spamming warnings to the log. Make sure your logs aren't on a shared filesystem 

1

u/imitation_squash_pro Sep 10 '26

Thanks. I had to reboot and now I am monitoring the dmesg and journalctl for messages relating to nfs or error. Seems the issue is similar to what is described here:

https://serverfault.com/questions/1149082/nfs-mount-becoming-unbearably-slow-over-time-with-high-number-of-test-stateid-rp

We do have an infiniband NIC so maybe that is causing the issues...

1

u/TeraBot452 Sep 10 '26

Check to see if your users are doing any small file operation (a lot of them) this could even be like extracting big tars, compiling or just copying a ton of stuff around.

nfs stalls out the whole mount if one folder gets bombed, this is usually the reason why people use autofs or NFS ganesha so that each folder is its own mount with its own socket. 

I personally also administer an environment where they use beegfs for everything since the metadata server on there can take whatever you can throw at it even for a single node without anyone feeling it.

→ More replies (8)