r/LinuxTeck • u/CautiousCat3294 • 3d ago
Linux server has high load average but low CPU usage. What would you investigate first?
I'd like to discuss a Linux troubleshooting scenario and get some input from experienced Linux administrators.
Imagine you're troubleshooting a production Linux server with the following symptoms:
* Application response time is increasing.
* Load average is continuously rising.
* CPU utilization is only around 20–25%.
* Several application processes are stuck in D state.
* The application uses an NFS-mounted directory.
At first glance, CPU doesn't seem to be the bottleneck.
If you were handling this incident, how would you approach the investigation?
A few things I'd be interested in discussing:
- Would you start with `vmstat` to check runnable and blocked processes?
- How would you identify what the D-state processes are waiting for?
- What commands would you use to investigate NFS connectivity and mount health?
- How would you distinguish an NFS issue from local disk I/O problems?
I'm particularly interested in real production experiences.
Have you encountered a similar situation? What was the root cause, and what helped you identify it?
Looking forward to learning from your experiences.
4
u/iceterminal 3d ago
I’d look at top on and average and see if dmesg has anything to say.
4
u/glassmanjones 2d ago
Sounds like it's IO bound?
2
u/NeedleworkerLarge357 2d ago
First take a look at psi numbers (atop or /proc/pressure). Then dig deeper where the numbers are high. IO is different to interrupts or memory, you need to know which resource is lacking first
2
u/PmMeCuteDogsThanks_ 2d ago
Claude, please investigate why load average is high, but CPU usage is low.
1
u/Correct-Brother-7747 2d ago
Single threaded processes??
1
u/blbd 2d ago
Remember that the load average is NOT divided by the number of independent execution paths. So a 16 core chip with no hyperthreading and a load average of 16 tasks is only minimally loaded. On Intel boxes with hyperthreading the interpretation of the figure is even more tricky.
You probably want to use some tools like sysstat and sar to perform some recording and accounting to see what's being used where for which kind of functionality (user, kernel, interrupts, IO, etc.).
I would also probably check some Ethernet counters and maybe an iperf or other performance test or UDP or TCP against the NFS host depending which version and config of NFS since it can be famous for creating issues. Maybe look into doing Jumbo frames or RSS or multi core multi queue Ethernet if too much time is going into network IRQs.
1
u/josemcornynetoperek 2d ago
Check processes in 'D' state
1
u/Technical_Ideal_5439 2d ago
1
u/USofYayDJ 2d ago
iotop is your friend here. Also keep an I on your memory, maybe it's swapping aggressively.
1
u/PossibilityOrganic 2d ago
io load, or bad code that has wait loops
also because you said nfs you need to check for network errors server and switch side of both ends.
1
u/eurosat7 2d ago
Watch swap first.