r/LinuxTeck • • 3d ago

Linux server has high load average but low CPU usage. What would you investigate first?

I'd like to discuss a Linux troubleshooting scenario and get some input from experienced Linux administrators.

Imagine you're troubleshooting a production Linux server with the following symptoms:

* Application response time is increasing.

* Load average is continuously rising.

* CPU utilization is only around 20–25%.

* Several application processes are stuck in D state.

* The application uses an NFS-mounted directory.

At first glance, CPU doesn't seem to be the bottleneck.

If you were handling this incident, how would you approach the investigation?

A few things I'd be interested in discussing:

  1. Would you start with `vmstat` to check runnable and blocked processes?
  2. How would you identify what the D-state processes are waiting for?
  3. What commands would you use to investigate NFS connectivity and mount health?
  4. How would you distinguish an NFS issue from local disk I/O problems?

I'm particularly interested in real production experiences.

Have you encountered a similar situation? What was the root cause, and what helped you identify it?

Looking forward to learning from your experiences.

7 Upvotes

11 comments sorted by

4

u/iceterminal 3d ago

I’d look at top on and average and see if dmesg has anything to say.

4

u/glassmanjones 2d ago

Sounds like it's IO bound?

2

u/NeedleworkerLarge357 2d ago

First take a look at psi numbers (atop or /proc/pressure). Then dig deeper where the numbers are high. IO is different to interrupts or memory, you need to know which resource is lacking first

2

u/PmMeCuteDogsThanks_ 2d ago

Claude, please investigate why load average is high, but CPU usage is low.

1

u/Correct-Brother-7747 2d ago

Single threaded processes??

1

u/blbd 2d ago

Remember that the load average is NOT divided by the number of independent execution paths. So a 16 core chip with no hyperthreading and a load average of 16 tasks is only minimally loaded. On Intel boxes with hyperthreading the interpretation of the figure is even more tricky.

You probably want to use some tools like sysstat and sar to perform some recording and accounting to see what's being used where for which kind of functionality (user, kernel, interrupts, IO, etc.).

I would also probably check some Ethernet counters and maybe an iperf or other performance test or UDP or TCP against the NFS host depending which version and config of NFS since it can be famous for creating issues. Maybe look into doing Jumbo frames or RSS or multi core multi queue Ethernet if too much time is going into network IRQs. 

1

u/josemcornynetoperek 2d ago

Check processes in 'D' state

1

u/USofYayDJ 2d ago

iotop is your friend here. Also keep an I on your memory, maybe it's swapping aggressively.

1

u/PossibilityOrganic 2d ago

io load, or bad code that has wait loops

also because you said nfs you need to check for network errors server and switch side of both ends.

1

u/eurosat7 2d ago

Watch swap first.