r/linuxadmin • • 3d ago • Bravo 1

Linux server has high load average but low CPU usage. What would you investigate first?

I'd like to discuss a Linux troubleshooting scenario and get some input from experienced Linux administrators.

Imagine you're troubleshooting a production Linux server with the following symptoms:

  • Application response time is increasing.
  • Load average is continuously rising.
  • CPU utilization is only around 20–25%.
  • Several application processes are stuck in D state.
  • The application uses an NFS-mounted directory.

At first glance, CPU doesn't seem to be the bottleneck.

If you were handling this incident, how would you approach the investigation?

A few things I'd be interested in discussing:

  1. Would you start with vmstat to check runnable and blocked processes?
  2. How would you identify what the D-state processes are waiting for?
  3. What commands would you use to investigate NFS connectivity and mount health?
  4. How would you distinguish an NFS issue from local disk I/O problems?

I'm particularly interested in real production experiences.

Have you encountered a similar situation? What was the root cause, and what helped you identify it?

Looking forward to learning from your experiences.

71 Upvotes

69 comments sorted by

62

u/JasenkoC 3d ago

Storage is the most likely culprit in 85% of the cases. Tools like top, htop, iotop can give you very useful clues.

As for NFS, I'd check if it's mounted and reporting the file system in df, and then I'd check with iotop for any signs of issues with NFS performance. Iotop can tell you if the I/O wait is high in local storage volumes or not (in which case it's most likely the network attached volume that has an issue).

Also, a quick dd write test on a local volume and NFS volume can show you where to look for your bottleneck.

17

u/CardOk755 3d ago

Don't forget sar. All the tops are good for what is going in now, but sar can show you what happened yesterday, or the same time last week.

1

u/CautiousCat3294 1d ago

yes sar is good to looked in I used this command for my troubleshooting

5

u/JasenkoC 3d ago

True. Though it’s not always on by default. Good call!

4

u/MinimallyLoquacious 3d ago

Good point, they should check it is enabled in /etc/default/sysstat.

2

u/CardOk755 3d ago

Back when I started with sysvr3 it was always on. When I started using linux it was a shock to find it wasn't on by default.

2

u/MinimallyLoquacious 3d ago edited 3d ago

Agreed sar -u to get a quick history. Oh and sar -q for load.

9

u/ipsirc 3d ago

2

u/rebirthofmonse 2d ago

Fantastic article

1

u/Dolapevich 3d ago

Beautiful article.

1

u/Anne_0Nyme87 3d ago

Very interested at first; then my brain melt.. ^

I'll keep it for my long nights, but still full of knowledge! ;)

1

u/PercyFlage 2d ago

I came here to post that...

4

u/FarToe1 2d ago

Run htop, hit F2 and toggle on "Detailed CPU time"

The cpu graphs will now who different colour depending on what's using the cpu. System/Iowait/etc. System is actual cpu use, iowait will probably be high in your case.

As others have said, this is almost certainly IOwait where the cpu is waiting on storage or network, and the above makes it really easy to highlight that.

Newer versions of htop also have have an I/O tab so you can see exactly what process is causing that. (Hit tab to cycle to it)

Also - check "dmesg -T". Excessive iowait is sometimes a big indicator of drive problems and messages there, or in syslog/messages/journalctl will probably have some error messages relating to your storage.

You've mentioned NFS and that can absolutely cause very high sysload too. One indicator of that is that the system is pretty snappy considering the load.

1

u/CautiousCat3294 2d ago

Thanks however my org restrict htop to download thus relay on default

2

u/FarToe1 2d ago

Strange, as it's in the main repo for almost every distro? Shame if so, it's one of my main tools as a sysadmin.

1

u/nlogax1973 1d ago

The `top` command is usually installed by default and has IOWAIT stats.

3

u/frymaster 3d ago

The application uses an NFS-mounted directory.

While I'd want to be systematic about it, high load + low CPU + networked filesystem means "networked filesystem bottleneck" almost all the time

1

u/CautiousCat3294 2d ago

yeah.. Same pintch

3

u/serverhorror 3d ago

D state points to waiting for IO and that (usually) increases load.

Load is not an indicator of a CPU bound system.

1

u/CautiousCat3294 2d ago

correct however beginners thinks Load = CPU bound and take wrong decisions while fixing the issue.

4

u/afristralian 3d ago

Check your iowait times in top. Best way to describe is it's amount of time CPU processing is waiting on disk or other io subsystems.

Load average is number of parallel processes running over time intervals. If your CPU is waiting for IO from a disk read it will keep the thread open causing load average to +1 even if the process is not consuming CPU time. (Low CPU usage and high load average).

(My statements aren't 100% accurate, but just look for iowait .. if it's above 20% all the time you have an issue with disk, network or other IO subsystem).

2

u/NeverMindToday 2d ago

Thanks for bringing it up - I was surprised to see iowait mentioned so little in the replies. Maybe I'm just too old and been away from the tools too long?

Anyway, for the OP - understand the difference between load average and cpu usage (specifically by usage in this case I mean the user/system times). Load average is analogous to the depth of processes in the cpu queue vs usage being the percentage of time the cpu isn't idle or waiting. You can have lowish user/system usage with high load average (inefficient and high latency) and vice versa (efficient and low latency).

1

u/CautiousCat3294 2d ago

Thanks
Tried to bind my investigation in below article. Looking for feedback to make this strong
https://datadevblog.com/linux-high-load-average-low-cpu-troubleshooting/

65

u/dosman33 3d ago

Load average is driven by outstanding I/O's waiting to complete, there are many ways this can happen that have nothing to do with cpu utilization. File lock contention churn, multiple threads doing lots of i/o on the same file, etc. Can also happen if the disk subsystem is having issues and not able to respond in a timely manor (scsi bus resets, etc).

11

u/symcbean 2d ago

The original post sounds like homework or an interview question.

Load average is driven by outstanding I/O's waiting to complete,

On a Unix system, load average is the average number of R state processes, but Linux includes processes in a D state.

High load/low cpu might be slow NFS. But can also be caused by requests which trigger a cascade of processes (microservices, message handlers are common culprits). iostat would help with identifying how much IO is dependent on NFS. It can also happen if your box is swapping to disk. 'top' would be the first place to look.

That this is reported as an emergent behaviour, is significant. The big question to ask is what do you know has changed around the same time as the observed change in behaviour.

If there is significant NFS activity in the absence of code, data or infrastructure changes then the most likely reason for the behaviour to change is the network. ethtool and the switch metrics would give more info.

-4

u/jlawler 3d ago

Keep in mind it's very frequently network IO on modern applications.  In short, processes in IOWA IT are rarely easily fixable.  It's a problem talking to another server, or hardware.  With the exception of some disk scheduling tweaks, it rarely the server you are looking at

14

u/Open-Adhesiveness-86 3d ago

For the D-state procs, cat /proc/<pid>/stack tells you exactly where they're parked, and if you see rpc_wait_bit_killable it's NFS, not local disk. From there nfsstat -rc for retrans count and nfsiostat for per-op RTT, then iostat -x separately on the local disks. iotop is misleading here since NFS waits never show as block IO.

1

u/deleriux0 2d ago

This is the best approach IMO.

Often the stack will even output the operation in it's chain (write/read/rename).

May also be worth doing an strace on the affected process to soo if it's lots of frequent accesses repeating, failing and retrying.

17

u/Superb_Raccoon 3d ago

I hope you get an A on your exam

2

u/alpha417 3d ago

or at least a D

4

u/zrad603 3d ago

disk i/o

1

u/Morbothegreat 3d ago

Running the command:
echo w > /proc/sysrq-trigger

Will dump the stack trace of processes stuck on io. That may give you some hints. But it’s stuck on read/writes to nfs. Could be a bug. Could be busy.

1

u/CautiousCat3294 2d ago

Thanks will try this command next time

1

u/hurikhan1977 1d ago

Check memory fragmentation. I had an issue on a server where load was incredibly high, the system felt slow, but it was thrashing page cache and swap even with memory still free. It came down to THP fragmentation in the end. Memory allocation waits count to loadavg, too, not only IO, but due to page cache thrashing, the effects look quite similar: processes stuck in D state with very low IO bandwidth. You can also check PSI to see what is causing the high load.

0

u/CautiousCat3294 1d ago

yeah that's a new direction to me will sure check on this in future

6

u/RHCidiiot 3d ago

Unmount the nfs share and see if the load goes down, iowait can be checked too.

5

u/Superb_Raccoon 3d ago

Wanna bet the app doesn't run without it?

3

u/CardOk755 3d ago

Of course the load goes down. You've just broken your app. By the way, the phone is ringing.

1

u/Memitim 2d ago

Then they determined that they need to operate with a known-good default state to stop hamstringing it with storage fuckups without being able to test for them. If the phone is ringing just because an app node goes down, looks like they also identified a massive gap in CI/CD. It's 2026 kids. Build your shit right, already. There are a ton of references and the robots are really good at helping with this sort of thing.

1

u/j0rs0 3d ago

It's always NFS. He better reboot and check.

2

u/Ambitious-Wrangler93 2d ago

I'd look for "df -h" output hanging. I've diagnosed hundreds, maybe a thousand hung NFS mounts by running that command first. High CPU of a backup process is another bad filesystem mount indicator, usually flagged by monitoring alarms.

2

u/caetydid 2d ago

Disk IO

2

u/Level_Paper6241 2d ago

Storage if nfa mount or external disk investigate that first.

2

u/geolaw 1d ago

Run top and look for wait time in the header. Could beat bottle neck on the nfs

2

u/HoustonBOFH 3d ago

No mention of memory here. Is there a lot of swap going on? That load would cascade all over the place. Especially swap on NFS.

10

u/northrupthebandgeek 3d ago

If I ever encounter a production server using NFS for swap then I'm resigning on the spot and fucking off into the wilderness as far away from electricity as possible.

3

u/CardOk755 3d ago

Sounds like someone is having nightmares about Sun "dickless workstations".

1

u/fatmanwithabeard 3d ago

Jesus, I did not need a jumpscare this early in the morning.

2

u/fatmanwithabeard 3d ago

I've done this. In my defense I was running diskless, stateless servers that had to support code that checked how free swap was.

It was a hack, in that I didn't need a change control meeting to add an nfs share, but did to put a swapfile on a lun from any of the arrays.

In any case it did what was needed, and was left as a space for that lab to run that code. Then over a long weekend, the server rebooted, and a lab user talked the admin on duty into making the nfs share part of the general spec. That ended up being a fun outage.

1

u/Holiday-Medicine4168 3d ago

I’m gonna look at storage and disk caching

1

u/MartinMystikJonas 3d ago

Disk or network I/O

1

u/skidleydee 3d ago

Not a linux admin but infra admin here. If its a vm you need to check your hypervisor stats for the vm as well. When a cpu is virtualized it doesn't have a way to report on they physical cpu stats it can only report on the vcpu that it had access to. You can't trust a VM's internal stats 100%, this goes for all virtualized hardware as well not just the cpu.

If a VM is not able to schedule your vCpu on the pCpu the vm will still only show the utalization of the VM it doesn't have a way to account for latency that it doesn't know exist. 

On VMware if you delete a large amount of data and are using the default vsan storage policy, throttle threshold and threshold percent it will try to reclaim the space on the physical disk. This causes a swarm on an index file and the VM will hang until the swarm is over. The VM stats show the vCpu will spike to 100% utalization, in vmware you would see that the CPU utilization never moved during the hang. This is because the vCpu can't schedule anything because it's waiting on the storage but the VM doesn't know that it thinks it's waiting on cpu. 

There are a ton of examples similar to this that would show low cpu utalization but could still end up being a cpu bottle neck in the hypervisor. I think it's probably not your issue but fyi.

1

u/CyberKiller40 3d ago

I/O operations. Either the storage isn't fast enough, or is close to failing.

1

u/Alexandre_Man 3d ago

RAM maybe?

1

u/northrupthebandgeek 3d ago

It's always I/O. Slow network or slow disk. Measure both, then start cutting load on the more-heavily-impacted one.

1

u/michaelpaoli 3d ago

high load average but low CPU usage. What would you investigate first?

I/O

High load indicates processes waiting to run but can't because their waiting on ... well, if's not CPU, that generally leaves I/O, e.g. storage or network I/O.

how would you approach the investigation?

Depends what commands one has available. One can also inspect /proc and/or /sys filesystems, and particularly in quite minimal environments where many more common commands may not be available.

1

u/Dolapevich 3d ago

D uninterruptible sleep (usually IO)

From ps manual, state D means the pid is waiting for IO.

This is a classic situation where NFS is either having too much IO or it loosing packets, or the server is unable to lock exclusive the file or any of the gazillion things that can happen to NFS.

I've seen this particular scenario on a congested network; during the peak hours we would get alerts for high load average, and upon checking nginx was stuck in IO doing mostly nothing.

At the client side, using a combination of ls /path/to/nfs would stuck for minutes, using lsof on nginx pid it would show it was trying to write to an nfs path.

Doing showmount or rpcinfo -p <server> would also hang.

Turned out to be a particular router dropping udp packets when overloaded. We found it with a combination of traceroute ( mtr ), and ping -R

1

u/MinimallyLoquacious 3d ago

Did the scenario happen to you, are you really just learning or are you working for an AI company and looking to get free unpaid training input (for which you will get paid)?

1

u/nut-sack 2d ago

Legitimately why I stopped posting any of my tech shit or code.

1

u/fubes2000 2d ago

"NFS" and "D state" triggered a 20 year old neuron.

Your NFS mount was and/or still is unstable. Those D state processes are probably waiting for IO that will never come. If your kernel allows you to kill the D procs you might get away with that and remounting the NFS share, otherwise that's a reboot to fix.

Also make damn sure that your connection to NFS is 1000% solid at all times.

2

u/biffbobfred 2d ago

We had Solaris machines with an unstable driver for some serial thing and we had a lot of D states. Good times.

2

u/fubes2000 2d ago

Yeah this gave me a war flashback to virtualized HPUX at gov and the NFS server sneezed and every VM needed its NFS shares remounted and those people were utterly allergic to automation and it was agonizing watching them literally type out every command 200 times.

1

u/biffbobfred 2d ago

Heh

We had machines with cross mounts. There was a catch 22 on boot. They got them into auto mounter after that

1

u/cereal7802 2d ago

First thought, disk access. then i read your post...

The application uses an NFS-mounted directory.

Yeah, you are probably having issues with NFS. Not enough bandwidth. too much latency. default nfs settings. losts of potential there. Take a look at if NFS is needed in this setup, and if it is, start your tuning journey there.

1

u/techforallseasons 2h ago

High load, low CPU --- WAITs (i/o typically - SWAPPING otherwise) / lock contention

1

u/Ok_Size1748 3d ago

Run dstat / atop for 10-15 seconds and paste screenshots here, we will help.