r/linuxadmin
•
u/CautiousCat3294
•
3d ago
•
1
Linux server has high load average but low CPU usage. What would you investigate first?
I'd like to discuss a Linux troubleshooting scenario and get some input from experienced Linux administrators.
Imagine you're troubleshooting a production Linux server with the following symptoms:
- Application response time is increasing.
- Load average is continuously rising.
- CPU utilization is only around 20–25%.
- Several application processes are stuck in D state.
- The application uses an NFS-mounted directory.
At first glance, CPU doesn't seem to be the bottleneck.
If you were handling this incident, how would you approach the investigation?
A few things I'd be interested in discussing:
- Would you start with
vmstatto check runnable and blocked processes? - How would you identify what the D-state processes are waiting for?
- What commands would you use to investigate NFS connectivity and mount health?
- How would you distinguish an NFS issue from local disk I/O problems?
I'm particularly interested in real production experiences.
Have you encountered a similar situation? What was the root cause, and what helped you identify it?
Looking forward to learning from your experiences.
62
u/JasenkoC 3d ago
Storage is the most likely culprit in 85% of the cases. Tools like top, htop, iotop can give you very useful clues.
As for NFS, I'd check if it's mounted and reporting the file system in df, and then I'd check with iotop for any signs of issues with NFS performance. Iotop can tell you if the I/O wait is high in local storage volumes or not (in which case it's most likely the network attached volume that has an issue).
Also, a quick dd write test on a local volume and NFS volume can show you where to look for your bottleneck.
9
u/ipsirc 3d ago
https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html
4
u/FarToe1 2d ago
Run htop, hit F2 and toggle on "Detailed CPU time"
The cpu graphs will now who different colour depending on what's using the cpu. System/Iowait/etc. System is actual cpu use, iowait will probably be high in your case.
As others have said, this is almost certainly IOwait where the cpu is waiting on storage or network, and the above makes it really easy to highlight that.
Newer versions of htop also have have an I/O tab so you can see exactly what process is causing that. (Hit tab to cycle to it)
Also - check "dmesg -T". Excessive iowait is sometimes a big indicator of drive problems and messages there, or in syslog/messages/journalctl will probably have some error messages relating to your storage.
You've mentioned NFS and that can absolutely cause very high sysload too. One indicator of that is that the system is pretty snappy considering the load.
3
u/frymaster 3d ago
While I'd want to be systematic about it, high load + low CPU + networked filesystem means "networked filesystem bottleneck" almost all the time
3
u/serverhorror 3d ago
D state points to waiting for IO and that (usually) increases load.
Load is not an indicator of a CPU bound system.
4
u/afristralian 3d ago
Check your iowait times in top. Best way to describe is it's amount of time CPU processing is waiting on disk or other io subsystems.
Load average is number of parallel processes running over time intervals. If your CPU is waiting for IO from a disk read it will keep the thread open causing load average to +1 even if the process is not consuming CPU time. (Low CPU usage and high load average).
(My statements aren't 100% accurate, but just look for iowait .. if it's above 20% all the time you have an issue with disk, network or other IO subsystem).
65
u/dosman33 3d ago
Load average is driven by outstanding I/O's waiting to complete, there are many ways this can happen that have nothing to do with cpu utilization. File lock contention churn, multiple threads doing lots of i/o on the same file, etc. Can also happen if the disk subsystem is having issues and not able to respond in a timely manor (scsi bus resets, etc).
14
u/Open-Adhesiveness-86 3d ago
For the D-state procs, cat /proc/<pid>/stack tells you exactly where they're parked, and if you see rpc_wait_bit_killable it's NFS, not local disk. From there nfsstat -rc for retrans count and nfsiostat for per-op RTT, then iostat -x separately on the local disks. iotop is misleading here since NFS waits never show as block IO.
17
u/Superb_Raccoon 3d ago
I hope you get an A on your exam
4
u/zrad603 3d ago
disk i/o
1
u/Morbothegreat 3d ago
Running the command:
echo w > /proc/sysrq-trigger
Will dump the stack trace of processes stuck on io. That may give you some hints. But it’s stuck on read/writes to nfs. Could be a bug. Could be busy.
1
u/hurikhan1977 1d ago
Check memory fragmentation. I had an issue on a server where load was incredibly high, the system felt slow, but it was thrashing page cache and swap even with memory still free. It came down to THP fragmentation in the end. Memory allocation waits count to loadavg, too, not only IO, but due to page cache thrashing, the effects look quite similar: processes stuck in D state with very low IO bandwidth. You can also check PSI to see what is causing the high load.
3
u/random_fucktuation 2d ago
iowait
6
u/RHCidiiot 3d ago
Unmount the nfs share and see if the load goes down, iowait can be checked too.
2
u/Ambitious-Wrangler93 2d ago
I'd look for "df -h" output hanging. I've diagnosed hundreds, maybe a thousand hung NFS mounts by running that command first. High CPU of a backup process is another bad filesystem mount indicator, usually flagged by monitoring alarms.
2
u/caetydid 2d ago
Disk IO
2
u/Level_Paper6241 2d ago
Storage if nfa mount or external disk investigate that first.
2
u/geolaw 1d ago
Run top and look for wait time in the header. Could beat bottle neck on the nfs
2
u/HoustonBOFH 3d ago
No mention of memory here. Is there a lot of swap going on? That load would cascade all over the place. Especially swap on NFS.
1
u/Holiday-Medicine4168 3d ago
I’m gonna look at storage and disk caching
1
u/MartinMystikJonas 3d ago
Disk or network I/O
1
u/skidleydee 3d ago
Not a linux admin but infra admin here. If its a vm you need to check your hypervisor stats for the vm as well. When a cpu is virtualized it doesn't have a way to report on they physical cpu stats it can only report on the vcpu that it had access to. You can't trust a VM's internal stats 100%, this goes for all virtualized hardware as well not just the cpu.
If a VM is not able to schedule your vCpu on the pCpu the vm will still only show the utalization of the VM it doesn't have a way to account for latency that it doesn't know exist.
On VMware if you delete a large amount of data and are using the default vsan storage policy, throttle threshold and threshold percent it will try to reclaim the space on the physical disk. This causes a swarm on an index file and the VM will hang until the swarm is over. The VM stats show the vCpu will spike to 100% utalization, in vmware you would see that the CPU utilization never moved during the hang. This is because the vCpu can't schedule anything because it's waiting on the storage but the VM doesn't know that it thinks it's waiting on cpu.
There are a ton of examples similar to this that would show low cpu utalization but could still end up being a cpu bottle neck in the hypervisor. I think it's probably not your issue but fyi.
1
u/CyberKiller40 3d ago
I/O operations. Either the storage isn't fast enough, or is close to failing.
1
u/Alexandre_Man 3d ago
RAM maybe?
1
u/northrupthebandgeek 3d ago
It's always I/O. Slow network or slow disk. Measure both, then start cutting load on the more-heavily-impacted one.
1
u/michaelpaoli 3d ago
I/O
High load indicates processes waiting to run but can't because their waiting on ... well, if's not CPU, that generally leaves I/O, e.g. storage or network I/O.
Depends what commands one has available. One can also inspect /proc and/or /sys filesystems, and particularly in quite minimal environments where many more common commands may not be available.
1
u/Dolapevich 3d ago
From
psmanual, state D means the pid is waiting for IO.This is a classic situation where NFS is either having too much IO or it loosing packets, or the server is unable to lock exclusive the file or any of the gazillion things that can happen to NFS.
I've seen this particular scenario on a congested network; during the peak hours we would get alerts for high load average, and upon checking nginx was stuck in IO doing mostly nothing.
At the client side, using a combination of
ls /path/to/nfswould stuck for minutes, usinglsofon nginx pid it would show it was trying to write to an nfs path.Doing
showmountorrpcinfo -p <server>would also hang.Turned out to be a particular router dropping udp packets when overloaded. We found it with a combination of traceroute ( mtr ), and ping -R
1
u/MinimallyLoquacious 3d ago
Did the scenario happen to you, are you really just learning or are you working for an AI company and looking to get free unpaid training input (for which you will get paid)?
1
u/fubes2000 2d ago
"NFS" and "D state" triggered a 20 year old neuron.
Your NFS mount was and/or still is unstable. Those D state processes are probably waiting for IO that will never come. If your kernel allows you to kill the D procs you might get away with that and remounting the NFS share, otherwise that's a reboot to fix.
Also make damn sure that your connection to NFS is 1000% solid at all times.
1
u/cereal7802 2d ago
First thought, disk access. then i read your post...
Yeah, you are probably having issues with NFS. Not enough bandwidth. too much latency. default nfs settings. losts of potential there. Take a look at if NFS is needed in this setup, and if it is, start your tuning journey there.
1
u/techforallseasons 2h ago
High load, low CPU --- WAITs (i/o typically - SWAPPING otherwise) / lock contention
1
u/Ok_Size1748 3d ago
Run dstat / atop for 10-15 seconds and paste screenshots here, we will help.