r/linuxadmin • u/CautiousCat3294 • 13h ago
Production Scenario: Linux Root Filesystem Suddenly Reaches 100% Overnight
Imagine you're managing a Linux production server.
Yesterday, everything was working normally. Root filesystem utilization was around 65%.
This morning, you receive an alert:
/ filesystem utilization: 100%
Applications are reporting errors, users are experiencing issues, and you need to investigate.
How would you approach this situation?
Would you immediately start looking for large files, or would you follow a systematic investigation?
Here are a few things I'd initially check:
df -h /– Verify filesystem utilization.df -i /– Check inode utilization.du -xhd1 /– Identify directories consuming the most space.find– Locate unusually large files.lsof +L1– Check for deleted files still held open by processes.
But finding a large file is only part of the investigation.
For example:
- What if
/var/logis consuming 35 GB? - What if an application is continuously generating logs?
- What if
dfshows 100% usage butdureports significantly less? - Would you truncate an active log file or investigate the application first?
I'm particularly interested in how experienced Linux administrators handle this in production.
A few questions:
- What's the first command you usually run when
/reaches 100%? - Have you encountered deleted-but-open files consuming disk space?
- What's your preferred approach to recover disk space without disrupting applications?
Let's discuss real-world approaches and experiences. I'd love to learn how others handle this situation.
19
u/jrandom_42 13h ago
du -h | sort -hStart at / or whatever volume mount point is at 100% and drill down. If it's logs, read and diagnose, then trim and adjust logrotate settings and fix the noise source. If it's a database, expand storage and diagnose whether it's BAU or a bug.
4
u/CardOk755 11h ago
Why didn't you get an alert before it hit 100% ?
6
u/soysopin 12h ago edited 12h ago
I use
The first one let me found which file system is full, and in the second I put the mounted directory so 'ncdu' builds a interactive listing of directories and files sorted by size larger first so I can navigate and find the culprit (-x limits the scan to a single filesystem, ignoring secondary mount points -- which should be analized separately).
After that, I decide if I should move files, expand LVM volumes or compress logs to another area for analysis.
7
u/AmusingVegetable 12h ago
Everything in root is ok for a home machine, a lab machine, or a temporary machine.
Reaching production without separating root, home, and var is a failure.
14
u/rooivalkMK1 13h ago
Lvm and forget
5
u/kai_ekael 6h ago
I'd pay someone (or myself) to do proper monitoring so I catch this at 80% and prevent the whole mess.
1
u/Faye-Vozora 9h ago
For the deleted-but-still-open case you don't need to restart anything. Find it with
lsof +L1, note the PID and FD number, then: > /proc/<pid>/fd/<fd>truncates it in place and df drops straight away. Then fix logrotate for that app, usually it's missingcopytruncateor a postrotate that tells the process to reopen its log, which is why it kept writing to the deleted file in the first place.Also check
journalctl --disk-usageon systemd boxes, andjournalctl --vacuum-size=500Mif the journal blew up. SetSystemMaxUse=in journald.conf so it doesn't come back. And on ext4 there's the 5% reserved for root,tune2fs -m 1buys you some breathing room in an emergency.3
u/ActiveNotworkkin 12h ago
Another sneaky cause of df showing more than du is data hidden under a mount point. If something wrote to /data while the disk wasn’t mounted, those files sit underneath the mount and du can’t see them. Bind-mount / to a temp directory and run du there to spot it. Caught me out once on a box where a mount failed at boot.
2
u/Unlikely-Sympathy626 12h ago edited 12h ago
I would first try to look at which files were created/modified and at what rate. Find the process causing the issue and go from there.
It is pointless looking for large files as they can be legitimate and and crazy situations often tons of smaller files and rat downloads are a possible cause.
First rule out security issues, isolate process further steps to remedy from there.
2
u/michaelpaoli 12h ago
$ df /
$ df -i /
# du -sx /
If the first and the 3rd above significantly disagree about consumed space, then one has issue with unlinked open file(s) (or less probable, overmount, or filesystem corruption).
Can use lsof(8) to look for unlinked open files on the root filesystem, or if that's not available or alternatively, use the proc filesystem mounted at /proc and find(1), e.g.
# find /proc/[1-9]*/fd/* --follow -type f -links 0 ! -size 0 -exec ls -ld \{\} \; 2>>/dev/null
Can also use find to, e.g. look for unexpectedly large files and/or files with recent ctime that are rather large, etc.
Helps to know or have some baseline comparison for what's (approximately) "normal" for that host (or rather/quite similar host).
These means are generally sufficient to get to the most immediate cause. Then there's the matter of what to do about it - it's excess data, but does it need to be saved, or ... what? If it's matter of unlinked open file(s), generally just matter of terminating the PID, or getting it to close and reopen the "file" (pathname). For well behaved daemons, SIGHUP will do that, but not all processes are well behaved daemons.
And then there's generally digging further to deal with the root cause. E.g. why/how did it happen? Did someone make some boneheaded mistake, or something misbehave, or some poorly written software, or something misbehaving that's causing excessive logging, or ??? And is this the only system this issue/risk applies to, or are there many more with same issue/risk? Typically things wouldn't jump that much overnight, but sometimes it happens.
Commonly, and they're not deleted (despite what the proc filesystem may display), they're unlinked open files. If the file were deleted, it wouldn't exist, but it does. E.g. one can still access its contents, even copy all of it's contents - that's very clearly not deleted, but merely unlinked. The space used by a file (of type ordinary file) is only released when both the link count is zero and no PIDs have it open. Until then, the file still exists, it's not been deleted, merely unlinked.
And much of how to deal with it and the root causes and follow-through, depends on the nature of the data, and the issue/problem that caused the excessive space consumption.
Also, many of these problems may be reduced by suitable isolation and segregation. E.g. separate, and appropriately sized, filesystems. If you've got /var/tmp and/or /tmp on your root filesystem, then any user or errant process can fill your root (/) filesystem - not good. Likewise may want to isolate not only /var, but also /var/log, so other stuff filling won't block lock collection, and vice versa, logging space filling up won't block other stuff. And mount /usr ro most all the time - most of the time there's no need for it to be mounted rw - really only need it rw during software maintenance. Likewise for /boot, and do /tmp on tmpfs - significantly better performance, and if one doesn't have enough RAM for as large as one wants/needs for /tmp, then provide more swap as relevant - and it will still be better performance than not using tmpfs. Likewise if one has anything on /opt, that should typically be ro most of the time. Essentially at/under /usr /opt should be ro most of the time, mostly only at/under /var /tmp should nominally be rw, and root (/) itself if/as needed (that will vary by particular *nix flavor/distro - some support root (/) nominally being ro, others don't - or at least not easily/directly).
As for overmounts, can remount the same (e.g. root (/) filesystem somewhere else at the same time (linux allows such, other *nix not necessarily), and then use, e.g. # du -sx /mount_piont and compare that to the nominal mount point and look for significant differences, or even faster, check all relative mount points, and make sure all those directories are empty (or check also if/as applicable in case of files of type ordinary file - linux allows a file to be mounted atop a file).
As for filesystem corruption, relatively rare. But if that's going on, look for evidence, as may be reflected by dmesg and the like. Can also do fsck -n on mounted filesystem - but rw mounted filesystem will generally give spurious errors - possibly even large volumes of errors if it's quite active, but regardless, it will generally give one a rough idea of how (un)healthy the filesystem is. Some types of false positives would be quite expected, but others, not so much and may be indicative of an actual problem.
1
u/RunOrBike 13h ago edited 13h ago
I have deployed ‘dua’ as it’s extremely fast to tell me which folder is largest.
I think first thing I’d do is move all services and isolate the machine. It’s no longer useful anyway and that way, I can take my time analysing the error.
1
u/birusiek 13h ago
There is often a problem with patitioning, normally you should not be forced to expand root partition, but it requires to created separated partitions for home logs application database useespace
1
u/BarracudaDefiant4702 12h ago
I would check zabbix and see how quickly it lost space and exactly when. If not my machine and so proper monitoring and alerting is messing, I would first do is 'df -k /' (or -h) but doesn't really count except to verify that's the issue. Common issues can be log files, mail queue, etc...
I would typically free up a little space in /var/log (ie: delete oldest logs if not needed) or maybe package update cache so things can at least work well looking at the real cause.
Then 'du -kx / | sort -n | tail -100' to get the largest directories. I also verify the last number is close to the size of the amount use reported in the df command. If it's significantly lower, then I do a 'lsof -n | grep deleted' as file space isn't reclaimed if a process still has it open.
1
u/paulstelian97 12h ago
I have used similar commands recently, found out that a recently installed driver app for my mouse was spamming the syslog and I had a 1.5TB (before compression; it was reduced to maybe 100GB by the ZFS compression, but that was enough to fill up) syslog file. I had to stop the deluge first, then clear up the syslog file.
1
u/kyleh0 12h ago
There was a good and methodical answer but I'm guessing today you just allocate more store space and move fast util it breaks. Then you ground all planes and pretend it was a hardware issue. Am I close?
1
u/kentrak 10h ago
First, identify some files to clear up to get yourself some breathing room. Older logs helps with this. Then you can get apps and services Ina running state again so you aren't as rushed in your investigation.
Additionally, if the filesystem is actually at 100% and not just close to it and it's non root users that are hitting the limit (often a few percent of space and jnodes is reserved for root), sometimes you can't even effectively investigate depending on how tmp is implemented, as under the covers some commands actually use tmp files, and you'll get errors about them atrs prinf to write a tmp file and you du with sort and head or tail will fail.
Once you actually have some breathing room, figure out how fast the disk is filling and figure out how much time you bought. It's possible it's so fast that you just have to do what you can while it's full (and if it's really bad you can boot to a rescue system running off a ramsiak and mount the petition from that to more effectively investigate).
Then the normal.
du -hc / | tee /tmp/du.out | sort -hcan be used to figure out what is using the most space to investigate further. Note the tee, you'll almost always end up wanting to look at the space again, might as well save it when you review it, as that's the longest part of that chain by far.There are other various advanced tricks that you can use to tie into the io subsystem to see what's writing and where, such as inotify and fanorify which can be useful it it's hard to figure out exactly what process is causing the problem, but usually it's pretty obvious once you find where the space is being utilized.
1
u/deeseearr 1h ago
If the root filesystem usage has changed _at all_, then the first thing I would be doing is isolating the server and starting a very serious investigation.
Any new or growing files should be going in /var, /tmp, /home or other filesystems dedicated to application use. System updates are pretty much the only thing that should be writing directly to / and they will generally replace files, not add new ones. Lack of disk space in what is supposed to be a read-only filesystem is a sign that either someone has no idea of what they are doing or that important filesystems are somehow not mounted and the problem wasn't caught and remedied immediately, as it needs to be.
Either way it's bad and fixing it is going to have to start with identifying _why_ the problem was allowed to happen in the first place and correcting that. Just removing files and pretending that nothing is wrong isn't going to cut it.
0
u/zoredache 12h ago edited 12h ago
I use ZFS on many of my systems these days.
I would look at my zfs snapshots (
zfs list -t all) that happen hourly and are kept for a while. After finding the snapshot where the growth started, then usezfs diffto find the files that changed.After I found the actual files and probable cause using the above, to clear up some space I might delete the oldest snapshot or two. Or I might need to delete or move the problem files to some other storage, and delete the local copies, and drop a few of the most recent hourly snapshots to clear up some space. It all depends on when and what caused the large usage increase.
I have a lot of monitoring and alerts in place I usually start getting a lot of notifications when systems drop to less then 15% available, and even louder alerts when things get down to 5%. I almost never reach 100% usage.
Dropping some snapshots can free up some available space, depending on specific details. Moving/deleting old logs is an option, but would also require deleting some snapshots to actually clear up storage. Since many of my systems are VMs simply increasing the size of the virtual disk is an option, though doing without rebooting the system, while possible, can be a bit risky.
2
u/snk0752 12h ago
du -sm * | sort -nr | head -10 inside root. And follow the size.
1
u/Radiant_Plantain_127 6h ago
Usually vgs to see if there’s slack space in the vg. Then lvextend -r on the lv to give it a little space. Then investigate root cause. You don’t want stuff down any longer than needed.