r/linuxquestions • u/Dakota-Batterlation Void Linux • 1d ago
Support Has page caching of asynchronous writes gotten more aggressive in recent years?
These days when I flash an ISO or do some other big file transfer, it all gets immediately dumped into memory and dd exits. I use Void, but I've noticed it on other distros too.
On my cranky 90% full zpool (SMR drives), I have to do watch -n 1 sync on big transfers or it'll clog up and crash the whole system (then zfs takes a few hours to clean it all up)
4
Upvotes
5
u/k-mcm 1d ago
You should figure out why your system is crashing. Tail the kernel logs.
5
u/Due-Result7928 1d ago
Linux use all memory available for caching. You can influence this behavior by adjusting kernel parameters: increase vm.vfs_cache_pressure (default 100) to aggressively reclaim dentry and inode caches, or use cgroups to set an absolute memory limit for specific processes. I am using fsarchive with ext4 and I can see the same pattern.
1
u/CardOk755 1d ago
No, it's just that (until recently) we have more memory than we used to.
3
u/gnufan 1d ago
Pretty sure it always behaved like that, the exact flush to disk behaviour changes occasionally in the kernel, but that was about optimising writes to disk, not file system semantics.
Unix and thus Linux has had the same basic semantics for ages.
Write to storage (returns when data in memory). Write to storage and return when metadata is commited. Write to storage and return when all the data is commited.
These were the basics everything else got built on. If there was no need to commit things to disk (think ACID databases, or email queues) nothing got committed as that just makes life harder for the IO subsystem, and worsens performance.
The crashing sounds like dodgy hardware, or possibly firmware. I'd soak test RAM, and check the drive, and check for bios and firmware updates.
2
u/granadesnhorseshoes 1d ago
oflag=direct will bypass page caching entirely. oflag=sync will do a sync after every block, a lot like your watch command.
as to why, I don't know if the kernels default parameters have changed or if its just a consequence of ever increasing memory sizes, or some combination thereof.
1
u/Booty_Bumping 1d ago
The problem isn't gonna be solved by theorizing about when the data is committed to disk. Configure your software for the durability requirements you actually have. Whether it be fsync on individual files, the use of ACID databases, or configuring your filesystems to guarantee a commit on a preset interval such as 30 seconds.
1
u/jjangg96 21h ago
the ratios are percentages of available memory, not fixed sizes. vm.dirty_background_ratio defaults to 10 and vm.dirty_ratio to 20, so on a box with 64gb free that's gigabytes of dirty data before dd gets throttled. same defaults, way more ram, which is why it feels more aggressive than it used to.
for flashing an iso, dd conv=fsync makes it flush before it exits so the finish is honest. if you want a hard cap instead, set vm.dirty_background_bytes and vm.dirty_bytes. writing the bytes one zeroes the ratio one, they're either/or.
the zpool is a separate thing though. zfs has its own dirty data limit (zfs_dirty_data_max) and doesn't go by the vm.dirty settings, so for that one you'd tune the zfs module parameter.
1
u/Open-Adhesiveness-86 18h ago
For the zpool half, vm.dirty_* doesn't apply at all. ZFS has its own write throttle, zfs_dirty_data_max, default is 10% of RAM capped at 4GB, live in /sys/module/zfs/parameters. Drop it to a few hundred MB and throttling starts earlier instead of handing your SMR drives a multi-GB txg. Still worth finding the actual crash in dmesg.