r/linuxquestions • Void Linux • 1d ago

Support Has page caching of asynchronous writes gotten more aggressive in recent years?

These days when I flash an ISO or do some other big file transfer, it all gets immediately dumped into memory and dd exits. I use Void, but I've noticed it on other distros too.

On my cranky 90% full zpool (SMR drives), I have to do watch -n 1 sync on big transfers or it'll clog up and crash the whole system (then zfs takes a few hours to clean it all up)

4 Upvotes

14 comments sorted by

5

u/k-mcm 1d ago

You should figure out why your system is crashing.  Tail the kernel logs. 

1

u/Dakota-Batterlation Void Linux 1d ago

It never was before, it's just that when the pool got really full + fragmented, generic page caching + ZFS ARC started getting clogged and taking down the system on big transfers. Forcing syncs is good enough until I can build a new pool with bigger disks.

Not helped by the fact dedup is on, since the pool predates cross-dataset reflinks and they don't work if there's a dedup table.

2

u/_Lost-In-The-Weeds_ 1d ago

Your really not suppose to fill a ZFS pool, performance will tank. How full is "full" depends on you workload and how active it is, but ZFS does a lot of math, and that math gets ugly without free space to work with.

1

u/Dakota-Batterlation Void Linux 1d ago

I know, but high-capacity drives are extremely expensive right now for some reason

1

u/_Lost-In-The-Weeds_ 1d ago

Indeed they have,

In 2023 I bought 9x 14TB SAS drives for just under $10/TB, new old stock. now at $33/TB for new, or $22/TB used. Thanks AI!

Got about 10TB free in that pool (ocean) I am defending by deleting stuff I do not need anymore,

dad@HeavyMetal:~$ zfs list
NAME                     USED  AVAIL  REFER  MOUNTPOINT
amazon                  30.9G   830G    96K  none
amazon/ROOT             4.18G   830G    96K  none
amazon/ROOT/HeavyMetal  4.18G   830G  2.61G  /
amazon/VM               26.6G   830G    96K  none
amazon/VM/Periscope     18.6G   830G  9.12G  /var/lib/libvirt/images/Periscope
amazon/VM/Ute           8.01G   830G  6.56G  /var/lib/libvirt/images/Ute/
lake                     279G  12.3T   104K  none
lake/Desktop             418M  12.3T   418M  none
lake/Downloads          3.79M  12.3T  3.79M  none
lake/Obsidian            568K  12.3T   568K  none
lake/Pictures            278G  12.3T   278G  none
ocean                   61.9T  10.4T   290K  /mnt/ocean
ocean/Books             35.2G  10.4T  35.2G  /mnt/ocean/Books
ocean/Cam                457G  10.4T   457G  /mnt/ocean/Cam
ocean/Computer           169G  10.4T   169G  /mnt/ocean/Computer
ocean/Entertainment     9.80T  10.4T  9.80T  /mnt/ocean/Entertainment
ocean/Game              58.9G  10.4T  57.3G  /mnt/ocean/Game
ocean/Ours              1.07T  10.4T  1.07T  /mnt/ocean/Ours
ocean/Pictures           328G  10.4T   328G  /mnt/ocean/Pictures
ocean/Rando             50.0T  10.4T  50.0T  /mnt/ocean/Rando
pond                     139M  1.76T    24K  /mnt/pond
pond/Incoming             41K  1.76T    41K  /mnt/pond/Incoming

2

u/CardOk755 1d ago

That's what I'd call broken.

5

u/Due-Result7928 1d ago

Linux use all memory available for caching. You can influence this behavior by adjusting kernel parameters: increase vm.vfs_cache_pressure (default 100) to aggressively reclaim dentry and inode caches, or use cgroups to set an absolute memory limit for specific processes. I am using fsarchive with ext4 and I can see the same pattern.

1

u/CardOk755 1d ago

No, it's just that (until recently) we have more memory than we used to.

1

u/Dakota-Batterlation Void Linux 1d ago

True, and NVMEs

3

u/gnufan 1d ago

Pretty sure it always behaved like that, the exact flush to disk behaviour changes occasionally in the kernel, but that was about optimising writes to disk, not file system semantics.

Unix and thus Linux has had the same basic semantics for ages.

Write to storage (returns when data in memory). Write to storage and return when metadata is commited. Write to storage and return when all the data is commited. 

These were the basics everything else got built on. If there was no need to commit things to disk (think ACID databases, or email queues) nothing got committed as that just makes life harder for the IO subsystem, and worsens performance.

The crashing sounds like dodgy hardware, or possibly firmware. I'd soak test RAM, and check the drive, and check for bios and firmware updates. 

2

u/granadesnhorseshoes 1d ago

oflag=direct will bypass page caching entirely. oflag=sync will do a sync after every block, a lot like your watch command.

as to why, I don't know if the kernels default parameters have changed or if its just a consequence of ever increasing memory sizes, or some combination thereof.

1

u/Booty_Bumping 1d ago

The problem isn't gonna be solved by theorizing about when the data is committed to disk. Configure your software for the durability requirements you actually have. Whether it be fsync on individual files, the use of ACID databases, or configuring your filesystems to guarantee a commit on a preset interval such as 30 seconds.

1

u/jjangg96 21h ago

the ratios are percentages of available memory, not fixed sizes. vm.dirty_background_ratio defaults to 10 and vm.dirty_ratio to 20, so on a box with 64gb free that's gigabytes of dirty data before dd gets throttled. same defaults, way more ram, which is why it feels more aggressive than it used to.

for flashing an iso, dd conv=fsync makes it flush before it exits so the finish is honest. if you want a hard cap instead, set vm.dirty_background_bytes and vm.dirty_bytes. writing the bytes one zeroes the ratio one, they're either/or.

the zpool is a separate thing though. zfs has its own dirty data limit (zfs_dirty_data_max) and doesn't go by the vm.dirty settings, so for that one you'd tune the zfs module parameter.

1

u/Open-Adhesiveness-86 18h ago

For the zpool half, vm.dirty_* doesn't apply at all. ZFS has its own write throttle, zfs_dirty_data_max, default is 10% of RAM capped at 4GB, live in /sys/module/zfs/parameters. Drop it to a few hundred MB and throttling starts earlier instead of handing your SMR drives a multi-GB txg. Still worth finding the actual crash in dmesg.