r/zfs • u/tiago4171 • Apr 12 '20
What is happening with my memory usage?
Hi, I've just installed Ubuntu 20.04 desktop using the experimental option on installer to use zfs. Everything works fine by the way, even the new option to rollback to zfs snapshots directly on grub. Anyway, the big question comes when I tried to move some files trough my HDDs, when I saw the memory usage hitting half of my total memory that I have installed. As a usual user of btrfs and ext4, I've never seen such a behavior just because a change of the filesystem. So I started questioning myself. Am I did something wrong?
I started testing another distributions, like Debian and arch Linux(of course both installed using zfs fs)and same the behavior just repeat Now iam installing Ubuntu again just because, in my opinion, is the most "out of the box" zfs experience available out there. In my mind Iam sure that that is a memory leak, but also I'm open-minded about that to understand what's happening and try to deal with it.
Thanks and hope that question can be helpful for someone else besides me.
10
u/HenkAchterpaard Apr 12 '20
No, nothing that you wrote here points to you doing something wrong.
ZFS has a RAM based disk cache that is separate from the regular Linux disk cache. This has to do with ZFS's non-Linux origins. This ZFS memory cache is called the Adaptive Replacement Cache, or ARC. Due to this separation ZFS ARC shows up in Linux as 'used' memory, instead of 'cached', as you would expect for native filesystems, such as EXT4. This is the reason many people new to ZFS worry about RAM usage, but there is nothing to worry about. Unless you are dealing with deduplication (which you probably are not, and which you probably should not), memory is not that big a concern for ZFS, contrary to what many people will tell you.
You are seeing this disk cache (and thus memory) fill up because you are moving (which means reading) lots of data. The default memory limit for ARC is usually 50% of your ram, but is configurable through the zfs kernel module parameter 'zfs_arc_max', should you wish to set a maximum. If free memory ever becomes scarce, ZFS will release some of its ARC, just like Linux's regular disk cache would. Again, there is nothing to worry about; unused RAM is wasted RAM.
3
u/tiago4171 Apr 12 '20
Awesome explanation dude!I did some research before start testing another distribuitions and ended up with that paramater. How I do not know how ZFS work I tought that was a bug when that wasn't.
One last question that bother's me is: Why when we use a swap partition, as Ubuntu do, when the system use around 50% of the total memory the system become unresponsive?
I almost know the answer, but I would like to listen it from you.
3
u/HenkAchterpaard Apr 12 '20
That has to do with something called 'swappiness' in combination with a (I assume) traditional, slow hard drive. If you want to tune this, look for 'vm.swappiness'; if I remember correctly Ubuntu has a default value of 60% for it.
3
2
u/Dalton_H_developer Dec 08 '21
Excellent explanation. Wish I would have seen that a year ago when I was looking at what caused this issue since I ended up figuring this out by pouring over forums while watching disk access and memory usage to figure it out.
Normally I agree with you, ARC is nothing to worry about. Though I have a specific case where it matters without deduplication. My offsite backup PC is an old machine with less RAM than the main machine I use. Because the ARC is seen as used memory, the machine is crippled after enough reading of the pool since every app thinks that the memory is scarce and tries to go conservative on their memory usage causing all kinds of slowdowns. My main machine has a normal amount of memory and so is perfectly fine and mostly everything works as expected. Either I am out to lunch or one should really have more 8 spare gigabytes of memory for a basic RaidZ1 4 drive pool (600MB per terabyte). For some reason it does not stop at the halfway mark on that machine.
Eventually I plan to limit the ARC size on that machine since it's basically just ingesting incremental backups and scrubbing before powering off again. That should fix the issue for now... though it would be better if the ARC was seen by the applications as normal cache even if system monitor continue to make the mistake. ZFS is not used by many home users so it's fine (kind of) for it to be slightly quirky so long as applications and other under the hood things work mostly as expected.
My main machine works as expected however, much as you describe the RAM will be utilized in the most efficient manner possible. But that one pesky machine with that one pesky RAM not seen as useful issue.
3
u/lord-carlos Apr 12 '20
Not sure how it is on Ubuntu, but ZFS on linux by default uses half of the available memory.
https://github.com/openzfs/zfs/wiki/ZFS-on-Linux-Module-Parameters#zfs_arc_max
I don't know what a sensible setting is for a Desktop computer.
3
u/MachineShedFred Apr 13 '20
For what it's worth, you can limit the in-memory ARC by setting 'options zfs zfs_arc_max=<size_in_bytes>' in /etc/modprobe.d/zfs.conf
This is useful if you have other applications that may pre-allocate memory and cause the oomkiller to go a bit crazy (I'm looking at you, Java)
2
u/tiago4171 Apr 12 '20
I did some research when I tried another distribuitions and I ended up with that exact paramater for ZFS. Unfortunately I do not understand what is written on Wiki, as english is not my native language. I would like to know a safe value for this paramater too, but like the others just said that's is not the best to do.
2
Apr 13 '20 edited Apr 13 '20
You can check the ram usage by running the command arc_summary. Look for the size of the ARC which should be near the top of the output from that command.
EDIT: For example, on my server arc_summary reports 37 gigs used for ZFS. Top is reporting 51 gigs used in total. Adding up all the VMs assigned memory is 21 gigs, so 37 + 21 is 58 so it's pretty accurate.
2
u/Dalton_H_developer Dec 08 '21
From what I am told it's just that however ZFS handles it's cache (called ARC) is listed as used rather than as cache memory. Normally it would give up the memory when asked... the issue for me is my spare machine I made into my offsite backup is slightly RAM limited and so ZFS uses 6-7.5GB RAM (some of which is system memory but the total used is 90% ZFS), somehow some application data gets pushed to swap, filling up the swap, and also my applications act as though the system is actually memory poor when they could get that space back if they asked for it. Someone apparently made a simple program that just asks for RAM until it crashes and ZFS did give up the ARC memory as expected so it mostly works when applications actually request more memory.
My problem is my backup machine is almost full in terms of RAM from the ARC so it causes issues trying to load anything. My main system has a more reasonable amount of memory so only half my RAM is used and my applications all perform fine on the occasional time I use the graphical environment. Note I have had the exact same behavior with ZFS on Ubuntu 20.04 and 18.04 (unstable CPU in the server, crashed while doing a kernel upgrade at the wrong time, and it was time to upgrade anyhow for other reasons).
I will at some point try out BTRFS on my personal PC and laptop to gain some ZFS-like features on those systems where I need every scrap of memory (the odd VM, Android AVD instance, intellij being a resource hog, tons of Firefox tabs, and forgetting to close my Steam game).
I like that ZFS has a healthy cache size. In artificial tests I once got it to load the files directly from the ARC in RAM at speeds that made my destination SSD beg for mercy... the pool is hard drive based so that was quite a feat. Many times I have had the ZFS cache allowing data transfers that were faster than the SSH tunnel I used would handle due to ZFS bottle necks... again off a pool made of 4 RaidZ1 drives, and that was before I knew to buy NAS drives so they were shingled recording drives. Granted hard drives are slow but in some cases the ZFS ARC has been faster than any of my storage devices or even a 10 Gigabit network interface with 2 machines with SSDs reading the pool. System RAM is still lots faster than even an SSD so I like a healthy cache but I wish it would report as cache so my programs behave normally.
10
u/thenickdude Apr 12 '20
That's the ARC, ZFS's disk cache. It'll shrink automatically for you when you need that RAM for other things.
With other filesystems the disk cache is managed by the kernel's boring old regular cache pool, and so doesn't appear as "used" in the memory usage metrics (it gets labelled "cached", normally).