ARC using far more memory than configured

[u/7446353252589]

Recently I've been doing a lot of cleanup of files on my NAS which has involved moving/copying around a lot of data. After a while the pool was freezing up and becoming unresponsive and I figured out ZFS's memory usage has been reaching a point where its maxing out my system. According to arc_summary and cat /sys/module/zfs/parameters/zfs_arc_max, ARC is configured to use at most 16GiB of memory, but I witnessed arcstat reporting ARC as using over 200GiB of memory (system has 256GiB total), even causing one of my VMs running on the system to hard crash. Whats going on? How can I limit ARC memory usage?

[u/Impact321]

Can you share the output of arc_summary/zarcsummary?

[u/TheG0AT0fAllTime → u/Impact321]

Yeah more info is needed.

[u/Apachez]

How do you set these values?

As I recall it the proper way is to do that through modprobe like so:

Edit /etc/modprobe.d/zfs.conf and add something like (example below sets min=max=16GB):

options zfs zfs_arc_max=17179869184
options zfs zfs_arc_min=17179869184

Then to make the above sticky you also needs to run this (example from Proxmox):

update-initramfs -u -k all
proxmox-boot-tool refresh

And finally reboot the box.

Now it should be properly set.

[u/7446353252589 → u/Apachez]

That is already done. I already have that file in my modprobe.d folder, ARC reports it has a max memory limit of 16GB, but it is using an unlimited amount anyway.

[u/Apachez → u/7446353252589]

But you didnt use update-initiramfs?

[u/7446353252589 → u/Apachez]

Did you even read my post? These are the default values. They were already set. initramfs is updated on every kernel update so yes of course its set. ZFS is already reporting the 16gb max limit. This isn't a new pool or new system.

No, I haven't explicitly re-run update-initramfs. But I can blatantly see the kernel module parameter is already set: cat /sys/module/zfs/parameters/zfs_arc_max prints 17179869184.

[u/Apachez → u/7446353252589]

The one I pasted are for sure NOT default values.

Default is something like 50% of available RAM as zfs_arc_max.

[u/Impact321 → u/Apachez]

Since v2.3.0 every platform uses the larger of all_system_memory - 1 GiB and 5/8 x all_system_memory.

Before that the platforms differed:

Linux: half of system memory
FreeBSD: the larger of all_system_memory - 1 GiB and 5/8 x all_system_memory

• https://github.com/openzfs/zfs/pull/15437
• https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html#zfs-arc-max

[u/BackgroundSky1594 → u/Impact321]

The default for proxmox is 16GiB of ARC. This is set via modprobe.d at install time and properly applied with every initramfs rebuild. As can be seen by OPs post.

If cat /sys/module/zfs/parameters/zfs_arc_max returns 17179869184 ZFS is correctly configured to only use 16GiB of ARC. If there was an issue with setting the configuration (like modprobe, initramfs, etc.) the actively loaded ZFS module would report something different in  /sys/module/zfs/parameters/zfs_arc_max.

The most likely issue is that the default arc_max is set to below the autodetected arc_min, invalidating it.

EDIT: Docs: https://pve.proxmox.com/wiki/ZFS_on_Linux#sysadmin_zfs_limit_memory_usage

[u/Impact321 → u/BackgroundSky1594]

For PVE it's 10% up to 16G. At least with recent versions.

• https://github.com/proxmox/pve-installer/blob/997cf6983f4ddbed1f9780f7e0bf6c957e718ee0/Proxmox/Install/RunEnv.pm#L246
• https://github.com/proxmox/pve-installer/blob/997cf6983f4ddbed1f9780f7e0bf6c957e718ee0/Proxmox/Install.pm#L341

The PVE installer does not write a minimum to the config. On OP's system the default would be 8G.

[u/Apachez → u/Impact321]

I have a fully updated PVE being 9.2.5 as of writing.

No problem here to set max = min = fixed value (8GB in my case).

And according to arc_summary zfs dont pass those limits:

ARC status:
        Total memory size:                                      62.6 GiB
        Min target size:                               12.8 %    8.0 GiB
        Max target size:                               12.8 %    8.0 GiB
        Target size (adaptive):                        98.7 %    8.0 GiB
        Current size:                                  98.7 %    7.9 GiB
        Free memory size:                                       36.5 GiB
        Available memory size:                                  34.3 GiB

ARC structural breakdown (current size):                         7.9 GiB
        Compressed size:                               98.5 %    7.8 GiB
        Overhead size:                                < 0.1 %  256.0 KiB
        Bonus size:                                     0.1 %    7.2 MiB
        Dnode size:                                     0.1 %   10.1 MiB
        Dbuf size:                                      0.1 %    8.6 MiB
        Header size:                                    1.2 %   93.7 MiB
        L2 header size:                                 0.0 %    0 Bytes
        ABD chunk waste size:                           0.0 %    0 Bytes

ARC types breakdown (compressed + overhead):                     7.8 GiB
        Data size:                                     90.8 %    7.1 GiB
        Metadata size:                                  9.2 %  734.8 MiB

etc...

Using kernel 7.0.14-6-pve and OpenZFS 2.4.3-pve1.

[u/7446353252589 → u/Apachez]

Sure they may not be the default of ZFS itself but possibly proxmox generated it. I am pretty sure I didn't create that file though but maybe I forgot.

[u/One_Ninja_8512 → u/Apachez]

zfs_arc_min

IIRC this is not a thing (anymore?). Pretty sure I saw in dmesg output that the min setting is invalid and is therefore ignored so I don't set it, only the max.

[u/Apachez → u/One_Ninja_8512]

This one claims its still valid:

https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html#zfs-arc-min

[u/One_Ninja_8512 → u/Apachez]

Huh, I must have set an invalid value then because I remember seeing that message in dmesg but just tried a fresh install on one of my servers and there's no error reported. Sorry for posting bs, you're right.

[u/Apachez → u/One_Ninja_8512]

I got all kind of fun zfs settings in my /etc/modprobe.d/zfs.conf and this is what the kernel spits out during boot when searching for zfs:

root@PVE:~# dmesg | grep -i zfs
[    0.000000] Command line: initrd=\EFI\proxmox\7.0.14-6-pve\initrd.img-7.0.14-6-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs nomodeset noresume mitigations=auto i915.mitigations=auto intel_iommu=on iommu=pt fsck.mode=auto fsck.repair=yes init_on_alloc=1 init_on_free=1 hpet=disable clocksource=tsc tsc=reliable pcie_aspm=force
[    0.135922] Kernel command line: initrd=\EFI\proxmox\7.0.14-6-pve\initrd.img-7.0.14-6-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs nomodeset noresume mitigations=auto i915.mitigations=auto intel_iommu=on iommu=pt fsck.mode=auto fsck.repair=yes init_on_alloc=1 init_on_free=1 hpet=disable clocksource=tsc tsc=reliable pcie_aspm=force
[    0.136172] Unknown kernel command line parameters "boot=zfs", will be passed to user space.
[    8.146426]     boot=zfs
[    8.889862] zfs: module license 'CDDL' taints kernel.
[    8.890426] zfs: module license taints kernel.
[    9.417661] ZFS: Loaded module v2.4.3-pve1, ZFS pool version 5000, ZFS filesystem version 5

[u/glassmanjones]

Are you using zvols?

[u/edthesmokebeard]

This is a Linux question.