higels

I just mount each SSD as its own XFS filesystem and use NGINX’s split feature to fill them up.

Not resilient on a system level, but refilling the cache is cheap.

ZFS was generally pleasant from an operability viewpoint once we ironed out the quirks, but the perf hit from no sendfile was too much.

Before I ditched ZoL for persistent storage for a few hundred NGINX caches, I saw this behavior too.

Setting zfs_arc_min to something like 50% of arc_max stopped it from dumping the ARC every 10 minutes.

YMMV.