Dragonflybsd's HAMMER filesystem seems to fit the bill nicely. There's even an option to limit the maximum amount of memory used for deduplication. Look up memlimit in the manual page: https://leaf.dragonflybsd.org/cgi/web-man?command=hammer&sec...