Hacker News Reader

Comment by 3np | Hacker News Reader

> whether or not the filesystem attempts deduplication

Unless I'm just parroting truisms here, dedupe is almost never a good idea and compression is a much more common case-by-case.

> So personally I'd encourage messing around (on something non-prod, of course). Measure measure measure! Does it actually make any difference.

100%. So much depends on the particular workload and resources.

> And making backups, even for ZFS backed filesystems. Sacrilege, I know.

Logical backups, specifically. We have seen more than once corruption and data-loss in replicated encrypted snapshots due to bugs in ZFS.

View on HN

Deduplication requires an absolutely enormous amount of memory, unless you have a dataset that has a huge amount of duplication for some reason (maybe a company or school where a lot of people have their own network storage and put the same large files in?) it is usually never the right choice to enable dedup.

It is cheaper to buy more disk than the memory to enable dedup (somebody could run the numbers). ZFS ARC also likes lots of memory and you get good performance gains with more memory allocated to it.

In other words, turn on LZ4 compression everywhere and don’t think about it otherwise.

Deduplication in ZFS can be pushed onto dedicated flash storage now instead of requiring gobs of RAM.

How to do this?

Great channel for side attacks btw.

Doesn't even need SSD's to open a side-channel: https://lore.kernel.org/linux-btrfs/CAKDzk=-HZardsLFH5c9HYre...

Without having physical access to my hardware, where every disk is FDE anyway?

Physical access not required. Dedupe side attack required timing measurement.

Oh got some hate there.. pissed off the zfs kids with a dose of reality.

Dedupe can be used to infer secret data. .. more news at 11

This is my understanding as well. These days we default to zstd, though.

Oh good, I have it off. I was pondering that feature years ago when I set it up. Couldn't remember where I landed.

Data loss bugs in ZFS?? Wow.

My personal approach to filesystem evaluation is pretty much "How long has it been since I've seen someone talk about a data loss bug". This is my first time hearing one for ZFS. BTRFS seems to have at least one post a year. Ext4 has almost none, but maybe that's because people trust it and blame hardware when unsure what the issue is.

Comparing zfs/btrfs with ext4 is not entirely fair. They're in different domains when it comes to scope and features. To some extent, silent corruption in ext4 is expected but in zfs it's a bug.

Reply on news.ycombinator.com