Hacker News Reader

On the other hand there are a couple of things in ZFS that can make a big difference. For instance: changing the page size to be the same size as Postgresql uses (8kb); disabling the writing of 'last accessed' timestamps; or whether or not the filesystem attempts deduplication. From what I remember (it's been a while) these can be configured on a per-dataset basis, where a dataset is a lightweight filesystem more akin to a directory, and many of them can be changed on a live filesystem.

Here, have at it: https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops...

So personally I'd encourage messing around (on something non-prod, of course). Measure measure measure! Does it actually make any difference.

It's a shame a bug crept in to an otherwise pretty much faultless filesystem. I'm sure there are some truths in there about commercial support or not running the bleeding edge or whatever. And making backups, even for ZFS backed filesystems. Sacrilege, I know.

> whether or not the filesystem attempts deduplication

Unless I'm just parroting truisms here, dedupe is almost never a good idea and compression is a much more common case-by-case.

> So personally I'd encourage messing around (on something non-prod, of course). Measure measure measure! Does it actually make any difference.

100%. So much depends on the particular workload and resources.

> And making backups, even for ZFS backed filesystems. Sacrilege, I know.

Logical backups, specifically. We have seen more than once corruption and data-loss in replicated encrypted snapshots due to bugs in ZFS.

Deduplication requires an absolutely enormous amount of memory, unless you have a dataset that has a huge amount of duplication for some reason (maybe a company or school where a lot of people have their own network storage and put the same large files in?) it is usually never the right choice to enable dedup.

It is cheaper to buy more disk than the memory to enable dedup (somebody could run the numbers). ZFS ARC also likes lots of memory and you get good performance gains with more memory allocated to it.

In other words, turn on LZ4 compression everywhere and don’t think about it otherwise.

Deduplication in ZFS can be pushed onto dedicated flash storage now instead of requiring gobs of RAM.

How to do this?

Great channel for side attacks btw.

Doesn't even need SSD's to open a side-channel: https://lore.kernel.org/linux-btrfs/CAKDzk=-HZardsLFH5c9HYre...

Without having physical access to my hardware, where every disk is FDE anyway?

Physical access not required. Dedupe side attack required timing measurement.

Oh got some hate there.. pissed off the zfs kids with a dose of reality.

Dedupe can be used to infer secret data. .. more news at 11

This is my understanding as well. These days we default to zstd, though.

Oh good, I have it off. I was pondering that feature years ago when I set it up. Couldn't remember where I landed.

Data loss bugs in ZFS?? Wow.

My personal approach to filesystem evaluation is pretty much "How long has it been since I've seen someone talk about a data loss bug". This is my first time hearing one for ZFS. BTRFS seems to have at least one post a year. Ext4 has almost none, but maybe that's because people trust it and blame hardware when unsure what the issue is.

Comparing zfs/btrfs with ext4 is not entirely fair. They're in different domains when it comes to scope and features. To some extent, silent corruption in ext4 is expected but in zfs it's a bug.

Also, compression. CPUs are more than capable of handling it on the fly, and the reduced dataset size helps IO.

This [0] is a brief post on MySQL, but most of it applies equally to Postgres.

[0]: https://www.percona.com/blog/mysql-zfs-performance-update/

I'd be curious if the addition of zstd compression has any effect on the results. Could be a fun follow-up follow-up :-)

For Postgres just follow the original doc:

https://openzfs.github.io/openzfs-docs/Performance%20and%20T...

compression=lz4 recordsize=32K

In postgres -> full_page_writes=off

But also experiment with recordsizes of 8k and 16k. The larger the recordsize, the more potential compression you get, which may be offset by additional delay caused by read/modify/write of the larger record (i.e. reading/writing 32k instead of 8k or 16k).

True, with platter storage i tend more to 32k with nvme more toward 16k. But yes testing is a good thing, but some settings should be just untouched if one is not absolutely sure it's a good thing, aka logbias=throughput.

> disabling the writing of 'last accessed' timestamps

atime is the most useless filesystem feature I’ve heard of. But on most Unix-like systems it’s enabled by default and the main thing it does is turn reads into writes, degrading performance. All the while ctime isn’t a standard Unix filesystem feature. And ctime to me always seemed infinitely more useful (and cheaper!) than atime.

</rant>

relatime has been the Linux default since 2009. It makes that be one write per 24 hours.

while it is unfortunate it requires a write, it is useful in some cases. for example, finding old objects in a cache, or today i used it to find out what firmware my kernel is loading by checking atime of /lib/firmware.

So it's useful as a debugging feature that you might want to briefly turn on once every decade.

Finding old objects in a cache is an issue specific to caches and should be implemented there.

For the latter you could enable it, reboot and get the same information right away, then disable it?

i did not see a clear list of all firmware files read by the kernel in dmesg, so the find/atime trick worked well.

> changing the page size to be the same size as Postgresql uses (8kb)

Mind, you shouldn't really change the record size without knowing your use-case. Make a PostgreSQL file system, set it's recordsize to 8k. That's a good idea.

Not so much a good idea doing that to all your filesystems. Stick to the defaults without a good justification to deviate.

A lot of end users run ZFS through TrueNAS/FreeNAS and I indeed found that community to spread a lot of misinfo and un-actionable advice. I liken them to gamers lapping up any snake oil solution to get 0.1 fps more. I would suggest anyone trying to learn about ZFS to avoid it, it definitely was counter-productive to me.

Reddit, random blogs and official documentation (often that by Oracle) turned out to be much better resources.

I think it comes down to the fact that the TrueNAS developers treat it as an appliance (which is a good thing, IMO). However, because so many people's only experience with ZFS is TrueNAS, when iXsystems says "Doing X on TrueNAS is a bad idea, that's not a configuration we support," it gets cargo culted as "doing X on ZFS is bad."

Using it through something like TrueNAS with sensible defaults is probably a better idea for the average person than trying to roll their own deployment, no?

Im not sure running TrueNAS counts as learning ZFS, which I think is GP's point.

If you want to use ZFS, then TrueNAS with sensible defaults is a great way to go. If you want to learn ZFS enough to roll your own deployment, change the configuration correctly and understand the changes you make then don't start with TrueNAS, and definitely don't listen to the TrueNAS community.

Or at least, that's how I read the GP comment.

I mean, pretty much if you want to learn anything you're going to have to put a lot of time into it. With ZFS it will be building different servers with different disk layouts and figuring out how to test and benchmark those options. Which just takes an absolutely massive amount of time.

I know, I was doing just that 5+ years ago.

Commercial storage vendors aren't really any different. Mostly the biggest difference here is they offer a very limited subset of hardware and disk configurations and tend to iterate over that very small configuration space. Step outside of that config space and they won't have much insight.

This remind me of the earlier days of Mac vs PC. Mac tended to have a very small hardware set of supported devices, and in many ways helped stability. PC supported pretty much anything you threw at it, which commonly had issues with crappy drivers and untested interactions.

> If you want to use ZFS, then TrueNAS with sensible defaults is a great way to go.

I evaluated TrueNAS Scale about 6 months ago and bailed on it very quickly. My install with their "sensible defaults" ended up having swap partitions on all my spinning disks. That alone was enough for me to decide against using it. One bad choice likely means other bad choices in my experience. They also trampled / reset my 'zfs_zrc_max' tunable every time a VM got started (I think) and it wasn't obvious what was doing it.

I didn't think there was any value in learning the quirks of TrueNAS over doing a setup from scratch, so I ended up going with Ubuntu LTS and doing all my own config / testing.

Reminds me of a game (MMO) I play on the weekends. Any time someone new joins, they always ask how come frame rates are so low. Then we begin the ”set your graphic settings to high” conversation. They immediately say we are full of shit. It’s already set to low and getting 10 fps.

Eventually, someone explains: “just try it, or stop complaining.”

30s later, they exclaim: “holy ** I’m getting 60 fps now!”

Sometimes, what looks like misinformation isn’t misinformation.

TL;DR: this game switches to a software rendering pipeline on “medium” or lower. You have to set it on “high” to switch to the hardware rendering pipeline.

Is that a visible option somewhere or do you only have one setting, the preset?

The preset is the only way to influence it.

Wow! Thats a great story

One of my favorite comments chains here was was something to the affect of:

Commenter posts a nice summary of how ZFS will try a quick compression algorithm to see if a block is even compressible and then try other more expensive ones to achieve better compression.

Another commenter asks how they know that.

First commenter replies that they wrote the code for it.

Nothing more than a ZFS user (to some extent my company relies on it even) here, so I just try and color inside the lines, but I'm ever impressed by it.

You probably mean [1].

[1] - https://news.ycombinator.com/item?id=36608024

That is it, thank you.

> So when you go to the issue tracker, it seems plausible to me that you're going to find issues where the devs can't explain what's happening because the user(s) might be doing something crazy.

No, that is not plausible. It's a filesystem. Behaviour in all cases for a filesystem should be predictable. There are zero things a user should be able to do that would make a filesystem non deterministic.

Tbh guessing is part of the way to solve a bug. You need to make a hypothesis and the test if it is true. If not try another.

Saying the zfs/btrfs teams have no idea means the person has no idea about software development process ... It is a system none fully knows a system that complex.

If they did, the bug wouldn't have happened.