Hacker News Reader

Comment by throwdbaaway | Hacker News Reader

Compressed ARC has been a thing in ZFS for a long time. So let's say you allow ZFS cache to take 50% of total memory. With 4x compress ratio from ZSTD, you will be able to cache 2x the total memory. It is amazing.

View on HN

Yes, but on the other hand, my toast relation now needs (e.g.) 4x the amount of blocks it would have had when compression was enabled. So we have a net 0 storage savings, except that we now need to request 4x as many pages from the OS to get the full toast data, those pages taking up 4x as much space in Postgres' shared buffers, also evicting 4x as many pages from the buffers as it would have otherwise.

Unless your read operations to the file system are somehow very cheap, I don't think disabling toast compression is a good idea.

LZ4 is quite fast to bail on data it can't compress, so why not just have LZ4 enabled on both?

I see your point. Perhaps the best way is to use TOAST compression for wide field values, and then rely on ZFS filesystem compression for others? For some context, in MySQL, both the table-level compression and page-level compression are problematic, while ZFS has served me well, thus I have a lot more faith in it.

> a Perhaps the best way is to use TOAST compression for wide field values

You don’t have to do anything. Toast compression is enabled for compatible types by default but never kicks in until a cell hits 2K bytes in size (by default).

You should not use ARC for pg data, since postgres has it's own cache, otherwise you just have two caches with the same data in it.

Unfortunately that's just how postgres works? I thought the rule of thumb is still to allocate 25% of memory for postgres 's shared buffer, and 50% of memory for OS buffer cache (i.e. ARC when using ZFS)?

That’s how postgres works, but ZFS lets you work around that by disabling arc caching on a dataset by dataset basis.

>Unfortunately

No, postgres know better what to cache then arc.

For a transaction processing workload (where hitting indexes is the all-important thing), that's exactly right. But if you have more of an analytic workload involving sequential scans, Postgres definitely expects the underlying storage system to be handling at least some of the caching.

Reply on news.ycombinator.com