I have a huge DB which would take many TB but now runs on a 500GB footprint. Thanks to Rocksdb/MySQL. Pondering a move to Postgress, but seemed like a step down in this regard.
I have a huge DB which would take many TB but now runs on a 500GB footprint. Thanks to Rocksdb/MySQL. Pondering a move to Postgress, but seemed like a step down in this regard.
If you want large datasets (petabytes) you really need to look elsewhere to something with better compression support or that tiers its data off to S3.
It's possible that with bcachefs we're like a decade away from "good fs in mainline kernel with fs compression" but right now it's not a great situation.
Otherwise, Timescaledb offers compressible table chunks using table inheritance. It's really pretty slick, depending on your use-case.
And a lot of data is already compressed (images, etc) so might increase your compute cost without any storage savings.
Filesystems are often the wrong level in the stack to do compression. You want individual control over what data is compressed.
I agree with this point, but I think those online compression algorithms add very little overhead, so it is mostly about if Postgres adds enough other benefits compared to your current DB.
The bigger issue potentially is that ext4 is faster than zfs and btrfs itself regardless of compression.
It's not just compute power, I was generalising with that statement. The real world speed hit can be as bad as read speeds an order of magnitude slower than if you disable compression.
You really only want to compress data where it makes sense to compress it.
data is compressed and decompressed in blocks..