Hacker News Reader

Comment by renewiltord | Hacker News Reader

We have a delta-encoded binary format for trade logs that we use and zstd still does a pretty good job on them for the last bit.

It's just a pity that storage providers won't implement transparent compression. We store our logs on S3 and on-prem and it's just annoying that we have to store them compressed on S3. Would be cool to just store without the compression stage and have S3 do that bit.

I haven't tried btrfs with the transparent zstd compression, but perhaps worth a shot.

View on HN

One reason this could be messy to add is that the S3 API includes support for reading arbitrary byte ranges within a file. There's an inherent conflict between that API and support for transparent compression. Compression performs best when compressing the entire file from beginning to end; when decompressing you'd similarly decompress from the beginning. To support byte range reads, the data could be compressed in short blocks, but those blocks will not compress as well.

One upshot of compressing outside s3 is that you save on data transfer. In theory I suppose the compressed data could be sent over to the client and decompressed but then it’s basically the same thing.

it’s also a conflation of two things that are each much more sustainable if kept independent. You don’t want a particular compression pattern becoming part of the S3 standard IMO

Yeah I’ve suggested to the s3 leadery folks many times to provide some sort of managed compression and other sorts of codec interfaces. They could do other things like convert between structured encodings as well. Unfortunately it’s never gone anywhere.

Isn’t that what S3 object lambda access points are for?

https://docs.aws.amazon.com/AmazonS3/latest/userguide/olap-c...

It’s too expensive. The thing is s3 has a ton of compute sitting around. The disks have a full system behind them on the storage nodes. A big question always was how can we do more with the edge storage? S3 select was one effort (although I don’t know if ever got pushed to the edge, it originally ran in the front end servers). But I always figured compression was a real obvious s3 edge storage operation, along with as I said format conversions.

Could you use a Lambda to compress with zstd on write and S3 Object Lambda to decompress the stream seamlessly on GET (and update the content length on HEAD)?

Yes. That adds costs though

Obviously, but the whole point of this exercise would be because you anticipated the reduced storage GB-hours saving more than the compute time, right?

zstd is quite slow at higher compression levels, if S3 would compress who pays the compute?

Maybe they could offer it as an extra, and spawn EC2 instances as needed and bill you.

I still wish s3 object select supported zstd on json.

Reply on news.ycombinator.com