Sure, but now you've added an extra layer of indirection which can have a significant impact on performance
I implemented this as part of the MinIO server. See "Seeking Compressed Files" here: https://blog.min.io/transparent-data-compression/
We choose a compressor without literal compression for a faster baseline, but the concept remains the same.
the index goes in there, no extra seek needed
yeah, 4 bytes for every megabyte
> s3 really isn't the right layer to implement compression. filesystems aren't either. it's better to leave it up to the application.
yeah, I'm sure you're right and Amazon have absolutely no idea what they're doing and like to spend unnecessary CPU cycles doing pointless work and add "significant bloat" to their metadata
... or, you're wrong (like in every previous comment in this chain)
this tweet is not talking about compressing customer data in s3, i seriously doubt that aws compresses customer data in s3 for all the reasons i've already listed. i am right and amazon does know what they're doing, which is why they don't compress customer data in s3.
4 bytes per megabyte becomes significant at scale when you have to keep it in ram, which you have to do if you want to avoid the extra IO.
Each part can be max 5GiB as per S3 spec. 5120 * 4 = 20KiB.
Even if you unpack to 8*2 bytes in memory when decoding, you are still not talking a huge amount of memory.
The on-disk space is ~0.0004% as blibble calculated, and should easily be offset by the compression achieved. In MinIO we don't store indexes for files < 8MiB, so for small files there is no overhead.
If the added metadata is a problem for whatever system you are looking at, then that is a characteristic of that system and not a general problem.
and you don't understand the algorithm if you think you need to keep the index in RAM, because you don't
as has been explained to you several times, you don't
> if you pollute the metadata cache with useless junk like this
the overhead is 0.0004% with 1 index entry per megabyte, and if that's too much that can be reduced by 10/100/1000/10000x that by changing the size
as we're clearly now going around in circles, I won't be responding again.