I doubt there’s a single byte stored to disk that is not compressed and encrypted at AWS. It’s transparent to the customer.
It might make more sense on their "deep archive" product maybe where the customer has to commit to a minimum storage retention and also pay a retrieval charge which scales with the amount of data recovered (hence paying for the CPU to decompress).
Since CPUs are fast enough to deflate in real-time now, your bottleneck for a read is your storage/network.
Reducing the bytes read from storage improves the IO latency.
Only temporarily with SSDs. With spinning rust, it also often paid off to compress data. We'd store large treebanks compressed, because decompression was much faster than disk reads.
Hardware compression is sadly not widely available, I think the only consumer product I know with it is the PlayStation 5.
The mainframes from IBM have had hardware zlib since Z14 iirc and in my small tests it is very fast compared to the CPU implementation
Example:
You have an existing 40 gigabyte file
It happened to compress well
You delete it and your free disk space goes up by 4 gigabytes.
You then write a new 40 gigabyte file that doesn’t compress well
Replacing an existing file of the same size just ate an extra 36 gigabytes.
How would you plan around that? SSDs should store the bytes given and don’t play fancy games.
Adding life to SSDs is a terribly useful feature
Given that, it's good sense to compress if at all possible simply to make the drive live longer.
And guess what - I just googled, tons of hits, and this has been done for a long time :)
So it makes sense, is done, and is important for modern SSD behavior.
I'm surprised (and shocked) that letting unencrypted data hit the disk is still common enough to make such optimizations worth it.
Even if you just stick the key in the server's TPM without any sealing, an encrypted disk makes it much easier to deal with e.g. drive returns (for warranty or fault analysis) or disposal.
Bitlocker supports on drive hardware encryption, and I'd be surprised if other major file systems didn't.
If I recall, it's a FIPS requirement for data at rest now.
Because the storage hardware might be loyal to someone who is not the owner of the data.
Because the hardware cannot be trusted to do it correctly. IIRC Bitlocker stopped relying on it for this reason.
https://www.howtogeek.com/fyi/you-cant-trust-bitlocker-to-en... (see the updates)
There's very, very little benefit in encrypting data at a filesystem level in a datacenter if you think about it.
And unfortunately compression and encryption are seemingly at odds fundamentally :c
"Even if you just stick the key in the server's TPM without any sealing, an encrypted disk makes it much easier to deal with e.g. drive returns (for warranty or fault analysis) or disposal."
Would you resell usable drives that you no longer want to use (e.g. because they're too small or your needs changed more towards SSDs) if unencrypted data was written to them at some point?
How much effort would you put into making sure that no broken drive that can't be wiped leaves the datacenter without shredding?
Any process you put in place will have gaps - e.g. through human error or malicious acts - and this provides pretty solid protection against that.
>Data compression via encoding algorithms enables a solid state drive (SSD) to write less data, which in turn yields higher write bandwidth. With a significant amount of data being compressible, performance benefits can be substantial.
https://www.intel.com/content/www/us/en/support/articles/000...
I would expect this to become part of newer generations of CPUs once it becomes popular.
My Intel i9 11900k seems to support it or has it built-in.
Maybe they charge at uncompressed rate but store it at compressed? Then they got even more money!
For any storage system like this you usually have a few bottlenecks. IO and Network are the obvious ones, followed by tiering (cache, fast io, slow io, ...) and at the very end CPU.
Now let's say network is your bottleneck. If you can send the data to the client in a compressed for then you get the compression ratio as additional bandwidth. And the user would get the data quicker! So compression to the network is a clear win.
But the common bottleneck is often IO, a high end SSDs with 1M IOPS at 4KB would _theoretically_ serve 4GB/s, a 40GBit link. That's without any redundancy over other overhead.
Again compression to the storage layer would decrease the total amount of IOs, thus making sure a customer gets data quicker.
Ok, let's say both are not the issue. The fastest compression algorithms compete with memcopy. So if you need just one copy of your data you might have been faster by compressing it.
Especially fast compression algorithms (zstd, lz4, snappy, lzo, ...) are worth the CPU cost with virtually no downsides. The problem is finding the right sweet spot that reduces the current bottleneck without creating a CPU bottleneck, but zstd offers the greatest flexibility there, too.
Oh for range requests.... Those large objects are likely split anyway, for easier error recovery (imagine 100MB into a 1GB transfer you notice that the file data was corrupted - not good). Once you work on blocks it's easy to do somewhat efficient range requests again.
It would be waste to _not_ use the CPU time.
Granted, the CPU powet consumption would drop with enabled power management, but that's only marginal gains as the CPU is likely busy anyway and the total duration of busy time might drop (race to sleep)
I have used this inverted logic to great extends. E.g. when expanding data center capacity I was usually swapping old harware for new hardware once it arrived. It is way better to have the older generations as spares and use the new capacity at the most utilized places.
Do not waste the things you pay for!
That said, a surprising number of video sources use sparse file type setups, and I've gotten pretty good compression (up to 60%) using LZ4 with NVR files from some brands.
But we are talking about multi-tenant systems here. You will get better performance and/or better prices if the system finds compressible data for other customers.
All the benefits hold, even for you with non compressible data, if there is an overall benefit.
Hitting a bottleneck less often then this will reduce your tail latencies, too.
It is just that _your files_ won't contribute to these improvements. But you get all the benefits as well.
this is pretty easy, you flush the compression buffer every megabyte or so and maintain an index
maybe 50 lines of code
I implemented this as part of the MinIO server. See "Seeking Compressed Files" here: https://blog.min.io/transparent-data-compression/
We choose a compressor without literal compression for a faster baseline, but the concept remains the same.
the index goes in there, no extra seek needed
yeah, 4 bytes for every megabyte
> s3 really isn't the right layer to implement compression. filesystems aren't either. it's better to leave it up to the application.
yeah, I'm sure you're right and Amazon have absolutely no idea what they're doing and like to spend unnecessary CPU cycles doing pointless work and add "significant bloat" to their metadata
... or, you're wrong (like in every previous comment in this chain)
this tweet is not talking about compressing customer data in s3, i seriously doubt that aws compresses customer data in s3 for all the reasons i've already listed. i am right and amazon does know what they're doing, which is why they don't compress customer data in s3.
4 bytes per megabyte becomes significant at scale when you have to keep it in ram, which you have to do if you want to avoid the extra IO.
and you don't understand the algorithm if you think you need to keep the index in RAM, because you don't
as has been explained to you several times, you don't
> if you pollute the metadata cache with useless junk like this
the overhead is 0.0004% with 1 index entry per megabyte, and if that's too much that can be reduced by 10/100/1000/10000x that by changing the size
as we're clearly now going around in circles, I won't be responding again.
Each part can be max 5GiB as per S3 spec. 5120 * 4 = 20KiB.
Even if you unpack to 8*2 bytes in memory when decoding, you are still not talking a huge amount of memory.
The on-disk space is ~0.0004% as blibble calculated, and should easily be offset by the compression achieved. In MinIO we don't store indexes for files < 8MiB, so for small files there is no overhead.
If the added metadata is a problem for whatever system you are looking at, then that is a characteristic of that system and not a general problem.
If you can stuff more bits in an IO operation, you’re winning.
Modify the compression level to try and keep the output buffer at 60% full.
On Linux PSI ( /proc/pressure/io ) probably provides more accurate information (the code already uses /proc/cpuinfo ).
Detail: in the fileio.c module there are lines such as:
if (oldIPos == inBuff.pos) inputBlocked++; /* input buffer is full and can't take any more : input speed is faster than consumption rate /
if ( (inputBlocked > inputPresented / 8) / input is waiting often, because input buffers is full : compression or output too slow */
This impacts a 'speedChange' variable.
Its potential values (an enum) are 'noChange', 'slower', and 'faster'. They are processed rather simply: if (speedChange == slower) { ((...)) compressionLevel ++;
if (speedChange == faster) { ((...)) compressionLevel --;
If the difference between standard S3 and S3 Glacier was just slower disk, then rate limiting the customer would suffice.
But if there’s a significant amount of compute thrown at data de-duplication, compression, and indexing, then it starts to clarify why there’s a pricing penalty for using Glacier with the same access patterns as one would use on standard storage.
That's not a good argument: they could lower their costs with compression, still charge the same, and make more profit.
And what if they can charge you the uncompressed size and only actually store much smaller compressed files behind the scene. That seems like having your cake and eat it too.
Like when it was easy to file share on dropbox by having the correct hashes. A GUID could summon a 1GB file.
Content addressing across accounts on private, AWS encrypted S3 buckets would run counter to their claims.
tl;dr:There's a real efficiency/security/insider risk trade-off here.
Edit: I should disclose that I work for a competitor. Don't intend any astroturfing.
That said, of course, customers can upload encrypted blobs of uncompressed data. But I’d call it an exception that proves the rule. Here service simplicity should win and those blobs may end up recompressed.
Since we already used a Snappy-derived method, each 1MB block is stored without backreferences. With this we only have to decode at most 1MB-1 extra bytes to respond with a specific range offset.