Hacker News Reader

Comment by gymbeaux | Hacker News Reader

No mention of how AWS/S3 approximates the size of a file to save CPU cycles. It used to drive me up a wall seeing S3 show a file size as slightly different than what it was.

If I recall correctly, S3 uses 1000 rather than 1024 to convert bytes to KB and MB and GB. This saves CPU cycles but results in “rounding errors” with the file’s reported size.

It’s discussed here, although they’re talking about it being “CLI vs console” which may be true?

https://stackoverflow.com/questions/57201659/s3-bucket-size-...

View on HN

This almost certainly has nothing to do with “saving CPU cycles” and is most likely just that whoever created the Cloudwatch console used the same rounding that is used for all other metrics in CW, rather than the proper calculation for disk size, and it was a small enough issue that it was never caught until it was too late to change it because changing it would disrupt the customers that have gotten used to it.

If anything a bit shift could do the 1024 division faster

S3 doesn't format the size for human display, the object's Size is returned in bytes:

https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.h...

What I meant to say was S3 uses 2^20 to convert from bytes to kilobytes rather than 10^6 (e.g. what Windows uses).

https://repost.aws/knowledge-center/s3-console-metric-discre...

> 3 console and Storage Lens use base 2 conversion (/1024) to report storage metrics, and CloudWatch by default uses base 10 conversion (/1000)

But KB is indeed 1000 bytes, and MB is indeed 1000 KB.

In case of 2^10 units the correct names are kibibyte (KiB) and mebibyte (MiB). Check https://en.wikipedia.org/wiki/Mebibyte#Multiple-byte_units

Yeah we have long standing confusion that for historical reasons KB and MB often means 2^10 bytes units so now when you see KB you really don't know what it means. Therefore I am a staunch supporter of unambiguous KiB and MiB.

I think I explained this poorly, and I appear to be mistaken about it being to save CPU cycles (though an entity such as AWS would absolutely be about saving minuscule CPU cycles that add up at scale).

I spent a lot of time researching this issue when I came across it years ago- I just remember that local file size and S3 file size was not matching up with anything and the takeaway was that S3 was calculating file size differently from Windows/Linux/macOS.

AWS uses base 2 (binary) for calculating file size, meaning 1MB is treated as 1,048,576 (2^20) bytes rather than 1,000,000 bytes (10^6).

And as you've said, one is MB while the other is technically MiB.

Downvotes notwithstanding

https://repost.aws/knowledge-center/s3-console-metric-discre...

Reply on news.ycombinator.com