Hacker News Reader

Comment by adastra22 | Hacker News Reader

It’s so bizarre that every time I upload something to S3 I have to jump through a hoop to make it publicly readable and Amazon displays a massive warning sign.

Like the only thing I use S3 for is hosting open source software binaries. Maybe there’s a use case for restricting access, but I don’t even know what that would be.

View on HN

Inadvertent public buckets leading to data loss is what created those hoops. Trying to take the ammo out of the footgun.

https://www.theregister.com/2022/12/14/aws_simple_storage_se...

But why would they upload private data to S3 in the first place? I’m just not understanding the context here.

S3 is just a general storage bucket, anything you could use a hard disk for, some app is using S3 for that instead. It's particularly common in serverless app backend architectures where no permanent storage exists at all, so volumes are not an option.

A good deal of hosting is done on ephemeral VM instances that get rebuilt, so you choose between volumes and S3, and S3 is a bit more flexible.

Many applications use s3 as a data store for... Whatever. Uploaded or generated files, intermediate outputs, as a key-value store for large blobs.

You can put all sorts of things on s3. The PDF containing your bank statement, your medical test results, whatever - S3 is plenty secure enough for that with sane configuration, and even by default with sane account management.

Because half the world is below average.

Below median, not average

Indeed, thank you for the correction.

A fair chunk of modern corporate data ends up in OLAP systems that are now more often or not stored using s3 (or their MS/Google equivalents) in 'Data Lakes'. The concept of 'private data' whilst using cloud providers is an interesting one, but there has been enough work done by all parties involved to ensure that all but the most sensitive data is now created, stored and analysed using the systems provided by "Big Cloud".

You really need to think carefully about whether you want to expose an S3 bucket publicly. There are probably some valid reasons out there, but if you're not an AWS expert, it's likely that you're making a mistake. If I find out the name of your bucket I could cost you thousands of dollars of egress tonight before you wake up in the morning. It's _especially_ likely to happen to hosters of open source binaries because of people absent-mindedly downloading artifacts from CI jobs. No malice required. They make the mistake but you pay the bill.

Hosting a file on a url anyone can access from anywhere is the whole point…

Public S3 buckets are not a good choice for that because of the "anyone could bankrupt you" reason above. That is generally NOT the use case for S3, and it's the reason why private is the default and there are alerts for public buckets. For public access, within AWS, you'd want to put CloudFront in front of the bucket and only allow external users to access CloudFront. However, there's a better option...

Outside of AWS, Cloudflare has a service called R2 which is just like S3, except you DON'T pay egress! It's the same thing, but without the "anyone could bankrupt you" aspect. You pay for the storage but you don't pay per download. I highly recommend this for hosting open source binaries. You can still keep a copy in a private S3 bucket for safe keeping if you don't trust Cloudflare long-term.

Thank you I’ll look into that.

Given the amount of data breaches that have happened from S3 buckets being accidentally publicly accessible, it definitely makes sense to me.

Reply on news.ycombinator.com