Comment by ericbarrett | Hacker News Reader

Comment by ericbarrett | Hacker News Reader

Parent Full thread

I really have to disagree here, at least with your conclusions re complexity. No doubt AWS S3 configuration is very technical and has a lot of docs and corner cases. However, if you are a company like Twilio, and S3 is front-and-center the source of your business (and if it's serving your SDK, it probably is), your S3 bucket permissions are clearly worthy of periodic review. And you should have this done by somebody who knows it well, or who learns it well in the process.

I worked for a company where our CTO identified the same thing (unauthorized modification of JS SDK served via S3) as a potential vulnerability. It took us half a day to get the bucket locked down to where only the root AWS IAM user—secured by physical 2fac—was permitted to change the SDK, and then only after "unlocking" the configuration changes. Since the SDK was not updated frequently, having a little manual ceremony was fine. If it had been a high-touch path we would have had to do a little more engineering to keep it safe, but not that much more.

That said, I don't blame Twilio for not catching this; shit happens at a company with years of legacy built up. I do blame anybody who says "S3 is hard" and throws up their hands. You're a professional! Read the docs, they're dry but quite extensive! Play around with it in a test bucket! It's no harder than learning any other moderately complicated professional system. If you can learn Rust, you can learn S3's API. If you can learn Kubernetes, you can learn S3 ACLs and the IAM authz model. It may not be as interesting as the topics you want to read about, but it will probably be even more useful.

The flaw in your logic is that "S3 is front-and-center the source of your business" is never true with AWS.

To do anything in AWS there's at least 3 or 4 vaguely connected services, including IAM and RAM as a completely separate UIs. Usually with S3 you also have Cloudfront or some other CDN. You probably also have Cloudwatch logs and CLoudtrail event tracking. You might have VPCs involved. Some or all of these things, in an org like Twillio, could have entirely different teams managing them.

The problem isn't that S3 is hard. The problem is, as OP suggests, the UX of actually doing anything non-trivial across half a dozen services is somewhere between abysmal and war-crime.

Each of these services are highly different concerns and don’t belong together, and are honestly not _that_ hard to orchestrate together with the liberal application of Terraform.

If you’ve got one team managing your logging who don’t talk to whatever team who manages your s3/cloudfront setup, then that’s your problem.

> and are honestly not _that_ hard to orchestrate together with the liberal application of Terraform.

I love Terraform, but the fact that you require an entirely different company's orchestration tools to make AWS "not that hard to orchestrate" is evidence of how poor the UX of AWS is.

But terraform is using the aws sdk behind the covers.

Most of the stuff available with terraform is available with CloudFormation minus a thing or two. (takes them a minute to catch-up to other teams new services/features)

You can use Cloudformation. Like, what do you want here? Some super console that nobody asked for because anyone who knows what they are doing knows why you want infrastructure as code?

Might be a war-crime, but people at the enterprise level like twilio, don't do anything at the UI level, and if someone was to do something at the UI level, AWS blocks public access _by default_

AWS console was only complicated as a complete noob. Once you understand how services work you actually realize is decent.

These only seem “vaguely connected” if you haven’t studied them together. IAM and S3 are intricately intertwined and you should understand both at at least an intermediate level to work with them in sensitive environments. (Note that IAM has permissions specific to each Amazon service, all documented in detail.) None of the rest of the systems you mentioned gate write access, though certainly if they’re in your production flow you should study and review them as well.

> all documented in detail

Thanks for the giggle.

If you have CloudFront in front of your bucket, the bucket shouldn't even be world-readable, let alone world-writable. Permission should be delegated to CF to read out of the bucket.

>That said, I don't blame Twilio for not catching this

I do.

Twilio is a multi-billion dollar company and there is no excuse for them not having proper security processes to catch stuff like this early. Even if we take the "S3 is hard" arguments at face value, this wasn't a 0-day or some complicated unpredictable exploit. This was an extremely basic misconfiguration on a mission-critical part of their architecture that would have been caught by even the simplest out-of-the-box penetration test or audit. For a company like Twilio to not be doing basic, fundamental stuff like that is a big deal and they certainly should be blamed for it.

your S3 bucket permissions are clearly worthy of periodic review.

better tooling makes this easier to do.

And you should have this done by somebody who knows it well, or who learns it well in the process.

No one knows S3 better than the AWS S3 team.

It took us half a day to get the bucket locked down to where only the root AWS IAM user—secured by physical 2fac—was permitted to change the SDK

Or a small feature team could spend a month or two improving the user experience so that literally every S3 customer doesn't have to have the technical acumen, knowledge of risks, and time to get this right. 0.5 days * a million users (to be conservative) is 500,000 days.

Reply on news.ycombinator.com