Hacker News Reader

Comment by whateveracct | Hacker News Reader

This looks like an AWS equivalent of the Amazon-internal secret management tool called Odin. Which is very nice because Odin was pretty much universally loved from what I saw.

View on HN

There's still a bootstrap issue with AWS Secrets Manager - you have to set up enough tooling to be able to call the API.

The killer-app of odin imho was/is the on-machine http server that let all manner of applications very easily get credentials.

You could just do something like

  MY_API_KEY="$(curl http://localhost:5000?key=my-api)"

and boom your shell script or whatever was very easily using secrets.

The fact that Amazon systems bootstrapped EC2 instances meant they could easily enable this org-wide. The odin back-end then owned all the lifecycles around those secrets including what kinds of hosts they would go to, if/how they would rotate, etc etc. It has its annoyances including the fact you have to use http to localhost which can get saturated if you're not doing things right, but overall it really made secrets-management a non-issue.

That is actually pretty scary from a security perspective because of SSRF attacks. Basically if I can get your service to make an HTTP request on my behalf then I can get at your secrets.

Suppose you have a service that fetches snippets from user pasted links (like Slack does). All I have to do is paste a link of "http://localhost:5000?key=my-api" and your server would return a snippet containing the secret.

Thats why Google Cloud and others require a special header to be set. Hopefully Odin does the same? https://cloud.google.com/compute/docs/storing-retrieving-met...

This is already an issue on EC2.

https://ionize.com.au/stealing-amazon-ec2-keys-via-xss-vulne...

This is exactly how things already work on EC2.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-role...

TL;DR;

    curl http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access

will get you the credentials for s3 access if you define a role named s3access and grant that role the associated permissions.

After leaving Amazon you really go on to understand how good the internal tools are at Amazon and how they probably have the best tools in the industry.

I had the exact opposite reaction after leaving Amazon. Brazil is a slow, bloated pile of crap that regularly blocked fixes, made it impossible to develop in a non-standard language, and regularly resulted in 100+ dev build blockages due to versioning issues.

Our team was abandoning Apollo for MAWS, so our deployments were bridged between the two, resulting in another twisted pile of incomprehensible script-glue.

Pipelines was ok (!)

Everything on public EC2 for managing deployments (CodeDeploy and CloudFormation in particular) were so broken and bug-ridden and non-performant that we couldn't reliably deploy production code with them.

I prefer open source solutions in almost every way. I actually can't think of a single problem that the Amazon internal stack solves that isn't better solved in the open-source world. Which isn't surprising, given the constraints involved.

Brazil + Apollo + Pipelines. I left (and later boomeranged) and was surprised that the outside world hadn't solved these problems nearly as well. Don't get me wrong, Brazil and Apollo probably are ready for a rethink and rewrite, but they do a great job.

if you went to another bloated enterprise company, that wouldn't be surprising. If you went to a high-profile consulting company and had that experience, I'd be shocked.

What would you change about them?

Hard to say (especially in as public forum like this).

Brazil's dependency model is, I think, very much worth rethinking. Just try to use it for NodeJS. Apollo is fine but slow and dated and needs love. Pipelines works for specific software models and not others (ever seen a pipeline with 500 envs?). Great design for building custom approval steps though (long ago I wrote a now-popular one).

Except SIM.

The Lotus Notes of Amazon

+1 Hahaha! This made my day.

As a former Googler, I know a lot of Googlers that feel exactly the same way. It would be interesting for someone who has deeply experienced both to weigh in.

Brazil/Apollo needs huge renovation though, especially when Containers are getting so much more popular.

Odin works well, it's just a hassle to share credentials with non-developers who don't have a dev desktop and therefore can't access Odin. It's an issue I've encountered several times recently with no clear solution, it seems.

Odin works really well but but only because it integrates so tightly with with the rest of the internal tooling. Without Apollo (another internal tool) to manage and orchestrate the deployment of packages that need Odin's secrets, it's a lot clumsier to use.

Surprised at all the folks openly discussing internal tooling.

There are a specific set of internal things at Amazon that public discussion of would be detrimental to your career. Saying what you think of Odin isn't quite the same as mentioning the release date of the next Kindle.

Of course, and I don't think anyone would or should get reprimanded for it. That said, I still think it's poor form and should be avoided unless absolutely necessary.

I miss having ODIN.

Reply on news.ycombinator.com