20 years on AWS and never not my job

The author calls it a 'joke' that Heroes are just unpaid Amazon employees, but reality doesn't become a joke just because it's funny. The asymmetry here is staggering. I find myself holding back private research because I don't want to provide free R&D for a value-extraction machine that is already efficient enough.

The author was at least dependency-driven in their contribution, but outside that kind of dependency, it's hard to justify contributing even 'in the open' when the relationship is this one-sided. Amazon in particular has done enormous damage to the economic assumptions that permissive open source once relied on. There's increasingly more projects adopting 'Business Source Licenses', precisely to prevent open work from becoming a free input into hyperscaler monetization.

These devs know Amazon is grabby and, at some point, the only dominant outcome their community contribution is upstream of is unpaid labor for a trillion-dollar entity that also diverts support and community engagement away from the original projects by funneling users into managed versions of the same software.

I understand people have a viewpoint here about not giving time to large behemoths. I'll counter with a story and perhaps a larger point.

Back in 2006/7 I had an idea for a project for which, in all enthusiasm, I setup a mailing list, but ended up never pursuing it. It's a very unique name.

In 2012, another developer landed on the same name for their project, but saw that the mailing list was taken up and reach out inquiring if he could take over, and I obliged because here's another person doing something in cryptography and open source, 2 of my favorite things then (and now).

The project was "scrypt" and the developer was Colin! :) I knew nothing about Colin or tarsnap then, IIRC.

Sometimes you just do kindnesses of which you're able, with people who you feel a sense of community with, without expectation of anything commercial. Karma adds up, and it's benefits are large, though hard to always articulate.

> in fact in one of Jeff Barr's AWS user meetups in Second Life

There's so much about that phrase that makes me smile. Easy to forget that Second Life was also one of the earliest users of AWS, S3 first. Jeff Bezos had personally invested in our 2005 round (a round that made Linden Lab a unicorn before that was a thing) and pointed us at Jeff Barr and the work coming from AWS.

In return, Jeff Barr started hosting AWS meetups in Second Life -- this was the era of lots of groups setting up Second Life outposts, from Jonathan Coulton to Reuters.

> In April 2024 I confided in an Amazonian that I was "not really doing a good job of owning FreeBSD/EC2 right now" and asked if he could find some funding to support my work, on the theory that at a certain point time and dollars are fungible

>I received sponsorship from Amazon via GitHub Sponsors for 10 hours per week for a year

For whatever reason, I remember being shocked that you were only charging $300/hr [1] which was what a mere L6 google engineer would make salaried. I hope they are paying you more nowadays

[1] https://news.ycombinator.com/item?id=30188512

I strongly disagree with the part about IAM roles for EC2

> a useful improvement (especially given the urgency after the Capital One breach) but in my view just a mitigation of one particular exploit path rather than addressing the fundamental problem that credentials were being exposed via an interface which was entirely unsuitable for that purpose.

What alternative interface does the author propose we use to securely exchange credentials? The only other approaches I can come up with involve allowing monkey hands to come into direct contact with secret materials. Outlook, slack and teams cannot possibly be more secure than IMDSv2. I think if you are manually passing around things like PFX files you've already lost the game.

The entire point of the IAM roles is to make everything a matter of policy rather than procedure. The difference here is insane when you play through all of the edges. IAM policy management is significantly easier to lock down than the alternative paths. I can prove to an auditor in 5 minutes that it is mathematically impossible for a member of my team to even see the signing keys we use for certain vendors without triggering alerts to other administrators. I've got KMS signing keys that I cannot delete with my root account because I applied inappropriate policies at creation time. This stuff can be very powerful when used well. Azure has a similar idea that makes accessing things like mssql servers way less messy.

Fantastic piece of lore. Fascinating to read the journey. But also hearing some of the names here (Tavis Ormandy is famous for his role on Project Zero, for instance) and knowing that even top engineers can bomb interviews for making poor choices.

Nothing useful to add except that I Like these blog posts from someone who actually did a bunch of things. Nice round-up of the past.

I had forgotten that you had to individually request AWS services early on. I checked my email history from 2007 and it's true, I was initially only granted access to "Amazon E-Commerce Service"! I got a separate email confirming that I had signed up for S3. Funny that they hadn't yet figured out that the automatic "package deal" is one of their biggest selling points.

The next service I signed up for was "Alexa Web Information Service". Web search as a service, back when "Alexa" was the search company they had acquired, not a voice assistant. By mid-2007 I was (finally) accepted into the EC2 beta. The rest, as they say, is history.

I remember many of these events as I was running FreeBSD a lot and subscribed to the mailing lists.

Why on earth would you give this monstrosity of a company so much free labour?

I get that volunteering is fun, but donating your time and competence to a hyper capitalist company is short sighted. I hope there was appropriate compensation, and I'm not including "early access".

He gave them so much free labor

A 20 year retrospective with no Hetzner or OVH numbers in sight is a bit of a tell. I run workloads across AWS, Hetzner, and a couple of smaller providers, and the gap is not subtle. For a small to medium web stack you are looking at roughly $350 a month on AWS versus 20 to 25 euros on Hetzner for similar specs, plus 20 TB of bandwidth included instead of being billed at 9 cents a gig after the first 100. What AWS actually sells at this point is not compute, it is the IAM model, the global footprint, the deep integrations, and the org chart consensus that nobody gets fired for picking it. That is a real product and worth a lot in some shops, but it is a very different product from what cloud meant in 2006. For the people who have actually moved a real workload off AWS recently, what was the part that turned out to be more painful than you expected?

Colin, if I remember correctly, you first ran Tarsnap servers on Ubuntu before you made FreeBSD work on EC2. At what point were you confident enough to switch to FreeBSD?

I was an early adopter and huge fanboy for AWS.

At some stage I realised AWS is extremely expensive, extremely slow, extremely ridiculously complex and also a parasitic attitude to open source.

I realised I should instead go all in on Linux on virtual machines on other platforms.

AWS I’m done.

Netflix is a big FreeBSD user and a big AWS user, do they run FreeBSD on AWS? Would be the obvious sponsor to me as they rely heavily on the infrastructure built by volunteers like Colin

Interesting how this history is about the edge cases and the unlikely risks that turn into real incidents. the systems scale faster than what we think about their safety.

I dug up my original AWS account confirmation email from 2006 a while (years) back. Now I need to go find it again to see if I was earlier.

Good domain name.

That attested EC2 instance rollout after ~2 decades was a nice joke LOL

AWS was the clear undisputed leader for years, but feels like it’s lost its way now.

It knew how to be the market leader and first to market with big launches. It’s now struggling to navigate a world where in more and more areas it’s falling behind. The big early misses on GenAI seem to have accelerated that.

A ton of momentum from earlier years keeps it moving, but that playbook only lasts so long.

I just want to contrast this article on AWS to its Azure counterpart- https://news.ycombinator.com/item?id=47616242.

2 companies have functionally similar products, but behaves completely different. One company makes technical decisions with security as the fundamental principal, while for the other company, security is not a consideration.

20 years of giving love to a soulless corporation