Laying myself off from Amazon
daniel.do
daniel.do
Poorly designed, failure-prone, brittle internal tooling was a time and energy sink to the point where even the most trivial deployment change was a nail-biter. Automated tooling and tests that were ostensibly created to make life easier were the number one pain point and constantly failed in obscure ways that required cutting tickets to teams responsible for the automation.
More often than not these tickets would be ignored and issues would persist for weeks beyond what's reasonable.
I'd actually force myself to write even trivial code on weekends to ensure my ability to develop didn't atrophy as a result of lack of use.
That, paired with the awful corporate culture and constant fear of PIP/job loss forced me to finally leave.
Generally a very unpleasant experience all around, sans the compensation and name on the resume.
I did not mention this in my blog post, but I have actually done similar. That's incredible. Thanks for sharing this comment. Our tenures were almost the exact same length too.
a) How to make it stick
b) How to maximize its efficacy/efficiency
Would be awesome.
I initially thought it was a misspelling of exorcism, and OP was making a joke.
The entire CodeBuild suite kind of sucks when you stack it up to options available on the market including GitHub Actions, GitLab Runners, Azure DevOps, literally almost anything.
If the internal analogue is worse than the CodeStar suite, then yikes.
FWIW, I don't miss Quilt, or VersionSets, or Apollo, or Hydra, or TOD, or NAWS Pipelines bridge. But Pipelines itself, brining all of those tools together is amazing!
Considering rather good quality APIs they expose to the external world, this is shocking to read.
Public facing products have competition. Internal tools have a captive audience that must use them, no matter what.
Most of the time its somewhere in between. You'll speak with the manager, speak with the team, you each review each other's artifacts, and if everyone's happy, you swap over. The balance of power is pretty nice. I've dropped out of the process many, many times after starting talking with management. Or even just learning more about what the team itself does (I couldn't live with myself if I worked on pre-roll ads, for example. So I noped out of that one).
Not familiar with Amazon culture so I don't know if that'd be considered acceptable behavior. I hope not!
The manager should put those bad things _in writing_ during performance reviews instead of trying to prevent their employees from leaving by dunking on them after they are informed about the fact.
And from what I understand at Amazon, it's kind of the same thing, but PIPs have been aggressively weaponized.
Whiskey team? Ok, probably somewhat stressful but doable. Wine team? Plush and cushy, with a line of people who want to be on that team out the door. Vodka team? Oh hell no. Etc.
... I'm kidding. Sort of. But not entirely.
There are so many data sources that are all queryable via API. There are a couple basic greasemonkey scripts already floating around but as an SDE it really isn't that hard to write something yourself... It does take a bit of experience to get a good heuristic on weighting the different data points but you can do way better than blindly guessing.
I will say as someone who HAS intelligently looked through this data for most teams at Amazon, there is an absolutely MASSIVE difference between the top 10% of teams and the median, and an even larger difference between than and the bottom 10%, which by all metrics appear to truly be hell on earth.
Tech survey results??
OP was making it sound like you could get the results for a specific team you'd want to transfer to.
Btw this happens in most companies, tons of tech debt. And those who created that st are off onto new projects recreating the exact same mess all over again.
It is a cycle that never ends.
Only chance is to join early and be in for the long run.
It's s lot of fun with the right attitude.
And hopefully no one else regrets it...
Thus is the circle of (engineering) life.
Just kidding. Please write maintainable code.
Not in the requirements, sorry.
Most people don't know how to do this, because it's actually hard, and I don't think it's being taught much in programmer education.
I certainly had to learn "on the job", and probably spent 10 years before I got good at it.
I learned the most by fixing bugs. After a while, I started seeing the patterns of why this bug had occurred, and started writing code do avoid the traps.
I prefer the term troubleshooter specialising in legacy code, but "code janitor" definitely has felt more appropriate at times.
I was an L5 with about 5-6 years of experience, 1.5 at that point at Amazon.
IME, the best managers are the ones who trust and enable their teams to make the right decisions. It would be a red flag if the manager was just giving out offers without allowing the team to provide their input.
Sometimes I get frustrated at my own job for some reason or another, but I appreciate learning that the grass isn't really greener anywhere else & even places where tech is the core business are having a lot of the same problems.
I figured this was pretty common. I've only had one job where I didn't feel the need to do this.
Well ... maybe?
N1 doesn’t really help, but it’s a start.
I'm battling something for weeks now which is a completely broken ass piece of shit inside AWS. I spent the first 3 weeks trying to get attention for this via support tickets and someone to take it seriously, resulting in escalating to account management. I've been on calls with the programme managers, been apologised to constantly and absolutely nothing has been done that is productive. Not only that they sheepishly suggested other customers were in the same shit.
My conclusion at the end is that critical bits of AWS are held together with only a couple of people who actually know how it works and they don't even have the ability to do fix anything...
Not that the consuming company I work for isn't a complete mess either but I expected better from AWS.
At least you have that.
My skills have atrophied and my pay is meh.
If you do the math on their TC to see how it actually works out from month to month, it makes sense. Unless you get more RSU's, you take a huge hit in pay, unless you weren't cashing in on them to make ends meet.
With the flat share price, there's no 4 year cliff anymore...
Another guy on the team intimated once in a while that he didn't like the team dynamics, but he played his cards pretty close to the chest. After he gave notice, on what we all thought was supposed to be his last day, we showed up in the morning just to find his computer and all of his equipment in a pile on a desk. We never did figure out if we or he were off by one or it was his middle finger to the team.
All the more mysterious because later that day we did discover his middle finger: The lead dev, who seemed to be one of the more reasonable people on the team, and the resident know-it-all both discovered that someone had fiddled with their configurations in ways that broke the system but produced obscure errors and hard to notice differences (like punctuation).
I took a couple lessons away from that project, and reinforced some that I'd already had. Perhaps the most important of which is: Bragging rights for being first mover are very, very short-lived. Once everyone else has copied it, the most accurate adjective for your version is 'old', or 'primitive', not 'first', and absolutely under no circumstances 'best'. Developers don't like bullshit. Don't pretend like your liabilities are assets.
When I quit Amazon, they ran to my desk and unplugged my dev machine, since they thought I was going to leave a software "time bomb". I thought that was the stupidest concern in the world, but maybe it does actually happen.
I never understood that “shut them out” reaction for people giving notice. Getting laid off, maybe. But I pick the time and place where I hand over my 2 weeks. I have all day or even a few days to do it, but I have known for a month or three that this was coming, I’ve known for days when it was likely to come, and I’ve known for hours or even days that it was happening today.
You don’t think if I was going to do something shitty that I would have done it first? You guys are really underestimating my intelligence, or at least my ability to plan. But then that’s probably part of why I’m leaving.
Also they say that liars think everyone else is lying, thieves think everyone else is stealing. So I wonder what this says about the paranoid person? Are they a danger to the company?
I am glad I am doing a master's in parallel because outside that I haven't written code in a while ...
I’ve seen work slow down across a few teams and it just leaves me in a place of boredom and complacency with a major technical itch. Despite this itch looking to be scratched, I cannot seem to find the time/energy to pursue coding regularly outside of work in the limited free time I have. Maybe if I were single, no kids, dogs…
I can relate. I worked for many years for one "those" companies that is a household name. The pay wasn't outstanding, and the job was not one that gave me much fulfillment, but it was at a marquee brand. My business card opened a lot of doors.
When that job finally ended (after almost 27 years), I started looking at working for other companies. I would have been a particularly good fit for startups, as I was financially secure (thus, didn't need to be paid too much), and have a huge arsenal of skills and experience, that can be quite useful to any small outfit that has "many hats" employees.
What I found, is that us older folks are quite unpopular, and got tired of slamming doors and passive-aggressive insults.
So I just decided that I was in early retirement.
Best decision I ever made.
I would have liked having the extra money, but the soul-sucking aspect would have probably killed me, by now.
I am in a good position. I have enough set aside to be OK, but not at an exorbitant level, and I get to write awesome code every single day. It's basically, my "dream job."
I feel terrible for over-leveraged young folks, having to deal with this kind of thing, early on. I feel even worse, for the ones that have "crossed over" into "old," and are still over-leveraged. They won't feel old, but they will be treated as such.
It's really humbling and infuriating.
May or may not be able to work with me, but always happy to expand my network.
Some of the older engineers I've worked with are among the most brilliant and productive people in the groups. In those cases I have seen bar none, they have been revered and respected among the other members.
Age discrimination is definitely a thing, but not everywhere. Maybe that rotted mindset of prejudice has taken hold in the "I'm saving the world broken by boomers by working at a corporation that sells advertisements on the internet" crowd.
Can't be helped. We will always look like that, no matter how hard we try to "fit in."[0]
Tension between younger and older folks is as old as humankind.
I think that a team is best served, with a combination of youthful enthusiasm and creativity, and experienced caution and completionism. I think that the whole from these teams, is greater than the sum of the parts.
If we have just older folks, stuff gets done, but it may not be that interesting.
If we have nothing but younger folks, we have ... FTX.
The difference, this time around, seems to be extremely young C-suite execs.
In "the old days," when we watched Archie Bunker rag The Meathead, companies were generally run by folks in their fifties (there were plenty of problems -not exactly the Halcyon days). These folks didn't have anything against older folks, except, maybe, that we were relatively expensive (but not, compared to these days). If they discriminated against older folks, it wasn't personal.
These days, we have people in their twenties and thirties, at the top, and their discrimination is personal. They also make it OK for their employees to have the same attitude.
What's that saying that the "olds" have? "The fish rots from the head down."?
[0] https://i.pinimg.com/originals/48/e5/30/48e53007e190cab89608...
I would expect Google to have better engineering discipline than Amazon or Meta.
So you either hope that others will be inspired to do better by your example, or you become a gatekeeper who loses respect because nobody likes to be lectured by someone who isn’t walking the walk.
olladecarne's story lines up very well with my own experience working on a post-startup-acquisition org at Google.
>adding a single line anywhere feels like playing Russian roulette
I worked at a company where all 4 founders were still there (15 years in) and that was very much our reality. I think the founders still cared, but that didn't matter - beyond a certain size (>100 employees), the majority of the people in the org didn't care the same way.
To an extent then, if the original authors are still around, they were either appreciated or feared, but you probably won’t know which just from the interview. If they split it may be good riddance.
I do have a couple of mystery cases where I believe that someone who previously thought very highly of their own work had a change of heart, and realizing what they’ve done and how hard it would be to fix it, have decided it would be easier to start over someplace else. That may be true but I would recommend that it’s a character building exercise if you at least try to clean up your mess before leaving. There will be other messes, from other blind spots.
It's not art. Art requires no function, it exists as a representation. (Professional) code, and software engineering generally, must produce valuable things, and sometimes the process of making those valuable things isn't "fun", but that's never been part of the equation. Aligning "fun" with "valuable" is nearly impossible, and when you try you're dooming yourself to almost certain failure.
This is really burnout, and the mental health effects that cause burnout. I'm sorry the author has gotten to this place, but wrapping it up in "passion" is a disguise.
Ugly code that works pays bills. Beautiful code that doesn't work is, in a very literal sense, worthless.
I'm not against writing clean code (there's utility in that), but I feel like some folks lose the plot and quit their jobs when they can't make "beautiful" things anymore, and I think that's more often a sign of burnout than it is a desire to "return" to creating art.
Surely there's also "ugly code that doesn't work" and "beautiful code that does work." And I'd argue there's a lot of in between as well. Unless the software literally does exactly what it needs to do, no more, no less then there's also the variants of "works well, somewhat well, etc."
what about good code that work? All places where I see your argument prevail, ended up to be not just 'ugly code', but unmaintainable code. And systems that cannot evolve at all.
Personally I don't call coding an art, but I've seen countless times how people people choose bad solution even though better one costs exactly the same, and in a long run - actually cheaper. And they _always_ use this argument - 'code is just a tool, if it solves the problem, it is good'. And then they either leave or have to spend weekends to write even more dirty code just to solve problem they wouldn't have in the first place if they spent a little bit more time thinking about the code.
Wars have been fought over defining beauty, but the important thing to note is that beauty is orthogonal to function. Clean code has clear criteria and value, beautiful code does not.
There's a limited number of solutions to a programming task, it's about finding the one that meets most constraints, it's about cost, effort, trade offs analysis and finally implementation.
But this is literally and technically not true.
Coding is mostly optimizing and compromising between all those factors, rather than staring at the wall waiting for inspiration expecting the solution is an act of poetry.
This is not to say that engineering is void of creativity, some sort of beauty, cleverness, etc, it obviously is, but it's not the same kind of creativity involved when painting or writing music.
Art has the goal of touching, moving and expressing the author's self.
That's just not the goal of writing a form, a serializer, a validator, a logger, a queue, a shader or a git hook.
In either case you didn't specify constraints and even with those constraints it's still technically false unless you're limiting yourself to a non-turing form of computation. I'll take all your tasks with their constraints and add no-op operations onto them to make them ugly but still functional. I'll add in all kinds of ill-thought out logic that will never be reached or is "functionally" useless.
And even an "artist" has these "real world" constraints which are "must sell before starving to death" or the work won't be "completed"
I read it more simply like, "Code is a craft that deserves careful attention."
We can find ways to do our writing jobs creatively, but fundamentally we type to get things done. We write to influence people. It’s not art, not even a little bit.
It’s obvious who gets it and who doesn’t.
However, I disagree that aiming to get enjoyment out of a coding job is necessarily a mistake. But it does often have to be weighed against competing factors like ease of getting a job and pay.
A business analyst once told me - "you programmers have it so easy. It either works or it doesn't!" as if the vague 2 sentence requirement from their spreadsheet that I have to turn in reality doesn't require any creativity to implement.
But what about the edge case they missed? How should I design the control flow of this? How should it handle errors? In a lot of cases, how should it look, because I only received a wireframe? Should I write this as a class or a function? Can I write this code in a way this is easy to read and performant? What if it's a really specific piece of business logic, that doesn't make sense unless you know the reasoning for it?
Good software is art the way a well designed car or watch is art. They exist to accomplish a function, but the path to get there often requires quite a bit of creativity and expression. Your assertion that art requires no function doesn't make sense.
Is a dish or menu designed by a chef not art? Is a house designed by an architect not art? I feel sorry for people who can't see the art in those things. And especially sorry for people who can't see the art in writing code.
But the difference is, all those things have sale values that DIFFER based on that creativity and beauty. Code does not. My customer does not care if my backend code is beautiful, and neither does my CEO because it won't make more money.
> Is a dish or menu designed by a chef not art? Is a house designed by an architect not art?
Of course I see the art in those things, and in response i'll pay a lot more for a Frank Lloyd Wright than a mcmansion. But the customer of my B2B CRUD app does not care if my code looks like a FLW or a mcmansion. They care if it works.
I wouldn't be so sure about that. I certainly prefer to use software that has a nice interface, is fast and responsive and has thoughtful features. I remember the first time my iPhone opened a pop up at just the right time asking if I wanted to share a wifi password with my Mac. Wow! Delightful. And I am willing to pay more money for such things.
>But the customer of my B2B CRUD app does not care if my code looks like a FLW or a mcmansion. They care if it works.
What is "works"? I'm not being disingenuous here. The software development process is often plagued by things like scope creep, unreasonable asks from stakeholders, cutting things for budget or time etc...
"works" is a subjective concept in your hypothetical customer's mind. Likely molded by you or your project manager setting expectations, pushing back on feature requests, etc...
My real point is, it's not just the value of the consumer or "customer". But the artist as well. It's the satisfaction you can get from designing a performant service that handles requirements and has the capacity for future expansion, etc...
Just because some people are philistines doesn't make the creation any less valuable as a piece of art.
'The real cycle you're working on is a cycle called yourself. The machine that appears to be "out there" and the person that appears to be "in here" are not two separate things. They grow toward Quality or fall away from Quality together.'
A program is not like a car, a chair, or a house, and if any of this were brought up in a design meeting for code, you’d rightfully be laughed out of the meeting.
But it’s fine to disagree about these things up to the moment where you attempt to slow progress towards delivery for these values. At that point, the point at which functionality is hindered in any way by your artistry, are you now a problem on a development team.
There are plenty of productive ways to deal with problems, but make no mistake, on any competent software team you will be disabused of this “art” notion, not the other way around.
You're taking it too literally. "viewing" in the case of software is "using". When I'm using a well designed, performant piece of software I can feel the art in it. When I'm using something thrown together I can feel that too.
However what we're discussing here is how the code and otherwise hidden implementation is designed and built. OP is not a UX designer, OP is a software engineer, and should not waste time building things that are "beautiful" in that capacity.
UX and systems design are almost entirely unrelated, and should not spend time in one another's domain.
Apple can only sustain the high margins they do because you’re wrong. Some people care a great deal. Some of those people have deep pockets. If none of your customers seem to feel that way it’s probably because they are Apple customers.
But you also have companies like Anker and at certain points in their history Sony, Samsung and perhaps LG. They got it. At least for a while.
People here don’t complain about Apple for silly reasons as much as they used to. But I’ve been laughing at people jealously predicting their imminent demise while cashing dividend checks ever since their shares were $40 pre-split (7x and counting IIRC). I’m going to get to retire at least a couple years early just on AAPL, even having done dollar cost averaging.
> B2B crud app
Ah. Businesses have a weird split brain problem and it’s difficult or impossible there. They want custom software for less than or around the price of off the shelf solutions. The OTS ones can be beautiful and occasionally get away with it. If you care about art of even ergonomics you’re gonna have a bad time. I encourage you to seek out a new vertical.
Are you suggesting that some people care a great deal about how pretty Apple's code appears, despite having zero way of knowing anything at all about that as a consumer?
You should think about writing a book :)
Do you really need creativity, though? Unless you’re working in uncharted territory and you’re just building some CRUD app like 90%+ of software in the world, you don’t. Software engineering patterns exist, and even so much of the innovation that supposedly came out in the last 5 years are mere iterations of the same patterns.
And the purpose of being creative with a solution is not creativity for creativity’s sake. Clarity, maintainability, scalability, should strictly be the primary goal of every line of code that you are writing. I wouldn’t debate that a code that is so clear and intuitive and easy to scale for so many people has achieved some degree of art, but that should be a side-effect, not the goal in itself.
I use a dozen different-looking web interfaces per day, often internally inconsistent in the same application. Here, it's OK Cancel, there it's Cancel OK, another one is Yes No.
There are two buttons on the "modal" dialog in a web page. One is a different color. Does pressing Return mean anything? Escape? Can I tab through them? Does the modal with one required text field (e.g. 2FA entry) focus that field? Does Ctrl-V work? Is it some bizarre custom design that can't keep up with a normal rate of typing?
Will it allow my password manager to autofill fields? I saw an interface recently using Angular that didn't even provide name or id on the text fields. Thanks.
The desktop app side of things is equally annoying. Either it's electron/nwjs, or it's attempting to look cool instead of just following the guidelines.
This is a bad time for users.
There’s nothing wrong with wanting to take pride in your work and to have working conditions that facilitate that. Get enough people together who feel that way and can find something useful to build and you get a huge competitive advantage. Software that is useful and performant is like gold but takes a lot of extra effort and passion to keep it that way.
You’re right about the “fun” part though. Making great software (and art) is hard work! “Satisfying” is probably a better metric to shoot for.
Software development isn't "engineering" by a long stretch. There is still another 40-50 years of development and consolidation before it could be considered the equivalent of other engineering practice.
I've lived through numerous lifecycles of "this time for sure" regarding software as engineering, from "modular programming" to "CASE" to "iterative development" to "UML" to (god help us all) "Agile".
What a lot of practitioners think of as "engineering" is actually "project management". Engineering is about defined practise, measurable inputs and outcomes, responsibility for signoffs and approvals that ensure that the defined practise has been followed.
I think good art (to me) requires some function. And code is art. Like architecture , or furniture design, or gardening, or dance, or many art where the design and precision and purpose is all tied into what the artist is doing.
I don’t think coding is pointless and think that coding for the purpose of being beautiful can be dangerous and painful to support or refactor. But I’ve read a lot of code and appreciated it on some aesthetic level and remember thinking, “that’s neat.”
art HAS a function. It sometimes appears invisible because it is we who are affected by it.
Art is anything that functions to manipulate a {conscious being}'s internal state (traditionally emotional state).
That said - I agree mostly with your point besides that one pedantic complaint :)
I want to say sorry at the beginning if I misread what the author (you?) intended when you wrote this. And I'm coming at this from someone who is not a developer and who, compared to the lofty Software Engineer salaries in this industry, feels underpaid and underappreciated for doing the operations/administration work.
With that said: I truly wish those who are leaving their roles where this kind of requirement exists--feels "forced"--would speak more loudly about the need to have actual people doing system administration work. This is especially true, I feel, in large companies who operate "the Cloud" and who ought to know better; someone needs to be doing that work and it can't always, or usually, be the people who are writing the features and implementing the updates to the core product you are selling.
But what seems like has happened is everyone in the tech industry has forgotten it, or named it "Site Reliability Engineer" with a job of 55% failure analysis, 35% coding, and 10% "are the infrastructure and products actually online and functional." And then this role gets looked down on and paid less because the people in the role--people like me--are not seen as "delivering" "value".
Which culminates in the proper software developers seeing it as a thing they are forced to do, resulting in disdain, and furthering the cycle.
(Worse, the incentives for improvements to production quality, and thus on-call quality, are utterly mis-aligned.)
Ironically, I say this as someone presently on-call for a bunch of stuff for which I am utterly unfamiliar with the code base of, and have no time to become familiar with. It's going predictably badly.
(And because I feel like this is bound to draw a strawman, steelman this: while there are definitely pages that might not get routed to BE eng in particular, assume we're routing pages to the person responsible for that system. I.e., in the face of infrastructural problems, those pages get routed to something like "infra eng", although IME there's very little that can be done with those in the middle of the night…)
If your system/product/service is down because you have a dependency on something that broke -- well it's up to that team to fix their code.
>If your code breaks something, you should fix that code. Who else should?
What if it wasn't my code, but code written by someone 3 years ago who quit because most people only work at the company for 2 years? And it's in a part of the codebase I've never touched. That's a much more likely scenario.
The problem behind that is that Amazon is a completely dysfunctional corporate hellscape. Like TFA said, you just don't have time or resources to actually fix things
Totally normal. Not crazy at all. Take ownership.
This however:
> And the worst part is, I feel like the product we were shipping was really bad. It was slow and kludgy. It was the opposite of the qualities I value in software products. I didn’t feel like I was adding any value, and I was miserable.
Yeah, there is absolutely no excuse for that. And if that is the case about the product you're working on, what I said just before about "enjoyable engineering" will never apply. Good for getting out of there.
In my experience there is almost no correlation between complexity of the project and time needed to do 'non-technical' work. Quite contrary, the most complex things I did as a dev required me to do a lot of coding - to prototype, to experiment, to load testing and so on.
And at the same time the easiest project in the larger corporations required me to write down design docs for trivial stuff and spent months in meetings to 'align' with people, who don't have time to even read your design doc, but want you to have bi-weekly meeting about it.
So no, I don't buy this excuse anymore. Unless you are building space ship or do very unique work, there shouldn't be any need to spend 80% of your time in non-strictly-technical work.
They don't have to have the highest commit counts, but if they're not the ones the rest of the development team look to for help solving critical bugs and advice about new features and designs etc., then they have no business evaluating technical concerns.
Just this week I witnessed some team having built feature without consulting another critical team, and now it has to be reworked and shipped at a much later date.
Even then, writing a prototype is often a quicker and more effective way to flush out disagreement than endless discussion.
To stay in the embedded world for an example, sometimes the actual code is just a few dozen lines with a handful of assembly instructions, or even less. But the danger and potential pain those lines of codes might incur on countless involved subsystems, if not properly thought out, even if seemingly "working" at first, can be immense.
[1] To give a sense of scale of complexity in the embedded world for example, the ARM Architecture Reference Manual alone is well over 10000 pages now. And that isn't any actual implementation, and only the "main CPU" itself!
Finding the right spot to make a 2 line change instead of the wrong spot to make 8 five line changes is seriously technical work
Once you scale up to large systems (ignore complexity) with hundreds of developers then you do and can not know it all. At that moment the churn starts.
And unfortunately, once you get big, regulators, lawyers, audits and other things become a drama as well.
[0] which I've found to be pretty-to-extremely-good compared with what I've discovered since leaving, and would actively welcome new perspectives on what flaws I'm missing.
> I know Amazon compensates very well
That's not been my understanding - I thought it was noticeably below other FAANG companies (at least until the pay hike earlier this year)? I have only just started reaching out to possible employers so I don't really have any points of comparison.
Point being, if you're not making more in year 3 than year 2 -- something is broke and/or your manager wasn't supporting you appropriately, or not hitting high enough on the performance ladder.
No, no, this sounds pretty much like every team I was on for the nearly-decade I was there.
This gives the lie to Jeff Bezos' whole "Day 1" philosophy. If you're at a Day 1 company you don't spend 40% of your time fighting with the crappy internal tools.
I just worked at a fang company, and here's an example - to search CI logs to see if a certain log happened frequently you would : WRITE A SCRIPT to download 100MB CI logs and search them for you.
Any startup can get this by piping their jenkins (etc) logs into ELK/Splunk/Sumo. And indeed this is exactly what I built at the last startup I worked for.
But again I was talking about a FAANG company not having a way to search logs (again, each individual log file is 100mb and there are 100k of them generated a day).
If it's not clear why that's entirely unacceptable, imagine your team is running the CI for over 10,000 engineers and you're landing changes to this CI system daily. Engineers are seeing all kinds of logs and bugs daily and you need to ascertain if these errors are new, unique to some subset of jobs, lead to failed jobs, etc.
You can also interpret it like - “always start from scratch. You don’t have anything”
I left this feedback a bunch of times with different people in the org and I really hope they scrap their janky "frameworks" and dev tooling. They should just move to more standard open source tools options that evolve and get better quicker than barely maintained internal tooling.
Open Source tools also have bigger communities and resources online to debug and solve issues, compared to mediocre documentation from internal wiki. If absolutely needed, make thin wrappers above open source tools. Some other teams within Amazon had the luxury of using better tools, but I bet a lot of people are in the shoes I was in.
Minimum 20 minutes to build my team's packages. Integration tests didn't run locally and you'd have to push. Same thing for LPT/CFN changes.
Tooling at Amazon is _incredibly advanced_, but no teams know how to use it properly.
I was at AWS 2008-2014, two years each in EMEA, APAC, Americas. I didn't contribute to code directly, but I built my own software demos (I was the tech evangelist for these regions).
My experience has been vastly different between these 3 "regions", and also when interacting with HQ in Seattle (which I visited very frequently).
There are certain things that unfortunately heavily affect the company - the internal tools were already considered quite bad back then, and I'm not surprised it hasn't changed much.
There are others, though, that in my view are even more important. I had a total of 4 managers, and three of them were amazing, average, and quite bad.
But in all honesty, I was probably not perfect all the time (real modesty, uh? :D), and it might be that the "bad" manager and me were just a poor match, or that I failed to understand things that made me perceive him as bad.
I will never work for someone else ever again in my life (I can afford to, both economically and career-wise; in fact, I just launched a VC fund in Europe), and I love that. But if I had to go back to a corporate job, my single best piece of advice would be: pick your manager wisely, especially if you're early in your career. Everything else is important, yes, but your manager is going to have a huge influence on how you develop, how happy you are, etc.
If you're curious, this is how I got hired by Amazon back in 2008 [0]. I find that post a bit naive in retrospect, but I also wrote it ~15 years ago.
[0]: https://simon.medium.com/2008-how-i-got-hired-by-amazon-com-...
I suspect the people who end up leaving those environments for a more normal F500 company are going to be in for a shock in terms of work ergonomics and pay. This guy was lucky that he was able to squirrel away so much tech-worker money to be able to quit without a backup plan.
My experience was similar. On the first day of orientation we were trying to build an environment to run some tests in. But some aspect of it was borked. I got to file a Sev 2 ticket my first day of work. The component that completed the dependency closure was borked (what was that, brazil? can't remember the name.) But the whole company spent two hours not building software.
I love that all teams advertise their capabilities through online services, but man.. it would have been cool if maybe they tested that update to the build system since we were all forced to use it.
The other big hilarious problem was when we ran out of IP addresses for new sites. Long story short someone made a mistake three or four teams up the chain and it only became obvious when we tried to allocate a new routable (public) IP address. The problem was in the DNS team, but we didn't use their service interface directly, so we had to wait while the team upstream of us responded. And then they had to file a ticket with the team upstream of them. And so on.
It always seemed to me that while they were trying to prevent their interfaces from being brittle, they missed out on some opportunities to understand what was going on by looking at data flowing from one team to another. I hope there's SOMEONE in the company doing that, but we never met them.
Not all my work relationships were toxic, but I saw quite a bit of toxicity while there.
I quit the day after my then new boss wore a MAGA hat to work and made comments about killing trans-folk (I'm f2m.) Our HR rep was out on vacation and the main office indicated they weren't interested in doing anything about it. I probably should have sued, but it wasn't clear I had a strong case (he said/xe said). At the very least I could have forced everyone to take another sensitivity training class.
Which is to say... There are some amazing people there. Also some complete shits. If you have a high tolerance for death threats, you'll probably do fine there. Some of the technology is amazing. Some of it is completely bonkers. When things are good, they're amazing and very good. When they're bad, they're hellish.
> 40% of my time trying to tame the bad internal tooling I was forced to use to submit my code, get it merged, deploy it, check logs, etc…
The tools for code submission, pull requests, pipelines, metrics, and logging are fantastic. Google is better. Most companies aren't.
I have never spent 40% of my time battling internal tools....
> 20% of my time in meetings
Developers complain when they're not invited to meetings, and they complain when they're invited. On my team we brutally introspect the value of every meeting, and if it looks like it's not delivering value, we find a new process.
> 20% of my time writing unit tests to hit the 100% coverage requirement of the codebase I worked on.
This makes no sense. This isn't a company mandate, every team is free to determine what code coverage percentage makes sense for them. Give this feedback to your tech lead, nearest Sr. SDE or PE -> 100% test coverage should never be "required"
> 10% of my time tracking down bugs in other team’s codebases for either internal tools or frameworks and trying to get them to acknowledge the problem by filing tickets.
So, software engineering?
In my experience, not at Amazon, long tenure employees get used to the quirky tools but the impact on new employees can be massive. Same with bad code bases, bad documentation and so on.
(I wouldn't say I love the CR process - making multi-package CRs is definitely flawed, and I had a Sage question open with the Builder Team for nearly a year where they admitted as much - but I certainly prefer it to Github's)
But I'm pretty new to the outside world and really keen to understand alternative perspectives. What's good about Github's process to you?
The reason why it's called a "pull request" is because you're requesting the owner to pull your changes in; I don't see what this has to do with the number of repos in the picture.
Right, that's my point, though - why is there no distinction between "ability to create a real, actual, complete branch on a repo" and "ability to create a 'fake' branch that only exists for the purposes of diffs for a change request"?
For contrast, the flow I was used to inside Amazon was: 1. Clone the repo locally 2. Make my changes locally 3. Run a command that creates a 'fake branch' on the main upstream repo, which is used as the reference for the change. This works even if I don't have push-permissions on the repo (in which case, I can still push my change once it gets approval by clicking a button in the UI, whereupon a service account will push "on my behalf")
Whereas for Github, the process (if you don't have push-permissions) appears to be: 1. Fork the repo to my own Github account 2. Clone that locally (equivalent of 1. above) 3. Make my changes locally (equivalent of 2. above) 4. Push my changes to my own forked repo 5. Run a command (either CLI or UI) that creates a Pull Request from my repo to the target repo
Sure, it's only two extra steps - but I don't see why that friction has to exist in the first place. It gets _much_ worse if your change is open for a while (which will happen to coincide with the case when you don't have push permissions - that is, when you're contributing to code that you don't own), because then you need to resolve rebase locally and push to your fork rather than just being able to update in the UI (Github UI doesn't support rebase-pulls, only merges).
But, yes, this does mean pushing things routinely a lot, not just when it's time to make a PR.
You could theoretically get the benefits of both approaches, though: have your own personal repo to which you push "in-progress" commits for durability and portability, but maintain Amazon's tooling which generates PRs with a diff between a _local_ commit and the target (by, behind the scenes, generating the ephemeral fork from which to Request a Pull), and permitting updates to that PR from local (not necessarily "pushed to an online repo") commits. That's _still_ advantageous over GitHub's model, because:
* If you don't want to have a personal repo, you don't have to
* Even if you do, the process of updating a PR is simpler and more flexible when executed purely with local Git commands rather than by manipulating a remote repo
I appreciate the perspective, thank you!
I battled those tools when I started. I watched them get better.
I've seen what new hires struggled with 5 years ago and what they struggle with 1 year ago.
Night and day.
The tools have gotten a lot better.
Here's the other ugly truth: That "40% of struggling with internal tools" may be saving the engineer 300% of time of having to implement the same from scratch themselves. Software engineering isn't all algorithms and data structures. A lot of it is just boilerplate code hooking up A to B. And better leave that boilerplate code to the internal tool that you have to figure out how to configure than implement it yourself.
Everyone below a certain level talks to new employees. Do I believe you take their concerns seriously and actively try to help? Based on my own experience with PEs as well as your comment history I think you absolutely do not.
In general my experience with PEs at Amazon led me to conclude that the vast majority of them are:
- entitled
- lazy
- egotistical
- less technically useful than the average l5 engineer
THANK YOU. I feel like I'm taking crazy pills whenever I see people bash Builder Tools - Pipelines in particular. Compared with what seems to be available in the Open Source world, it's fucking stupendous, and has silky-smooth integration.
Oh yeah I love the multiple hours we have spend every week 'introspecting' processes, just to throw out one of the dozen we'd already defined and add another one. And this 'introspection' typically boils down to the loudest, most ambitious mouthbreathers forcing their BS down everyones throats so that they can jot down their amazing process contributions in their promo doc. Brutal is the right word.
Quit, and go make sourdough bread, shit.
But it's the difference between an individual carpenter making a rocking chair for himself and his family, or maybe making a couple to sell to his friends, and being a structural engineer.
Bikeshedding is not a necessary antipattern to the process, but large software projects absolutely need group collaboration, and a discussion of processes, tools, and best practices.
Based on your comment history (which is consists of about 0% technically interesting topics, and about 100% pro-amazon 'tales from a super-senior principal engineer guyyyss') tells me you don't like engineering, you like politics and beuracracy. Which is okay!
Lots of good internal tooling. When something critical to us breaks, we usually cut high severity ticket to get other team's attention.
We rarely have actual customer affecting issues in production, there's no PIP threat, promotions feel achievable within a couple years if you really want them and the extra responsibility, automation is very helpful and we're constantly looking for ways to reduce our operational load.
Yes, testing could be better. Yes, we could have more engineers. Yes, oncall could be less stressful. Yes, I spend 10+ hours on meetings every week. Yes, I write a lot of documents and do a lot if ops to debug issues. I have probably merged just 10-20 lines if code in the last couple months.
But EC2 dataplane is very complicated. Every small change needs to be given a lot of thought. Success and scale bring responsibility. Plus, I've learned so much here. I'm never bored.
And my work life balance is good overall. Slightly less than 40 serious pomodoro hours a week except when oncall.
Ec2 is a very stable system and we keep making it more stable for our customers. Do use it. It's not brittle at all. Expect occasional instance failures and design around that with cross-AZ redundancy and you should be golden.
Is 100% really practical? Probably in life or finance critical code.
What about those "should never happen" cases? Incredibly hard to trigger because you only know from gut feeling it can happen in some really weird way.
I know it would feel very nice to have 100% coverage, but my gut feeling is that it will also affect the code base in a negative way. Developers might skip catching errors that would be very hard to reach.
* Build times * Testing times * Test Flakiness * High quality fast unit and integration tests * High quality fakes * Fast, high fidelity non-prod environments * Time from commit to production, number of cherry picks/rollbacks * Good Coverage, but focus where it matters.
If any of these go out of whack it needs to be a P0-P1 to fix because it really is debt that grows to become unmanageable (not to mention repelling to the best people).
In most companies, even big tech, this is difficult because non-coders set priorities and allocate the resources. Even once-upon-a-time engineers who make it up the org chart find themselves preoccupied with other things and invariably sacrifice these things for features, compliance, conformity, toolchain complexity/sophistication, security, outage/incident scar tissue, etc.
(And maybe rightly so as there are few successful businesses that manage this consistently)
1) Work is work. The percentage breakdowns here don't seem too far off from any other company. It would be interesting to see what the author feels the percentages should be. My gut feeling for anyone willing to walk away from a job is that they probably do need a break for a few months.
2) If you're looking for reasons to be unhappy, or trying to justify unhappiness... you may be depressed. Switching jobs may help, but if you walk around smelling shit all day remember to check your own shoes. Everything is going to smell like shit until you get the help you need to get back to being healthy. Getting help is hard, but it really makes life (and work) a lot easier.
3) Amazon... I've written about this in the past, but I really have a love / hate around my time there. Looking back, lots of talent, lots of cool projects. But I just felt like Amazon was perhaps the worst place I ever worked for people giving praise or sharing appreciation. Amazon... I can't put my finger on it exactly, but it did often feel like their mission was to suck all the fun out of the room.
Also, I found the PIP process to be overly dramatized relative to my experience. I had a PIP imposed on me a couple years before I willingly left, and I "graduated" out of it with support and cooperation of my manager. Everyone involved seemed to want me to succeed.
Also, when I left, I opted into a second PIP, because HR and my manager wanted me to leave on good terms. Doing so let me get severance while quitting (it was ~3 months worth based on my 5+ year tenure). I'm not sure if this option still works though.
Not to say it was all bad - some of it was certainly necessary. Especially for security - IMO the systems of Amazon are great with security, but not necessarily the spirit of the company. Without all that bureaucratic annoyance, there 100% would be existential leaks or breaches coming out of Amazon with how fast and loose everybody treated security in the name of "Deliver Results" .
Also, I enjoyed my first two managers at Amazon, but my third manager was a tool, which is a systemic problem at Amazon.
I'm glad to have Amazon in the rearview mirror of my career.
Pile that money up high (in something like index funds) while you're earning the big bucks, and there comes a time where you can choose whether you want to keep doing it, or move to the beach and surf and still be young enough to enjoy it.
Engineering at Amazon is not for a fainted heart, and internal tooling is definitely the source of most people's miseries. On multiple occasions, I had to fix external tools at the expense of making progress on my own work, in the name of exhibiting a "deep dive" ethos.
On another hand, however, I think Amazon is a good place to work for people who will eventually get into startups.
i suspect that, if you work on a "slow and kludgy product", the responsible thing to do would have been to write a memo laying out why it sucks and what needs to be done. I've seen a rare few folks actually get heard and get the mandate to do what needed to be done. but I know it can feel overwhelming for most and ultimately its the responsibility of the GM/PM to know whether this is true or just the whining of an underling with no context.
also kudos for having the guts/integrity to do this before thanksgiving. make it count!
This is very stressful, in particular when what there's to do is clear and glaring.
I can only wish you the best. 6 months ago, I was in your shoes. Today, I have my own startup building applications I am passionate about. I have never been happier. I’m grateful for everything that I’ve learned and endured so far.
I just wanted to tell you that I hope you stay positive even during the dark days and that you maintain your integrity. Truly, I wish you the very best.
Regards, Daniel :)
Things definitely move at a slower pace than at new startups, but systems within AWS still change at a pretty fast pace for such a large organization. A fair number of employees don't stay past two years because of how compensation works, so they don't stay long enough to observe tooling changes, and are left with the assumption that tools are sacred. The reality is that it takes about two years between major rewritten versions of most tools. Then there is a leap forward, stasis for a while while feedback is gathered and the limits of the existing tooling are found, then in roughly two years there is another leap forward, etc. This churn requires work to keep up with, so some teams also fall behind if their product team prioritizes features over keeping up, so they may also be stuck on old versions of tooling for even longer.
So in summary I'd say yes in the short term some of the tools appear sacred. In the long term pretty much every internal tool or framework is discarded and replaced by newer versions on a fairly regular cadence.
Can you elaborate? You mean a new service gets someone promoted?
You have as much impact and are exposed to as much scale (maybe not in terms of TPS but in terms of data, etc.), but you usually have much lighter development processes and a much more frequent release cycle. Your ops will also be a lot lighter.
I'm an early career software engineer and am curious - why do people say working "at scale" like it's a good thing? Why is that desirable? What are you doing differently from someone who doesn't work "at scale"?
Someone who's dealt with websites at scale before is going to be able to skip the first 20 of that process, so if you're a startup dreaming to get big, you find someone that's worked "at scale" eg Amazon or Google to design and build your systems and avoid some pain points like the website going down because the system can't keep up.
The actual functionality is a very small part of software development.
Making it work within its constraints (memory/CPU/network/storage in the small scale, ability to handle peaks while maintaining uptime at a larger scale), as well as dealing with the "non-functionals" like AuthN/AuthZ, obervability, performance, etc is actually the "hard part".
Then add in business "non-functionals" like "minimize CapEx/OpEx", "meet external SLAs with measurable and reportable KPIs", "meet corporate and regulatory standards and requirements" etc.
All of that is much larger than the actual function being developed.
Does such a company exist?
That being said, I agree that the internal tooling at Amazon is not that great. It's actually a place where newer employees tend to clash with older Amazonian because it compares unfavourably with GitHub actions or any modern CI/CD while being significantly better than anything that existed in the early 2010s.
I've heard a lot of testimonies of SDEs maintaining pipelines that are making them miserable.
Literally the only advantage I'm aware of for OSS systems is that they allow you to use whatever language, dependencies, build system, etc. that you want - which, yes, fair, if that's something that you need, then that's a deal-breaker, but for the 99.99...% of internal cases that Pipelines works for, it (seems to me to) work flawlessly and smoothly.
What are some unfavourable comparisons that I'm missing?
As far as the other tooling goes, I often find that they are very Java-oriented (which is fine considering the AWS background) and everything in Python feels janky.
One point that I've found _really_ strange after transitioning from Pipelines to other CI/CD (I'm using Drone and Argo, but I think this applies for Flux as well) is that infrastructure-definition repos define the _image-version_ that they deploy to a particular stage, rather than just defining the source repo (and automatically deploying the latest image-version which has passed all preceding tests). That seems...odd. Unless you hack together your own automation[0], that means you need two actions to get new App Code out to an environment:
* Push the App Code commit (and have a new image-version built)
* Make a change to your Infra Code, updating the desired image version
Am I missing something?
---
I'm surprised to hear that you consider AWS to be Java-oriented. When I think AWS, I think TypeScript - to the extent that the primary reason I started learning TypeScript was to be able to write CDK more fluently (and, then, to write Lambdas without a whole bunch of Java type boilerplate - though Python also works there).
[0] Which I've done, but the fact that I had to hand-build it suggests I'm doing something that the tool author's discourage
On the other hand, I've actually found myself missing features from amazon internal tools at the jobs I've had since. One example: I remember a pipelines feature where you could look at any change and quickly tell exactly how far that change had made it in your pipeline. At my current job, determining exactly when change X deployed to region Y has been a multi-step exercise in manually comparing commit hashes.
In fairness, I guess this is the tradeoff you get for being forced into "one and only one way to do it". The benefit is that all the tools are likely to play nice with one another "all the way through the flow", because they only have one representation format to be compatible with. The downside is, well, there's only one way to do it; and if that way happens to suck for you (as, for instance, for an ex-coworker who needed to build multiple different versions of the same code against different base OS images for...reasons), the tools are going to hinder more than they help.
But then I saw how some teams (predominantly, but not exclusively, based in Seattle) worked, and....yeah. Overall, not great.
There is no central SRE teams, SysAdmins, etc. There is no throwing it over the fence.
There is nobody better to fix a production issue than the person who wrote the code which broke. Plus, you are more motivated to write better code.
And good on you for doing it and great you don't have obligations to provide for others. What a freeing decision you made for your self!
heh. try one of the teams with no coverage reqs…
I have zero tolerance for anything else at this point.
Be the change you want to see.
(i dont work for amazon, but I can imagine it's similar everywhere)
It's not something that is fixable at all, let alone by a lone engineer.
That said, I also want to be more tempered than the author, I don't find the tooling to be that bad. Most teams are on full Native AWS now and the few non-native tools actually do make your life easier because of how well they're integrated in the whole Amazon ecosystem.
Some tooling does suck, but it's more like 2% of my time spent fighting bad tools, not 40%.
Also the guy you’re replying to has no idea what he’s talking about. If I had tried to do that my manager would have said “that’s not the work we have planned for you, that’s not the work our team does at all” and if I had just done it anyways (not that I think I could have) I’d have gotten fired.
You say it would literally be faster to stop feature work and fix the broken thing which is a laughably naive view on how these companies operate. That's just not an option unless you do it all on your own time in which case you will still be incredibly limited on what you can change and even if you make a significant fix there will be no extra reward for doing so. The incentives are all wrong.
He's a digital nomad who took off to Mexico (and Belize) and seems a bit lost in where he is going (he says it himself). I've been there myself (except I moved to Vietnam), so I understand it.
This is a lot deeper than just writing tests or 40% internal tooling issues. I also suspect that being remote, he feels like his hands are tied in a company that isn't used to working remotely. Or at least, if I was struggling with tooling, I'd work to find a way to make it better.
I also never complain about writing tests. I can't tell you how many times tests have saved my bacon or resulted in writing cleaner code.
Are you trolling?
It is absolutely possible to set the coverage threshold in Jest to 100% for all categories and they are set to that in the codebase I worked on. I would spend hours trying to that last 0.02% covered sometimes.
It does make me wonder why it would take you hours to do that though... was the code that complicated to test for some reason?
It's not ridiculous to expect some, *configurable*, amount of test coverage for newly generated code, is it?