How good engineers write bad code at big companies

My issue with the article is maybe more that the perspective and framing seem to want to lock out idealism from the conversation and encourage people to become ruthless, hard-nosed pragmatists in order to survive in this environment. Is that actually good, effective, or desirable? Again, lots of questions.

https://en.wikipedia.org/wiki/Marx%27s_theory_of_alienation

I got into this career because I like mucking about with this kind of work. Programming is sort of like building thought sculptures. Ok, maintaining them too. I spend a lot of my adult life working with these systems, so of course I care about their quality.

It is unpleasant to work on a hack-filled creaking monstrosity of a codebase. My skills, such as they are, don't help as much as it feels they should, because I rely heavily on understanding a system based on how it sort of has to work given what I know about the constraints, and hacky systems constantly confound my expectations. That means I have to spend way too much time and effort learning the idiosyncrasies and unpredictable corner cases, and so have less time to make the changes I want to make. It's like a carpenter working on a house where nothing is straight, nothing is level. Or a cook given flavorless or half-rotten ingredients to use. You spend more time picking out the bad (or picking out the good and discarding the rest) than actually cooking.

Any my scope is limited. Some changes are just not feasible to make, because there are too many hidden and/or unnecessary dependencies buried in the code. If you change one thing, everything falls apart. If you patch it together well enough to more or less work, you're faced with a long tail of subtle bugs that you'll be battling for a long time to come. There's no conceptual integrity.

Furthermore, working with a team that accepts a never-ending accumulation of creeping technical debt is also unpleasant. You can't trust that a positive review means anything. If you try to do something right and take pride in your work, your teammates will be resentful that you're not closing your bugs fast enough. Who cares that it means less maintenance overhead in the future if nobody's planning on sticking around, or people are depending on things being constantly broken so that they can spend their days knocking off relatively easy whack-a-mole style bugs?

Bad code is actually fine. It happens. But I want to know why the code is bad, whether the reasons are defensible or not. If they aren't, I know that the codebase is going to be littered with similar crap, and that my job is going to suck. That's what bothers me so much. This is true even if I'm the author of the bad code. Why didn't the reviewer point it out? Is it like the article said, and there's nobody competent enough to spot glaring issues? That is also a component of a job that sucks.

exactly!

If a senior engineer is responsible for the problem, but he wasn't given room to develop a proper solution in the first place, then he cannot be blamed for the outcome. And yet, this is exactly what happens.

If a system had been made well, and functions well, same senior engineer does not get the credit for not causing problems!

This is what happens when management/stakeholders don't understand what is going on, but have an expectation that is not well communicated to the engineers (and also inversely too, where engineers are not communicating the situation properly to management - or they refuse to hear it/ignore it).

The arguably unproductive part of my brain craves to reach a day where I can say literally everything straight - if this would be a true description of the circumstances, then say exactly that.

Unfortunately, in addition to not only not being a “team player” this would also be really mean to people that had different time and resource pressures when they wrote the original flawed implementation. So both burning bridges and being kind of a dick.

At the same time, there’s sometimes people that develop stuff in such a way that I will never agree with them and a few people like that are enough for me not to want at a particular team/org even if it’s maybe like just 5 to 10% (and even then often it’s about them being inherently mean/nitpicky/whatever, rather than purely technical stuff).

Obviously the irony of the above isn’t lost on me so I just bite my tongue, focus on being professional and fixing the things I can and let the rest be without picking fights that will lead nowhere - otherwise I’d be a hypocrite and the situation couldn’t be described as anything other than “a bad culture fit” from an outside perspective.

But I’ve seen a lot of really shit code and it baffles me that alarm bells don’t go off more often and people just trudge along.

Then again, you do see a lot of drama in plenty of open source projects, so maybe that’s just human nature.

"You’re a senior engineer - why did fixing this take so long?” This is also maddening: it's not a question arising from technical indight or a management strategy ala Drucker, ishikawa, or any other serious approach. It's a guilt trip: it's like saying why did you dress so bad at this party? It's manipulative. And nobody likes these smeer tactics.

The complaints in this thread seem to ignore cases like that and I’m not getting the sense that it’s because everyone is spectacular at avoiding rabbit holes.

Now add "Why can't you use/ask AI agents to fix it?" which is being shoved through your throat by management.

Bad, over-complicated code is code that requires you to form an unnecessarily detailed mental model of it in order to answer a particular question you may have about it.

People rarely read code in its entirety, like a novel. There is almost always a specific question they want to answer. It might be "how will it behave in this use case?", "how will this change affect its behaviour?" or "what change should I make it to achieve this new behaviour?". Alternatively, it might be something more high level, but still specific, like "how does this fit together?" (i.e. there's a desire to understand the overall organisational principles of the code, rather than a specific detail).

Bad code typically:

* Requires you to read and understand large volumes of what should be irrelevant code in order to answer your question, often across multiple codebases.

* Requires you to do difficult detective work in order to identify what code needs to be read and understood to answer the question with confidence.

* Only provides an answer to your question with caveats/assumptions about human behaviour, such as "well unless someone has done X somewhere, but I doubt anyone would do that and would have to read the entire codebase to be sure".

Of course, this doesn't yield some number as to how "complicated the code is" and it depends on the question you're asking. A codebase might be quite good on some questions and bad on others. It can be a very useful exercise to think about what questions people are likely to seek answers for from a given codebase.

When you think about things this way, you come to realise a lot of supposedly good practices actually pessimise code in this sense, often for the sake of "reusability" of things that will never be reused. Dependency injection containers are a bête noire of mine for this reason. There's nothing wrong with dependency injection itself (giving things their dependencies rather than having them create them), but DI containers tend to end up being dependency obfuscators, and the worst ones import a huge amount of quirky, often poorly-documented behaviour into your system. They are probably the single biggest cause of having to spend an entire afternoon trawling through code, often including that of the blasted container itself (and runtime config!), to answer what should be a very simple and quick question about a corporate codebase.

To be fair, though, I don't trust modern labor either if they can't figure out how to NOT vote for a rapist pedophile real-estate billionaire. Twice. Including a pandemic and a coup.

What a mess we find ourselves in.

But management depends on your manager; at scale it becomes likely there are bad apples in every management tree. Incentives may not align with what you want or need, with work From Home policies getting shrunk. Even money sometimes is a point of contention.

I think this is a very telling statement, but perhaps not in the way you intended. I would agree that management only cares about results, but I would posit that maybe that's a good thing. If you don't have ground-truth knowledge of a problem, you must rely on either the word of someone who does, or metrics that can be used as a yardstick.

When all a manager has to go on is someone's word, it can be really hard for them to gauge the depth, severity, and impact of the problem being expressed to them— and without any metrics, they have no way of tracking progress on resolution. In a modern codebase, you could spend YEARS on improving maintainability and still not "finish". The key (that I've found, personally) in this situation is to give the manager some form of metric to describe the problem. If you can establish a number to measure what you're advocating for, and quantify the consequences of not doing it into actual business impact, I've talked managers into taking my suggestion more often than not.

If you set incentives that say that being sloppy and leaving landmines for the next group of people is the way to get promoted, guess what? the management is bad. Often because they are also looking at their own self interest, and expect to leave the consequences to whoever comes after them. This isn't new to big tech: You'll find this all described in books about corporate dysfunction written in the 90s.

It's all traditional principal agent problems, which just get worse and worse as you add layers of management, as the principal has agents upon agents underneath, all using the misaligned incentives. One either wants t avoid getting fired while doing the minimum, or sacrifice the health of what is around them for a good enough promotion packet/review. And since there's no reasonable way for individual objectives to align well with long term objectives, people leave landmines. When there's enough landmines everywhere, you are always better off in greenfield development. And at that point, doing any maintenance, or being stuck in a project that isn't getting fed a bunch of capital to grow it is career suicide. All about bad incentives, set by bad management.

Management doesn't know how good the code is. So they can only look at results.

Like you say, this easily ends up rewarding people writing quick and dirty solutions.

This problem is easy to describe and hard to solve. Yet some teams do it much better than others. One solution is that management also writes code.

The thing is, good code is a form of a good result. You need to solve the underlying problems (which manifest as impact) but if you used code to get there, that code if well designed is extensible, reusable, then you pay low maintenance on it and that same code can be used to solve other problems (ideally).

It's a difficult judgement call to make though. If your org doesn't have the right technical leadership and the performance management structure doesn't reward it then you get what you see.

It's a lesson that's always learned far too late when it becomes slow and costly to deliver something new because you've amassed so much tech debt so it's often cheaper to start from scratch (which is saying something).

I see this all fwiw in big tech myself and with my peers at peer companies.

Management cares about what their management cares about. So this boils down to what the CEO cares about. The CEO cares about what the board cares about. The board cares about share prices going up.

I do believe that e.g. retaining engineers is something that helps the business. It's stupid that someone ramps up for 2 years and then goes to a different job just as they start becoming really effective. It costs companies a ton of money which they could instead just use to get people to stay. But I'm not on Google's board. It's only when Google board decides that fostering the right engineering culture is important enough for the business that something is going to change. And so far- they don't (s/Google/BigTech).

Re: Incentives- Obviously(?) the incentives are not right. So when you say "plenty of incentive" what it really means is incentive to ship sloppy code and get promoted. Or the incentive to go from Google to OpenAI and get a pay increase or whatnot.

So it's the McNamara fallacy?

And management is probably right: code is a mean, not an end

It is us, as software engineers, to make that better code brings better results

Some benefits of better code can be:

  - less infrastructure cost
  - more speed / efficiency / whatever metrics available from the users
  - easier to integrate new features
  - less issues, less time spent in maintenance etc

Very political stuff

I try to do good code. My team is rarely cited in that list of "bad players". Management does notice.

Corollary: our roadmap is not messed up. Product managements notices.

Also, good code must mean robust infrastructure. We have very few incident. Almost all our changes are successful. Incident and change managers notice.

Anyway, my point is : if good code brings no benefits to the outside world .. how is it good, exactly ?

I moved teams a few years ago, and the very first thing I did was push hard to re-structure the schema they (sorry, pals) slapped together without much thought. It took some fair amount of arguing, and maybe a PR that I had reviewed by only a single person and pushed through because it's easier to ask for forgiveness, but we got there in the end.

Luckily, we were able to do this before the code started hitting production traffic; it would have been significantly more difficult to fix once we started getting real data into the system.

I feel that is a very likely scenario.

AI makes these types of developers much more dangerous because they will accept anything the AI generates tha looks like it works, and they’re experienced at pushing nonsense through code reviews.

AI also provides more of a “productivity” boost to these types of developers because unlike everyone else they actually spend the majority of their time typing code.

Ideally by the time your at code review this is not a question. It sucks for everyone for it to come up.

I think people also avoid rejecting for this reason becasue of that

You say that until you are tasked with doing impossible - three lines, all perpendicular, five green, two anti-green, seven in ten or more dimensions, any color; while customer only uses purple lines.

Last guy that worked on it committed seppuku. Rest of team is in mental ward. Your only team member is guy that programmed his entire life in PHP, and doesn't know backend's language. Just teach him.

Documentation, is spread between Jira, wiki, Markdown, ftp server and some napkins.

CI stands for continuous Indians. You send code to India, where a team will assemble it. It may take anywhere between a few minutes or few hours. But it beats GitHub actions. Make sure to inspect artifacts, the Indian team has a habit to add some of their ""bug fixes"" covertly.

But you gotta finish it by Thursday. Good luck.

So maybe not every line of code needs to be even more beautiful than the last, but there's clearly a balance to be had. And yes, sometimes the business people are right. Sometimes they are wrong, however.

That's too generalised. A fast food cook can't spend time to make things perfect. A tiny, fancy Japanese place will spend time to manually craft a perfect dish and you'll wait while watching the whole process.

I suspect that you can find something similar in every category you mentioned.

Not the case in commercial programming. If you manage to pull a heroic and still deliver something that does not fall over in a near-impossible deadline and with a lot of pressure then you are actually doing a huge disservice to yourself because the leadership will think "welp, obviously he can do it in those conditions" and next thing you know, next time around it will be even more difficult.

"Give them an inch, they will take a mile". Sadly this proverb very often applies to business people.

Most commercial programmers are extremely squeezed. I started daydreaming for another profession lately but yeah, ain't happening in my 40s with a very unstable financial situation.

I've read your sibling comments. It seems like you were on the other extreme and it does seem to me that now you are overcompensating by being too sympathetic with business and management. Whiplash effects are very understandable while one is still trying to find their balance. Still, don't give those people too much credit.

Very true. And experiencing this has absolutely been a useful lesson to me. I remember feeling pained over some code I wanted to write in a more robust and principled way in front of a big deadline, and after a bit of friendly push and pull with my manager, I agreed to console myself by throwing in some TODOs and FIXMEs in the comments instead of getting carried away overengineering or burning rubber just beautifying my style. I remember this ritual feeling painful at the time, though it helped me cross the finish line sooner. A few months later, it was clear that ~80% of that code would be good enough for the next year or so— by which time other parts of the application would have evolved as well. I even ended up glad I'd deferred a few of those changes just so the small refactors could be informed by things we'd learned in the meantime.

Sometimes it goes the other way, or course! Many times I've been glad for some extra care I put in early, or regretful about some I didn't. But you're right that that balance always seems to change in retrospect, one way or the other.

Then you know what to work on to become an even better engineer. My recursive function for improving code has clear exit conditions. One of them indeed includes aggressive timeboxing, so on that point we're aligned.

I feel you are trying to make an argument against better code. I get it, there are very fussy programmers out there who can indeed polish stuff until the heat death of the Universe (and still be unhappy) but bringing up extremes is not very interesting in a discussion IMO.

> Is it still bad after 5min? After 30? After an hour? After a day?

Generally, time spent on code has near-zero correlation with the quality you'll get. That correlation only becomes stronger when you have a lot of trust in the person writing the code (including yourself).

As above, don't try to make managers look like the reasonable people. I've met some and I adored them but the 99% are egotistical snowflakes that will fire you the first time you say "no".

---

Nobody is saying "accommodate the insufferable nerds and give them two weeks to do what junior Joe will do in 2 hours and it'll be pretty good" here in this entire thread. People are saying "maybe we the programmers should have SOME power in some decision-making as well". I am OK with my influence being somewhere between 10-15%; that's expected, I am neither a manager nor a financial stakeholder. What I am not OK with is when that number is 0-1% and sadly that has been the norm during my consulting and contracting period. And it's the main reason due to which I am trying hard to end this period right now and settle somewhere).

Especially with enterprise code you're hammering your thoughts into a shape roughly compatible with someone else's so its no wonder overtime with the constant revolving door of people that without careful shaping things can get nuts.

Just as we might agree on universally panned bad movies, but disagree on cult followings or one of us for a particular reason can't stand a popular movie.

That is to say we can all agree on extremes, but just because something isn't extremely bad doesn't make it good in everyone's eyes and that's where the contention is.

Then this leads to people setting tight deadlines. It's a vicious cycle.

Reasoning with unreasonable people is not impossible but drains too much time and energy... and you don't know if they will not come back to you next week and hit you with "You know what, I actually think that your idea is not as good. Let's get back to mine and...".

I have danced this dance. As much as it pains my younger idealistic self to say this, you'll know how well you'll work with somebody after the end of the first week. All the signs are there, we just choose to ignore them because you can't bail during your first month after all.

Maybe higher compensation does work out for FAANG, but it does not at other larger employers, at least not for software developers. I am highly dubious about this at FAANG too though. My perspective is as a former 15 year JavaScript developer and a lot of what I saw coming out of FAANG in JavaScript land just wasn’t impressive.

I do see a lot of impressive things in the world of JavaScript but it’s almost always open source from small teams or single developers.

At the end of the day it comes down to who the decision makers are and how they are incentivized to act. As a simple example - company X has product C, and they set a goal of increasing usage of feature F (of product C). Currently this feature F completely sucks and users don't want to use it - so the idea is to improve it and thus increase usage.

There are 2 ways of increasing usage:

1) Make the feature F more useful/better.

2) Force/push your users to use feature F, by aggressively marketing it, and pushing it within the product surfaces, making it non-optional, etc. and other dark patterns.

Option (1) is hard to do - it requires deep understanding of the product, user needs, the related tech, etc. It requires close tactical collaboration between product and engineering.

Option (2) is easy to do - it requires ~zero innovative thinking, very surface-level understanding of the problem, and relies purely on dark patterns and sketchy marketing tricks. You can almost completely ignore your engineers and any technical debt when following this approach.

If your decision makers are imposter PMs and marketing/sales people - they will almost always choose option 2. They will increase the 'apparent usage' of this feature in the short term, while reducing overall customer satisfaction increasing annoyance, and reducing the company's overall reputation. This is exactly how many 'growth' teams operate. Short term benefit/gaming of metrics for long term loss/reputational damage. Their success metrics are always short-term and linked directly to bonuses - long term effects of these kinds of strategies are ~always completely ignored.

I think that this may eventually become better now that there isn't so much dumb money around (no ZIRP) and with AI assistants taking on some low-effort work (enabling companies to lay off incompetent engineers). But it will take many years for companies to adapt and the transition won't be pretty.

I have communicated the business value in addressing tech debt, as in I have pointed out how many critical customer features got delayed by it and we churn customers we really can't afford to.

Still got overruled. The tech lead stepped in and threw his weight around with zero explanation to the CTO or the CEO. Who needs to hear what the engineers on the ground have to say, right? They cannot communicate!

Man, tearing down strawmen is easy. You should do better and see nuance. A lot of engineers communicate very well and are still ignored.

He deployed our applications using Kubernetes and refused to implement libcluster. There was something else, too, but I can't recall what it was. It was seven years ago.

> Yeah, this is weird...

I kept telling this developer that you're supposed to test your private functions through your public interfaces, not expose your private functions and hope nobody uses them (which they did), but that fell on deaf ears. He was also a fan of defdeligate and used it EVERYWHERE. Working with that codebase was so annoying.

Everyone I hired is still there.

I'm fine with passing the baton, I enjoy mentoring.

Knowing those patterns is very helpful as a way to think about design problems, as long as you have the common sense to realize applying the pattern "by the book" is often overkill and you can just take some ideas out of it.

That article conflates as "Pure engineering" both reducing a software system to a small set of cohesive concepts, and architecture astronauts, when those are polar opposites.

> Companies burned hundreds of thousands of engineer-hours migrating from monoliths to microservices, or from HTTP service calls to event-sourced architecture, or from event-sourced architecture to full CQRS, and so on.

Is that what you're talking about? imo this hardly counts as an attack on CQRS itself. The issue is rather with enterprise companies forcing the migration of large codebases based mostly on hype.