Hacker News Reader

Comment by hayst4ck | Hacker News Reader

This is jaded and angry, but I think there is another archetype called "the politician." They play the game extremely well, providing poorly thought out solutions quickly to problems they haven't dug into the complexity of. They lay operational traps everywhere in their quest to get things done fast (like directly embedding config data in code to avoid fixing the config format). Once they've picked the low hanging fruit and left too much complexity to maintain, they hop to another company where they sell themselves on all they accomplished leaving out the absolutely unmaintainable mess they left behind.

View on HN

I usually think of an at-work "politician" as somehow taking advantage of relationships or social forces.

What you describe sounds more to me like the ""10X engineer"" (with extra scare quotes for good measure).

Many people have been skeptical of the idea of the 10X engineer. One take is that if your normal engineers are less than 1/10th as productive as another engineer and all of them are merely human beings, there might be some common organizational problems that your normal engineers are all being stymied by. Another take is that your 10X engineers can only achieve their high level of productivity through taking on more technical debt than others understand or would tolerate, leaving messes behind for the eventual maintainers to fix.

Or maybe they just type faster.

In the workforce - the real answer is that the 1x engineers really really aren't operating at full capacity. You assume all parties give it their all. They rarely do.

Get through college

Do what you have to in other to get the job.

Do what you have to in order to keep the job.

10x people do some very basic things. They care. They read about code. They have side projects. They improve their skills because doing so is fun.

In a company that hires "the best" and pays very well you shouldn't see a 1x 10x divide, assuming they hire well.

In those that don't, those that only want work done and don't particularly care? That's where the dichotomy presents itself.

There is nothing inherently wrong with either group. Work gets paid, you don't have to be a magical 10x engineer to be worth the salary. And there's no shame in coding as a job instead of as a hobby.

"Hiring well" or hiring "the best" isn't a panacea. People change. Before I had kids, I was working hard, working late nights, and some of my work was all-consuming. After, I've definitely had periods where my main focus was on showing up to meetings and making sure I got at least pretty okay on my performance reviews, while actually focusing on, y'know, not work. If the company doesn't punish people for becoming half as productive, a lot of the folks are going to become half as productive, even if they were "the best" at some point. Especially true when morale at a company drops. Canceling projects, being assigned to boring projects (the next year will focus on Sarbanes Oxley compliance, guys!), having your company or work dragged through the news, it adds up, and people get detached.

I don't think there's anything wrong being this 1x engineer. In fact, the 10x engineer is probably only being paid a touch more for doing a lot more work and having a lot worse work-life balance.

> In fact, the 10x engineer is probably only being paid a touch more for doing a lot more work and having a lot worse work-life balance.

Sometimes, sure.

But often (especially later in a career) overperforming people operate a bit more like senior partners. They have jr staff / understudies to handle crank turning. They can give direction and set things on a solid path without necessarily putting in a ton of hours.

In a healthy org, this is the "architect" role. Or sometimes the "solver," or some combination thereof. I'm sure you're familiar with the adage about the guy who charges $10k to mark the problem, $1 for the time and $9,999 for knowing where to make the mark.

If these 10x engineers really exist, they're rare. Building an engineering department with this type of employee isn't sustainable.

In 20+ years I've only worked with maybe 2 genuine examples of the good 10x engineer, and far more examples of the bad "10x engineer" who flies around quickly reimplementing patterns that they implemented at other companies, whether they're suitable or not. Once the low-hanging fruit is all gone and only the hard stuff (and hubris) remains, they leave. Usually, after getting pissy because other engineers have inherited their crap have started pushing back and questioning their design choices. I've inherited systems built by people like this and it's not pleasant.

You also better be willing to hire constantly if you want a few more 10xers in your back pocket. If hiring a regular "good engineer" is hard, then hiring one of these is harder. You need to find them, convince them to interview, present them with an interesting challenge, have a great interview process that separates wheat from the chaff. And finally, pay them a ton and don't let them burn themselves out

Oh, and admit that replacing a 10x engineer is going to be basically impossible, not just because it's hard to hire them, but because each one that leaves your takes far more experience and knowledge with them when they go (than the 1x engineer). If the 10x moniker was really accurate, it would be like losing a team of 10.

Sometimes it's better to have a reliable team of 6 engineers in at least 2 timezones who get along well, sync up once a week and have a really good lead/manager. This is sustainable, easier to hire, onboard, retain. And less detrimental if you lose one.

> the next year will focus on Sarbanes Oxley compliance, guys!

How is it more boring than any other kind of project you'd do in a large finance institution?

More often than not, it's people that care about how to create abstractions that amplify their productivity.

That is only possible if they can spend some time building abstract stuff that cannot be explained to non-technical stakeholders, which is not possible with sweatshop engineering methodologies like Scrum, or when you promote the smiling bozos that completed their 5 person-minutes Agile crash course to management.

Abstraction is not a panacea. It's often the root cause of significant technical debt. Abstracting as late as possible is often the best course of action, as it give you more information about how to abstract from the initial implementation.

Improper abstraction does usually lead to significant technical debt, but if you are lucky you can bank your short-term successes and leave the tech debt to someone else.

Yes, that's why it takes effort to learn how to properly create abstractions that solve more problems than they cause.

I would add to this; that a certain amount of bad abstractions are preferable to underdoing it. The learning from failure (when there's sufficient investment in the outcome) is far more valuable than avoiding inefficiency.

Learning better ways is a process - so long as you remain robust in the face of failure and grow from it.

Most people think it's the people that are special or underperforming, but my view is probably a bit more contrarian. I think that some environments work in such a way that only a few can succeed while others make success possible in a network effect or on more manageable scale. I don't think that environments that produce the former really mean to; rather I think it's the leaders that are unable to craft incentive systems, pool resources, or effectively communicate that end up creating these kinds of ecosystems. It's really a fundamental misunderstanding of the people part of the job, which turns out to usually be the hardest.

I disagree. You should work on bottom up abstractions, base yourself on first principles. Amending abstractions built like this are cheap and straight forward. This is opposed to top-down, gluing already made monstrosities.

I've seen 10x engineers. They do exist. However I've never seen them in isolation. It's more like teams of 10x engineers. I've seen teams of people who just get 10x more work done than equivalent teams elsewhere, usually because they consist of a bunch of amazing engineers and are enabled by management to get things done.

I'd generally agree to this. The most recent job I've worked at have amazing management buy-in and empowerment by management to just "get things done fast" that ends up working amazingly well because the freedom we've been given translates to great results.

This is a little catch-22 in getting started though. A great team held back by management hesitation can demotivate and cause attrition. Management buy-in for ineffective teams will run faster by bypassing many org hurdles fall over by their own inability to execute solutions that pass the test of time. It seems like these rare "10x" teams are hard to form in companies that can't consistently attract and retain very talented ICs

Motivated + enabled. 110%.

I will say. In terms of man hours, often those teams take a multiple of effort from people around the company to support.

They just, consume a lot of time and attention. From various business analysts, vice presidents reviewing their output, etc etc.

I think that's because a good amount of 1x engineers could be 10x engineers if given the right kinds of support. When leadership makes sure they have that support, they become 10x engineers. If you only ever see clusters of 10x engineers, it's more likely to be about the factors around them than the engineers themselves.

I think it's both. You get a 10x engineer and a good environment and they bring in other 10x engineers who want to work with them.

If you have never seen a 10x engineer you either have never worked with truly talented people or you haven’t worked on actual hard problems.

There aren’t just 10 or 100x engineers , there are even infinityX engineers when you work on actual hard problems because some of those can only be solved by handful of people.

> some of those can only be solved by handful of people

Ultimately it can probably be solved by other people, but it’d take multiple years.

I think the 10x is not so much because those engineers are particularly fast, it’s just that anyone else working in the domain is 10x slower.

Like if I tried to build a compiler, I’d quickly become known as a 0.1x engineer.

This reminds me of an old essay: https://www.joelonsoftware.com/2005/07/25/hitting-the-high-n...

The thesis is that the "10x engineers" don't output a linear multiple of the same kind of work their peers do. Instead, they implement solutions that their peers wouldn't or couldn't over any length of time.

This matches my experience.

The 10x engineer identifies common subproblems across different problems, and then creates a layer of reusable code which becomes a library. Then uses this library to write code 10x faster.

The 10x engineer tries to standardize problems and solve them en-masse.

The 1x engineer sees every problem as a different problem that requires a different solution.

The 10x engineer talks about abstractions, the 1x engineer talks about concrete implementations.

The 10x engineer believes in investing in productivity. The 1x engineer wants a visible solution right now.

The 1x engineer never has time because they're too busy making themselves less productive by owning more and more code to maintain. The 10x engineer is always looking for code to remove and refactor, and making the solution as small as possible.

The 10x engineer will talk about programming language features and libraries that promote reusability, the 1x engineer only talks about tangible stuff.

The 1x engineer talks about their car and weekend, the 10x engineer talks about technology.

> The 1x engineer talks about their car and weekend, the 10x engineer talks about technology.

Call me a proud member of the 1x engineer club. When I close my laptop at the end of the day or even when I’m having lunch with my coworkers, few people would know that I’m an “engineer” at all. The list of things that I care about more than technology in my non work life is a mile long. Technology is a means to an end - to support my addiction to food and shelter.

This is fine for a certain class of work, and disastrous when applied to another class of work.

To use an analogy a physician's assistant[1] doing brain surgery could be a big problem. if you need the latter. Also, you're probably overpaying if you hire the latter to administer Tylenol ...

[1]: exclude the special class of surgical PAs xD

Spoiler alert: most of the 2.7 million developers in the US aren’t “solving hard problems”. I interact with the people who are writing the services for the largest cloud provider. When I meet them for lunch, they are seldomly talking about work.

For awhile, I was working with the service teams involved with the various AI services trying to publish an open source demo around their services. When we left for lunch we were talking about travel, cars, our families, places to eat, etc.

Most of the millions of plumbers in the US are not solving plumbing puzzles, they're ensuring that plumbing systems work.

I bet the well adjusted ones don’t talk about plumbing in every sentence.

There is so much more to life than technology - this is coming from someone who did hobby programming in assembly language on four different processors by the time I graduated - in 1996.

I sincerely doubt the 10x brain surgeon talks only about brain surgery in their downtime. I would assert that a true 10x person is smart enough to maintain good work-life balance.

There is an infinite pool work you can do. Smart, real 10x’ers understand what is a priority and are smart enough to pack their shit at the end of the day, go home and do nothing at all with their work.

Maybe 10x are just people whose hobby overlaps with work?

I check all of the 10x engineer boxes from above, or at least I think I do, and that speaks as if I am a 10x engineer but I genuinely don't think I am. I think 10x engineer is something different. 10x engineer usually doesn't care about the things from above.

I think it is more about the ability to grasp vast amounts of complexity in relatively short amount of time and be able to sustain it. Business and market wise this is what is going to make a difference. Some engineers aren't even able to tackle such complexity because of so much layers of it which is why at some moment it just becomes ... too much. Others that can do it, it would take them considerably larger amount of time to do it.

I've been a 10x engineer, I've been a 1x engineer, and I've been a 0.5x engineer. There isn't some consistent set of things, it's a ton of circumstance.

Now, as a leader type, I would prefer to hire 1x engineers to 10x engineers, for a whole litany of reasons. 1x engineers are largely predictable, where as 10x can be 10x in any wild direction, because they're more or less deciding for themselves a bunch of shit they really shouldn't be.

10x engineers like to decide for themselves things like how to go to market or which features to build in what order, without doing any of the requisite research or information gathering necessary. It's infuriating because 10x engineers think passion or some arbitrary definition of "Correctness" gives them power, when in reality they're just operating with a heavily limited set of information (gee, I wonder how they had all that time to build stuff, maybe because they weren't attending meetings???).

Sorry, this turned into a not-very-coherent rant, but the point is 10x engineers aren't worth hiring as anything other than literally employee #1, and even then it's a huge dice roll.

As an 11x engineer that is also a rockstar engineer I approve the message

It’s interesting how obvious this is for other domains. For example, imagine you are hiring for an orchestra, and are told just to increase the violin section by five to make up for the lack of good violinists.

> What you describe sounds more to me like the ""10X engineer""

I think 10X engineer is supposed to imply the average engineer - maybe at the person's level?

We all know there are plenty of engineers who do negative work, and plenty who do almost nothing.

So depending on how you think of it - it might not be that impressive.

The 10x is a transition from knowing how to solve problems quickly and efficiently, to being burned out because too much of the solving problems quickly and efficiently. Like a bright burning filament bulb lol. OK I'm probably exaggerating but I have encountered this.

No it's more like at some organizations the culture is very apathetic and the mean dev there has very low throughput. So you can be "10x" merely by caring to the level a professional should. And ironically in those situations it's the low throughput devs creating all the tech debt, with their copy pasta and infinite one off solutions.

The scare quote 10x engineer I'd consider a faux 10x engineer. I have seen them -- and working through their code is not pleasant. But I've also seen people who just operate as such a high level that their code is top notch from the get go, they think clearly about the problem and then have a clear solution. Those are the real 10x (or whatever multiplier you want to use).

> Many people have been skeptical of the idea of the 10X engineer. O

Nah, of course 10x engineers exist. A 10X engineer saw that hundreds of engineers were writing MPI code to process data and struggled with error handling and therefore came up with a map-reduce framework and its underlying infra. The same infra also helped bootstrap a whole industry. In this case, we are talking about 10^6X engineer. Another 10X engineer got fed up with MapReduce's abstraction and decided to borrow more concepts from functional programming and also invented a little data structure called RDD. This is a 10^5X engineer. A 10X engineer saw that his org build ad-hoc solutions to provision hardware and therefore decided to build an abstraction layer called EC2. Oops! That's another 10^6X engineer. A 10X engineer was not happy that writing GPGPU code was slow and error prone so he decided to worked out a library called CUDA. That's another 10^4X engineer, no? A 10X engineer was fed up with people implementing all kinds of broken consensus protocol and coded out a little practical implementation of Paxos called Chubby. All of a sudden the world of engineers woke up to the idea that consensus can be centrally managed and can be robust. That's a 10^4X engineer, right? A 10X engineer hates that his org write ad-hoc and crappy code on Zookeeper, so he packaged his wisdom into a little library called Curator with two clean concepts: framework and Recipes, and thousands of engineers' life just got immensely better. That's at least a 10^3X engineer, right? A 10X engineer was not sure why it takes an artist years to write quality compilers, be it frontend or backend, so he decided to come up with this little framework called llvm. That's a 10^5X engineer, right? Oh man, I think I can write a book about the examples...

Were each of these really achievements by a single engineer, or a team? One guy wrote CUDA?

Ian Buck wrote the initial version, right? I'm sure multiple teams polished the aforementioned software. It's just that it was this one or two engineers came up with the idea, built the the first working version that the other fellow engineers thought impossible or too expensive or unworthy to build, and the rest was the history.

> that the other fellow engineers thought impossible or too expensive or unworthy to build

This is not an exhaustive list and you have just picked the points that support your weak argument.

I thought $\exists$ does not require an exhaustive list, but $\forall$ does.

The initial version of MapReduce was conceptualized and implemented by two people (Jeff Dean and Sanjay Ghemawat). Those were the only names that appeared on the paper: https://static.googleusercontent.com/media/research.google.c...

Look up Jeff Dean:

https://research.google/people/jeff/

> Many people have been skeptical of the idea of the 10X engineer.

Why is 10x anything such a skeptical concept? We see it everywhere.

We see it in quality, impact, and effort.

Linus wrote git in days, and it was way better SVN.

Messi is probably 100x better in terms of stats and skills and 100000x better in terms of earning when comparing to an average soccer player

Apple's iPod is 10x better than Zune and makes 100000x more money.

A great entrepreneur is probably 10000x better (more impact more money) than a good one.

It is not strange that one engineer would be 10x of the other engineer in certain aspects.

> Linus wrote git in days, and it was way better SVN.

Linus designed and coded some data structures, and some basic utilities to manipulate and store them, in just a few days - based on a fundamentally better model and as a replacement for an existing product they could no longer use. Designing and developing git into a usable product and system took a whole lot longer and involved thousands of people.

The examples you cite are all similarly flawed gross simplifications or mischaracterisations.

I urge you to reconsider your reasoning.

Instead of being pedantic, it would be better to describe how the examples are flawed, so we could have had a discussion. But you think "nah let's be pedantic".

I urge you to reconsider not being pedantic next time

And what Linus initially wrote was definitely a usable product. The product has evolved more since then, but that doesn't mean the initial version doesn't work. Actually most of the original code still exists.

Back to the original point, this is the example of a >10x engineer (by impact, coding speed, innovation) who conceived, wrote a popular software in days, grew the community, and grew it to the insane popularity today. Only a handful of engineers in the world could ever achieve this kind of things.

I'm going to ignore the personal attack and get right to the point. If you redefine '10X' to involve impact, speed, or innovation, then you're going to invite external factors of chance (popularity, adoption, etc) and the work of others in building the tools the individual engineer uses - albeit perhaps more effectively than others. Further, they did not 'write' a 'popular software' in days - it was an early prototype that took years to gain the momentum it has now, as I was saying. Trying to credit Linus for all of git is very much like trying to say that the author of a book on which a critically acclaimed movie is based is an amazing filmmaker. You simply can't credit single people for the work done by more than one person. Nuclear though they may have been to the work being there, their impact is far less than you would suggest. There are >10X teams and times and there are engineers who fit particular >10X teams and who are more likely to fit others, but there are no lone >10X engineers.

A 10x engineer writes code, and a lot of it. We can debate the quality, but the 10x part is about concrete output. Political Staff+ engineers live in meetings, documents, presentations, standards, etc. with little if any of their own output.

I know that the idea of the 10x engineer is somewhat eyebrow raising, yet in most fields I ever have been in there where those people who would get shit done n times as fast/efficient or throughly as others, all while yielding better results — be it Camera or Light people on the film set, programmers, designers and architects, writers.

The differenciator rarely was the pure skill at the profession, but an ability to mentally plan ahead and maybe even see the final "thing" in their head. So instead of going try and error or just using what worked last time those people tended to be able to see the clear outlines of the whole thing in their head before they do it, which has the benefit of not having to do things you know will not satisfy. Couple that with a willingness to not waste your time and then you get someone who looks magically efficient to people who "just do their job".

In my eyes those people are not inhumanly good, it is just that most other people are quite average both in their skill and in their motivations.

And even if you are average and just do your job you can easily compare against those people by investing persistence and time.

It's easy to get 10x engineer - just hire 9x shitty ones and you're done.

There is also the "positive" version of the politician. They have enough technical credibility, social skills, relationships with senior management, to remove organizational log-jams. Other companies might hire a consultant to come and tell them what they already know they need to do (but can't organizationally agree to do) - but with this archetype, you can call them in and they will cut out the bullshit, somehow without upsetting everyone.

I think that description matches the articles "architect" archetype:

> The Architect is responsible for the direction, quality and approach within a critical area, both today and stretching into the multi-year future horizon. They combine a deep knowledge of technical constraints, user needs, and organization level leadership.

The reason I labeled it politician is because they are primarily using their position for self-gain. A general definition for politics is any time the inequality company > team > self is violated (at least in terms of compensated work). I generally agree with that definition. That is why I used the word politician.

I have found myself in this archetype before. Management wanted a feature, went to the architects and got some INSANE estimate that couldn’t be justified by the revenue that would be coming in. Management felt like they were being told to fuck off with crazy estimates and they weren’t completely wrong.

I was asked to review the proposal by a VP, came up with some simplifications, presented it back to the same architects and we delivered it for 10% of the anticipated cost and the feature was profitable in year 1. It’s had no defects because it was a simple design.

Sometimes you need people who can speak to managers and engineers with credibility to get things done.

Most people have a very cynical view of politics and I can emphasize. However, I try to stay more positive about it. Politics, for me, is about making decisions in groups too large to just talk with everyone. In such circumstances “bridge people”, like davewritescode, who can communicate with multiple factions are very valuable for progress.

Sadly, especially in large “safe” enterprises people don’t want to learn this. Developers blame managers for “not understanding software” and managers blame developers for “lack of business understanding”.

Since we're being anecdotal: I think that claim is jaded and angry, and makes me wonder what kinds of companies people have worked at. Honestly, I've only encountered this once or twice, and I've been a career engineer since 1989 at places like IBM, Intel, and Microsoft. Dozens of positions and groups, and I've never had to play politics. Maybe I've just worked at "good" companies (irony noted), but in my experience it was less about politics and more about getting shit done, and if you failed to get shit done, you were alerted to that very quickly. If you did get shit done, and got it done well, you got promoted, and were given even more shit to get done. Upper management at those big three had their game tight.

Just like there are loads of average developers, there are loads of average companies.

You can get lucky and work at all great companies in your career. Most of us will not have enough jobs in our lives to get a representative sample.

There is also a question of having different standards. Bad companies might just be bad for someone who has access to the top companies, but otherwise great for average workers.

Point taken. My comment is a good example of "tech privilege".

im actually amazed you never ran into politics at work. you never needed to navigate around egos to get what you want, you do every thing purely on merit, every decision is only technical?

my whole career as been filled with stuff like people forcing their pet projects on the company, people hiring their friends and dealing with their in crowd

> you do every thing purely on merit, every decision is only technical?

Yeah, honestly. The vast majority of my hard decisions I made were based on schedule and resources: having to kill things or move people around. The only "political" anomaly I witnessed was that it seemed the employees at the Israel Design Center at Intel were getting promoted to principal at a much faster rate, and with much less years and experience, for what didn't seem like actual merit. But that didn't impact me other than some jealousy, because the line for principal promotion in my division(s) were very long with really really smart people. And I've seen many principals get shown the door for poor performance.

One possibility is that I never reached a level where politics mattered, or teams were small enough that it didn't have time to foment. And there certainly people who lived to become managers because they thought it was a status symbol, but I was happy for them to do the job.

Another could be that I'm just... well, daft. Both are possible. :)

Is there somewhere I can send a resume? Hah

Politics I’ve seen almost everywhere but nepotism and cronyism are horrible. I’ve experienced that firsthand at a startup, when a new CIO hired his unqualified buddies into leadership roles. One was a young female and he was a much older divorcee and it made my skin crawl.

Intel was overran by MBA types until they got their most recent CEO, and it'll take years for them to undue that rot.

If you worked in a software group at Intel after 2015 and didn't play politics, I straight up don't believe you.

I never worked for a software group at Intel. And I started there in 1989. I'm not out to convince a non-believer like yourself. Just sharing my experience.

EDIT: Intel certainly had a lot of dumb projects over the years, from POTS videoconferencing boxes in the 90's to its attempt at an iMac-like modular cloud server box. But that was largely paranoia pushing them to explore every possible corner.

But again, your experience is not my experience, and I'm not the one telling you that you are lying.

Politics is in the eye of the beholder I think, and even though toxic politicking at work is definitely real, I suspect a lot of what is accused as "politicking" is just sour grapes for people with poor people skills, and people who refuse to consider soft skills to be a necessary core competency, especially at senior and staff levels.

Take some of my practices for example: I lead and design many projects, and if I know a particular element of my proposal will be controversial for a particular stakeholder, I will schedule a review with them privately before to make sure the solution I'm proposing actually works for them, and also make any needed changes before getting in front of a larger audience. This ultimately just wastes less time for everyone and makes product better.

But for someone cynically inclined I'd be easily accused of some kind of backdoor politicking.

A lot of "politics" is just communication and soft skills.

Most startups today I would imagine fit this bill.

This is almost exclusively what I've encountered in most of the "elite" engineers. It's frustrating because they're the ones saying coding and job hopping is easy. They're the ones in tech interviews who downplay your achievements.

In my experience, rather than leaving for another company though, they're often a C-suite favorite and get to start some new project (often it's whatever they want to do as long as it's remotely justifiable). They get it up to 80% complete so they can hand it off to another team to do the last 20%.

Since the project is whatever they want to do, they use very unusual languages or design choices that the new team has to pick up and fix. A lot of times it's whatever the big new thing is. (For example, expect statements like this: "Everything is obsolete legacy code now that Electron exists!" "No need to keep this standalone offline test database, setup shell scripts, or maintain the environment setup guide because Docker will automatically solve all of these cases!")

They continue getting new projects because these 1 or 2 people "did the project 80% complete in 1 month, and the last 20% is taking the handoff team of five at least 3 months to do."

Bonus points when it is expanded from some sort of hackathon, or some fake story about how some person completed the 80% portion over the weekend. In the former, all the connections and APIs are hard-coded and faked for their demo. In the latter, they had actually been working on this for the past couple months instead of doing their assigned work which made their official team look even less productive.

I was on the teams always picking up the last 20%, but I got moved onto the 80% team once. I was amazed at how much code I could blow through with time to spare when I didn't have to worry about all the edge cases and minute details.

Unfortunately, I got temporarily moved onto a company-wide prod support team which then got cut due to covid. Hooray.

The story reminds me of the so-called 80/20 rule.

> Microsoft noted that by fixing the top 20% of the most-reported bugs, 80% of the related errors and crashes in a given system would be eliminated. Lowell Arthur expressed that "20% of the code has 80% of the errors. Find them, fix them!"

More relevant part:

> It was also discovered that, in general, 80% of a piece of software can be written in 20% of the total allocated time. Conversely, the hardest 20% of the code takes 80% of the time.

https://en.wikipedia.org/wiki/Pareto_principle#In_computing

It sounds like you were often stuck with the toughest 20% which actually took 80% of the effort. I also heard that of that 20%, there's another level of 80/20, where a further 20% takes 80% of the rest of the effort, and so on.

Anyway, I hope you find a better situation!

Next level politicians do a really broken implementation first cycle so they can deliver a solve next quarter for eye popping data driven results. I've personally warned people about terrible designs, saw them get implemented, they cost the company millions, but then they "improved" it next quarter to save the company "millions" ... They got promoted, I got called names like "too negative"...

Great company.

I am guessing most/a lot of us have seen similar situations (unfortunately). (Sometimes, though situations like that are unavoidable -- given that demos can become product overnight or a deadline is unattainable no matter what you shave off. If I had a nickel for each time I had seen any of the above happen...)

yes there are shortcuts that one takes for a demo, but these were multimonth projects extending existing systems, it's not like the optimal solution was going to take much longer (if at all), we just needed an architect that understood computer science

I think my biggest failing in my career is in being able to convince people that just because they don't understand something doesn't mean it's harder to code. Many O(n^2) solutions take just as long to code as O(n), for example...

> They lay operational traps everywhere in their quest to get things done fast (like directly embedding config data in code to avoid fixing the config format).

Specific to your example, rather than the sentiment, I think embedding config in code is highly valuable when you don't have a lengthy deployment cycle and have direct access to the source. It gives your compiler more information which can help prevent bad configurations (which are the cause of failures more often than anything else from what I've seen). Developers also have a better shot at feeling how badly configurable a particular component is when the configuration is code. It's much easier to hide the overly-configurable systems under a rug when the configuration is far away from implementation, in a DSL, in a different repository, or only visible at deployment.

I agree, configs in code are not a bad thing necessarily if they don't change often or at all. Sprawling config files are a significantly higher challenge to get your bearing with.

I’ve been trying unsuccessfully to get buy in on config as code.

Instead we have property / yaml files, read into anaemic “classes” which do not perform validation themselves. Validation duties are also foisted onto other parts of the system.

Stacked onto that we sometimes have file name based convention for defining a hierarchy of configs, to save on repetition between environments.

Config as code is fine as long as it results in a fully realized config that is then consumed by other code. All the big companies have almost certainly done it.

I would make sure when implementing it you have someone who is very opinionated review all configs to ensure that the same behavior is always implemented in the same way until a linter can enforce idioms. I would avoid explicit conditionals and explicit loops, instead relying on implied conditionals and implied loops. I would absolutely not use any "live" data, meaning the code should be relatively pure without any dependencies.

Pystachio is not a bad choice in python land:

https://github.com/wickman/pystachio

That was a bit ambiguous and there is a greater "should config be code" debate, which the "yes" people will always win eventually because it's so pragmatic.

Config will always move towards code as systems become more complex and scale up. At some point code-config will have an output intermediary data structure that is consumed by what consumes configs. I don't really have a problem with that.

I am talking about things like querying a database and inserting what would otherwise be a row in the database in the code that does the query itself. So if you need to do debugging, you go do a query against your database and the row isn't there. It sounds absurd, but I have seen it, and caused an outage because of it.

There are people who are bad at every job.

That doesn’t mean that the people who are hiring people into that role are looking to have someone do it badly. Or that anyone who wants such a role is automatically suspected of having nefarious motives.

These role archetypes are descriptions of realistic, reasonable expectations for high level IC roles. It’s a useful, positive contribution that will help engineering directors and senior coders better negotiate clearly defined roles and responsibilities.

It’s just not constructive to say ‘but yeah, sometimes when you employ someone at this level and you don’t clarify the role or hold them accountable, they do terrible things and it enables charlatans to prosper!’

Sure, of course it does. But this article is a useful and constructive model for helping prevent that from happening.

What you frame as an addition to the article actually represents a cynical undermining of its entire value. I think that’s a sad way to engage.

Their crappy code might be generating millions of dollars in revenue while you fix it and curse their name

True! But it might also grind the company's ability to release features to a halt creating a stagnant stock price and an underwhelming IPO because they chose to avoid the hard work of migrating, refactoring, and simplifying, or setting a culture of maintainable code preferring more easily marketable (and probably more fun and less failure prone) feature work that also looks much better on a resume.

Counterexample: Twitter. Every indication is that the code base was horrible. They found product market fit, got funding and went public.

Amazon codebase was also famously barely held together by duct tape for the first few years

I'm not trying to say that you can't incur technical debt if you pay it off. What I am trying to say is that a culture of technical debt stagnates growth.

I would point to twitter as an example of the point I was trying to make, so I don't think it's a very good counter example:

  11/07/2013 $44.9 117,633,000 $45.1 $50.09 $44
  10/06/2022 $49.39 68,511,080 $50.98 $51.55 $49.29

Practically no sustained growth in almost a decade. Poor code and operations hampers the ability to prototype new features and destroys innovation.

Yes, Twitter's stock has basically gone nowhere, just like its product. (I'm a TWTR bag holder. Hoping to get a pay out from Musk soon!)

Do you really think it’s because of the software or because of the product? I bet no one has said that if only Twitters software was better it would gain more customers and advertisers.

All indications are that Twitters architecture is now top notch. There are entire sections of “Designing Data Intensive Applications” (the Bible for system design interviews) about Twitters current architecture.

I think as systems get more complex and bogged down, it's harder to innovate. As innovation is stifled by build problems, dependency problems, political problems, operational problems, the amount of code you have to read to do anything problems, etc, I think engineers start to say "could I be more productive, and therefore theoretically get more reward, elsewhere?"

I think as soon as the innovators start to jump ship, growth momentum is lost and it's nearly impossible to regain that momentum. Who does interviews? Who sells candidates? What is the marketing? What kind of sense of morale will potential hires perceive?

If a large refactoring project did take place, refactoring projects are generally very high risk and low reward. That is the type of project that most senior devs will avoid like the plague. It is not fun engineering. It is drudgery and toil.

So a history of poor software results in required cleanup/developer pain, which results in high value devs leaving for greener pastures, which results in a perception of high inertia non-growth, which then makes it harder to sell people on the prospect of "high growth" stocks which results in the tanking of average dev quality (and morale for that matter), which then makes it even harder to hire innovative devs or leadership, which ultimately results in stagnation.

Senior leadership, both managerial and technical, will play hot potato until the costs of not addressing the real, highly unpleasant, problems is too high and by then its too late. Things continue to get worse between the start of the effort to fix problems and the actual fix of the problems and a lot of people will choose not to suffer that.

I didn't work at twitter, but I watched this process happen. To me, there is a clear relationship between systemically unaddressed and festering technical debt and ability to innovate with direct personnel consequences.

> I bet no one has said that if only Twitters software was better it would gain more customers and advertisers.

But whats the cost of setting up a new service? Of creating a new interface? Of testing a new feature? Of creating a new payment system, etc. All of these are software problems. How efficient would an organization be if every team had their own repo? If every team had their own monitoring systems? If every team implemented their own release process? If every team re-implemented the same behavior over and over again because higher up leadership couldn't allocate resources to problems that every team has, what effect does that have? That inefficiency is innovation opportunity cost.

Imagine you acquire a company you think can provide an innovation to your own. What is the time/resource cost of integration and what effect do you think dev environment would have on success of the acquisition?

I think technical debt clearly has compounding interest. I think companies get into a position where the debt becomes debilitating particularly through pathological short term thinking. At some point the compounding technical debt exceeds growth and it's over.

If Twitter has a sane architecture and all indications is that it does now, creating a new feature shouldn’t require back end refactoring. You have core shared services and you build another service on top of it. I know how at least one of the newer services work at AWS and how many existing back end existing services it depends on and that if anyone outside of AWS had the resources, they could build it themselves by calling publicly available APIs.

I seriously doubt Twitter is one large monolithic app

And let’s not pretend that Google for instance is any better at product development. It’s failed at everything it’s done not related to advertising - ie Stadia. It’s one success hides a lot of failures.

https://thenewstack.io/how-airbnb-and-twitter-cut-back-on-mi...

> Originally, Twitter ran their public APIs through a single Ruby-on-Rails application (“Monorail”), which had grown into one of the largest Rails codebases in the world. And so it was increasingly difficult to update. By 2014, Twitter went the route of microservices, migrating the API service to a set of 14 microservices, running on an internal Java Virtual Machine (JVM)-based framework (“Maccaws”).

> “While the microservices approach enabled increased development speeds at first it also resulted in a scattered and disjointed Twitter API,” Cosenza said. Independent teams designed and built endpoints for their specific use cases with little coordination. This led to fragmentation and, inevitably, a slowing of developer productivity.

Microservices don't make a lot of sense to me unless you can print money or have a public infra offering. It points to an engineering org that doesn't know what they are doing. Turning O(N) code complexity (virtual complexity) into O(N) operational complexity ("physical" complexity) points to an org that doesn't understand where their complexity is coming from or how to minimize it.

> creating a new feature shouldn’t require back end refactoring.

The argument I was making is that it doesn't matter if it gets better. Once momentum is stopped by poor architecture, inertia is too high to get moving again.

> I know how at least one of the newer services work at AWS and how many existing back end existing services it depends on and that if anyone outside of AWS had the resources, they could build it themselves by calling publicly available APIs.

Yes they product-ized their services which is absolutely brilliant. I think the fact that amazon can print money plays a very key roll in why they won while other companies lost. They scaled early in the game rather than scaling too late and reaped incredible rewards for doing so.

> And let’s not pretend that Google for instance is any better at product development.

Google is famous for developers "just shuffling protobufs between services." While it's generally said derogatorily, that means that developers are generally not in the weeds and minutiae of what they are trying to accomplish, but are actually working on the higher level concept they are trying to achieve.

> It’s failed at everything it’s done not related to advertising

gsuite, maps, search, adsense, analystics, gcp, youtube, play, android, pixel, chromebook, news, docs, project zero, etc.

I don't agree that failure is the rule. Cutting lackluster offerings rather than pouring investment in them shows at the very least a sane short term strategy with the weakness of potential harm for long term consumer trust. That's a reasonable trade off to make with good arguments for both sides.

> Microservices don't make a lot of sense to me unless you can print money or have a public infra offering. It points to an engineering org that doesn't know what they are doing

A little company I’m sure you have heard of may disagree. This was way before AWS was thing and was an edict for Amazon Retail

https://nordicapis.com/the-bezos-api-mandate-amazons-manifes...

Your information about Twitters current architecture is way out of date.

https://thenewstack.io/how-airbnb-and-twitter-cut-back-on-mi...

https://www.quora.com/What-were-the-technical-limits-that-Tw...

> Yes they product-ized their services which is absolutely brilliant. I think the fact that amazon can print money plays a very key roll in why they won while other companies lost. They scaled early in the game rather than scaling too late and reaped incredible rewards for doing so.

That’s not the point. The point is that new internal initiative don’t start from a blank slate at any large tech company. There is already infrastructure in place to build on top of even when that underlying infrastructure isn’t publicly available.

> gsuite, maps, search, adsense, analystics, gcp, youtube, play, android, pixel, chromebook, news, docs, project zero, etc.

- gcp - losing money

- addsense - more ads

- GSuite - a distant second to MS

https://www.saasgenius.com/blog/why-office-365-is-overtaking...

- Android: it came out in the Oracle trial that Google only made $27 Billion in total during the first six years. Google gives Apple 18 billion a year to be the default search engine on iOS. Apple makes more money from Google in mobile than Google makes from Android

- Pixel: depending on who you believe, Google sells 1.2 million phones a quarter. If it were a separate company, it would be an abysmal failure

https://www.engadget.com/google-pixel-q4-2021-best-sales-qua...

That’s about the same number Apple sells in 2 days in a down quarter.

YouTube: while YouTube contributes 10% to Google’s revenue, most think it’s still not profitable once you take into account bandwidth, storage, and revenue share with creators.

Project Zero is not a profitable product - nor was it meant to be.

> This was way before AWS was thing and was an edict for Amazon Retail

Yes, you are telling me the point I was making. Product-izing your microservices is a justification for microservices.

> 5. All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.

That is the property that makes microservices make sense. Without that property, technical leadership is setting themselves up for failure.

> Your information about Twitters current architecture is way out of date.

The only point I was making about twitter architecture, of which I know far too little about, was that as you stated previously "Every indication is that the code base was horrible" leads to loss of momentum and inability to innovate, so it's not surprising their growth stagnated for, what I believe, are primarily technical reasons. It doesn't matter if they got better if they went through a period which stopped momentum.

> That’s not the point. The point is that new internal initiative starts from a blank slate at any large tech company. There is already infrastructure in place to build on top of even when that underlying infrastructure isn’t publicly available.

The point I'm attempting to make is that if underlying infrastructure is poor, that's not a fertile ground to grow new product so innovation becomes stifled because resistance to new functionality is too high. If maintenance costs are too high resources must be spent maintaining rather than innovating. I've seen developers stumble around for a week because they put `30` instead of `10` in a config file. I've watched an entire team waste months of effort because production was not advanced enough for what they wanted to do. I've watched an entire engineering organization lose hours every day to a poor dev environment. Time was spent fighting infrastructure, not creating features, so it's not surprising when new products and features weren't created and growth didn't meet expectations.

> google stuff...

We have different definitions for failure. You said google wasn't able to develop products. Google is clearly able to develop new products/services, maybe not monetarily successful ones, but google as as a company has a disproportionate effect on my life and how my life works past showing me some ads.

> Yes, you are telling me the point I was making. Product-izing your microservices is a justification for microservices.

This was not about Amazon selling its micro services. This was only about AWS Retail developers internally.

> Every indication is that the code base was horrible" leads to loss of momentum and inability to innovate, so it's not surprising their growth stagnated for, what I believe, are primarily technical reasons

This was in the early days. They moved away from the Ruby monolith years ago.

> We have different definitions for failure. You said google wasn't able to develop products. Google is clearly able to develop new products/services, maybe not monetarily successful ones

For a for profit company, the only definition of success is does it make a profit.

Edit: I change my mind slightly. We are talking across each other. From a technical standpoint and being able to get products to market, Google is great. From a business development standpoint/program management standpoint, Google sucks. Google’s saving grace is that it has one great revenue stream.

Maybe if Twitter had better product development capabilities - not computer development, business development, they may have found something that sticks.

I also just don’t think Twitter is as compelling of a product for most people as a Facebook and an Instagram

That’s a great technical example. How many times have you encountered an engineer who wraps up their tool’s interface and exposes its functionality through a YAML config file when really it should have just been left as a programmatic interface?

The hubris of the engineer is in believing they’ve thought of everything anyone could ever want to do with their tool. Imagine if a shell only let you set your PATH via a static config file filled with strings. It’s probably fine 99% of the time but at some point someone somewhere is going to start programmatically building paths.yaml. Yuck.

This describes the “enterprise architects” at my company. They read a few white papers, make nice PowerPoints, send the work to some offshoring company. The systems produced either don’t work or have serious problems but in the meantime they have already moved into another project and leave cleaning up the mess to other people.

I'm not sure if I'd name them "the politician", because I think they are not doing the harm consciously, but I've met several of them. Pair it with the "CV builder" ,that just throws new technologies at problems to add them to their CVs, and now your codebase is a minefield.

I'd wager the "product manager" equivalent of this is even more widespread. They constantly disallow tech debt sprints or work to happen, in order to get that nifty new feature out the door quickly in and will tackle the cut corners and tech debt "later, after MVP" or somesuch nonsense.

Then by the time that debt must be paid, they're off to another team|org|project.

I've seen it way too many times.

How does that work exactly? I can absolutely commit an unmaintainable mess directly to master, but I’d be very quickly called out by my team.

Very large portions of some very major companies have <4 years work experience. Couple that with incredibly large engineer to management ratios and that inexperienced devs are often placed under more experienced ones, and the ability to call out unmaintainable code starts to diminish.

A lot of these prolific devs will not see the code base complexity for the "tragedy of the commons" it is, and bemoan the people and teams who start to try adding "regulations" (meetings/code review) to protect the common because it slows down their ability to meet their management given goals.

There is also a magic phrase that frequently disarms attempts to prevent bad code from entering a code base "this is just temporary." It's easy to put off fixing the "temporary" code forever as other things become higher priority.

Often times someone will inherit a staff engineer's code and therefore won't have been able to stop it. Sometimes staff engineers are able to create an entire service with minimal review that is then "gifted" to a team to refine and maintain.

In a lot of these situations, the feedback mechanisms (yearly peer review) will limit who can give feedback about who, not to mention they are not anonymous and staff engineers are usually highly connected.

Easy - you rope other engineers in to work on the implementation. Then if it works out you claim the credit, if not either scrap the project due to “change of direction/reprioritization” or throw them under the bus.

It works because management is bought into the politician's bullshit. So if another team member tries to call it out, then the manager labels the other team member as "hard to work with" (https://lethain.com/hard-to-work-with/) instead of holding the politician accountable.

The politician is totally a thing at large FAANG companies!

It took a while for people to figure out their BS and when it's finally unearthed, two years has passed and they are ready to move on to another team or company and repeat the cycle.

Sometimes they even get a promo before their rube goldberg machinery is finished and someone else had to take the fall.

You have to be a politician at a tech companies. Your promo docs depend on it. How can you show “scope” or “cross team impact” without some amount of politicking?

Example: Dennis Nedry

Prior experience: Solution architect at Jurassic Park. Present: Dinosaur food

This is 100% accurate. In fact, I think I've met a couple during my career.

What is the difference between code and config, other than one is well-typed?

Config could be put into a database and queried while code doesn't have a sensical database representation.

Even when config is code (which is almost guaranteed to happen as a config requiring system is developed), at some point in the process there is a representation that could be meaningfully stored in a columnar database before it's consumption.

Code is generally considered stateless, asking for an input and providing an output that is determined only by the code and the input. Embedding state inside of code is generally a pretty grievous sin, although some places are vastly more sinful than others.

Reply on news.ycombinator.com