Hacker News Reader

Comment by cek | Hacker News Reader

21 years at Microsoft led me to have great disdain for "Software Architects".

5 years at Amazon led me to fall in love with the concept of a community of Principal Engineers. The Amazon Principle Engineering tenets are amazeballs.

Exemplary Practitioner - Principal Engineers are hands-on and lead by example. We deliver artifacts that set the standard for engineering excellence, from designs to algorithms to implementations. Only by being close to the details can we earn the respect needed to be effective technical leaders.

Technically Fearless - Our company's startup culture does not admit the luxury of conservatism. Principal Engineers tackle intrinsically hard problems, venturing beyond comfortable approaches when necessary. We acquire expertise as needed, pioneer new spaces, and inspire others as to what’s possible.

Balanced and Pragmatic - Principal Engineers are pragmatic problem solvers. We apply judgment and experience to balance trade-offs between competing interests. We simplify processes and technologies while advocating a long-term view.

Illuminate and Clarify - Principal Engineers bring clarity to complexity and demonstrate smart ways to simplify. We frame each problem in its customer and business context and boil it down to its essence. We probe assumptions, illuminate pitfalls, and foster shared understanding. We accelerate progress by driving crisp and timely decisions.

Flexible in Approach - Principal Engineers adapt our approach to meet the needs of the team, project, and product. We solicit differing views and are willing to change our minds as we learn more. We recognize there are often many viable solutions, and that sometimes the best solution is to solve a different problem, or to not solve the problem at all.

Respect What Came Before - Principal Engineers are grateful to our predecessors. We appreciate the value of working systems and the lessons they embody. We understand that many problems are not essentially new.

Learn, Educate, and Advocate - Principal Engineers are constantly learning. We seek technical knowledge and educate the entire organization about trends, technologies, and approaches. We combine vision and discretion to drive fruitful and even game-changing technology choices.

Have Resounding Impact - “Deliver Results” is a low bar for a Principal Engineer. Without seeking the spotlight, Principal Engineers make a lasting impact that echoes through the technology, the product, and the company. We amplify our impact by aligning teams toward coherent architectural strategies.

View on HN

It's typical of engineering organisations to come up with this kind of list, the main reason being that as an industry we're uncomfortable with experience, seniority and progression. We're collectively suspicious of it. We want to codify and clarify it, endlessly. We demand the right to define it but when given the chance we always set the bar unrealistically high.

To recognise someone as a Principal Engineer, they should be definitively and objectively exceptional in every way, right? Their depth of knowledge in all technical areas is unmatched, they deliver innovation, domain knowledge, expert teaching and mentoring, strategic vision. They're growth hackers and also they design and deliver timeless systems and ever-green architectures. They're research scientists, and also inspiring leaders who (with no formal seniority) are able to align large groups of engineers via respect alone. How else can they be deserving? I wonder, does any other discipline in the org require this?

Whenever we come close to settling on a sensible way of recognising the value that experienced engineers (that are well connected to the domain and the business) can bring, it takes only a few years before that route is smashed by a new iconoclastic bunch of thought-leaders.

The parts of the org that are outside engineering now just let us get on with sabotaging ourselves.

I'm not familiar with Amazon, but the equivalent level at FB (according to levels.fyi) represents just 1% of engineering. So this isn't only "experience, seniority and progression", it represents something really unique.

It's hard to look at a list like this and say "oh I know what to do now", but it is useful as a rubric; is this person meeting most of the criteria? Just a few?

Principal at Amazon is the first level above senior software engineer, so closer to staff than principal at Google/Facebook. It isn't a particularly uncommon level to reach at Facebook or Google.

Principals at Amazon are 1.8% of the SDE population, so no, the comparison to "Staff" engineer is unfounded.

Amazon is just more stingy with promotions.

Yes, and so their "staff" engineers stay Sr. Engineers. Isn't that the same point I made?

In a field where the spread of outcomes is as wide as software, it seems reasonable that the right tail of practitioners would be very impressive.

In practice, no company requires their equivalent of principal engineer to hit 100.00% of the defined marks, but rather a comfortable majority of them with no glaring weaknesses.

Principal engineer roles should be filled on ability and impact not seniority/experience.

Bear in mind, an Amazon Principal Engineer is intended to be a 1 in 100 engineers position. The pay is generally above 600k, with the max pay dependent on the position.

> as an industry we're uncomfortable with experience, seniority and progression. We're collectively suspicious of it. We want to codify and clarify it, endlessly.

Are we wrong to? We've all had the experience of being held back by poor decisions from a senior person who simply didn't have the skill. I watched a 200-person company full of smart people single-handedly destroyed by a "software architect" who just made a lot of, frankly, stupid choices that all of the freshly-graduated engineers could see were stupid.

Frankly I've found the vast majority of "experienced engineers" aren't actually any better at their jobs than fresh grads. There should be very high, objective standards before we allow one person to overrule the rest of the organisation.

This is a bit too buzzwordy for me.

> Resounding Impact

> game-changing technology choices.

> aligning teams toward coherent architectural strategies.

> lead by example

> pioneer new spaces

> inspire others as to what’s possible.

> pragmatic problem solvers

> bring clarity to complexity

> illuminate pitfalls

> probe assumptions and foster shared understanding.

This perfectly illustrates why ppl dismiss software architects that they hide behind silly buzzwords without delivering anything tangible.

This reminds me of the discussions back on wiki.c2.com of https://wiki.c2.com/?ArchitectsDontCode and https://wiki.c2.com/?ArchitectsPlayGolf

There's also https://wiki.c2.com/?NonCodingArchitectsSuck for a rant.

Can you think of more concrete language that applies equally to all organizations? If so please do share.

Sure, there are the Dilbert PHB-esque abuses of trendy lingo through mindless cargo culting. But the bigger and broader your goals, the more abstractly you have to describe them lest you end up writing a full-on textbook.

Having said that, I have to disagree that your examples are buzzwords. Buzzwords are usually contextless, like saying "synergistic impact" without any hint of what that means.

As just one example, "Probe assumptions" is not a buzzword. It has a clear meaning within the context of software engineering. For example it is often assumed by new grads starting a company that a startup just has to use Kubernetes and microservices. But that is an assumption that is worth probing. Horizontal scaling only becomes a bottleneck long after the millions of users point. Microservices are meant to help giant engineering teams isolate the things different teams work on, but in a startup with only one team you don't have that bureaucratic overhead in your org, so you probably don't need it in your code.

> Microservices are meant to help giant engineering teams isolate the things different teams work on, but in a startup with only one team you don't have that bureaucratic overhead in your org, so you probably don't need it in your code.

We had these discussions many times on our team. Everyone understands that there are tradeoffs and "pitfalls" associated with any decision. Isn't this just normal rational thinking? Is your assumption that ppl that aren't principals don't consider pitfalls. Do non-principals employees really think that there are no pitfalls to using microservices.

My experience is that people rationalize what they want. If they want microservices then you're going to get microservices.

It's not all nefarious, I think if many of them truly _UNDERSTOOD_ the difficulties of microservices it would give them pause (there's a difference between knowing and understanding), but they want what they want, they're optimistic, and they tend to back into the reasoning afterwards.

That’s a junior mentality. There’s nothing more dangerous to a resource constrained startup than a highly talented engineer who thinks this way.

I've seen many engineers reach for a microservice architecture by default.

Why? Because they read HN, reddit and lots of blog posts, and thought that's how you do things now if you start from scratch, no legacy architecture to consider.

Not everyone thinks this way, but it's a frighteningly common thing.

Is your assumption that ppl that aren't principals don't consider pitfalls.

No such assumption on my part. But it might be safe to assume that the better you are at questioning assumptions, avoiding pitfalls, and thinking of the big picture (think breadth-first vs depth-first traversal of the problem space), the more likely you are to advance to higher levels of responsibility.

Another example. Suppose you are adding a new field to your payment processing system. You have considered engineering pitfalls. How will that change affect other teams? What about other departments? Will Accounts Receivable suddenly have a 10x workload of invoices to chase because that field change wasn't considered in their workflow in all cases?

There's nothing magical about what senior vs staff vs principal vs whatever engineers do, it's the same skills amplified and applied to the benefit of many people across an org, rather than limited in scope and awareness to a single project.

> Everyone understands that there are tradeoffs and "pitfalls" associated with any decision.

I haven't seen this to be true. Especially as companies get larger.

Sometimes it's because they really believe option A is better in every dimension than option B. (But often are mistaken.)

Sometimes it's because option A means their team has more work and can hire more people.

Sometimes it's because they understand their own ideas really well but aren't understanding other people's arguments.

Sometimes it's because they just don't respect the other people in the room.

But assuming that everyone is going to rationally jump initially to the thing that's best for the long-term health of the organization isn't something I've been able to count on.

It's corporate mythos. Something that worked for someone long long ago was so successful that it became the corporate form of a folk song.

When we down near the bottom hear "lead by example", it's facile to find the nearly useless obvious veneer to it, but if we could attach the original odyssey that led that engineer to tell their story ... we might agree with the lesson. But retellings upon retellings compress, simplify and gloss the original story into a cliche.

Each thing in this list sounds obvious, but if one can picture the opposite of the advice to be false, that reflection might actually lead an individual to growth.

That’s something Amazon does well:

Principal engineers are expected to give talks about their experience, which are heavily attended in person and widely shared.

It’s not perfect, but it’s less abstract when one or two concepts at a time, they explain why those truism through their personal experience.

“Earn Trust” is a truism — but being able to search for talks by the keyword “earn trust” where a high level engineer discusses different situations from their career and how they handled them is useful.

There was one key entry there, actually coding. Non-coding architects are rife with the possibility (but not certainty) of being too disconnected from the reality of the systems they are working on.

I guess Principal Engineers are also really good at corporate bullshit speak

These seem like totally different roles. And the hours I've spent staring at the AWS console trying to figure whether I need "Elastic Beanball3" says to me that Amazon could use maybe just a little architecture. But I'm sure the Beanball Principal Engineer is quite proud of their resounding impact.

The console is a UX/UI issue, not a "architectural" one. Most "architects" nowadays are just mere users of what was built by the engineers and principal engineers of amazon.

AWS services don't mesh nice with each other, and the reason "software architecture" exists in the first place is so that disparate teams and technologies play nice with each other.

The disparity within AWS is the result of common inevitability that arises with managing technology built on over a decade of scaffolding. For sure there is a central planner within the company trying his best to put everything together into a cohesive whole. Technical debt is the most common name for this issue.

An "Architect" can't solve this problem anymore than an "Engineer" can fix the problem of technical debt because neither person can see the future. It's like pointing at everything wrong with a piece of technology and saying an "Architect" could've fixed that in hindsight.

"Architects" are not the solution to technical design and technical debt.

An engineering manager can place all engineers in a room and ask them to come up with an "architecture" that is the same if not better than anything a single "Architect" can come up with, and this "Architecture" will never solve the problem of technical debt for the same reason why we can't predict stock prices.

Technical debt isn't the root or even the most common cause of software components not playing nice with each other.

In fact, this problem is probably more prevalent in greenfield over-engineered solutions than old legacy-ridden ones.

(And really much of what we call "technical debt" is just the scaffolding we put together over the years to make stuff be compatible.)

I think if you ask, most anecdotal evidence from people who have worked with architects shows that companies with "software architects" don't necessarily have less "technical debt" then companies without.

Theoretically, those "software architects" would put a stop to "over-engineered" solutions, but why do these companies still have technical debt?

Most likely because technical debt is not solely just "over engineered" solutions.

> In fact, this problem is probably more prevalent in greenfield over-engineered solutions than old legacy-ridden ones.

Yup. In that context, it is usually called Second System Syndrome (when discussing the person that is laboring under it's influence while in the act of designing) or Second System Effect (when discussing the resulting system):

https://en.m.wikipedia.org/wiki/Second-system_effect

Hard disagree. Thinking you can separate form from function is the evergreen mistake of our industry.

But you can. The separation is critical to the entire computing industry. The entire reason why higher level programming languages exist is to separate the form (assembly) from function.

Usually there are "costs" to doing this and much engineering work is done to reduce this cost as much as possible. Either way, what you say is categorically wrong. Separating form and function is not a "mistake" given that it is so fundamental to the entire industry.

Ah, but I would argue those are better described as form and function being part of the designs. Done wholistic, it is why the new apple hardware is able to take some very impressive gains.

You can argue that some interfaces are separate products from the services. I agree with that. But each has its design and each is targeted on form and function to something.

That is, you don't separate them. You make a new product, that has its own form and function.

My remark is based on what you said. You called the separation of form and function a "mistake" and I said this is categorically wrong. My statement still stands, you have not addressed it.

I neither agree or disagree with your remark about "Wholistic" designs but your example is extremely inapt.

The entire MacOS is programmed using a an interface separate from function. If they didn't program the Operating system in a higher level language that can be compiled into separate architectures then the move to ARM would be an even more complex engineering maneuver. The entire reason why Apple was able to move to new hardware without completely rewriting the software is because Form and Function are separated.

My address of it is that things don't separate to distinct things, often. They, instead, made new forms that have their functions. Or vice versa, if you want.

This is true for content as much as services. In simplistic cases, you can point at something and say "there is the content." Often, though, you cannot. The distinction between the two is a huge blur.

You are bringing up, essentially, that abstractions can be layered. I agree with that, with each layer having is own forms for its functions.

Applied to the example here. I am not saying they are identical. I'm saying that they are best considered together. Such that our industry typically passes the buck on UI and UX, to the detriment of everything.

No. I am bringing up the fact that form and function are always separated in our industry and that this is fundamental to the entire industry making your initial statement categorically wrong.

It's obvious that abstractions have costs and it's obvious that sometimes things are designed holistically. But these obvious things are off topic. The point here is that your initial statement is completely and utterly wrong wrong wrong.

Please stay on topic. Are you still claiming your initial statement is right? or are you now claiming it's wrong? Don't change the topic into the industry "passing the buck to UI and UX" this is not your initial statement.

My opening statement is short, so I'm assuming some charity in reading it. My complaint is the idea that you can make a hard distinction between function and pass that to another layer is a huge mistake in our industry. As noted by us conversing in a web browser. :)

Are they separate things? I mean, yeah. But they interplay with each other such that near every attempt to force them apart is full of ugly edge cases such that I assert we would benefit from people seeing that form and function typically work together in building things that work.

>My opening statement is short, so I'm assuming some charity in reading it. My complaint is the idea that you can make a hard distinction between function and pass that to another layer is a huge mistake in our industry. As noted by us conversing in a web browser. :)

Ok, so you are basically implying that your opening statement is wrong and what you meant to say is that abstractions do not have zero costs and that the industry is unaware of this. Just admit you're wrong instead of relying on the political stratagems used by "architects" to stay relevant.

I would argue that everyone is fully aware of this concept. It's a necessary evil done to reduce complexity otherwise we would all be programming in assembly language. As for AWS specifically, isn't it proof of what you say? The technical details are leaking into the interface showing that function and form can't be fully separated.

If you're saying the architecture of AWS should have been holistically designed to support a more intuitive interface, I would say you're describing another problem all together. A holistic design is a synonym for "waterfall," and there's a good reason why it's avoided in the industry.

The problem with waterfall is that the current needs of the market is always a moving target. No architect or even UI/UX designer can predict the needs of the market in the future thus things are continuously put together agile style (aka ad hoc). This produces technical debt which is something neither Architect nor the UX/UI designer can solve. AWS definitely suffers from some debt and the UI reflects this, but again, this is NOT because they lack "architects" as every company has this problem whether or not the company has "architects."

With my statement following the claim that UI/UX is not an architectural issue, I'm not sure how you are getting this reading.

I maintain that separating the ui/ux from the architecture is a fools' errand. Any other reading of my statements, while almost certainly my fault, don't make sense here.

The way I took it is that you said your original statement in a broader context along the lines of "in computer science form and function should never be separated." And you used that statement to apply it to UI/UX and architecture.

This statement is patently false, especially for UI/UX. Even at the lowest level there are platform agnostic API's (opengl) that are divorced from CPU/computer architecture. At even higher levels, javascript/css/html is a graphics API that is divorced from even the operating system. Separation of form and function is the foundation of software. It cannot be a "fools errand" when all of UI/UX is designed with the separation of form and function in mind.

It is therefore fundamentally impossible for you to stand by a notion that is completely false. More than likely your statement is just a mistake and you didn't think it through. Such is one of the downfalls of "Architects"... they are so concerned with high level truthisms that they fail to even build the correct high level picture of the computing world.

For software systems, the devil is, unfortunately, in the details.

You still seem to be taking a very non charitable read. I aluded to the browser, specifically because of the hard split between content markup and stylistic markup that is of very arguable benefits.

It is alluring. And when it works, it feels magical. Yet I still prefer TeX over many of the document systems I have been subjected to use. Markdown being a strong modern contender that embraced the ascii art nature of plain text files. Throw in the modern resurgence of jsx and the hard won separation of scripting from content is not even fought anymore.

To the ui/ux point, directly, I'm not even sure what you are arguing. The APIs that you list are all notorious for being bad leaking abstractions. And is why people make products over them, introducing new forms and functions for a different set of users. Just look at how you shoe horn shaders into those low level APIs.

But again, in the context of this story, I meant that you can't separate out the ui/ux concerns of a product completely from who owns that product. So, for AWS, they own both service and console. If one is bad, it is almost certainly from lack of prioritization in the team that owns the whole product. Often behind the guise of "this is not the architecture's concerns, but the front end."

Jumping to another metaphor to try and say it differently. In a bike, you can separate the drive train from the steering, but it will be a largely perspective based separation. Such that the entire bike is ultimately part of the steering and the power distribution. Yes, this is a simplistic look at it, but no more so than thinking you can fully isolate one from the other.

(Now, I do have to confess that Tufte is finally really resonating with me. And I fully agree that much of this is as useful as so many platitudes. Ask me about front end versus back end, someday. :) )

> To the ui/ux point, directly, I'm not even sure what you are arguing.

Yeah I can see that given how off the point you are in your posts. Your initial response is that the separation of form and function is a mistake.

I’m saying it’s not a mistake, separating form and function via abstraction is intrinsic to the industry and I use UI/UX as an example. Your retort is that UI/UX abstractions are notoriously leaky.

How does a leaky abstraction move the conversation in any direction? Most abstractions are leaky, everyone know this, but despite this we still need to program our computers via a high level language designed to separate form from function. Otherwise everyone would use assembly language. The leaky ness is known but to manage complexity we have to separate form and function.

So your statement is completely and utterly wrong. Your responses fail to prove your point and go off topic into unrelated metaphors, tufte and Tex. Stay on topic.

To be fair, most of AWS's services are also mere users of other AWS services

Naw. The term "mere" was to de-emphasize the importance of the term "architect" which doesn't apply to other AWS services.

The parent poster declared that AWS needed some "architects" which I think is kind of arrogant given the fact that most "architects" don't even have the ability to engineer the tools built by AWS. They are just "mere" users of the tools.

I would argue that naming very high-level things (like your top-level services) is very much an architectural decision.

Agreed. Only Software Architects have the special ability of naming things. Engineers and managers don't have this special knowledge.

When I went to Standford to get my PhD is Software Architecture I was required to take at least 2 semesters of classes on the theory of naming things in software. The biggest point the class made was proving mathematically how one should never name anything after a plant, Greek/Roman god, gemstone, animal or molecular structure.

I hate with a passion services that have cute clever names. It makes it impossible guess what they do. It makes it easy to overload the service with things that don’t belong.... if the service was named “OrderService”, if somebody wanted to add a “spell check” endpoint people would start asking questions if that service was the right place. But since the service was named “CaptainCrunch”, who knows if spell check belongs...

Same goes for team names too... clever team names are obfuscation.

Agreed, the post you replied to had a sort of different point though. I took two semesters of naming things as part of my PhD in software architecture. Do you know anyone with such a PhD? Do you know of any course in any university that teaches people how to name stuff? Did you need to take such courses to arrive at your philosophy on naming things?

Mathematics does calculations, not decisions.

The post was sarcasm that flew over your head. You can’t even get a PhD in software architecture because the entire field is a sham title created by industry. There is no theory or science behind “software architecture”

Either way you are wrong about math. All decisions can be mathematically modeled. It’s called a conditional expression, and you use it to model/calculate decisions when you program.

Different conditional expressions make different choice in the same situation, then you need to choose what conditional expression should be in the program.

Different numbers produce different results in the same arithmetic expression.

Here's the concept of conditional expressions explained via a mathematical text: https://math.libretexts.org/Bookshelves/Mathematical_Logic_a....

UI can only show stuff in different place, with different colors and fonts, but it won't solve the problem. Explaining architectural purpose of a tool is an architectural job, neither UI designer nor salesman can do it, especially salesman.

There’s a concept in CS called an abstraction layer and the UI can literally change the thing underneath. I can create a UI with one button that when you press the button it triggers 50 actions to happen. It is not just about moving fonts and colors. This the exact same thing that a programming language does. A programming language abstracts away assembly language by having one programming expression execute several assembly commands. This is the job of abstraction layers and also the job of UI/UX.

Second explaining architecture is not solely an architectural job. It’s mainly the job of the person who writes documentation. The “architects” job is pretending he knows some “theory” of architecture more then the engineers actually coding the “architecture” when no such “theory” even exists.

Then architect should tell the purpose of the tool to the documentation writer, otherwise the documentation will read "press Ok button to install %toolname%" that tells nothing how to decide what the tool is and does and whether the user needs this tool. That's the problem at hand.

This isn't specifically the job of the architect. Any idiot can explain how the tool they built works. So the software engineer or the product manager can do this as well. Explaining how something works is not a magical skill reserved for software architects.

What is a skill reserved for architects? If you find yourself spending 100% of your time creating a powerpoint to explain things in a way that makes you sound smart and takes all the credit away from engineers then what you did there is a skill exclusive to architects.

The architect is the only idiot, who knows the tool as a whole and, more importantly, its role in the bigger world. Other idiots know only how bits are fiddled in a small corner of the tool on which they worked during their probation, they don't know the big picture, at all. In any case documentation in big companies today is outsourced to people with zero programming experience, and they obviously can generate only absolutely vapid text with traces of fuzzy associative connections with the topic.

So literally you’re saying the architect is the only person who can write good documentation.

Also the architect is the only person who did not write a line of code yet understands the tool better than engineers who work, code and solve problems across the entire stack.

That’s not only arrogant. It’s delusional.

He doesn't know how local variables are named, but what's needed for documentation is purpose and nature of the tool, and that's what engineers often don't know, because it doesn't help with bit fiddling. And a coding architect is an architect too.

> trying to figure whether I need "Elastic Beanball3"

Just wait until you have to try to figure out how much that's going to cost you...

As far as I can tell, the state of the art is to run it for a month and see what the bill is.

After many years at big tech companies, I am now working at my first that has "Architect" as a separate role. While many of my colleagues in that role are great people and very smart, after a few years of living with it, I strongly believe that it's an "org smell" and I will seek out companies without it in the future. It creates completely the wrong dynamic to have the hands-on engineers, no matter their seniority, not be fully empowered and expected to make these decisions. And no matter how experienced and smart the architects are, if they aren't working hands-on with the systems and code on a regular basis, they can't make the best decisions. Software architecture as a separate job from software engineer is bad for everyone involved.

It gets worse. The transition from "programmer" to "architect" results after a few years with the architect losing track of where the rubber meets the road. This results in worse architecture decisions that are made by a similarly experienced programmer.

This effect is so strong that if you're interviewing someone who became an architect, make them write code in the interview. If they don't do it really well, you don't want them in ANY role. Because now they can't program, they can't architect. All that they can do is sound smart while they make bad decisions.

I don’t buy this analysis. The principles of architecting code and system design are timeless. There are definitely people out there who don’t “code really well on command”, but can ask the right questions to design a solid system.

The principles of architecting code and system design are timeless.

That I emphatically agree with. Some of my favorite programming books are decades old.

There are definitely people out there who don’t “code really well on command”, but can ask the right questions to design a solid system.

There are?

I can't say that I've ever met one.

The problem is that people don't know what they don't know. A non-programmer can know exactly the business problem to solve and exactly how they want to solve it. But they can't tell the difference between what is easy for a programmer and what is hard.

They don't have to learn a ton of programming to fill that bit in. But if they never fill it in, that limitation will keep them from being able to be a good software architect.

It will be me. I am systems architect and last production code that I wrote was probably 10 years ago or so.

I am probably rather crappy programmer. I was crappy perl programmer, crappy php (starting with version 2), crappy in delphi, c, c++. Probably okay in Python, but mostly because Python allowed to concentrate on solving problem and not fighting the language.

Me been crappy was probably because I never really enjoyed to write code, as interesting part is figuring out solution to a problem. Rest is implementation details :) You can say if you want that I am simply too lazy to write code.

Me been crappy in writing code doesn't mean that i am bad in understanding code or understanding what is it to write code the easy way or the hard way.

In majority of cases when programmers come to me with design that will be easy for them to implement, turns out that they have a bunch of big problems that they missed, mostly due to lack of experience. Most of the time solutions that they come up with complicate things further and add even more problems.

And this is the point of time when I assist them to simplify whatever complex design they made to something much more simple and doable.

I saved many man/years of development time by showing programmers easy ways that they didn't know that they exists

Your self-assessment matches the self-assessment of other systems architects that I've seen. Suffice to say that my assessment was so different that I never again want to work at a company who have people who think that is their job. And I've seen a wide variety of programmers from a wide variety of companies agree. As a random example look up the thread for someone who had decades of bad experience with architects at Microsoft who is massively happier at Amazon (which actively avoids the architect title and mindset).

Of course I've never met you, and you may be better than the bad examples that I'm aware of directly and indirectly. But a priori I have no reason to expect your self-assessment to be more reliable than past experience.

Totally understand you At my last job I was "granted" a team of architects. One of them got self-terminated after a few weeks as he was completely useless and he knew it (he got even more senior job in different company), one was okay and one required constant supervision because he was astronaut architect (he left after half an year).

Architects that know what they are doing are hard to find (my last positions were open for 6-9 months before i arrived. me hiring extra architects is similarly long process). From the other side, hiring manager/interview loop needs to have proper skill set to evaluate person and to make sure that this is what they look for (last time i got interviewed by principal architect, vp r&d, and a bunch of hands on dev leads from different departments across the company).

"make them write code in the interview... Because now they can't program"

This comes across disrespectful and presumptious.

Firstly, do you believe that if you have not coded for 4-5 years you become useless just becauss you don't know the latest %fad% framework?

Secondly, the whole phrasing just conveys disrespect for cadidate, where the only responce to 'jump' is 'how high'.

If someone is interviewing for non-code postition they might tell you to take a hike.

The idea that a contrived 30 minute coding session tells you more about a candidate that studying code they've previously written or debugging together doesn't make awnce and is not supported by evidence

We don't ask surgeons: "since you can do heart surgery in 8 hours, surely you can reattach a toenail in 10 minutes during an interview"

"make them write code in the interview... Because now they can't program" This comes across disrespectful and presumptious.

It is the interviewer's job to check the possible reasons to not hire someone and validate that the they aren't going to be problems. One of the possible reasons to not hire someone whose last job was "architect" is that they have lost contact with coding. If they have, then from experience I can tell you that they won't want to go back to coding, and their architectures are losing contact with reality. They are a liability.

This observation is not original to me. The first time that I saw it made explicit was when I took interviewing training while I was at Google. Google's experience is that a certain fraction of architects avoid coding. That fraction makes really bad hires. So they explicitly told people to look for that in the interview so that don't wind up accidentally hiring them.

But if someone's title is "architect" and they can code, then that isn't a concern. Companies do tend to promote good programmers to architect, and you don't want to miss out on those programmers.

Firstly, do you believe that if you have not coded for 4-5 years you become useless just becauss you don't know the latest %fad% framework?

This is a straw man argument. I said nothing about "latest fad framework". The exercise is to write some code, not write code in any particular framework or language. Go ahead and be offended at something that I didn't say, but please remember that I didn't say it.

Secondly, the whole phrasing just conveys disrespect for cadidate, where the only responce to 'jump' is 'how high'.

If an interviewee objects to writing code in the interview, how are they going to respond when they're asked to write code on the job? Seriously, if they get offended in the way that you just did, that's a good reason not to hire.

If someone is interviewing for non-code postition they might tell you to take a hike.

I do not want to work in an organization where "architect" is a non-coding position. Seen that, left in horror. So if I'm interviewing you, you are almost certainly being hired for a coding position.

The idea that a contrived 30 minute coding session tells you more about a candidate that studying code they've previously written or debugging together is idiotic.

It doesn't tell you more, but it tells you something different. In particular it tells the difference between someone who shows their own code versus shows someone else's code and lies about the source. (Yes, I have seen that.)

"Companies do tend to promote good programmers to architect, and you don't want to miss out on those programmers."

Everything in your responce sounds to me like in your view, architect is 'extra senior' developer, whereas I understand it to be a different job. Isn't what you are describing a principal engineer?

"In particular it tells the difference between someone who shows their own code versus shows someone else's code and lies about the source. (Yes, I have seen that.)"

To eliminate that possibility, you can ask questions about the code, how it was designed, etc. Presumably if they stole the code, they are not going to know it inside-out, like the author would.

Tip, the word is "response", not "responce".

In my view, software architecture should be part of the job of a principal engineer, and the role "software architect" should not exist as a separate thing.

Separately, you would be amazed at how many developers will happily give you code that they wrote a year or more ago, and then not remember their design decisions. So not knowing it inside and out doesn't imply that they are not the author.

I interviewed a very senior architect astronaut who literally couldn’t write code to compute the average of a list of numbers.

Wasn’t that they got hung up on the details of whether the average of ints should be an int, float, or double, or how to handle overflow, but just couldn’t get started at all.

After that, I stopped shying away from wanting to see code from even senior technical candidates.

(It was not a standard question for senior roles; I got a strong whiff of “this guy might be entirely full of it; let’s have a look-see”.)

i am very senior architect astronaut. it will take me some effort to computer the average of a list of numbers in c/c++ as i didn't touch those languages in many years.

from the other side, while reviewing designs (not code) of various components I can spot locations where code handles sockets incorrectly or a couple of dozen of scenarios that system will fall apart under because engineers don't think that far.

value that I bring is not in averaging list of numbers. my value is 25 years of making mistakes, learning from them and knowing how to avoid them.

Algorithm interviews should always let the candidate choose language to avoid these bad excuses, or in worse case even fall back to pseudo-code. If you can't calculate the average of a list of numbers in pseudo, then you are out regardless of how many years of mistakes you are bringing to the table.

in pseudo i can. or in python. avg(list) :)

anything more complex, not sure. i touched last time linked lists in 2001. never learned algorithms and in fact didn't even graduated from school. and guess what - everybody is okay with all of it. because this is not the value that I am expected to deliver to the company.

edit: when people are interviewed for system architect positions, algo/coding interviews are not part of the process. because it's not skill set that company is looking for.

For the record, with that interview, even saying “in python, I’d use the standard library avg(list), because it would be ridiculous to re-implement provided and tested ecosystem functionality” would have served him well in terms of indicating he wasn’t an utter fraud. That alone wouldn’t land the job, but it would be a strong positive.

depends on the company. i once (15 years ago) was interviewing with a company and they essentially asked me to whiteboard some regex like pattern matching flow. i did it, and after completing it to their satisfaction i said that in real world i wouldn't do it, as it's waste of time and will use lib* whatever. I didn't get this job :) a few years later, i discovered that few guys on one of the teams were from this very company. what was standing out about them, it's that they always had to reinvent any standard functionality/library with peak of it been their own xml-rpc protocol that they came up with, because xml-rpc wasn't good enough. I guess ethos of this company and all people that they hired, was to reinvent wheels, bicycles, etc and I didn't fit that ethos well .

this was also the only whiteboard coding interview in my life

To be fair, xml-rpc truly is terrible. To start with, XML is extremely verbose and you add a lot of network overhead for that, plus a lot of parsing overhead, which makes rpcs significantly more expensive than they should be. Use https://capnproto.org/ instead. (Based on protobuff, which is an open sourced version of what Google uses internally.) As far as I'm concerned, the only valid reason to use xml-rpc is because you're interfacing with someone else's system and they have chosen to use it.

Moving on, here is an excellent example of an important architecture system that almost everyone gets wrong. In any distributed system you should transparently support having rpc calls carry an optional tracing_id that causes them to be traced. Which means that you log the details of the call with a tracing id, and cause all rpcs that get emitted from that one to carry the same flag. You then have a small fraction of your starting requests set that flag, and collect the logs up afterwards in another system so that you can, live, see everything that happened in a traced rpcs. To make this easy for the programmer, you build it in to the rpc library so that programmers don't even have to think about it.

You then flag a small random small fraction of rpcs at the source for tracing. This minimizes the overhead of the system. But now when there is a production problem that affects only a small percentage of RPCs you just look to see if you have a recent traced RPC that shows the issue, look at the trace, and problems 3 layers deep are instantly findable.

Very few distributed systems do this. But those that do quickly discover that it is a critical piece of functionality. This is part of the secret sauce that lets Google debug production problems at scale. But basically nobody else has gotten the memo, and no standard library will do this.

Now I don't know why they reinvented xml-rpc for themselves. But if they had that specific feature in it, I am going to say that it wasn't a ridiculous thing to do. And the reason why not becomes obvious the first time you try to debug an intermittent problem in your service that happens because in some other service a few calls away there is an issue that happens 1% of the time based on some detail of the requests that your service is making.

It happened 13 years ago and it was exactly same but with different xml syntax :) It was way before microservices, etc. purely point to point.

It was way before microservices, etc.

It was not way before microservices at Google. But it was before there was much general knowledge about them.

Sadly the internal lessons learned by Google have not seeped into the outside world. Here are examples.

1. What I just said about how to make requests traceable through the whole system without excess load.

2. Every service should keep statistics and respond at a standard URL. Build a monitoring system which has scrapes of that operational data as a major input.

3. Your alerting system should supports rules like, "Don't fire alert X if alert Y is already firing." That is, if you're failing to save data, don't bother waking up the SRE for every service that is failing because you have a backend data problem. Send them an email for the morning, but don't page people who can't do anything useful anyways.

4. Every service stood up in multiple data centers with transparent failover. At Google the rule was n+2/n+1. Meaning that your service had to globally be in at least 2 more data centers than it needed for normal load, and in every region it had to be in at least one extra data center. With the result that if any data center goes out, no service should be interrupted, and if any 2 data centers go out the only externally visible consequence should be that requests might get slow.

Now compare that to what people usually do with Docker and Kubernetes. I just have to shake my head. They're buzzword compliant but are failing to do any of what they need to do make a distributed system operationally tractable. And then people wonder why their "scalable system" regularly falls over and nobody can fix it.

Exactly. Maybe there are companies facing problems so unimaginably vast that architects who can’t write any code are critically needed. (Not who don’t which would be okay for a lot of companies, but who can’t.)

To me, this seems more like an executive chef who doesn’t know (or has forgotten) how a knife works rather than an Elon Musk or Henry Ford who doesn’t know how to weld a body panel.

my current job is company with 1B in sales, with code base of one of the the main products started about 20 years ago, few additional acquired and semi integrated companies/products, high hiring rate of young and eager developers that know how to code very fast but don't know how to keep thing from blowing up in production environment.

what i do spans from feature set and scope negotiations with product (because devs don't have either patience or knowledge to do so), design reviews of any semi major changes or additions, helping with designs, revamping sre/devops practices and company infrastructure, figuring out what land mines we got in our systems in the past and how to remove them and more generally what changes in architecture are needed in order to carry company forward given it's rapid growth.

in overall I participate or oversee activities of few hundred developers across few continents and equal amount of sre and infra/tool people and use three spoken languages in order to do so (could use another one, but never got to learn it.)

hence, i am not musk or ford that don't know how to weld. i know welding, it's just not the best way to use me. my job is to make sure that rest of assembly process will be smooth and car won't explode when it hits the road (and it's not something that welder does).

Oh, I am sure that you do lots of work. And your company is organized in such a way that your role is critical path for lots of things.

But if a company the size of Google can get by without software architects, why does your company need to be organized in such a way that you require that role?

Back when I worked there, one of the universal shocks when people go to work for Google is that the overall architecture was light years ahead of anywhere else. The person usually given credit for that is https://research.google/people/jeff/. Based on results, he may well be the best software architect alive.

If you want to learn from his example, I recommend that you start with "a running prototype beats a whiteboard design".

hence, i am not musk or ford that don't know how to weld.

What sheer irony.

BOTH Musk and Ford knew how to weld, and considered that knowledge essential to being able to do their jobs. Sure, they didn't do a lot of welding day to day. But how could they make correct decisions about how to build new machines if they didn't know how those machines are put together?

>But if a company the size of Google can get by without software architects, why does your company need to be organized in such a way that you require that role?

I have a few notes here:

0. Google maybe is exception. But i never worked in google, so i don't know how good it is really.

1. Not all companies can pay same wages and get same talent as Google

2. Every time there is talks about GCP/AWS/Azure a lot of people say that all of those services seems to designed by completely different people and don't mesh together nicely.

3. I worked in one of the FAANGS , and while it's widely known for it's excellent engineering it was a massive dumpster fire. There were excellent people that did amazing things on lower infra level, but in the moment that you move a bit higher up it was disorganized mess of multiple teams not knowing how to collaborate properly what resulted in outages, sometimes rather big, which were not noticed from outside mostly due to size of production environments that allowed to shift users to still functioning parts of the system. Those outages were mostly result of not having somebody who will look at system end to end and will identify potential cascading failures or weak points in general.

>BOTH Musk and Ford knew how to weld, and considered that knowledge essential to being able to do their jobs. Sure, they didn't do a lot of welding day to day. But how could they make correct decisions about how to build new machines if they didn't know how those machines are put together?

I didn't say that I don't know how to code. I said that I won't pass coding interview. Companies that I work in have coders much better than me, but I am much better than them in understanding how system supposed to work end to end and how to prevent it from collapsing under various unpleasant scenarios.

I did work at Google, and their engineering architecture is insanely good. Their product design, not so much. Both aspects are visible when you use their site.

I acknowledge that Google is able to hire incredible people. They also got a lucky break in hiring Jeff Dean early. However it is my considered belief that when you call Google an exception, you're reversing cause and effect. When people who come up with the architecture have to write code, work with their system, and see from experience what is wrong with it, they produce better architectures.

It is not whether coding produces more value than architecture. It is that continuing to code informs their architectural decisions.

For a similar idea in a very different industry, when Robert Townsend was CEO of Avis back in the 1960s he turned the company around. He made it profitable, and made it grow.

One of the things he did that he says made a huge difference was to make it a rule that everyone worked the rental desk. Didn't matter whether you were the CEO, VP, big manager or whatever. One day a month you stood behind the rental desk and had to deal with live customers. And having that regular experience meant that problems in the organization that would otherwise go missed became instantly visible. Just like how actually having to code and debug things in your architecture makes architecture problems visible that otherwise you'd discount. It made Robert Townsend a better CEO. The corresponding exercise would make you a better architect.

See https://hbr.org/2010/02/make-the-change-be-an-undercov for more.

>I acknowledge that Google is able to hire incredible people. They also got a lucky break in hiring Jeff Dean early. However it is my considered belief that when you call Google an exception, you're reversing cause and effect. When people who come up with the architecture have to write code, work with their system, and see from experience what is wrong with it, they produce better architectures.

Totally agree with this. I personally started with installing trumpet winsock and netscape navigator on win3.11. I did customer support while been developer. I went through most of positions that are usually part of software development org and dealt with with most aspects of software. And I consider myself lucky to been able to do so.

But the unfortunate truth it's that in majority of companies that develop software (lets get outside of silicone valley for a moment), developers tend to be very far removed from production environments and from clients. As example, I was hired to work on a new, from scratch product, in one of biggest suppliers of software for telecoms (8B in sales). Over there "core engineering" was tasked with developing "core functionality" and there were totally separate delivery organization that will take this code and adapt it to client needs and will write installers/deployment systems, figure out how it actually integrate, run, etc. Clients weren't accessible to core engineering and internal product management couldn't talk to them either.

Also, the overall approach was that it's perfectly normal that software will crash and leave everything in inconsistent mess, hence there were mandatory tooling to be developed for manual operation to "fix things".

I spent first few months arguing with my boss that it's not normal and that we must deliver software that can be installed and work without crashing left and right. I got to built first devops org in company, first private cloud lab, pretty much first in company integrated ci/cd, jira+confluence (because company used internally developed "agile" tools that were unusable. and i actually got visit from internal auditors questioning me why we spent money on something that already exists in company), lobby to create track for engineers to interact with clients so they will understand what is real operational environment of software they are writing and what are the real requirements so they will be able to do better job, get training for developers so they will learn that there is something outside of "enterprise java" (as anecdote, when i presented rabbitmq as part of architecture, i got asked to what java standard it corresponds and when i said to none, i was told that they don't really understand why would I use it). Project was amazing success and delivered more in less time than was common in the company. (manager got fast tracked to vp/svp/gen manager overseeing 2B in business within 4 years. I got relocation to states :).

my point i guess, it's that most of software development in most companies is disorganized mess with developers been completely detached from realities production environments. And when I am hired, pardon the pathos, my job is to bring order into the chaos or light into the darkness or whatever. In cases when I am out of my depth in some areas (i well aware about limitations of my knowledge) , i usually have enough pull to hire some domain specialists that I can outsource to them designs/implementation.

In other words you're saying that "software architect" exists as a role in these organizations because bad organizational structures create problems that it is a band-aid for.

And I'm saying that the band-aid doesn't work that well, and you do better if you fix the organizational structure to not need it. And furthermore, as a developer, those organizations are no fun to be in and you can't pay me enough to want to deal with them. Particularly not when I have the option of finding better organizations to be in.

Which adds to the problem. When good developers choose not to be in your company because your company is a disaster for developers, you don't get good developers. Which reinforces the idea that developers aren't that good, and causes them to try to put guard-rails around developers to prevent the problems that bad ones cause. Thereby creating a negative reinforcement cycle.

Good point, architect is indeed compensating function that comes to cover for many deficiencies in organization. Those deficiencies are:

. Developers that are mature but lack proficiency because they don't care about software development

. Inexperienced developers that lack, well, experience to make proper designs

. Absence of domain specific knowledge when company tries to shift it's technological gears or switch to new domain/market all together

. Problems in communication and planning between dozens of teams/services that build together 1 product that must work in unison perfectly, yet it's hard for each team to know architecture/plans of all of the other teams and properly consider it during development

. Absence of ability to see "the big picture" by individual teams and take it properly into consideration

. Just a general jack of all trades that will compensate for bad product/project/program management, will establish proper processes, etc.

As part of this you occasionally can revamp organizational structure and leave company in much better shape, but it requires a lot of work that you, as architect have no authority to do :)

> And furthermore, as a developer, those organizations are no fun to be in and you can't pay me enough to want to deal with them

One of my last jobs was in company that I literally sweared to never work for. They gave me a shiny title, 50% raise of already top salary in industry that I had, yet what really sealed the deal was manager who wanted to try to do something new. Even though it was at times horror show, at the end of the day it was probably most fulfilling position that I had due to a sheer volume of things that I managed to change. And we also delivered totally new product, first in industry beating all the competition to market, including companies that were supposed to be market leaders in this area.

>Which adds to the problem. When good developers choose not to be in your company because your company is a disaster for developers, you don't get good developers.

The issue is that really good developers are rare. Developers that you can delegate to them and sleep well at night are a rare commodity on the market. I been working in dev for 25 years, both in startups and in established companies and I maybe know only dozen of people like this. Unfortunately, most of the developers on the market are in it only for money and don't really care about what they are doing.

Also, as anecdote, at this ^ job place that I described, in order to work around bad reputation of the company I suggested to create daughter company with a different name that will be operated independently in a different location in order to try to get better developers. We were a few steps away from closing on office lease before management bailed out because they were afraid that it will upset employees of the company

To the deficiencies that you list, I have to say that your organizational structure can create better developers, or make them worse. Delegating all responsibility for the bigger picture to architects and hiding developers from those issues will make them worse. On the reverse, I admit that not every developer can learn to do architecture well. But when you enable developers to take those responsibilities, the ones who are capable of it will step up.

My rule of thumb for developers goes like this. I mentally divide developers into junior, mid-level and senior. This has nothing to do with their titles. The breakdown is that a junior developer is still learning basic programming consistency and good habits. They generally struggle on programs that go over a couple of thousand lines. A mid-level developer has internalized consistency but is unable to remain aware of the big picture while focusing on details. They tend to top out and struggle at 10k-20k line projects. A senior developer is aware of the big picture while dealing with details, and remembers details while dealing with the big picture. They tend to top out around 200k line projects. And a principal software engineer adds an awareness of subtleties about how the big picture is likely to evolve, and how to guide that in a long-term positive way. These people can manage architectures in the millions of lines.

On this scale, I rate myself senior. To move to the next level, I'd need to spend more time at large companies with very large codebases. But dealing with the associated politics is not my preference so I'm happy where I am.

Judging by what I saw of environments with software architects as a role, I guess I shouldn't be surprised at this.

However I know a heck of a lot more than a dozen programmers that I consider good like that. OK, throw out top-notch organizations like Google, and just go for relatively small companies that I've worked at. Let's narrow it farther and just do places that I've worked in the last 10 years. I still have no problem getting past a dozen.

Why? Because I choose to work at organizations where software architecture is owned by developers. With other developers who have made the same choice.

(Incidentally, 3 of those programmers were co-founders at ZipRecruiter. Who hired a lot of mutual acquaintances on my list, then grew to be a billion dollar company. Somehow I don't think that this is coincidence.)

I most definitely enable developers to take responsibilities. I don't want to do more then what I need to do. I also prefer to help developers grow and take more responsibility, as it makes my life easier.

BTW, principal software engineer in many places == architect. Titles are very country dependent, company dependent. When startup where I worked was bought by US company I became principal. When I was acquired by German, I was retitled... don't even remember to what.

>Judging by what I saw of environments with software architects as a role, I guess I shouldn't be surprised at this.

I wasn't always working in companies that had this title. Yet, this still remains true. Also we may have different standards and operate in different environment on different types of products :)

It seems to me that in most organisation, the equivalence to 'rental desk' is facing the customer, not having to write some code. So we should be taking developers/architects/etc. and bringing them to interact with end users.

Unfortunately it's not always possible. I worked on OSS [0] for telecoms. Consumers of OSS are dozens of different teams across telecom. No telecom company will externalize those end users to vendors. The best you can get is meeting with with "enterprise architects" who collected requirements from all the teams and wrote RFI to which, if you answered good enough, you will be granted in person meeting to explain them better how your stuff is working.

[0] https://en.wikipedia.org/wiki/Operations_support_system

The engineers you work with have to write code that does the equivalent of averaging a list of numbers, many times per day. If you can't (in any language) handle this then I question your ability to understand whether your choices make it easy or hard for the engineers to do their jobs.

in some languages i will write the code. more complex algos not, because i don't know them. and never cared to know them. in reality engineers that i work with, write code that does whatever_library.avg(list_of_numbers). if i will see engineer that many times a day implements code that averages list of numbers I'll have a serious conversation with him about more efficient development practices

and maybe you are questioning my ability, but engineering teams that I work with - not. They come to me so i'll help them with problems that they have.

This "average of a list" is a red herring, because in practice one gets some variation of dynamic programming, trees, graphs, etc.

And for these kinds of more complex algorithms problems I would not expect most architects to be able to solve them, since they're utterly irrelevant for their jobs. Heck, a huge amount of software developers can't solve them and bitterly (and at least partly legitimately) complain about them to whoever will listen.

> The transition from "programmer" to "architect"

Transition? If only it was at least that.

I've worked with architects whose job was the result of taking courses in software architectry at college and then they took a job as one. Apparently there are organizations looking for these people.

Now I can't get the picture of Amazon employees in dark robes repeatedly reciting the tenets out of my head...

That being said, where's the proof that having PEs is better than having architects? If we look at results, I don't see any evidence that Amazon's creating better software than Microsoft on any quality axis.

In 2020, the Amazon Principal Engineer community added a ninth tenet. Much effort went into crafting the new tenet, as was the case with the others. The new tenet may be my favorite of them all.

LEAD WITH EMPATHY

Principal Engineers shape an inclusive engineering culture where others are heard, feel respected, and are empowered. We are conscious of how our words and demeanor impact others, especially those with less influence; we take responsibility for that impact, intentional or otherwise. Our work builds productive relationships across teams and disciplines, and across a wide range of life experiences.

This sound like a symptom of having too many smart people with too many diverging architectural ideas at the same place. How do the mgmt get them to march in the same direction? Easy, find the top 1% and create a mythological narrative around them. Then when a smart engineer in a team have a solid architectural idea he/she can defer to the Unicorn Principal that will "advocate" another "approach" that "balances tradeoffs" with "competing interests".

Sort of like the much hated ivory tower software architect.

Just one question - 'elastic beanstalk' - who the fuck came up with that name?

That Amazon characterization of a Principal Engineer is great, and approximately how I've viewed the ideal role of the title.

Sometimes the title means being very highly skilled and experienced IC, or having deep historical knowledge of a particular system or organization, or being a niche technical expert. I like what Amazon's description adds to it expressly (which many have sometimes done unofficially), such as leading by example, engaging, and seeing humility as a quality of the role.

I share your disdain for "Software Architects." I'm curious on your reasoning though. Can you expand on why you hold disdain for "Software Architects"?

For me, regarding the ones who don't actively code, it's something along the lines of "everything compiles on a whiteboard."

It really comes down to the tenets I posted above from Amazon. At MS there was a propensity for folks in these roles to just pontificate. We called them "Architecture Astronauts". As an example, they were rarely practitioners (they rarely wrote code).

Reply on news.ycombinator.com