Staff Engineer Archetypes (2020)
lethain.com
lethain.com
Great company.
I think my biggest failing in my career is in being able to convince people that just because they don't understand something doesn't mean it's harder to code. Many O(n^2) solutions take just as long to code as O(n), for example...
What you describe sounds more to me like the ""10X engineer"" (with extra scare quotes for good measure).
Many people have been skeptical of the idea of the 10X engineer. One take is that if your normal engineers are less than 1/10th as productive as another engineer and all of them are merely human beings, there might be some common organizational problems that your normal engineers are all being stymied by. Another take is that your 10X engineers can only achieve their high level of productivity through taking on more technical debt than others understand or would tolerate, leaving messes behind for the eventual maintainers to fix.
Get through college
Do what you have to in other to get the job.
Do what you have to in order to keep the job.
10x people do some very basic things. They care. They read about code. They have side projects. They improve their skills because doing so is fun.
In a company that hires "the best" and pays very well you shouldn't see a 1x 10x divide, assuming they hire well.
In those that don't, those that only want work done and don't particularly care? That's where the dichotomy presents itself.
There is nothing inherently wrong with either group. Work gets paid, you don't have to be a magical 10x engineer to be worth the salary. And there's no shame in coding as a job instead of as a hobby.
Sometimes, sure.
But often (especially later in a career) overperforming people operate a bit more like senior partners. They have jr staff / understudies to handle crank turning. They can give direction and set things on a solid path without necessarily putting in a ton of hours.
In a healthy org, this is the "architect" role. Or sometimes the "solver," or some combination thereof. I'm sure you're familiar with the adage about the guy who charges $10k to mark the problem, $1 for the time and $9,999 for knowing where to make the mark.
In 20+ years I've only worked with maybe 2 genuine examples of the good 10x engineer, and far more examples of the bad "10x engineer" who flies around quickly reimplementing patterns that they implemented at other companies, whether they're suitable or not. Once the low-hanging fruit is all gone and only the hard stuff (and hubris) remains, they leave. Usually, after getting pissy because other engineers have inherited their crap have started pushing back and questioning their design choices. I've inherited systems built by people like this and it's not pleasant.
You also better be willing to hire constantly if you want a few more 10xers in your back pocket. If hiring a regular "good engineer" is hard, then hiring one of these is harder. You need to find them, convince them to interview, present them with an interesting challenge, have a great interview process that separates wheat from the chaff. And finally, pay them a ton and don't let them burn themselves out
Oh, and admit that replacing a 10x engineer is going to be basically impossible, not just because it's hard to hire them, but because each one that leaves your takes far more experience and knowledge with them when they go (than the 1x engineer). If the 10x moniker was really accurate, it would be like losing a team of 10.
Sometimes it's better to have a reliable team of 6 engineers in at least 2 timezones who get along well, sync up once a week and have a really good lead/manager. This is sustainable, easier to hire, onboard, retain. And less detrimental if you lose one.
How is it more boring than any other kind of project you'd do in a large finance institution?
That is only possible if they can spend some time building abstract stuff that cannot be explained to non-technical stakeholders, which is not possible with sweatshop engineering methodologies like Scrum, or when you promote the smiling bozos that completed their 5 person-minutes Agile crash course to management.
Learning better ways is a process - so long as you remain robust in the face of failure and grow from it.
This is a little catch-22 in getting started though. A great team held back by management hesitation can demotivate and cause attrition. Management buy-in for ineffective teams will run faster by bypassing many org hurdles fall over by their own inability to execute solutions that pass the test of time. It seems like these rare "10x" teams are hard to form in companies that can't consistently attract and retain very talented ICs
I will say. In terms of man hours, often those teams take a multiple of effort from people around the company to support.
They just, consume a lot of time and attention. From various business analysts, vice presidents reviewing their output, etc etc.
There aren’t just 10 or 100x engineers , there are even infinityX engineers when you work on actual hard problems because some of those can only be solved by handful of people.
Ultimately it can probably be solved by other people, but it’d take multiple years.
I think the 10x is not so much because those engineers are particularly fast, it’s just that anyone else working in the domain is 10x slower.
Like if I tried to build a compiler, I’d quickly become known as a 0.1x engineer.
The thesis is that the "10x engineers" don't output a linear multiple of the same kind of work their peers do. Instead, they implement solutions that their peers wouldn't or couldn't over any length of time.
This matches my experience.
The 10x engineer tries to standardize problems and solve them en-masse.
The 1x engineer sees every problem as a different problem that requires a different solution.
The 10x engineer talks about abstractions, the 1x engineer talks about concrete implementations.
The 10x engineer believes in investing in productivity. The 1x engineer wants a visible solution right now.
The 1x engineer never has time because they're too busy making themselves less productive by owning more and more code to maintain. The 10x engineer is always looking for code to remove and refactor, and making the solution as small as possible.
The 10x engineer will talk about programming language features and libraries that promote reusability, the 1x engineer only talks about tangible stuff.
The 1x engineer talks about their car and weekend, the 10x engineer talks about technology.
Call me a proud member of the 1x engineer club. When I close my laptop at the end of the day or even when I’m having lunch with my coworkers, few people would know that I’m an “engineer” at all. The list of things that I care about more than technology in my non work life is a mile long. Technology is a means to an end - to support my addiction to food and shelter.
To use an analogy a physician's assistant[1] doing brain surgery could be a big problem. if you need the latter. Also, you're probably overpaying if you hire the latter to administer Tylenol ...
[1]: exclude the special class of surgical PAs xD
For awhile, I was working with the service teams involved with the various AI services trying to publish an open source demo around their services. When we left for lunch we were talking about travel, cars, our families, places to eat, etc.
There is so much more to life than technology - this is coming from someone who did hobby programming in assembly language on four different processors by the time I graduated - in 1996.
There is an infinite pool work you can do. Smart, real 10x’ers understand what is a priority and are smart enough to pack their shit at the end of the day, go home and do nothing at all with their work.
I think it is more about the ability to grasp vast amounts of complexity in relatively short amount of time and be able to sustain it. Business and market wise this is what is going to make a difference. Some engineers aren't even able to tackle such complexity because of so much layers of it which is why at some moment it just becomes ... too much. Others that can do it, it would take them considerably larger amount of time to do it.
Now, as a leader type, I would prefer to hire 1x engineers to 10x engineers, for a whole litany of reasons. 1x engineers are largely predictable, where as 10x can be 10x in any wild direction, because they're more or less deciding for themselves a bunch of shit they really shouldn't be.
10x engineers like to decide for themselves things like how to go to market or which features to build in what order, without doing any of the requisite research or information gathering necessary. It's infuriating because 10x engineers think passion or some arbitrary definition of "Correctness" gives them power, when in reality they're just operating with a heavily limited set of information (gee, I wonder how they had all that time to build stuff, maybe because they weren't attending meetings???).
Sorry, this turned into a not-very-coherent rant, but the point is 10x engineers aren't worth hiring as anything other than literally employee #1, and even then it's a huge dice roll.
I think 10X engineer is supposed to imply the average engineer - maybe at the person's level?
We all know there are plenty of engineers who do negative work, and plenty who do almost nothing.
So depending on how you think of it - it might not be that impressive.
Nah, of course 10x engineers exist. A 10X engineer saw that hundreds of engineers were writing MPI code to process data and struggled with error handling and therefore came up with a map-reduce framework and its underlying infra. The same infra also helped bootstrap a whole industry. In this case, we are talking about 10^6X engineer. Another 10X engineer got fed up with MapReduce's abstraction and decided to borrow more concepts from functional programming and also invented a little data structure called RDD. This is a 10^5X engineer. A 10X engineer saw that his org build ad-hoc solutions to provision hardware and therefore decided to build an abstraction layer called EC2. Oops! That's another 10^6X engineer. A 10X engineer was not happy that writing GPGPU code was slow and error prone so he decided to worked out a library called CUDA. That's another 10^4X engineer, no? A 10X engineer was fed up with people implementing all kinds of broken consensus protocol and coded out a little practical implementation of Paxos called Chubby. All of a sudden the world of engineers woke up to the idea that consensus can be centrally managed and can be robust. That's a 10^4X engineer, right? A 10X engineer hates that his org write ad-hoc and crappy code on Zookeeper, so he packaged his wisdom into a little library called Curator with two clean concepts: framework and Recipes, and thousands of engineers' life just got immensely better. That's at least a 10^3X engineer, right? A 10X engineer was not sure why it takes an artist years to write quality compilers, be it frontend or backend, so he decided to come up with this little framework called llvm. That's a 10^5X engineer, right? Oh man, I think I can write a book about the examples...
This is not an exhaustive list and you have just picked the points that support your weak argument.
Why is 10x anything such a skeptical concept? We see it everywhere.
We see it in quality, impact, and effort.
Linus wrote git in days, and it was way better SVN.
Messi is probably 100x better in terms of stats and skills and 100000x better in terms of earning when comparing to an average soccer player
Apple's iPod is 10x better than Zune and makes 100000x more money.
A great entrepreneur is probably 10000x better (more impact more money) than a good one.
It is not strange that one engineer would be 10x of the other engineer in certain aspects.
Linus designed and coded some data structures, and some basic utilities to manipulate and store them, in just a few days - based on a fundamentally better model and as a replacement for an existing product they could no longer use. Designing and developing git into a usable product and system took a whole lot longer and involved thousands of people.
The examples you cite are all similarly flawed gross simplifications or mischaracterisations.
I urge you to reconsider your reasoning.
I urge you to reconsider not being pedantic next time
And what Linus initially wrote was definitely a usable product. The product has evolved more since then, but that doesn't mean the initial version doesn't work. Actually most of the original code still exists.
Back to the original point, this is the example of a >10x engineer (by impact, coding speed, innovation) who conceived, wrote a popular software in days, grew the community, and grew it to the insane popularity today. Only a handful of engineers in the world could ever achieve this kind of things.
The differenciator rarely was the pure skill at the profession, but an ability to mentally plan ahead and maybe even see the final "thing" in their head. So instead of going try and error or just using what worked last time those people tended to be able to see the clear outlines of the whole thing in their head before they do it, which has the benefit of not having to do things you know will not satisfy. Couple that with a willingness to not waste your time and then you get someone who looks magically efficient to people who "just do their job".
In my eyes those people are not inhumanly good, it is just that most other people are quite average both in their skill and in their motivations.
And even if you are average and just do your job you can easily compare against those people by investing persistence and time.
You can get lucky and work at all great companies in your career. Most of us will not have enough jobs in our lives to get a representative sample.
There is also a question of having different standards. Bad companies might just be bad for someone who has access to the top companies, but otherwise great for average workers.
my whole career as been filled with stuff like people forcing their pet projects on the company, people hiring their friends and dealing with their in crowd
Yeah, honestly. The vast majority of my hard decisions I made were based on schedule and resources: having to kill things or move people around. The only "political" anomaly I witnessed was that it seemed the employees at the Israel Design Center at Intel were getting promoted to principal at a much faster rate, and with much less years and experience, for what didn't seem like actual merit. But that didn't impact me other than some jealousy, because the line for principal promotion in my division(s) were very long with really really smart people. And I've seen many principals get shown the door for poor performance.
One possibility is that I never reached a level where politics mattered, or teams were small enough that it didn't have time to foment. And there certainly people who lived to become managers because they thought it was a status symbol, but I was happy for them to do the job.
Another could be that I'm just... well, daft. Both are possible. :)
If you worked in a software group at Intel after 2015 and didn't play politics, I straight up don't believe you.
EDIT: Intel certainly had a lot of dumb projects over the years, from POTS videoconferencing boxes in the 90's to its attempt at an iMac-like modular cloud server box. But that was largely paranoia pushing them to explore every possible corner.
But again, your experience is not my experience, and I'm not the one telling you that you are lying.
Take some of my practices for example: I lead and design many projects, and if I know a particular element of my proposal will be controversial for a particular stakeholder, I will schedule a review with them privately before to make sure the solution I'm proposing actually works for them, and also make any needed changes before getting in front of a larger audience. This ultimately just wastes less time for everyone and makes product better.
But for someone cynically inclined I'd be easily accused of some kind of backdoor politicking.
A lot of "politics" is just communication and soft skills.
In my experience, rather than leaving for another company though, they're often a C-suite favorite and get to start some new project (often it's whatever they want to do as long as it's remotely justifiable). They get it up to 80% complete so they can hand it off to another team to do the last 20%.
Since the project is whatever they want to do, they use very unusual languages or design choices that the new team has to pick up and fix. A lot of times it's whatever the big new thing is. (For example, expect statements like this: "Everything is obsolete legacy code now that Electron exists!" "No need to keep this standalone offline test database, setup shell scripts, or maintain the environment setup guide because Docker will automatically solve all of these cases!")
They continue getting new projects because these 1 or 2 people "did the project 80% complete in 1 month, and the last 20% is taking the handoff team of five at least 3 months to do."
Bonus points when it is expanded from some sort of hackathon, or some fake story about how some person completed the 80% portion over the weekend. In the former, all the connections and APIs are hard-coded and faked for their demo. In the latter, they had actually been working on this for the past couple months instead of doing their assigned work which made their official team look even less productive.
I was on the teams always picking up the last 20%, but I got moved onto the 80% team once. I was amazed at how much code I could blow through with time to spare when I didn't have to worry about all the edge cases and minute details.
Unfortunately, I got temporarily moved onto a company-wide prod support team which then got cut due to covid. Hooray.
> Microsoft noted that by fixing the top 20% of the most-reported bugs, 80% of the related errors and crashes in a given system would be eliminated. Lowell Arthur expressed that "20% of the code has 80% of the errors. Find them, fix them!"
More relevant part:
> It was also discovered that, in general, 80% of a piece of software can be written in 20% of the total allocated time. Conversely, the hardest 20% of the code takes 80% of the time.
https://en.wikipedia.org/wiki/Pareto_principle#In_computing
It sounds like you were often stuck with the toughest 20% which actually took 80% of the effort. I also heard that of that 20%, there's another level of 80/20, where a further 20% takes 80% of the rest of the effort, and so on.
Anyway, I hope you find a better situation!
> The Architect is responsible for the direction, quality and approach within a critical area, both today and stretching into the multi-year future horizon. They combine a deep knowledge of technical constraints, user needs, and organization level leadership.
The reason I labeled it politician is because they are primarily using their position for self-gain. A general definition for politics is any time the inequality company > team > self is violated (at least in terms of compensated work). I generally agree with that definition. That is why I used the word politician.
I was asked to review the proposal by a VP, came up with some simplifications, presented it back to the same architects and we delivered it for 10% of the anticipated cost and the feature was profitable in year 1. It’s had no defects because it was a simple design.
Sometimes you need people who can speak to managers and engineers with credibility to get things done.
Sadly, especially in large “safe” enterprises people don’t want to learn this. Developers blame managers for “not understanding software” and managers blame developers for “lack of business understanding”.
That doesn’t mean that the people who are hiring people into that role are looking to have someone do it badly. Or that anyone who wants such a role is automatically suspected of having nefarious motives.
These role archetypes are descriptions of realistic, reasonable expectations for high level IC roles. It’s a useful, positive contribution that will help engineering directors and senior coders better negotiate clearly defined roles and responsibilities.
It’s just not constructive to say ‘but yeah, sometimes when you employ someone at this level and you don’t clarify the role or hold them accountable, they do terrible things and it enables charlatans to prosper!’
Sure, of course it does. But this article is a useful and constructive model for helping prevent that from happening.
What you frame as an addition to the article actually represents a cynical undermining of its entire value. I think that’s a sad way to engage.
Amazon codebase was also famously barely held together by duct tape for the first few years
I would point to twitter as an example of the point I was trying to make, so I don't think it's a very good counter example:
11/07/2013 $44.9 117,633,000 $45.1 $50.09 $44
10/06/2022 $49.39 68,511,080 $50.98 $51.55 $49.29 All indications are that Twitters architecture is now top notch. There are entire sections of “Designing Data Intensive Applications” (the Bible for system design interviews) about Twitters current architecture.
I think as soon as the innovators start to jump ship, growth momentum is lost and it's nearly impossible to regain that momentum. Who does interviews? Who sells candidates? What is the marketing? What kind of sense of morale will potential hires perceive?
If a large refactoring project did take place, refactoring projects are generally very high risk and low reward. That is the type of project that most senior devs will avoid like the plague. It is not fun engineering. It is drudgery and toil.
So a history of poor software results in required cleanup/developer pain, which results in high value devs leaving for greener pastures, which results in a perception of high inertia non-growth, which then makes it harder to sell people on the prospect of "high growth" stocks which results in the tanking of average dev quality (and morale for that matter), which then makes it even harder to hire innovative devs or leadership, which ultimately results in stagnation.
Senior leadership, both managerial and technical, will play hot potato until the costs of not addressing the real, highly unpleasant, problems is too high and by then its too late. Things continue to get worse between the start of the effort to fix problems and the actual fix of the problems and a lot of people will choose not to suffer that.
I didn't work at twitter, but I watched this process happen. To me, there is a clear relationship between systemically unaddressed and festering technical debt and ability to innovate with direct personnel consequences.
> I bet no one has said that if only Twitters software was better it would gain more customers and advertisers.
But whats the cost of setting up a new service? Of creating a new interface? Of testing a new feature? Of creating a new payment system, etc. All of these are software problems. How efficient would an organization be if every team had their own repo? If every team had their own monitoring systems? If every team implemented their own release process? If every team re-implemented the same behavior over and over again because higher up leadership couldn't allocate resources to problems that every team has, what effect does that have? That inefficiency is innovation opportunity cost.
Imagine you acquire a company you think can provide an innovation to your own. What is the time/resource cost of integration and what effect do you think dev environment would have on success of the acquisition?
I think technical debt clearly has compounding interest. I think companies get into a position where the debt becomes debilitating particularly through pathological short term thinking. At some point the compounding technical debt exceeds growth and it's over.
I seriously doubt Twitter is one large monolithic app
And let’s not pretend that Google for instance is any better at product development. It’s failed at everything it’s done not related to advertising - ie Stadia. It’s one success hides a lot of failures.
> Originally, Twitter ran their public APIs through a single Ruby-on-Rails application (“Monorail”), which had grown into one of the largest Rails codebases in the world. And so it was increasingly difficult to update. By 2014, Twitter went the route of microservices, migrating the API service to a set of 14 microservices, running on an internal Java Virtual Machine (JVM)-based framework (“Maccaws”).
> “While the microservices approach enabled increased development speeds at first it also resulted in a scattered and disjointed Twitter API,” Cosenza said. Independent teams designed and built endpoints for their specific use cases with little coordination. This led to fragmentation and, inevitably, a slowing of developer productivity.
Microservices don't make a lot of sense to me unless you can print money or have a public infra offering. It points to an engineering org that doesn't know what they are doing. Turning O(N) code complexity (virtual complexity) into O(N) operational complexity ("physical" complexity) points to an org that doesn't understand where their complexity is coming from or how to minimize it.
> creating a new feature shouldn’t require back end refactoring.
The argument I was making is that it doesn't matter if it gets better. Once momentum is stopped by poor architecture, inertia is too high to get moving again.
> I know how at least one of the newer services work at AWS and how many existing back end existing services it depends on and that if anyone outside of AWS had the resources, they could build it themselves by calling publicly available APIs.
Yes they product-ized their services which is absolutely brilliant. I think the fact that amazon can print money plays a very key roll in why they won while other companies lost. They scaled early in the game rather than scaling too late and reaped incredible rewards for doing so.
> And let’s not pretend that Google for instance is any better at product development.
Google is famous for developers "just shuffling protobufs between services." While it's generally said derogatorily, that means that developers are generally not in the weeds and minutiae of what they are trying to accomplish, but are actually working on the higher level concept they are trying to achieve.
> It’s failed at everything it’s done not related to advertising
gsuite, maps, search, adsense, analystics, gcp, youtube, play, android, pixel, chromebook, news, docs, project zero, etc.
I don't agree that failure is the rule. Cutting lackluster offerings rather than pouring investment in them shows at the very least a sane short term strategy with the weakness of potential harm for long term consumer trust. That's a reasonable trade off to make with good arguments for both sides.
A little company I’m sure you have heard of may disagree. This was way before AWS was thing and was an edict for Amazon Retail
https://nordicapis.com/the-bezos-api-mandate-amazons-manifes...
Your information about Twitters current architecture is way out of date.
https://thenewstack.io/how-airbnb-and-twitter-cut-back-on-mi...
https://www.quora.com/What-were-the-technical-limits-that-Tw...
> Yes they product-ized their services which is absolutely brilliant. I think the fact that amazon can print money plays a very key roll in why they won while other companies lost. They scaled early in the game rather than scaling too late and reaped incredible rewards for doing so.
That’s not the point. The point is that new internal initiative don’t start from a blank slate at any large tech company. There is already infrastructure in place to build on top of even when that underlying infrastructure isn’t publicly available.
> gsuite, maps, search, adsense, analystics, gcp, youtube, play, android, pixel, chromebook, news, docs, project zero, etc.
- gcp - losing money
- addsense - more ads
- GSuite - a distant second to MS
https://www.saasgenius.com/blog/why-office-365-is-overtaking...
- Android: it came out in the Oracle trial that Google only made $27 Billion in total during the first six years. Google gives Apple 18 billion a year to be the default search engine on iOS. Apple makes more money from Google in mobile than Google makes from Android
- Pixel: depending on who you believe, Google sells 1.2 million phones a quarter. If it were a separate company, it would be an abysmal failure
https://www.engadget.com/google-pixel-q4-2021-best-sales-qua...
That’s about the same number Apple sells in 2 days in a down quarter.
YouTube: while YouTube contributes 10% to Google’s revenue, most think it’s still not profitable once you take into account bandwidth, storage, and revenue share with creators.
Project Zero is not a profitable product - nor was it meant to be.
Yes, you are telling me the point I was making. Product-izing your microservices is a justification for microservices.
> 5. All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.
That is the property that makes microservices make sense. Without that property, technical leadership is setting themselves up for failure.
> Your information about Twitters current architecture is way out of date.
The only point I was making about twitter architecture, of which I know far too little about, was that as you stated previously "Every indication is that the code base was horrible" leads to loss of momentum and inability to innovate, so it's not surprising their growth stagnated for, what I believe, are primarily technical reasons. It doesn't matter if they got better if they went through a period which stopped momentum.
> That’s not the point. The point is that new internal initiative starts from a blank slate at any large tech company. There is already infrastructure in place to build on top of even when that underlying infrastructure isn’t publicly available.
The point I'm attempting to make is that if underlying infrastructure is poor, that's not a fertile ground to grow new product so innovation becomes stifled because resistance to new functionality is too high. If maintenance costs are too high resources must be spent maintaining rather than innovating. I've seen developers stumble around for a week because they put `30` instead of `10` in a config file. I've watched an entire team waste months of effort because production was not advanced enough for what they wanted to do. I've watched an entire engineering organization lose hours every day to a poor dev environment. Time was spent fighting infrastructure, not creating features, so it's not surprising when new products and features weren't created and growth didn't meet expectations.
> google stuff...
We have different definitions for failure. You said google wasn't able to develop products. Google is clearly able to develop new products/services, maybe not monetarily successful ones, but google as as a company has a disproportionate effect on my life and how my life works past showing me some ads.
This was not about Amazon selling its micro services. This was only about AWS Retail developers internally.
> Every indication is that the code base was horrible" leads to loss of momentum and inability to innovate, so it's not surprising their growth stagnated for, what I believe, are primarily technical reasons
This was in the early days. They moved away from the Ruby monolith years ago.
> We have different definitions for failure. You said google wasn't able to develop products. Google is clearly able to develop new products/services, maybe not monetarily successful ones
For a for profit company, the only definition of success is does it make a profit.
Edit: I change my mind slightly. We are talking across each other. From a technical standpoint and being able to get products to market, Google is great. From a business development standpoint/program management standpoint, Google sucks. Google’s saving grace is that it has one great revenue stream.
Maybe if Twitter had better product development capabilities - not computer development, business development, they may have found something that sticks.
I also just don’t think Twitter is as compelling of a product for most people as a Facebook and an Instagram
A lot of these prolific devs will not see the code base complexity for the "tragedy of the commons" it is, and bemoan the people and teams who start to try adding "regulations" (meetings/code review) to protect the common because it slows down their ability to meet their management given goals.
There is also a magic phrase that frequently disarms attempts to prevent bad code from entering a code base "this is just temporary." It's easy to put off fixing the "temporary" code forever as other things become higher priority.
Often times someone will inherit a staff engineer's code and therefore won't have been able to stop it. Sometimes staff engineers are able to create an entire service with minimal review that is then "gifted" to a team to refine and maintain.
In a lot of these situations, the feedback mechanisms (yearly peer review) will limit who can give feedback about who, not to mention they are not anonymous and staff engineers are usually highly connected.
Prior experience: Solution architect at Jurassic Park. Present: Dinosaur food
It took a while for people to figure out their BS and when it's finally unearthed, two years has passed and they are ready to move on to another team or company and repeat the cycle.
Sometimes they even get a promo before their rube goldberg machinery is finished and someone else had to take the fall.
Specific to your example, rather than the sentiment, I think embedding config in code is highly valuable when you don't have a lengthy deployment cycle and have direct access to the source. It gives your compiler more information which can help prevent bad configurations (which are the cause of failures more often than anything else from what I've seen). Developers also have a better shot at feeling how badly configurable a particular component is when the configuration is code. It's much easier to hide the overly-configurable systems under a rug when the configuration is far away from implementation, in a DSL, in a different repository, or only visible at deployment.
Instead we have property / yaml files, read into anaemic “classes” which do not perform validation themselves. Validation duties are also foisted onto other parts of the system.
Stacked onto that we sometimes have file name based convention for defining a hierarchy of configs, to save on repetition between environments.
I would make sure when implementing it you have someone who is very opinionated review all configs to ensure that the same behavior is always implemented in the same way until a linter can enforce idioms. I would avoid explicit conditionals and explicit loops, instead relying on implied conditionals and implied loops. I would absolutely not use any "live" data, meaning the code should be relatively pure without any dependencies.
Pystachio is not a bad choice in python land:
Config will always move towards code as systems become more complex and scale up. At some point code-config will have an output intermediary data structure that is consumed by what consumes configs. I don't really have a problem with that.
I am talking about things like querying a database and inserting what would otherwise be a row in the database in the code that does the query itself. So if you need to do debugging, you go do a query against your database and the row isn't there. It sounds absurd, but I have seen it, and caused an outage because of it.
Even when config is code (which is almost guaranteed to happen as a config requiring system is developed), at some point in the process there is a representation that could be meaningfully stored in a columnar database before it's consumption.
Code is generally considered stateless, asking for an input and providing an output that is determined only by the code and the input. Embedding state inside of code is generally a pretty grievous sin, although some places are vastly more sinful than others.
The hubris of the engineer is in believing they’ve thought of everything anyone could ever want to do with their tool. Imagine if a shell only let you set your PATH via a static config file filled with strings. It’s probably fine 99% of the time but at some point someone somewhere is going to start programmatically building paths.yaml. Yuck.
Then by the time that debt must be paid, they're off to another team|org|project.
I've seen it way too many times.
You can change between these every review cycle, job, career, all you want - no one in management is keeping track of your "archetype". They are keeping track of your impact.
Lastly, you are the "person in the box" once you are senior enough. Take the archetypes, look at your org, and see which they need. Whatever is missing will be the most value to the org. How you see yourself accomplishing that box is the real challenge of executives, not this endless "what color parachute are you" self evaluation.
The intent of this article (as I understand it) was to help people navigate the emerging nebulous role that "staff engineer" was, particularly as it started to become a more mainstream concept a few years ago (whereas prior to it, the dichotomy was largely that senior was "the" terminal IC role and you had to transition to management past that).
Many engineers on the upper side of the senior spectrum have clearly been operating above and beyond what is typically considered senior level, but not in the capacity of a typical manager archetype, so this was meant to help people understand how to identify/categorize these people as well as help people understand what kinds of challenges to look at after you've become a "full-fledged" senior engineer. Resources are useful, whodathunkit.
My advice for those eyeing for a staff+ level would be to avoid making "hot take" categories, everyone can do that and if they're coming from a position of inexperience w/ the subject matter, there's a good chance it's a bad take. Instead, try to derive action items from each of the archetypes, and incorporate those into your work. A good point I heard is that no staff engineer qualifies as a single archetype exclusively; the role is usually a mix of all archetypes, with different people leaning more strongly one way vs another.
It's very easy to make sour comments like "oh staff people just spend all their time in meetings, what good are they for". It's quite another to be that staff engineer and have to balance technical depth acquisition, alignment among disparate groups and the multiplexing required to operate at high levels at both. It's easy to chalk it up to politics, but it's quite another to be able to visualize power structures as tools, and use them as such to accomplish large scale engineering goals. There are classes of engineering challenges that only really arise at the staff+ engineer level, and engaging with them can be rewarding in their own right.
They are the glue that gets buy-in and agreement between people talking past each other. They demonstrate how a healthy organization can lead a team by setting guidelines for collaboration, communicating, being constructive, and removing barriers or silos between key stakeholders. They can resolve disagreements and unblock political stalemates with their unilateral authority and uncanny conflict resolution skills.
But I'd call this person The Wolf (named after the Harvey Keitel's character in Pulp Fiction.
Does this person get to ride off into the sunset in a fancy sports car, with the babe from the junkyard?
Years ago, one of them had a Macbook Pro while the rest of us were dealing with garbage Dell laptops that barely worked due to aggressive security scanners. He got his turned off by IT.
That's a management function.
If I took a Staff Engineer role and my job was to compensate for organizational dysfunction and fill voids left by underperforming managers, I'd be leaving that role very quickly. Either that or requesting a formal title change (or promotion) into an actual management position.
Hiring a Staff Engineer when you really need better managers isn't a good solution, especially because a Staff Engineer wouldn't have the necessary org structure authority to be an effective manager-of-managers.
Eh. You might not fit it, but I really like that role. I don't want to be out of technical conversations and I don't love reports, but I do really like facilitating a path forward. At the moment I'm working outside of an engineering function, but I do it when wearing that hat, too.
> a Staff Engineer wouldn't have the necessary org structure authority to be an effective manager-of-managers
If that were the job, then yes. It is a soft-power role, though, not a hard-power structural role.
Limited authority absolutely can be a problem, but usually due to intransigence, at which point the hard-power elements of the org (who are typically limited in their available attention span) can be escalated to.
iirc this kind of role, and other glue are actually really great Scrum Master && Project Manager skills.
I'm not suggesting that the role isn't helpful or that some people wouldn't like it.
This is the problem:
> At the moment I'm working outside of an engineering function,
If the company needs a firefighting cross-team manager, get a firefighting cross-team manager and empower them accordingly.
But hiring Staff Engineers to combat management dysfunction is a misunderstanding of the Staff Engineer role.
So say, in such an organization, two teams each with a Staff Engineer are taking different and conflicting approaches, and neither Staff has been able to pursuade the other to their side? Often in this scenario it's extremely valuable to have a technically capable third-party mediator to help bring the two together.
Sounds like the bright line is the problem.
> That's a management function.
Not at a technical level it's not. If there are honest disagreements about technical direction, you need a strong technical voice and vision to break the logjam. That's not a management problem, although it certainly requires excellent soft skills (requisite for any good Staff+).
It takes both the soft skills and the technical knowledge to get consensus, or at the very least, consent.
Those people are an absolute necessity, and I think it's very tough for someone to be effective in that role if they don't have a solid technical understanding, but I also don't think that an engineer has the clout to wrangle people the way a _therapist_ needs to. In a small organization, maybe, but at a decent size, mid-sized company and larger it will need to be someone working on the business side, not development or IT operations
These positions can be very comfortable if the person isn't directly responsible for anything, but they can use their leadership-adjacent position to come in and take credit for successes on anything they touch. I worked at one particularly miserable company where the CEO's "Right Hand" engineer would be dispatched any time a project was running behind schedule. Lo and behold, the work would be delivered slightly after the deadline by the slightly behind schedule team, but the credit for the "turnaround" went to the CEO's "Right Hand".
These positions can also be awful if the company puts unreasonable expectations on the Staff Engineer but doesn't actually give them the authority to execute. If you've been tasked with delivering a large initiative but you haven't been given any employees to do it with, your only option is to influence other teams to work on it with you. When it comes down to it, people are going to work on things for the person who determines their raises and performance reviews, not a floating Staff Engineer who needs people to help them.
This is why "Team Lead" and maybe "Solver" are the only two of these archetypes that are sustainable for a company, IMO. Letting Staff Engineers sort of float around in the company and spread their knowledge sounds good in theory, but in practice you need to carefully assign them into the org structure like everyone else.
My manager handles the overhead of having reports, which is large at my company. But people only work toward my goals if my manager motivates them to, and most of his mental energy goes towards a separate fiefdom of his, not my team. And I have never had any control whatsoever over who is on my team - except in a couple serendipitous situations, where I made much better choices than the ones that were made for me.
Overall I feel forced to work with entrenched incompetence and semi-competence, but at least I am rewarded fairly well for doing well in spite of this.
That's how they make sure the problem stays fixed.
If it's not that urgent, they might guide others in a more asynchronous fashion on how to get to a fix
1. I may know enough to know the problem and the rough fix. I may not know how to fix it in their codebase as quickly as they do.
2. See above. I want them to internalize the fix.
That said, I ain't afraid to dive in and get it done. I remember not understanding some Java build system somewhere. So I fixed the code, tested it quickly, and hand deployed the jar... then I asked how to fix it for real ;)
My boss was amused.
This is largely dependent on the company in my experience. I've been a Staff at one company where I had to actively protest meetings. I would get in trouble for not showing up. I had to point out several times that 1-2 hours a day of non-meetings was not enough to be effective no matter what level a person is at.
At other (typically smaller) companies, I do feel like Staff engineers can really have a multiplying effect. Based on the OP, I personally fell into "The Solver" category and would bounce around helping out teams who were stuck on something particularly troublesome.
> are they really essential to the success of large engineering orgs
I don't about know this one. In a large org, I was at my most ineffective. I do think they are essential to small-to-mid sized engineering orgs though, even if it only means they help out by knowing all the ways *NOT* to do something.
I'm also the one that sets certain best practices in code repositories, and I've learned that identifying a best practice and assigning it to someone else to introduce doesn't work very well. At those times I have to dive into the code, restructure things, and then present, so that the other coders can actually refer to it as an example. This in addition to helping team members know what is in-bounds and out-of-bounds for their story, and helping with code reviews. This is roughly the "tech lead" role. This has some sort of force-multiplier impact just by doing things like guiding the metaphorical ships away from the metaphorical icebergs.
Finally, there's the classic "architecture" role, where we identify tech paths for the org in general. Recognizing when a well-targeted effort can replace an inefficiency with something smoother. Like I had to take it upon myself to set up elasticsearch and kibana correctly, and structuring the code to report the correct dimensions. Or structuring how our graphql queries worked so they didn't pound our graphql server. Or placing an in-memory cache on one layer of servers (this basically saved our launch). That's also stuff where it's hard to calculate "force multiplier".
So I dunno. I'm talking about an eng organization of 50-100 people, so maybe that's not big enough to be valid for your question. But the point is that while the role is necessary and is a "multiplier", it's difficult to calculate because it's not as simple as just being 5x or 10x faster than another developer for a well-defined task.
Nonsense it's as old as the tech industry, and the defence industry it grew out of.
The name comes from the military, where it's even older.
I was curious about this, because I also think of the staff engineer as a newish thing, and I don't believe I've ever actually met one.
I searched my old email for the term and found that the first reference I have was in January 1996, in a HotWired HotFlash newsletter:
"Java - so-named because it evokes liveliness and speed - is the brainchild of Arthur van Hoff, a senior staff engineer at Sun Microsystems. He's the HotJava engineering lead, has designed most of the Java applet API, is the implementor of the current Java compiler, and is the author of the book "Hooked on Java." Get the scoop behind the hype, when Arthur van Hoff joins us in Club Wired on Friday, 12 January, at 12 p.m. PST (20:00 GMT)."
After that there's really nothing until the term starts appearing in LinkedIn notifications around 2010. I guess that just means it wasn't used in the circles I moved in, so only became visible to me when I joined LinkedIn.
I don't think of the tech industry I have been working in as being related to defense much at all - I trace my computing roots back to the home computer era of the '80s. Perhaps there have been parallel streams which are now mixing.
On the other hand I have never cared about titles, so perhaps it's been there all along and I just didn't notice.
Maybe you’re now reaching that level yourself so now interacting with those people?
Like saying ‘how come nobody’s ever discussed prostrate exams until I got to my 40s’.
> I don't think of the tech industry I have been working in as being related to defense much at all
Microprocessors come from defence, as does Silicon Valley.
We should rename all the accountants "staff" though.
As the GP said these titles have existed for a long time. But for whatever reason there's also been a progression in terms of them being super common in social media discussions.
The cynic in me says that it is driven by a certain subset of Internet Famous Devs progressing through their career.
From my experience job descriptions/title that lean into "architect" tend to attract people that can theorize about how software ought to be built all day, but haven't written a single line of code in ages.
It doesn't matter what you call them, they've always been there and always will be. Unless your engineering org is not run by engineers, in which case, you should join a company that has staff engineers :)
As a senior engineer, or just "software engineer", you can be a brilliant architect, but if noone sees that outside your immediate team, and the only way for you to get bigger impact is to become a manager, with no time for actual code, then your impact will be minimal.
Positioning yourself as staff engineer, and selling yourself as "architect oriented staff engineer" to the different managers and leads whose buy-in you need to do cross-team architecture, is a force decoupler for you as a technical person.
I'm a fundamentally technical/code monkey person, I love writing and shipping code. But I'm in a position right now where, in order to get my technical goals accomplished, most of my work is managerial. Once I was able to frame things that way (I could write code all day long, and never would I be able to achieve X, but if I play the game of calling myself principal engineer and leveraging that "marketing brand", then I actually can), all the remnants of apprehension I had kind of fell away.
Overloading the Staff Engineer title with these archetypes is going to lead to identity confusion in the job hunting landscape.
If you're searching for Staff Engineer positions by title, now you need to figure out if your personal archetype matches what they're looking for. You might be able to get it from the posting, but maybe not.
This also confuses salary bands. You might have some really effective management type Staff Engineers making loads of cash because they're management, but someone who's just a Solver might be on the far lower end of pay because they aren't management. These examples might be reversed if the Solver is able to solve really hard technical problems.
https://careerkarma.com/blog/how-to-become-a-staff-engineer/
...which seems to be saying that a "staff" engineer is indistinguishable from what I'd call a senior developer.
As far as I'm concerned, "staff" is people that are permanently and directly employed. It doesn't mean "senior". If it's found it's way into the cycle of job-title inflation, that makes me sad.
They generally denote a management-level seniority, and they can be given staff, department head status, and a discretionary budget.
It is basically the next level past “senior”. Whereas a senior will normally report to a line manager, staff engineers will often report to a middle manager and have managers as their peers.
A normal range for a senior is 5-10 years of experience (but still many people will have more experience than that but not be “senior” depending on company), whereas staff is 10+ years. Obviously these are not wholly sufficient conditions but they’re common buckets, especially for external hiring.
There is some title inflation going on where staff engineers are typically acting more as tech/team leads as opposed to senior engineers. IMO it’s a good title because “principal” has connotations of doing really big-picture stuff that may involve some resting on laurels. Also the bar for “principal” varies widely across companies - at Microsoft it’s the same as “Staff” at other places, at some places it’s like the top <1% of IC engineers, others are probably even more lax than MS.
The last time I was involved with job-title hierarchies (a long time ago), nobody was a programmer, everyone was an analyst. The people who did telephone support were called support analysts. I graduated to senior sales support analyst (roughly, doing random free work for the prospect, to close a sale).
We assumed we were superior to "engineers": they were the guys that swapped-out hard disks and cards, twiddled the fuses to do performance upgrades, and lifted heavy boxes. We were better-paid. I think of them in boiler-suits, but actually they wore business suits like me. I liked the engineers.
[Edit: I guess maybe the difference might have been that we (analysts) had degrees, and they didn't - they got lots of training on the hardware. The attitude "we" had was obviously degree snobbery. I say "we", because the engineers were regarded rather as I would regard a gas-fitter, by just about everyone.]
Interesting nomenclature given that in many countries you can't even call yourself an engineer without an accredited degree and being licensed by the proper regulator.
Personally I think Software Engineer is a better descriptor of what we do than Programmer/Analyst, though of course not everybody with the title is actually engineering systems.
I call myself a programmer, or sometimes a "software developer". But even the latter term is fishy, to me; it has connotations of "research and development", as if there's something blue-sky and innovative about building a commercial website.
Incidentally, my dad counselled me to NOT go into programming as a trade[0]. I ignored his advice, and I did OK.
[0] Yes, I consider myself a tradesman, not a "professional". Professions are trades that are policed by a professional body; there are professional bodies for programmers, but hardly any programmers belong to those bodies, and if you list your membership on your CV, it'll probably reduce your chance of getting an interview.
> The Solver and Right Hand bounce from fire to fire, often having more transactional interactions with the folks they’re working with on any given week.
It's worth noting that Ben Purgason from LinkedIn, identified Firefighters as stage 1 and beyond throughout his article on the five stages of SRE:
> At this early stage, SRE is essentially fighting fires whenever the need arises while simultaneously trying to automate the process of fire suppression. With each piece of reliable automation written, the time saved on fire suppression is freed up for use on permanently fixing problems or for forward-looking priorities such as monitoring and alerting.
https://www.usenix.org/system/files/login/articles/login_win...
My observation has been that problems arise when you have roles that clearly default to firefighting without the larger organizational recognition that this is a signal that the default mode need to evolve or improve continuously. That is, that the stages beyond stage 1 (firefighting) that involve developing the roles, teams, tools, and ops in concert with one another are latent and need to scale with the organization, products, services, and customers.
There's always something that is highest priority, and having a floating person(s) who can continually identify and provide support for whatever that thing is isn't a sign of a dysfunctional organization.
Good point. That's right.
> There's always something that is highest priority, and having a floating person(s) who can continually identify and provide support for whatever that thing is isn't a sign of a dysfunctional organization.
Yes, I do think that "floating" is an appropriate term here. I've also seen "consulting engineer" used in this context within large multinational organizations.
However, I would point out that too much floating may be a sign of diminishing returns on matrix management.
Disregard levels. Disregard titles. Fix what needs to be fixed. Unit tests, CI, docs, there's no such thing as grunt work. Real leadership happens in the IDE and only code matters - everything else is overhead.
Tell that to the person writing my paychecks.
Unless the arguments are about "how to do the code correctly" for some value of correct that's imagined to be beneficial in the future.
a: we can write code to do $x undifferentiated heavy lifting
B: can we throw money at a third party to solve $x
The answer is almost always B:
The question is always will writing this code give us a competitive advantage/help us go to market faster/impress the VCs to give us more money.
Or
“Does it make the beer taste better?”
I’m just as proud of the code that I had sense enough not to write as I am the code that I did write.
Providing technical leadership is weird and makes people hard to rely on?
Your IDE thing probably only applies to a 3 person startup where one or two of them are coding.
I agree with you: fix what needs to be fixed. But that can be human things too, not just code.
And not using Kubernetes…
Specifically the problem with me is I don't have deep expertise in any area. This is fine in most cases but there are areas where expertise is important. And more often than not titles and levels are a reasonably proxy for expertise in a large organization. Turns out asking "who's the expert in X" really, really doesn't work once you disregard titles and levels. Disregarding titles and levels is a good way to turn engineering into a popularity contest so "who's the expert in X" becomes "who do I know best who recently said something like X to me"
lolwut?
Do people actually believe these kinds of things? Like what comprehensive leadership happens solely in an IDE?
> there's no such thing as grunt work
It sounds like this messaging for people who are on a career track that will never not be grunt work. And even if the goal was to lead someone into being the best code grunt possible, how do you lead them to that point through only the code?
Most don't, no. But you'll definitely find a higher percentage of people who do at places like FB.
What do people complain about at FB that they can't seem to get done there that seems really obvious that you should? You'll find similar issues. Maybe not like google, but unique to FB.
More seriously, please listen to what every other prominent staff+ engineer here, in blogs, on Twitter, even within FB, etc. has to say about the job. There are a great many takes, but by far the most common thread is that the most important part of the role is almost never about code. Many complain about not having time/energy to code any more. The whole point of having this as a role/title is to give leverage to those who have proven they can use it for the good of the organization. That means maximizing the ratio of effect to code written, by themselves or anyone else, and there are a great many ways to do that. "Code is all" is the hallmark of the very junior engineer who hasn't finished climbing that first rung yet - a very important rung, yes, but still only one of many. TBH it comes across as something between sour grapes and Tall Poppy Syndrome.
Coding ability only affects the leveling guideline between junior and mid. Everything else is about “scope”, “impact”, “managing ambiguity” and in the case of Google “writing a messenger app”
Of course these are the official guidelines.
The tricky part is making all that code add up to something meaningful, though, right?
A thousand developers diligently and efficiently working away in their IDEs building completely the wrong thing is a terrible waste, and not something that can be solved by just having a staff+ ‘fixer’ come along behind and clean up.
Code is just an implementation detail that solves a business problem. I can't disagree more with this take.
Edit: Just noticed it's only about "staff"-engineers. I wondered why there were only 4. However, i think the "solver" isnt part of the staff either.
Staff archetypes is a meme. Here's an archetype for you: gets whatever needs to get done done. The idea of an archetype is harmful, it limits what people do and can become an excuse to not do hard grindy work.
EDIT: On a second read, I guess you do end up doing one of those things for an extended period of time. But my point still stands: don't box yourself in
staff is sometimes used on road to principal, in some firms there are no principals but just staff, etc.
i think _generally_ the industry is doing swe, sr swe, staff, sr staff, principal
sometimes there is no sr staff, sometimes there is no staff and just principal, sr principal...
All the other archetypes can be independent, as in the company can pay them to do their mission even if the executive is not engaged.
I suspect it is time we stopped seeing companies as lead by an executive as the tip of a spear and more as a crowd with an agreed direction.
Edit: I would also suggest that Staff Engineer is the next step on what I would call Software Literate Companies - it's not that we need execs who are not day to day technical supported by people who are. It's that we need execs who are day to day technical and not have ones who cannot code.
You can assume different archetypes at different times. That's the best thing about being staff. The staff engineer is a clear leadership role with expectation to perform each of these archetypes at varying degrees. The person might prefer to be one of these archetypes but doesn't shy away to pick up others if the situation demands. Of course, they and people in their vicinity would want to play their strengths.
Some people just like to organize and clean things up.
I have been asked if staff engineers are around so that they can pick up projects noone wants to do. This entirely defeats the point of staff engineers.
I would want my staff engineering bunch to either pave the path or mentor the teams to do it themselves. Making it easier for the teams and future engineers could be a better strategy and effective multiplier.
If there is organisational mission critical software lacking any of this then it is a p0 and I would count it under high impact work rather than gardening. Besides depending on company size, some of those responsibilities should belong to platform teams.
0) the wizened emeritus engineers staffed with special 'futures' or 'next-gen' bluesky type projects
1) engineers who may not be overly technical anymore but are really good at negotiating the organizational hierarchy and might know where and how to poke and pull at the cybernetic organism to nudge it in a desired direction
Having these distinctions help clarify exceptions from both consumers of the employees work and aspirers who want to do a certain type of role or not.
watch!
blasts through twenty inches of solid boilerplate