What makes a senior engineer? Writing software vs. building systems
codewithstyle.info
codewithstyle.info
To me, what a non-senior engineer and a senior engineer do are so different that they ought to be different career tracks, but this progression seems depressingly organic and inevitable. The more experience you get, the more people come to you for answers. The more people who come to you for answers, the more involved you are in the big things. The more you involved you are in the big things, the less time you have to code.
It seems crazy to me that you would spend so much time and effort mastering the arcana of coding only to let it go unused later in your career. It's like if a virtuoso musician were told that he can't play music anymore. He needs to move on to conducting, and after that composing. And he can't just stay at his current level and keep doing what he's been training for years to do. If he wants a future with this orchestra, he needs to exhibit career growth. The expanding business has no place for stagnant employees.
Some business somewhere ought to make an effort to let the virtuoso coders keep doing their thing, and break out the engineers who want to go into system design as a separate career track.
One analogy that comes to mind is wuxia/murim novels (i.e. stories set in ancient times east asia, involving martial arts sects). Among the strongest characters, there's the sect leader type that is involved in politics most of the time, but there's also the recluse master training in the mountains type. Naturally, there's also plenty of characters clinging to titles, overconfident fools, etc.
I think this analogy is interesting because there's a clear parallel between the two worlds in terms of how titles are perceived, the craft that they purport to represent, and their place in the world. Titles can represent both status and mastery, but only loosely at best, the craft can be honed irrespective of both titles and the state of the world, and it's only through increasing a sphere of influence that one can have a sizable impact in the world and its citizens.
There needs to be a third track. IC (training in the mountains), EM, and IC (planning and design). A lot of orgs would call that last position a PM, but I have never worked with a PM competent enough to actually carry out that kind of work without ICs stepping up to do most of the PMing themselves.
In my experience, PMs are the corrupt bureaucrats that serve as antagonists in wuxia stories.
Who is going to pay for what? Who's responsible for when a certain part goes wrong?
If you figure out the politics well, you've mostly constrained the system design to something straightforward
— Melvin E. Conway
it's not a complete solution, as 1/I've seen this cause games of musical chairs happen that essentially stall the work and 2/ creating silos make cross-cutting concerns fail to be addressed...
Not ideal either, IMO, ICs involved in planning and design should get out there and talk to customers themselves!
This is an important point, and should be more obvious than it is. Projects succeed or fail more as a function of individual personalities than of whatever their job titles are. Easy to forget given that people who share a job title are usually treated as interchangeable components of the machine (at least in big companies and at least in my experience).
If your manager will not accept way X. It really doesn't matter if X is optimal. You'd better find Y, or convince the manager.
Here's my problem with these roles. Going by the definitions from Will Larsen in your link, I am currently expected to be: a tech lead, an architect, and a solver. I engage primarily with tech lead and solver work. I also frequently have to dig in to "regular" IC work tickets. No single individual has the capacity to do all of these things well but the business will see you succeed in some degree with tasks associated to these different roles, fail to recognize that these roles are distinct from each other, and unrealistically expect you to do it all.
I am working with my management on this problem, specifically about expectation setting (because it's a mess, everyone, and I do mean literally almost everyone, has a different idea about what my role confers), but it's frustrating.
I think this is critical to note because an additional complaint by many senior engineers I know of is that they don't have enough influence when managers make dumb decisions or their opinions about the direction of the product aren't valued more than less-experienced product people. That is to say, I don't think the deliniation between IC and Manager work in practice because the influence of even a lower-level manager will often supercede an experienced IC, much to IC chagrin, except ICs also don't want to be in all the meetings managers are in where such influence is exerted.
That's because coding skill, once you reach a certain baseline, is good enough and other things like architecture matter far more than raw ability.
Unless you are an outlier like John Carmack, improving your coding ability really does not improve your ability to provide business value. The difference between a junior and senior engineer is really the difference in knowledge about what can go wrong, and that knowledge is fundamentally about architecture and edge cases, not coding ability.
> Some business somewhere ought to make an effort to let the virtuoso coders keep doing their thing, and break out the engineers who want to go into system design as a separate career track.
That does happen, mostly at large companies that can spare the resources. I know a number of people who have been around for decades and make a lot of money doing that.
What do you define as part of coding ability?
You're making design decisions anytime you code things. Sometimes its smaller scale but sometimes it's larger. Edge cases and consequences of your code are abound in every line you type.
Trying to divorce these things from coding ability feels really artificial. If they aren't coding ability, what is? Raw typing speed?
Not understanding the architecture means not knowing what you do at all!
If you don't know what you're doing you should be strictly prohibited form doing it anyway. Or else you will very likely cause damage.
Or do you think it's viable that some programmers require constant hand holding with almost every line of code they write because they otherwise don't know what and how to implement anything correctly?
The model "coding monkey" just does not work; as everybody knows…
Junior eng: screenshot of code (with some help chat)
Eng: screenshot of code (with no help)
Senior eng: screenshot of design doc
Staff eng: screenshot of calendar with a wall of meetings for 5 days
There are a lot of organizations where people can be sucked down that career path. If you find this happening to you, switch roles or switch jobs. Because it is an anti-pattern and a dead end.
My dream job is to have none-to-few meetings and to have my head in code all day.
I do have something close to this right now, but I really hope that in 10 years I can still be writing code.
But you will notice that you are just one person. Its impossible to do all of that alone. Then you realize that you need to work as a team.
I know enough about the breadth and depth of computer science to know how insane it is for any one person to think they've run out of things to learn.
We've got only one life, and as much as I like programming and engineering, and fp, maths, logic I really hope life has also other things to show me and enrich me, and I hope I moved my interests elsewhere in a decade.
* manipulating lists with functional programming * manipulating lists with OOP * manipulating lists with a non memory managed language * manipulating lists with a hard type safe language
and will start being interested in * guiding people to manipulate lists as simply as possible to MAKE SOME MONEY
Proof languages, dependent types, other category theory -inspired abstractions... For example I've been really enjoying reading this journal lately: https://www.cambridge.org/core/journals/journal-of-functiona...
Its not about learning. Its about creating. After a while, you will have created a majority of the potential things you can create and everything will start to look the same. If you are a CS researcher and will remain as such, your perspective may be applicable. In the actual industry that apps, services are created and provided to the users, its not. So as long as you stay in research and not go into actually making apps, you'll be fine. That is, if constantly learning and not creating does not start to bother you...
This sounds like the famous quote: “Everything that can be invented has been invented.”
It’s as wrong now as it was a century ago. We’re surrounded by opportunities, things waiting for someone to invent them. If you don’t see them, maybe you’re not looking hard enough. I certainly work with plenty of people who feel like we’ve basically solved it, our system works fine. It doesn’t. It’s shit, they just don’t feel like building anymore.
I’m in my third professional decade, for what it’s worth. Far from leveling off in the long run, an IC’s capability is exponential, if you keep trying.
Thats for science and invention. Not engineering. Less, the current tech space in which we are catering to the public. Even when there are new inventions, it takes decades for one of them to materialize.
You could definitely work in a large research organization, do research with large funding - the research that your funders want you to do, of course. Then you can spend a few decades researching something, and maybe you can invent something new. If that would satisfy you, that's also a career option.
It also definitely includes trying to solve business problems by not writing code; by saying "this is a business problem not a technical one, how about trying <some non-coding approach>"
Not to disagree with you, just giving a contrast.
Anyone's job can be described as "solving a business problem" or "adding value", but that's not usually the best way to describe it unless you're in a room full of MBAs. (And even a bricklayer's job goes beyond "merely laying bricks".)
IMO, it goes (or should go) without saying that my work output is valuable, and that I do what I do for a reason, and that "writing code" does not encapsulate everything there is to know about the job.
To be fair, the problem had to be solved using code anyway (why I was originally brought in), so another team wrote that, but it took several man-years. My interim 'fix' bought the company a lot of time.
This isn't a great example as the code had to be written anyway, but you get my point I hope.
A good surgeon will also very often advise a patient that surgery is not indicated.
A surgeon is a perfect example against your argument -- the surgeon's job is to achieve a positive patient outcome, not to cut someone open. Sometimes achieving the first is best done by cutting someone open, but when it is not then it is the surgeon's job to say that a different approach is needed.
Of course, this attitude may be a reflection of the fact that I have never written software for a domain in which I have expertise. Maybe if I were working on a product that I myself used, I would be able and happy to help solve business problems. As is, that is better left to a SME, and I am happy to solve technical problems.
Engineer (junior through principal) - screenshot of code
Architect (mid-career through principal) - screenshot of system diagram alongside code that implements part of the diagram
Technical Fellow (principal+, considered equivalent to directors, but still IC) - screenshot of system diagram alongside overbooked calendar
Manager (mid-career+) - overbooked calendar, maybe an IDE on a light day
And also product managers and business analysts and others.
I find distributing knowledge just as important as using it. Help others help themselves. It’s not for everyone! But if you enjoy it, you should consider going that way, even if you still enjoy coding. Or maybe not “even if” but “because”. If you stop coding, you’ll quickly get out of touch.
I (also) code in my free time. The most interesting technologies I learn when I’m not coding for work.
Senior teachers are principals, department heads, etc.
And most companies want the brilliant engineers to be the ones making those decisions, hence the pay incentives guiding the brightest towards that direction.
Which reminds me of this game: https://www.saynomo.re/
It makes me much happier, though, which is also important.
Some of us are even polygots.
But really... As you go on in life, what you do, and why you do it matter more... because everybody knows how. Duh. /s (But this is really the way of thinking.)
College grads can do it barely, and 5 years in they’re fine. What do you do with the rest of the people who, 10 years after that, want to do the same things but get paid 50%+ more?
“Virtuoso” coders aren’t worth much (relatively), because it’s not that rare, and honestly bugs aren’t really a big deal. Heck, you can write some pretty terrible code, and as long as it’s not some specific flavors of bad, your company will be totally okay for a very long time.
Single virtuoso coder - or just a coder who just actually puts the effort to keep the mess contained - can be the glue that keeps the project actually usable and who in-turn educates others to be better programmers.
They may not produce +50% more code than their less senior peers but the code that's in total written will be of much higher quality. That's how I see it. It's like having a great musician in a band - everyone will strive to play better to meet their standard.
I've worked a lot on networking stacks, protocols and servers, and it found to be extremely specialized. It's unlikely one can an engineer with 5 years of "generic coding experience" which would be immediately productive in the field. I assume the same goes for implementing operating systems, databases, compilers (like LLVM), graphics engines, etc. Sure, a junior engineer that is new to the field could probably also implement something - but it would likely end up less reliable and less extensible and maintainable than if someone who has worked on a similar system before.
At that level it's however also all a bit of a mix between coding/implementation and design. While there's no real "system design" since the piece of software might just be a single component, there are still the usual design questions involved like "how does this component fit into the environment, like the APIs the operating system offers", "how will it be used", "what operational requirements need to be fulfilled", and "how can it be made to be as efficient as possible". Having a good amount of experience on how components will work together will be very helpful.
My larger point is that they're not worth as much as the coder who can also project manage/lead/mentor/etc, because the latter coder can both be a "virtuoso" (or like 95% of one) and also make the people around them better, and that's substantially more valuable.
The thing folks who "just want to code" don't realize is that their "value above replacement" is never going to be high enough to justify the desired pay increase over time that most folks expect as they advance in their careers.
That said, it's probably more fine if you want to be 10+ years into your career and accept the pay comparable to those more junior coders. Some people do get this by negotiating fewer hours, for example.
And not for nothing, but leading/mentoring is highly achievable for any "virtuoso" coder; not wanting to help others and pitch in on design is a personal choice, as most "virtuoso" coders would probably be great at both if they put in the effort. It just means holding meetings more and coding less, which is outside of their comfort zone and therefore "scary" in some sense.
They could be doing so many other great things, I wish they would please try to get over their insecurities and fears! I will help, as that's my job because I did get over those things and transition into this role from a coding role.
Yet I don't know hour-wise how much they'd have to put in having meetings if they already kinda know how to build the system. I don't think that would necessarily be scary to them, just annoying if they had to be constantly distracted by having to mull over details that aren't that important.
To me a virtuoso is the consultant who knows how to build all parts of the project who is not there to lead the team but will help out anyone in the project - and will take care of it being built properly.
There is no level of skill at which anyone will ever just be able to do an entire project on their own; no matter how good you are, you’re always going to be better with others.
I still like to code, but today I do it mostly in form of some kind of presentations, this might be informal presentation for developers/operations for given use case, this might be presentation describing deployment/infrastructure/architecture, but in general purpose is to guide less experienced engineers to use efficient methods of solving problems instead of just smashing code together.
Based on my experience, communication barriers are biggest issue in big corporations and it's crucial to communicate your knowledge to others in approachable way, even if solution is technically not optimal. It's described so it can be improved later. However sophisticated solution that is not documented, may technically work better, but it's maintenance nightmare.
Has anyone ever liked situation, where you get alert about production incident and you have no idea how to fix it, because you don't understand what happened? This is a people problem, not technical problem, because you cannot understand everything and either someone will explain system to you or you may have no idea how to react. People that understand people problems will ensure that there's always proper documentation and processes to follow, so no-one is stuck. Technical master will create greatest technical solution, but eventually systems fail and that's where the problem begins, if others don't really understand them.
That's why software is a shithole compared with less abstract machines.
As software is eating the world this becomes a greater problem with every year passing. And the even greater problem behind that is that we won't be able to fix the already created mess, ever. The current shithole is almost unfixable as it would require decades of fundamental clean up…
Exactly the above cited attitude brought us into this situation.
It's like all doctors would says: "Well, up to 30% of our patients die while treated, but who cares? You can't get better medical care anyway anywhere because just nobody cares—and it would be actually to expansive to put more caution into this." ¯\_(ツ)_/¯
If that would be the common sense among almost all doctors the patients that make it would be "totally okay" with that likely too; as they would not even know that their chances with most treatments could be much better, if only the doctors would be more willing, and of course capable, in the first place.
The "code is good enough if it works", and "bugs in software are god given" attitude is imho the root cause all evil in software development.
As long as this does not change broadly humanity is doomed to have only bug ridden security nightmares, that barely work, instead of proper "virtual tools". It's even a wonder that actually anything computer related works despite this "who cares" attitude among the majority of coders.
Because while modern medicine has advanced a lot, AFAIK there are still some cases where the quality of medical treatment is exactly how you described. For one, there aren't really any processes in place to ensure doctors actually keep themselves up to date with the latest medical knowledge, so the doctor you see might be say 20 years out of date. And yes, for this example, it would be expensive to fix this, say you enforce a rule that 20% of the doctor's time must be spent on something other than treating patients or revenue generation. So nobody's gonna do this.
At least software doesn't often lead to deaths. At least not directly.
It's about the value you are able to bring, not howmany days you worked.
I'm less clear on if it's value. There's "direct value", like, I literally helped make $X, or so. And then there's "what my employer wants done", which might very well be worthless. For the latter, I could argue that I am valuable to them, from their perspective, as I am doing what they desire … even if ultimately it is not worth something.
That is, I've plugged away at many valueless projects, things that the company wanted done, but ultimately won't bring value. Nonetheless, I can bring my experience to bear on the problem and move a mountain or two.
The times I have brought value have usually been quick and unexpectedly. Like, just by being aware of what's running & catching a $20k/month mistake. Or understanding how something is done, to unstick / enable someone else. But it is often not part of my goals, what I am doing day-to-day, or something I get to claim/matters in performance reviews.
And sometimes, you don't get to know what your value is. I remember at one company we had this really onerous customer: completely custom thing was built for them, it was always lots of back and forth, weird corner cases, just lots and lots of custom stuff. We thought we were mostly bending over backwards for someone for little gain. Turned out they paid quite well, but for the longest time, we had no idea: finance didn't share that information with us. And this is true of most companies I've worked for: the financial details are completely opaque, unshared, and I couldn't tell you if I'm bringing value or not, ultimately.
There's a lot of people with the same 1 year of experience repeated several times, who then call themselves 'Senior' based only on number of years.
When talking with my oldest collogues, those that started out in the 70s, "Senior Engineer" was apparently reserved for, well, very senior engineers. People could have worked 10-15 years, before getting promoted to that.
These days, it's almost something which happens automatically after 5 years. I've even seen people get angry and frustrated if they haven't gotten that promotion after as little as 2 or 3 years.
But to their defense, A LOT of companies just use the various titles to place them in the correct salary bracket. I've seen examples of hiring more junior people than the title would indicate, simply because they were good candidates / seemed to have potential. And then they were stuck without salary promotions for 2-3 years, as their work output didn't necessarily warrant any higher salary.
Lots of weirdness in this industry, compared to the others I've worked in.
ime the only thing you can count on the term senior meaning is that you're in a different pay band then other levels. other then that the term is meaningless.
They do the same low-level code work, but I'll hand it to them that they are some of the best engineers at the company.
The difference is that principals left the company for a couple years to come back for the higher pay. Staffs were promoted while staying at the company.
I just happen to be at the top of our small (less than 10) team of in house developers, so tricky things are escalated to me and I do my best on them.
- expert at programming
- dev ops
- business / system analyst
- team lead or project / people manager
- automated test engineer
- product owner
It's almost feels like they're searching for a unicorn
One of the first steps to title inflation was people started to include internships and their time in college as "years of experience" . So this meant new grads could have 5 years experience in their minds. Part of this is fair because if the industry is going to start a 12 week boot camp person at 0, there should be _something_ for the 4 year bachelor.
Worse, your title - and your quality of life at Amazon - is entirely dependent upon what your manager thinks of you.
It matters way more than at other companies.
In my experience, the litmus test is frequently one that the author of the article can easily pass, which makes them an XXX, by default, and the article is really just about how awesome XXXs are, or the author assumes they have the authority to bestow the mantle of XXX unto others.
The word "senior" means "Has more responsibility/experience." That's about it. It can mean "Manages/directs others," but doesn't have to be that. It can mean "Is an architect," but that doesn't mean they are.
I have watched senior blacksmiths at work (awesome, BTW), and they got their hands dirty, and their aprons spotty, just like their juniors.
I was a "Senior Manager," but was only a first-line manager. The title was really just there, so I could have an office, and the folks in Japan would take me slightly more seriously. Banks can have hundreds of "Vice Presidents," that work in cubicles, and don't have any direct reports. It's really just a title.
I'm definitely a "senior engineer," these days, and I architect stuff, but I don't manage or direct others, and the scope of what I do, is smaller, because, every time I count myself, it comes up "1."
It's really up to each organization to define "senior," and to set up a professional environment that implements their definition.
Litmus tests are fine, I guess. I fail them all deliberately, because I'm an ornery sumbich. I won't join any club that would have me as a member.
It's a huge change from my 25 years in industry at 'smaller' / less prestigious companies where most promotions happened because it's the only way my boss could get me a 5% instead of 4% raise, or whatever. Or in other companies where they gave you a promotion INSTEAD of a raise. In all my previous companies, it happened just because somebody gave it to you, not because you met any real standard.
There's a rubric of requirements. Technical proficiency, leadership, mentorship, cross-functional impact. If you want to be cynical about it, sure -- it can mean you drive projects just for the impact, you lead internal conferences, you hold meetings, you mentor, whatever, all for the self-serving benefits of promotion.
And you definitely get the whiff of that occasionally from certain initiatves people take on.
But I generally find it pretty positive and inspiring to see how it encourages people to grow, and how it drives the tone of the organization. For example I've never worked somewhere where it was so encouraged to take on tasks outside of your comfort zone. To suck at things, and learn, and get better -- versus staying in your own little hole and getting very good at your specific role, but no more.
This. "An expert is a person who has made all the mistakes that can be made in a very narrow field." - Niels Bohr
I agree with your assessment for the ones who were promoted to it.
I gained crazy insights but was unable to published because it was done on company’s dime.
We call that “Trade Study”.
Good times, good times.
(This was before Phoronix’s popularity)
What’s crazy was that ReiserFS came out ahead for the read/write data flow composition modeling that we were using.
* More emotionally mature
* More pragmatic
* More business aware
* More recruitment aware
* Better at whatever process you are running
* More security aware
* More performance aware
But more than all of this, ideally there would be an objective progression ladder to ensure they have actually done all of these things not just assume they have because they worked somewhere for 10 years and was called "Senior"
It is not until you get to a lead/architect level that the responsabilities change. The system building comes into focus.
> To them, creating software is just one of the steps. First of all, they question whether the software needs to be built in the first place. They ask what problems would it solve and why it’s important to solve them. They inquire who will be using the software and on what scale. They think about where the software would run and how they’re going to monitor whether it’s working properly. They also decide how to measure whether the software is actually solving the problems it was supposed to solve.
Based on this definition I would suggest the author is using "senior" as a modifier to include all positions above junior rather than the typical titling. Moreover, the amount of times I've been given any time at all to actually think about the system things will run on, or how we will determine if it's performant, I can count on one hand. If this is the author's experience he is very lucky to have a company that cares about end-to-end performance.
If anything a senior engineer will complete tasks better rather than faster by having a wider and deeper understanding of the system.
Many of us are quite capable of writing libraries, and likely do so in an internal-only fashion, but have no interest in trying to be the nth new library that accomplishes the same thing during our free time.
Skill vs. effort vs. drive are all very different things.
What i'm saying is that, from the perspective of an employer, i'd be rather impressed by someone who can show me one or more libs, that he wrote and people are actually using, than someone who was given the title 'senior' by whoever with whatever motivation to do so.
Nothing in that article. It only means the person (probably) fits the 8 or so bullet point skills that management has said a senior engineer needs to have. And it's company specific.
If you want to be promoted, then either, ask your manager, or apply for a senior job somewhere else.
I don't get how these articles get so much traction. They are like fortune telling, just spout off some non-specific high-level items and let the audience fill in the blanks.
I too find that seniority should involve understanding systems and business e.g. and the author layed it down nicely.
A larger view of things, considerable foresight are second, but close companion. They tie into being able to understand people and things. They give you the ability to avoid things that would be problems in the long run. And allow you to build stuff that will last.
I'd say that everything else trails behind. Technologies come and go. Fads come and go. Anything can be sorted out as long as you can build things for the purpose at hand without straying outside the scope and build them in a way that will last.
As an example, what's beyond code quality is writing library code where doing the right thing is easier, and is aligned with the system. If, for example, one uses a design pattern whose anti-pattern is misaligned with the overall system, then it is easy to do the right thing locally and the wrong thing systemically.
As another example, the "best practices" are actually sound engineering principles applied specifically and locally. In a different architecture, with a different constraint and use-case, how those principles applied may sometimes look different than what "best practice" says, but not so strange if you understand the engineering principle underlying the "best practice".
Lastly, I would say, "systems thinking" always includes people: the people writing the code, the people making sure the infrastructure is running, the people using the system as end-users, the people supporting the users, the people running the business, the people paying the business, the people depending on the technology. Incorporating that means dipping into a product perspectives where people talk about things like, ux, or "job to be done", or "lived experience" rather than "requirements analysis" or "observability".
Systems thinking is broader in scope and responsibility. You're going to have to cross the rubicon eventually (unless all the software projects that hire you fail gloriously before they achieve any significant scale, I guess).
Code quality, best practices, etc. involve the same values and priorities as systems thinking, but is scaled down to a bus factor of 1. (i.e. the code you, personally, write or review).
Going beyond n=1 requires building mechanisms and guard-rails to prevent a failure of the individual level from affecting the larger system in a bad way. Thinking about it early won't significantly kneecap you from writing good code.
If it interests you, go for it.
The only reason you might not want to do so is if it doesn't interest you right now. You'll likely get there one day, once a project you care about scales up enough.
So this is a managers view of it.