My thoughts about the Principal role | Hacker News Reader

My thoughts about the Principal role

galiglobal.com

220 pointsantonmry113 comments

I have personally not found the staff/principal level very enjoyable or satisfying. You end up being a hub for the larger team, helping everyone do their jobs. I'm not just referring to the coders.

It's very draining and I resent that the industry classifies this type of work as technical IC work. A small subset of these higher "IC" roles do get to focus on technical IC work, but for most people a staff/principal role is basically going to be that of a co-manager with no authority.

I've been considering down-leveling or getting into some software niche where technical expertise is actually valued and existentially necessary to the business.

I am in a PE role right now and I 100% agree that the PE is the hub. I like that. I see my role as being a catalyst. I make things move along faster and more efficiently. I like seeing growth in others on the team.

Also, it’s ok if you don’t like that. If you would rather just be in there getting your hands dirty solving that one hard problem with no interruptions, then I think you should do that. You’ll be happier. Probably the increase in happiness is worth it even if you have to “pay” for it with a slightly reduced salary.

A question for all the PEs here. When you want to reflect critically on what you do, and actually measure your performance and/or bottom line contribution, what do you measure and how? What do your bosses measure? Surely warm fuzzy feelings and other placebo effects are not enough. Imagine a situation that some underlings resent about your uselessness or lack of responsibility. What criteria/facts could possibly make you agree with that?

As a manager, I’m looking at how my principals are helping the team move past big hurdles. Their ability to write code is a given, and they still do it (though not as much as they did, or at least not the same type of code). But are they mentoring and pair-programming with junior team members? Can they take what the BA/PM ask and turn it into meaningful technical tasks (that’s part of my job too - I try to lead that exercise, but need them as a sanity check and to stay on top of new things I might miss). Can they talk to a customer in a useful way (not too technical, not condescending) and do whatever problem is being discussed? Are they a go-to resource for other teams? Are senior leaders dropping in to ask them questions?

I’m definitely not looking at the raw number of Jura tasks here close. But I’m not looking at that for junior devs either.

Edit - usually, the more junior drive are more in awe of the principals. I’ve never encountered one who outwardly seemed to think the principal wasn’t contributing. More than likely, the principal was actively helping them get their own tasks done.

This. If I’m not helping junior devs get their work done, I am failing as a PE.

Helping is activity, how do you evaluate the outcomes?

Where I work scrum masters are supposed to “help teams improve”. That’s measured, in one way, by looking at the current rate of performance of a team and projecting it out. Then a scrum master needs to improve the team’s performance by some amount that exceeds the cost of having them there.

Some examples, number of on time deployments, number of story points, number of bugs... I’m not saying this is perfect, but that’s how some people are measuring.

So if a PE is responsible for improving juniors, I suppose a PE can be judged on the size and complexity of projects their juniors are able to take on. Maybe the time to promotion and number of juniors getting promoted?

Unfortunately, it tends to be a bit "squishy" - task completion (by the PE or by junior team members), team velocity, etc can all be used, but using them exclusively is just asking for the team to game those metrics. Much better to have regular 1-on-1s with each person and just discuss progress - team progress, individual project/task progress, and any career goals. If somebody is actively seeking promotion, we'll use a job progression matrix (HR puts them together) and talk through the differentiators and what that employee can do to fill the gaps.

You definitely need to be with a team that knows how to use principal-level ICs to be effective. Most teams need to be trained/coached (by the principal) in how to do that. :)

I quit my job as a PE in AWS, and now work on writing performance-critical code in a scientific context. I couldn’t be happier with the decision.

If you’re considering getting out, I would encourage it. Getting back to working on primarily technical problems, rather than social/communication scaling ones, has been great for my overall happiness.

Have you been able to maintain a similar level of compensation?

I started my career in scientific computing, and was paid a pittance compared to what a PE is paid at a big tech company.

I’m wondering if moving to scientific computing is a luxury afforded by putting in time at AWS.

If one started at Amazon before 2016 or so, and put in a full 4 years to get 100% of their new hire stock vesting, they’d easily have more than $1 million just in stock. That plus all your other savings from the time you put in can make relaxing in a lower paid role for the next N years until you’re ready to retire an attractive proposition.

Depending on where you live (and whether you own property) that may not be a lot if you are the only breadwinner with a family to raise.

Edited my comment. The $1M is essentially on top of whatever you’d normally be able to save. If that is a lot more than $0 then most people would be in a pretty good position to take a lower paying job for as long as they want then retire comfortably.

> epending on where you live (and whether you own property)

No, it's a lot for 4-5 years anywhere. This is on top of a pretty decent salary. For most working folk, putting this much aside in cash-equivalent over entire career is hard.

GP is talking about going from a high base salary to decent base salary, while using the extra comp (estimated here at approx 1mm) to smooth the impact of the drop in base. This is very manageable.

Yes, that is exactly my point.

On the other hand, if you spend your early career as a researcher making five digits a year in the Midwest, then move to SV in middle age to increase your earnings, you probably won’t ever catch up and be able to live very comfortably given the cost of housing.

This is just a possible counterpoint, learned the hard way, to the common advice to “pursue your passion.” My advice to those who have the option: make some money first, then pursue your passion.

The sibling comment pretty much nailed it. I saved well over 75% of my income as a PE and have more than enough money accumulated, so I'm not worried. My wife works as well, and even though we're in a pretty expensive area (Seattle) and have a child, we're very comfortable.

I get paid $110,000 per year, and have great benefits. It feels like plenty to me, even though I don't save nearly as much anymore.

I agree it's a luxury from putting in time at AWS. I don't think I'd be as comfortable (psychologically, anyway) if I had gone straight to academia.

Curious what a role in non-academic scientific computing looks like. How did you get into the field? what kind of companies have this kind of role?

Oh, I work at the University of Washington in the astronomy department - so, definitely academia. I am just classified as “professional research staff,” which distinguishes me from students and faculty.

As I mentioned below, I got into this field by emailing a professor out of the blue, essentially saying “Hi, I am a software engineer interested in working on astronomy software. What’s your world like?”

That’s a dream of mine (astronomy and SWE) but alas earning money is a higher priority right now and unclear when that will change

It's still quite strange to me that we're at a point in tech where technical skill becomes a penalty.

This is not to say that non-technical skills are not important and often more important, but for both myself and friends "getting paid well" and "working on hard problems" have become conflicting goals.

There was a brief while, at the beginning of our current tech boom where strong technical skills were extremely valuable. What initially attracted me to software was hanging out with some absolutely brilliant software folks working on very hard problems, and getting paid better than anyone else I knew.

I'm perpetually disappointed that after many, many years of honing my technical skills I find that if I want to use them 9-5 I have to choose to be paid considerably less, and scour far and wide to find those jobs. I've now decided that it's best to treat my passion for technical problems the same way I would any other non-work related hobby. Similar to the way that being an avid reader and book lover is nice, but ultimately not that important if you want to work at Barnes and Noble selling books.

This is generally the way things go. That it wasn’t true at some point is the anomaly.

The generalized version is that anything widely desirable about a job other than money acts to push down wages. For example, if a position is widely respected, well most people like to be respected so wages are going to be less than if it wasn’t widely respected. Similarly, if there’s a lot of people that enjoy spending time with children then jobs that involve spending a lot of time with children aren’t going to pay as well as they would have otherwise. See also, games programming.

After leading a team for about a year and a half, I see why managers get paid more. Simply, it's less fun, especially for someone who likes to solve hard technical problems.

I think I'm okay at leading a team but moving me to a staff engineer position would be better for me and the company and I think my leader knows this. The experience has been enlightening and I have a lot more respect for non-technical managers now.

How did that decision impact you financially?

Were you able to get something in a similar range as what levels.fyi lists for an Amazon Principle Engineer ($640,000 [0])?

Or did you become financially independent from the Amazon comp which allowed you to not care about the new lower salary?

[0] https://www.levels.fyi/company/Amazon/salaries/Software-Engi...

I saved most of what I made from the Amazon comp, and now have more than enough to feel comfortable.

Now, I get $110,000 per year. That's plenty. I have a family, but my wife works too in tech and has a good income.

The marginal value of the extra income from working in big tech just wasn't worth it for me.

Thanks for the reply!

If you had it to do over again, would you chose to go into the area that you’re especially interested in and sacrifice ever getting the high FAANG comp?

Or do you feel that having that savings invested has given you the long term security to feel comfortable, but perhaps if you’d made ~80% less over your career you’d be less satisfied?

That’s something I think about often. Sometimes I do wish I had plunged towards passions, but I know a lot of people who burned out by doing that.

I think my path was probably a good one; it really is very psychologically nice to not worry at all about money and be able to advance my career in any direction I want now without feeling like I am being selfish, or neglecting my responsibilities as a father, or whatever.

I also certainly learned a lot at Amazon (although the thing I learned most was “how to be effective inside Amazon,” which isn’t a particularly great skill in the long run!). So it was not so bad.

> I also certainly learned a lot at Amazon (although the thing I learned most was “how to be effective inside Amazon,” which isn’t a particularly great skill in the long run!).

Can you speak more to this? Why is being effective inside Amazon different from being effective at any other large organization?

Oh, there's plenty of overlap between those, and I'm sure lots is applicable to other large organizations.

But I also learned things like:

- How do I most effectively advocate for headcount within Amazon's yearly budget process?

- Who is the right person to go to in AWS Networking when we think we have a possible performance problem in the network stack?

- What is AWS AppSec really looking for when a design is going through a security review? (and what are they not looking for)

These have a little applicability, but mostly they're the inside knowledge that just comes from time and experience inside the organization. You learn the shibboleths that open doors or get people to listen to you on the first try. Sorta useful, but mostly not very interesting to me personally.

How did you get into scientific computing? This is something I've been interested in but not sure what paths look like.

I've long been interested in astronomy - I majored in it in college. So I just emailed a professor out of the blue. I looked for one whose personal page said a lot about "astronomy software." We met for coffee, and it went from there.

I think it's pretty easy to find tons and tons of scientific groups that are absolutely dying to find more professional programmers - at least in physics and astronomy, but probably in lots of disciplines. Feel free to email me if you'd like to talk more.

> You end up being a hub for the larger team, helping everyone do their jobs.

This is 70% of my job as a Sr Staff Engineer and I love it. But yes, it does take a certain temperament to enjoy it or find it satisfying. I like collaboration and pairing and helping other ICs grow their careers. And I love teaching, which this role allows me to do in abundance in all sorts of ways. Sure, I enjoy sustained technical work as much as the next engineer - but usually that work is better done by someone else who can use it to learn and grow and can own it down the road. Otherwise it becomes another piece of company knowledge that lives only in my head, and there's too much of that already.

I think there is still a bit of a perception that IC levels higher than Senior are about "Senior but with more interesting technical problems". This is largely false. In most organizations those roles are about empowering other people. You can see this if you read the stories Will Larson's new book, Staff Engineer: Leadership Beyond the Management Track. Done right, it can delivery a ton of business value. Yes, it may be a continuation of the IC track, but it's a qualitatively different job. I think that could be stressed more in career guidance from management or when promoting folks into these roles, so that we don't keep promoting our most effective programmers into jobs for which they're not really suited and won't enjoy.

How would you recommend that an engineer network with principal/staff engineers like yourself in their respective companies? Asking as you mention that the role is about enabling / empowering others.

Context: I am going to be starting a new job as a senior software engineer in a mid-size company in a few weeks.

Thanks

> I think there is still a bit of a perception that IC levels higher than Senior are about "Senior but with more interesting technical problems". This is largely false.

Anyone know of a career path where this is actually the case? Lately it seems like everywhere I turn it's meetings and PowerPoint. Ideally I'd like to keep working on harder and harder engineering problems until I die of old age.

Work on small teams doing hard things. Best if the team is mostly experienced people, too.

Big companies may work on harder problems, but usually with bigger teams and you end up doing a lot more communication.

> I think there is still a bit of a perception that IC levels higher than Senior are about "Senior but with more interesting technical problems". This is largely false. In most organizations those roles are about empowering other people. You can see this if you read the stories Will Larson's new book, Staff Engineer: Leadership Beyond the Management Track.

I think that's _one_ way to do it, but it defeats the purpose of having an IC path. It's basically "ship code while being a lite-manager". There's space for more types of staff+ engineers than "manager-lite" versions of the role.

I'm in a semi rebellion with exactly the description you provide, because I find it's short-sighted. We need to allow these roles to be anywhere along the spectrum from manager-like to purely technical. The best description I found of these more senior roles is that they give you license to build the role you wish to have. Kind of like being a tenure professor.

Here's my own self-defined role definition: focus on exploration and research along with mentoring my peers into becoming the next version of themselves. On the other hand, I have no interest in managing the day to day process; it gets in the way of my primary goals.

That's currently my worry as I grow, PE is the next level up, but I don't know if I want to get this far removed from the concrete code deliverables and software architectural decisions.

Does your employer have a separate software architect track? We have basically 3 tracks... developer, architect, and engineer. Developers write application code and the principals end up in that "hub" role mentioned up-thread. It's a good role, if you enjoy it. Architects drive corporate-wide tech direction, build new products from scratch (alongside developers), and some spend a lot of time working on standards and processes. And engineers are the DevOps/CloudOps guys - they code, but it's more scripting things and wiring systems together (vs building business apps) and they're on call. But, between the 3 general roles, there's pretty much a job for everybody.

I'm in an SRE role now, big change from a UX developer/manager (did that for 15+ years), but also fun and challenging, just in different ways.

Edit - All three tracks end up in some variety of "hub" role. Senior architects end up in meetings with VPs and other senior leaders. And writing documentation and diagrams. Engineers end up with lots of customer-facing time, which can be stressful if you don't thrive under pressure. Kind of comes with seniority. If you don't "play well with others" there's nothing wrong with staying in a true IC role - but you need to have the self-awareness and communication skills to make it known. And it definitely caps out a lower compensation level (ignoring some very niche skills).

So don’t let yourself get removed completely from the deliverables. Keep in there. Take some tickets off the queue and work them so your skills stay sharp. If you are worth your salt, this contribution will be noticed by other ICs on the team and your boss. Just realize that the IC role isn’t primary for you. Provide useful feedback in reviews to keep dumb stuff from happening. Make sure your team is working on the right things.

And yes, the PE contribution is harder to measure and the risk is there that others won’t see it. If you aren’t comfortable with that risk, then don’t go for the PE role.

Which is OK! It doesn’t have to be “up or out”. For me PE is as high as I want to go. I have no desire to deal with the stress and risk of moving up past PE.

I’ve also intentionally moved from a PE at my precious job back to being 100% IC in my current company and then a few years later back to PE where I am now. I did that to build up a slightly different, more marketable skill set.

At the end of the days it’s hard or impossible to be such a good programmer that you can create as much value through programming as someone else can through enabling several to many other very good programmers to do better work.

There may be some less than candid framing in calling the roles “IC” but it does make sense that all ladders lead to multiplicative contributions in one or another.

My friend explained it this way: up through senior, you’re concentrating on making yourself and later your team more efficient. As you transition through to staff/principal, you’re working to make your department and whole company more efficient. Sometimes that means unblocking someone so their work can continue. Other times that means getting two teams together to work out their differences. Even if you’re not getting your hands dirty as often, you’re making a much bigger difference than you would if you were spending all day looking at code.

Reframing it that way made a huge difference for me. The things I’d thought of as chores, or necessary evils, are really my opportunity to use the knowledge I’ve accumulated to make whole teams of people work better. That’s pretty cool.

I did a “principal” role with another name in a CTO organization. It got old quickly because you had no control, and it was difficult to actually accomplish anything.

The org evolved to a strategic devops role where we absorbed strategic platforms. “Principal” tech chops plus actual control of moats in the org translate into getting shit done.

IMO, unless “principal” is primarily a way to pay you more, you’re better off actually owning stuff than being “IT Yoda”. The same skills you think you’re avoiding in a management role are critical (and harder) for a technical leader.

Totally. It's also ridiculous difficult to change companies since your brain becomes pretty much the context of your current organization. It's IMO harder to apply to other companies as explaining your day-to-day requires a lot of context and nuance about your company that do not fit a resume.

Further to that, the interview process of staff/principal are usually akin to a senior dev with technical grinding except that isn't your day to day and hasn't been for a while.

> a staff/principal role is basically going to be that of a co-manager with no authority.

The expectation is to be a leader without being a manager.

In many disciplines, to earn higher salaries you will have to produce/influence output at a level that might not be possible by one person. The way to get that kind of output is to mentor junior engineers.

That's been my role as a senior staff scientist. And at first glance, you'd wonder, well who in their right mind would want that? But it actually has its merits.

With authority comes a chain of authority. The manager is just projecting the authority of the manager above them, and so forth. That can translate into being a manager without being a leader or an authority. Also, if you rise to the expectation of being a leader, you might have a chance to actually get good at it, something that many good administrators never experience.

I've also been a manager. It was dull, and I knew that I would never be competitive at it, so I went back to the technical track.

Some of this advice is great, like getting out of the way, guiding people with how to think about trade-offs and doing daily coding. But it doesn't feel like 'Principal' level advice, at least in terms of Big Tech and the blog notes the author isn't sure what the distinction is with Staff.

The author complains that it's hard to be in the critical path at a senior level. This lacks self-awareness. It's always hard to be in the critical path. Shipping on time is one of the toughest deliverables of the software engineer role, and one that many people struggle with. Accurately estimating development costs including wall time vs. actual time someone has to work on a project is a very important skill. It's not acceptable for senior engineers to abrogate responsibility for this, especially if they claim to be mentoring other engineers.

Senior engineers own the business outcome and must weigh costs of all kinds, from security risks to technical debt. As scope increases, the feedback loops get longer and longer. A new engineer can tell if they did well with a comprehensive unit test. A junior engineer can tell if they did well with a performance or integration test. A senior engineer can tell if they did well with an A-B test in the market. A staff engineer can tell if they did well by seeing market share grow.

In Big Tech, senior staff and principal roles carry the idea of doing something to 'shock the world' - that is, successfully shipping innovation that people were afraid to, for example, because it seemed risky. Greasing the wheels of communication between teams and helping people avoid common mistakes is fine and a good thing. But there is limited business value in building consensus around the latest "architecture" or framework or language or whatever, however nice it feels to enjoy the social status as the person turned to for this kind of question. Step change innovation is the real value add and this article hardly touches on it.

Do the architects actually… build things? The description of the work there is more like a program manager. Which is an important organizational job which doesn’t have any power, but also isn’t the person doing the work, and so doesn’t have the understanding to design it.

Interesting feedback, thanks! What I describe in the article are the challenges of the role based in my experience. What you describe it's the business as usual work. Innovation, deliver and push key initiatives, etc. are the reason to be promoted to Principal in first place. They are important but they aren't what I find more challenging in the role.

> A senior engineer can tell if they did well with an A-B test in the market. A staff engineer can tell if they did well by seeing market share grow.

I don't get this, these are marketing and business skills. I don't understand why tech roles don't leverage the full power of engineering and instead engineers are steered towards tangential specializations. This isn't true across the board of course, but how can companies not see the value of letting some engineers become the technical experts without needing all these related skills? Knowing advanced CS and information theory can help, if anything to keep the other engineers up to date (and I think it would involve technical leadership, just not in terms of managing others), but it's not valued at all and without those other skills it seems engineers just stay as "contributors" (i call this bottom feeders).

Principal role = expensive lean manager. They want to help others achieve something without actually getting their hands dirty. Of all the principal engineers I’ve worked with, the vast majority behave like motivational coaches/lean managers. Examples:

- There’s an outage on a Sunday morning. A huge Slack thread starts. Devops and devs on call manage to fix the issue. Principal engineer gets involved ACKing other’s people comments, adding thumbs up emojis, and at the end says “Big thank you to all the people involved” (plus a rocket emoji)

- There’s a decision to be made regarding i18n for DB tables. Architects, senior devs and the principal engineer get together to talk about potential solutions. The principal engineer acts as a lean manager: let people talk in turns, does support good ideas (I mean, he’s not clueless ofc), does try to gather everyone in to a common solution... but never gets his hands dirty as in: “what do you think about solution X? I could work on a POC to see if it’s worth it”. If, after months, it turns out that the idea to be implemented does work, then he says “All the props to the team!” (No way, really? It’s obvious that all the props should go to the team! You, principal, did nothing!). If it turns out that the idea actually does not work, then “no one is to blame, let’s do it better the next time. Let’s get feedback and blah blah”. As I said, if something works or if something doesn’t, it’s never on the principal (but hey, he gets to make twice as much as the most senior dev in the company!)

- Microservices or modular monolith? Same as above. The principal engineer knows his stuff, but just as much as any other architect or senior dev, with the difference that the architects and senior devs are the ones who are going to do 99% of the work (both im terms of choosing the final decision and implementing them. The principal is just there to “support”... but that’s actually quite useless tbh) and they make significantly less money than the principal engineer.

> There’s an outage on a Sunday morning. A huge Slack thread starts. Devops and devs on call manage to fix the issue. Principal engineer gets involved ACKing other’s people comments, adding thumbs up emojis, and at the end says “Big thank you to all the people involved” (plus a rocket emoji)

Just want to call out how accurate this is. The last place I worked was full of this kind of person (as managers, wecdid not have a principal title), and the description, right down to the emojis is bang on!

If I had such a principal on my team, I’d be happy about that. It means that they’ve contributed to an overall team dynamic where the team does not depend on them personally to perform under pressure.

Far worse is a team of on-call juniors who stand around jabbering cluelessly until someone decides to call in Principal Pat who fixes it in 15 minutes during the 7th hour of outage.

If the team is on the right track, providing non-technical support and encouragement is exactly what a principal/director should be doing, to reinforce that the team is performing and earns the credit.

The principal “owns” the annual downtime and problem-loss figures. The team gets credit for this weekend’s fix.

In other words: “A leader is best when people barely know he exists, when his work is done, his aim fulfilled, they will say: we did it ourselves.” —-Lao Tzu

Not a "Principal", but you guys would rather be followed up by a business person who would come up with their revised requirements and timelines, each day or the other? Or obey the commands of the technological genius that somehow always have the answer to every question, for others to implement exactly as told?

Leading by using technological experience and refraining from prescribing "the meds" all the time, is lighter work, so one can stretch over much more than otherwise. But nobody becomes master of everything, and those places that have those, truly suck without knowing it. Having to go into all the depths, someone else'll need to take point..

You're right about how the role ends up being implemented, but it shouldn't be this way.

As someone adjacent to this role, a lot of the time this sort of "hands off" behavior is incentivized; the reason the principal is not jumping in and taking point on making the final decision and implementing is because that severely reduces the agency and ownership of the senior devs below them (and in many companies risk undercutting the latter's career progression, if it depends on them showing that independent ownership).

Interesting; at the places I've worked, "architects" are more senior and highly paid than "principal engineers". This description is totally unfamiliar to me.

I'm in a PE role, and I have to tell myself every day not to go do the POC because that's not how I can maximize my impact. If someone else can do that work, I'm going to try and lean on them.

Strategy is hard. Much harder than I thought it would be. I am unable to make progress on strategy when I'm deep in tactical work.

I do jump in when needed, but every time I get my hands dirty I fall behind on strategy.

Your PE may indeed be useless, but it's also possible they are pretty good at their job. Good strategy is boring and seems obvious when presented because it creates clarity across a diverse company.

This is a fine line for me as a PE. It is good to help the team grow. However, I like to do the PoC sometimes so that I can keep my skills sharp and my opinions grounded in reality.

If I just sit in my ivory tower dishing out advice, the advice will eventually become useless.

+1. And if there are several ways to design or implement something, it is 1000x better to do the one proposed by the people who will do the work. They’ll understand it better, have more context, and be much more motivated.

Unfortunately this has the side effect of making the principal look like an overpaid rubber stamp and cheerleader to the best less-senior ICs who usually have their shit together. :)

I’ve been at this place before even if I don’t have the title formally.

What I have learned to do is to actually do my own flavour of the PoC concurrently with the team. That way I have more material to discuss and am able to give deeper insights like “have you considered X or what happens when Y?”.

It helps getting to know the problem better while letting the responsible get to own the product.

> I have to tell myself every day not to go do the POC

What is a "POC"?

Side Rant : It's really helpful if an acronym is at least expanded in the first use. I don't find the acronym in the article or the parent comment. I am not sure why and how this is a norm but it's frustrating as someone who wants to understand a conversation.

Proof of Concept

Piece of Crap?

Sorry about that. POC was introduced in the parent comment, but as you said without expansion.

If anyone has a hard time empathizing with unexplained acronyms, read a bit of https://www.reddit.com/r/churning/. I think the jargon is okay for the nature of that community (taking advantage of credit card sign up bonuses) but following discussion there can be really confusing until acclimated.

Example sentence: "CSR CL can be lowered. Need $5k CL to PC it to most cards, $10k to PC a(nother) card to CSR. Recon might only be able to drop it to $5k, you might have to SM to go lower than that."

at my work, I'm a principal and my boss is a partner. he spends a lot of his time doing organizational stuff and coming up with plans to solve problems, but he also still writes PoCs. he'll take the downtime he has between meetings, solve the really hard parts of the problem and hand it to the team for incubation. I do the same thing now - I pitched an idea which requires a fair bit of pure research, and I'm going to get a rough prototype going that vets the idea, then build out my team to make it better and turn it into a robust product.

the trick is to keep the PoCs lean enough that it's easy for your reports to flesh out and change the architecture as they need, but developed enough that it's working code, so people can run it and start working from a grounded product.

The PE should also be doing a lot of behind the scenes (from senior dev pov) strategy work with leadership team w.r.t identifying long term technical risks and ensuring right investments happen, grooming tech talent bench through perf/promo processes, seeding/sponsoring new tech initiatives etc.

All this is informed by the observations and insights that come from the "support" work they do in those outage huddles or design discussion meetings.

The technical thought leadership work they do with the business/product leadership team is where they really earn their paycheck.

Ok, but are you implying there is no value in that? Wouldn't you need such a person, and wouldn't you need that person to not be clueless and know their stuff to be able to do a good job at it?

It is supposed to be a super-smart engineer who is extremely vertical in multiple disciplines, can solve complex problems, and can navigate politics.

I worked at Intel. A principal engineer in the process or architecture was vastly different than a principal engineer elsewhere. For the most part, you become a principal engineer by sticking around, doing lots of extra work, and making at least several major contributions in your career that impact multiple products' bottom line or viability. As a bonus there is a weird deferred-income strategy they use that helps you avoid taxes, your regular bonus multipliers skyrocket, and you are expected to work 24/7.

However, I witnessed several abuses where:

- You can become one if you are friends with an upper manager who scoots you ahead in the line. Several times I saw this type of PE later pushed out by new management, because it was painfully obvious they were in over their heads to everyone else.

- You can become one if you are in a small group that wants to justify its existence. Every group needs a principal engineer, and without one, it is a risk the group will be dissolved. Some managers of tiny groups like their ego trip and want to continue being big fish in small ponds.

- And you can become one if your work a site that hands them out to puff up its image. In the latter case, there was one non-US site that was almost 40% principal engineers, and used that as their claim to fame. "We have the most principal engineers at our site" said the VP. Duh! You are also the VP that approved all of them so you could say that!

So while Intel's definition makes sense and is generally true, it is abused about ... ~35% of the time?

Disclaimer: Intel is an up or out company. You move up, or you move out. It is hard to hover in certain high-pressure divisions. I was passed over twice for PE and shuffled between divisions, and eventually exhausted to the point where i was "out". So i'm a tad bit bitter.

I consider principal engineers / architects "astronauts". They've spent so much time floating around they've forgotten what solid ground feels like. They're both disoriented and atrophied when it comes to engineering. It's a trap. You'll eventually have to come back down to earth, and the longer you spend in this role the harder it will be.

I don't know that I even really buy into the idea of a "principal engineer". Even in a really large company where you have a shit ton of people working on one bit of software, principal engineer doesn't seem like a well defined role that you need, because once you scale past the point where one person (a "lead engineer" or whatever) can manage the technical side for everything, the next "level" up in whatever structure you're working with by definition has to defer to their subordinates for technical expertise. That makes it a management role, not an engineering role.

I think once you're in "principal engineer" territory, it's your job to divvy up the work and hire people to make their own decisions and take responsibility.

If I was hiring specifically for that role I'd probably go with engineering manager or technical product manager or something.

Having said that I think you could make the case that principal engineer works if you're doing that stuff as well as taking responsibility for a subset of the app and continuing to do engineering work. Because I think there's a fairly large range of team sizes where you probably don't have 40 hours of managing to do per week (probably a range like 10-30? Maybe higher? Probably depends a lot on the app too). It's a nice hack to deal with the fact that you can't give someone two job titles.

Btw I think that strategy is underutilised. I've worked with far too many people that are doing 15 hours of really useful stuff and 25 hours of making everyone else's life more difficult because they're a bit constrained by their narrow job description. I really think more devs and designers should have side gigs as little mini PMs, POs and BAs instead of the usual strat of scaling up as quick as possible and running face first into the consequences of Parkinson's law.

Maybe that's your experience but the decisions and discussions you're not privy to is where their value is added. An IC4 isn't there to write implementation code as junior engineers can do that.

It's common in the consulting industry to have a wide range of titles - [senior] manager / principal, director, managing director, executive director, vice president, associate partner, etc. Depending on the firm, literally all of those could refer to a similar job.

The reason I've seen put forward for this is to make it hard to compare titles, so that a person can feel like their title matches their seniority.

I've seen this happening in tech too, especially with senior and "lead" roles (especially being "a" lead vs. "the" lead). At some places, I think the staff / principal / distinguished / etc engineer role is along these lines too, basically a way for a more experienced person to feel like they don't have the same job as a 25 year old.

It's also a way to provide a technical career path without having your super experienced people all stepping on each others toes.

I've seen that in Video Games versus Advertisement, Art Director is a high seniority position in the former while it could be an internship role in the latter.

This, to the extent that after 13-14 years in the industry, I have no clue what differentiates any of these titles in theory, let alone in practice at individual organizations.

> An architect? A big part of my work is to improve systems and platforms. Listen to problems and propose solutions. But I don't feel like the guy who does a plan which must be blindly followed by others. I don't want to be a gatekeeper who says what can be used and what can't.

The only area where I feel like you need to be super careful is around architecture. How many chefs can bake the same cake before you have a total shitshow on your hands? In my experience, this depends on the complexity of the cake. The more complex, the fewer chefs can be involved at the same time.

The last place you want design-by-committee is when you are trying to model a very complex problem domain. Have your elder council determine the types, facts & relations for your solution, then let the team go wild on top of it. The people developing the schema should understand what normalization is and why it's so critical to long term success. 100% of business apps can start as excel spreadsheets documenting types, facts & relations. There is zero excuse to not start here [0].

Without some common understanding of what the fuck you even call things (and how they are related), you don't need to have a bunch of distributed design meetings about logic & UI that will be built on top of those concepts.

[0]: "Show me your flowchart and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowchart; it'll be obvious." -- Fred Brooks, The Mythical Man Month (1975)

From a practical perspective, most companies large enough to support PEs as described here, are going to have misaligned hacks that made sense once upon a time but are no longer sensible.

A big part of a PEs job is to keep the train moving despite nonsensical design decisions, organizational politics/empires, and legacy code bases/use cases that no one wants to touch. A good PE will help incrementally resolve/improve the above situations.

> A good PE will help incrementally resolve/improve the above situations.

100% agree. What I posted above is an idealistic perspective in which you get to greenfield an application from zero. Most real world situations involve some degree of legacy domain implementations which must be bridged into the future.

Being able to rebuild the ship from the inside while it is sailing to the new world is an incredibly valuable skill set. Many developers get frustrated and demand a total rewrite (I used to do this). It would be really hard to make forward progress if you start off with a new ship in Spain every time you encountered the slightest bit of friction.

Unfortunately what you're describing never actually occurs in practice, unless you're simply rebuilding an existing piece of software, which has only technical improvements (ie. better performance, fewer bugs, etc).

The cases where you are starting from zero are also the cases where you don't understand the problem domain to a sufficient degree. You can try to get clients involved earlier, but that is very dangerous:

- It's hard to get a sufficient diversity of early users.

- Clients are incapable of prioritising what's important. Everything feature they think of (even the things they won't actually use) will be essential to them if you ask.

- The actually essential features will be so obvious to them that they won't even think of mentioning them.

- They don't understand abstractions: any concept you try to define with their help will become amorphous and therefore useless.

- Engineering is fundamentally subtractive: with zero code a product could eventually do anything. As you add code, you are restricting what it can do so that the results are the "useful" ones. For example. when you create a "user" table with an "email" property, you are restricting users to have one email address. You cannot make progress without these restrictions, but if you actually spell out those restrictions, you'll get massive pushback.

This is why so many engineers like building things for other engineers - because they can be their own client.

It's also why it's impossible in most cases to go from zero to a well-architected product in one go.

This is... kind of precisely the distinction between technical leadership in a medium sized team vs in a really large team. It's one of the hardest challenges I work with senior devs to overcome as they move up to larger and larger scoped roles, up to principal engineer positions. Because what they have learned that got them that far stops working as they develop into more expansive roles and responsibilities.

Basically, the problem solving approach of 'I'll go away and work out how we're going to do this, then bring back a masterplan', while it offers a good path to successful project delivery early in your career, does not scale.

What worked for you when you were building a system with one or two teams working on it for three to six months is not a viable approach for systems which will have five to ten teams working on them, over multiple years.

You have to adopt approaches that decentralize decisionmaking - else you'll become a bottleneck; you have to adopt approaches that are adaptable to information that will be uncovered later and to requirements that will shift; you have to adopt approaches that acknowledge that you are never going to be 'full stack' enough to be able to make smart decisions about every single part of the solution - the mobile apps, the backends, the financial system integrations, the frontend caching, the analytics and reporting... and you need the expertise of different teams to deliver that.

You basically have to unlearn the pattern of 'let an experienced person set out the whole plan' that has been successful for you so far as you move to higher level leadership roles.

As a principal/staff/whatever in a big engineering org, your role is much more about making sure that the right people are having the right conversations and negotiating the right boundaries and not getting caught up on artificial constructs they think are being imposed by you from above. You need to leverage all the intellectual capabilities of the senior engineers on every team, not tie their hands by thinking you know more than them.

Same as as you become more senior, you shouldn't be writing critical code - eventually you also get too senior to be making critical architectural decisions.

The issue is that at a certain scale no one person is smart enough, knowledgeable enough or has enough time to actually design a system properly with all the necessary detail. Those who try create an even worse mess than a design by committee.

>The people developing the schema should understand what normalization is and why it's so critical to long term success.

I'd expect everyone except a junior engineer to understand this if the system they work on support normalization well (ie: map reduce hates joins for example). If only your most senior engineers understand basic concepts then you have much more serious issues than design by committee.

>100% of business apps can start as excel spreadsheets documenting types, facts & relations.

Except in a large system no one person knows what all the types, facts and relations are. So you either delegate the work and listen to feedback, or you end up missing critical information or having the wrong information.

There's some sort of PE analog to the Brooks quote I've been working on, but I'm not quite there yet. Something like "Show me your solution and conceal the org chart, and I shall be mystified by your decisions. Show me the org chart and I won't need to see your solutions, they'll be obvious."

It's not quite as determinative, though. See the data and you often really don't need the code, but even understanding the org chart isn't enough. You also need history, and some other stuff.

Still, there is a general analog, in that it's a big danger in this role to think that it's about the produced architecture rather than the structure of the business, which "capturing in data and shared terminology" is a very good starting point for.

You're getting close to Conway's Law

Well, that's part of what I meant by "it's not quite as determinative". Conway's law constrains solutions but it doesn't determine them; it's not a 100% guidepost the way "letting me see your data" almost does 100% show what the code needs to be, at least at some level of abstraction. I feel like there's some stronger formulation that may more strongly narrow down how the architecture has to look than just the org chart.

My quality of life as a senior staff eng depends on a couple things. Be curious what others would add:

- *good manager*: at my company, I’m paired with a manager over a focus area. If they are good we'll form an effective partnership. They’ll help protect focus time, take on lots of planning and people management work, while I focus on the teams technical leadership and roadmap.

-*Important company priority*: if what I’m working on actually matters to the company, we'll get the resources we need to execute. Without this, our team will be frequently poached and we won’t achieve much.

- *strong stakeholder relationships*: I regularly meet with and take feedback from stakeholders. They can see where we’re going on a tech level and incorporate us into their plans. We can be an asset to the company’s different, customer facing product lines, not a hermitted group of devs.

- *strong inter-disciplinary collaboration*: we have all the disciplines we need on the team. Whether it’s data, UX, eng, or something else, we’re not playing politics to negotiate with another management structure on every little task our team needs. It just gets done a a virtue of this discipline being a teammate...

- *A road to prod*: we ship early and ship often. We don’t have anything in our way external to the team for shipping. Also we’re not working on a theoretical thing, it’s actual prod code we help with!

-*hiring great people*: we hire amazing people, and usually we don’t have to worry about their quals. Or worry about them being a jerk. Also we meet to ensure they’ll be a good fit for the team.

-*a clear area I own*: I want to make sure that if I’m given technical authority over an area, there’s not another overlapping staff/principal eng who’s also expecting to own some of that space. It’s clear what the groupings are and who does what to avoid politics and ego clashes between alpha geeks.

-*active burnout prevention*: the culture and management work against my hard-charging style and strongly encourage me to walk away from work during vacations, weekends, and evenings...

I'm senior staff, and I think you really nailed it. In my case I'm in a more exploratory research team, so a lot of my stuff doesn't make it to production, but I develop the techniques that will hopefully lead to the next generation of products.

Maybe I'd add a couple things:

* Always learning.

* Visibility and interaction with a broader range of people. For instance I work for a big multi-national, and I greatly enjoy cross site collaborations.

One thing we have made good experiences with recently is, having two separate meetings for managing engineering initiatives:

1. Engineering Meeting - led by Principal Engineer. With pure focus on technical questions. We go through open issues, PRs and discuss technical questions, engineering decisions like API design. Stakeholders from other teams are invited to participate.

2. Planning Meeting - led by a Engineering Manager. Report on progress, discuss priorization and decide work assignment a.k.a. "sprint planning". The principal engineer can take part in this as a team member.

This helps me (as a principal) to stay away from people- and project management and purely focus on engineering topics (inc. writing code), which is where I see as my core competence.

25 years of experience, and I never wanted to get above Staff Engineer. The work is just not something that I enjoy. I love coding, and I like working directly with my team. When I applied to new companies, I purposefully applied at the Senior engineer level, not Staff or higher.

Sometimes I wonder if I should have just put on my big boy pants on and put in the extra work to get promoted, something my bosses actually asked me to do. And more especially when I see my peers become at a higher level, director, VP etc. But life is short and I never would have stayed in the business if I had to care about things that I fundamentally don’t care about. Almost dying a couple of years ago really helped gel my thoughts on that.

Not only is it hard to define what the principal engineering role is, it will be different at almost every company.

The same goes for staff eng and even to a lesser degree senior eng.

Interestingly it's sometimes harder for companies that already have the role defined than for growing orgs that usually do this JIT as really strong devs top out of senior and realize/decide they don't want to be become managers. I've defined this type of role twice now by using the skills and behaviours of the specific individuals (i.e. things they're already doing), then expanding the scope and sphere of influence for growth areas. It's worked pretty well because typically you should be promoting someone when they're demonstrating performance at the next level, rather than give them a promotion and then see if they're successful or not. The role needs to be vague enough to allow for individual and strategic changes, but defined enough to provide some type of identity. This is hard.

And even more harder is that within a company, those titles are vastly different depending on what team you get hired on to. You could be hired on at a senior level for a given team yet be considered a standard software developer or even associate on a different team.

Case in point, an engineer was hired on in my current company at a Senior role, being even considered for Principal and was subsequently moved to my team. They are now gauged at almost an Associate level when compared to the rest of the engineers on my team and their titles.

The special problem with Sr Engineer is that it’s most people’s title from year 5 to year 40. If you keep improving in your craft then it can be a very wide band.

I for one have no interest in being a tech lead or a staff engineer as all the non technical work exhausts and frustrates me endlessly. I’m still striving to improve every year at engineering and architecture.

This post is good in that it doesn't try to define what a principal engineer is, but just give the author's opinion on what their job is.

One thing I realized with getting jobs with increasing degrees of ambiguity and autonomy is that you can _somewhat_ make it what you want. If it turns out that what you want aligns with what is needed, then things go super well.

The other thingis that titles are just a construct. They're nice badges that help rewarding people by giving ego instead of money. being a principal in a startup is different from being one at Amazon is different from being one at Facebook.

This is spot-on from my perspective. The comment I’ll add is that a true Principal is different in kind, not just in experience or knowledge, from a very senior developer. A Principal embraces the interdisciplinary nature of their role. They tackle the hardest and most awkward challenges to the business because they see them coming before you do. They naturally mentor and develop junior talent, but are self-aware enough to know that they would have less impact if they were saddled with managing a team. Not every senior engineer has what it takes to become a Principal.

“If after one year, you can't leave a team without impact, you've failed in your work.”

A top managerial professor I knew defined a manager’s role to be exactly this. Principal or manager or whatever you are called doesn’t matter in the end, since once you go beyond “senior” level positions your role largely should be facilitating, empowering, and advocating for the members of your team. In reality, these roles unfortunately deteriorate into countless meetings with other managers and political struggles within a strong current of corporate compromises. Ignoring the latter part usually results in bad outcomes for one’s career at this level.

In my opinion, a true, technical principal role only works when there is another very capable person that fulfills the role of shielding the principal from the inevitable political, hierarchical forces that will head their way particularly from the sales and marketing departments.

> In reality, these roles unfortunately deteriorate into countless meetings with other managers and political struggles within a strong current of corporate compromises.

It is unfortunate that talented programmers turn their backs on leadership roles requiring inter-personal skills, propagating the stereotypes that prevent programers from progressing in their careers.

> In my opinion, a true, technical principal role only works when there is another very capable person that fulfills the role of shielding the principal from the inevitable political, hierarchical forces that will head their way particularly from the sales and marketing departments.

Needing such "adult supervision" is harmful for programming profession in the long run.

This is not about who is the “adult” … it’s about creating an environment for success given the expectations of the job.

Technical roles like a true principal at a high level require deep concentration and focus … being interrupted every 15-30 minutes to attend a meeting, answer a phone call, strategize a political response, respond to a slack message, handle complaints from critical clients, attend budget meetings, create reports, etc. is not the right environment.

> I don't feel like the guy who does a plan which must be blindly followed by others

When I became PE, my boss said that if you have to successful then you need to get teams listen to you by influencing instead of an authority.

At that moment, I had no idea how to go about this. But I realized one thing: "Knowledge is power". If you know the tech-stack, product & domain then you are well equipped for this role.

> Am I a manager? No. [I] let the team self organize with my help

Isn't that what a manager is, or should be?

I look at it as being situational. Sometimes you end up with a team whose natural styles all mesh and they don’t need much leadership. Self-organization is great here; a manager’s role should be to give advice and deal with corporate politics (IMO the politics/budget angles are the difference between a principal/staff engineer and a manager).

Other times you get teams whose working styles are very far apart and have a hard time finding common ground. In the latter case, self-organization leads to communication breakdowns; so you have to force some organization on the team.

Ideally you would just hire a team with a diverse but complementary set of skills, but you go to war with the army you have.

Also, although managers appear to work for a team or specific area, they usually interface across many or all in a department. The self organizing team will usually do so as if they are completely independent which is rarely true, so they need to organize within constraints of which they are totally unaware. Example: getting a different team to priortize work upon which your team depends will involve larger development plans, product management and the "politics" you reference. I've never met a technical lead who wants to do this and few who are good at it. This is a good thing.

Job titles are overrated. Often they are not an indication of skills and/or achievements, but rather political ability to networking and managing up in the company.

I have worked with Principals, Architects etc ... they were more an obstacle than facilitators. Most of them, if they were to interview, they wouldn't even be able to get a job as a senior engineer.

At that level it's not what you know but who you know. Your ability to interface with the rest of the org and get things done is where you deliver value. If you see senior people as such obstacles then it suggests you've not worked for a larger org or perhaps the obstacle isn't those other people.

hehe I guarantee you I have worked indeed in many large FAANG and not, tech companies. And it is pretty much the same everywhere.

I think when you get at a certain level, people tend to rest and vest, thus producing little to no tangible value for the company. But, since their networking is strong, they keep sticking around.

Large corp => too many "architects" => meetings => ... and nothing gets done.

Ever asked yourself why big corp innovate through acquisition?

It seems like "Principal Engineer" is the new "Architect".

This is a pet peeve of mine due to how common it is in all sorts of tech career advice, but I can never take a tech blogger/influencer seriously when they are so flippant about amounts of money that most people will never see in their entire lives.

If you look at salary info on a site like levels.fyi for top tech companies the difference between a senior vs staff vs principal engineer salary is pretty mind boggling. E.g. looking at Google right now, the annual total comp goes from $353k at senior level to $486k at staff level to $992k at the principal level. Just a cool million dollars a year on the table, and the author is saying things like "Nobody complains about a salary raise but optimize to career based on the salary is a quite bad idea if you are already covered." The whole piece is about how they just really like selflessly helping other engineers, and it's got nothing to do with pay or status.

It's not like we're just talking here about the difference between upgrading your car from a Toyota to a Lexus, or staying at a slightly nicer hotel when you go on a vacation. This is the ability to never worry amount money again for the rest of your life after just a couple of years in the role. It's the ability to retire 20 years younger or pay off your kid's 4 year ivy league education in just a year of savings or fund your own charity that helps thousands of people, or whatever else you might want to do in some of your wildest dreams.

I can't help but see it as anything other than a very obnoxious flex/virtue signal whenever people in elite tech roles make claims like this about how salary is so much less important than how much you're learning, and other things of that nature.

An IC4 role is generally one where you can give a problem to the IC4 and have them figure out how to break it into work and get things done. They don't need hand holding like junior engineers but they also know how to get input from experienced peers and make informed decisions. You're not payed to solve run of the mill problems at this level but to leverage your experience and intuition. Collaboration is critical.

I’m confused - you’re describing a Sr Engineer

If you’d like to learn more on this topic Will Larsen has great read about being Principal Engineer. Here link to page that has link to his book https://staffeng.com/about/

I had that job title once. It was a way for my boss to justify paying me more, I just wrote code.

Job titles don't mean anything to me but they do to HR.

Nice. Here is my attempt following a few years partnering as an eng manager:

https://dev.to/solidi/what-is-a-principal-engineer-anyway-55...

Question for Principal Engineers and Senior Managers, Directors here -

can you shed some light on how to influence senior engineers and PEs, and directors? I am rising in my career track as a people manager, but I see that PEs, directors don’t appear convinced or influenced by me. Please suggest some learning resources, books, blogs, YouTube videos etc.

Reply on news.ycombinator.com