Ask HN: How to become a ridiculously good back end staff eng? | Hacker News Reader

Ask HN: How to become a ridiculously good back end staff eng?

77 pointsandher36 comments

A bit of a click-baity title, apologies. I'm a senior eng and want to really push myself to better levels. My non-tech skills (comms, collab, leading) are one of my strengths and I'm constantly trying to make them better. I constantly keep feeling that I don't know enough tech stuff, there's a ridiculous amount to learn and foundational blind spots and holes that keep cropping up.

One of the questions I try to ask myself is, if I joined a new team or company, what skills would I need to really help the team succeed quickly and effectively.

A lot of ya'll have been around the block a lot, and either been or worked with such folks. Your insights would be super appreciated

View on HN

There are a lot of not-sexy but highly impacting skills in systems programming.

For example, the most productive person I've ever met, is a master of shell script, all the various shell tools (with obscure flags and usage modes), has deep knowledge in the use and application of makefiles, is a perl wizard and knows lots of tricks for virtualizing/replicating things in non-obvious ways.

When this person wants to automate a task, he can do it immediately, without having to think about 'how do I xxx'. It just flows from his fingertips because he knows all of these 'basic' things. Since so many things in SW are automatable, this gives him a non-linear advantage against nearly all his peers. He can dedicate 50% of his time building tools that act like a multiplier for the 50% of his production work and gets 500% done.

This seems to be an amazing quality. Would you say that his work is sharable across users or are they too specifically tailored for himself (maybe a bit too "write only") or very specific to a task?

He writes very generalizable stuff, and has an uncanny sense for picking the right blend of extensibility and concrete behavior. That is, he doesn’t get bogged down in ‘what if’. but at the same time his stuff usually can be extended without too much trouble.

The only drawback is he documents very little, and as a consequence you have to be willing to dig through code to get the value out of his work.

Sounds like something to aspire to!

What are some of his tasks that call for perl?

Ship code that works, do it quickly, and do a lot of it.

A common mistake is to focus too much on the technical aspects and not enough on shipping. It’s easy to get lost in over-engineering the hypothetical perfect system that doesn’t ever shift to customers. You don’t want to be that person.

Instead, focus on simplifying implementations and shipping simple, trustworthy code as quickly as possible. A reputation for getting things done will go a long way.

Beyond that, focus on area under the curve. The more code you ship and the more tickets you work through, the more experience you’ll acquire. Cut through the time wasters and get coding as much as possible. Over a decade you can accumulate multiple times more experience than someone who moves slowly and puts in the bare minimum. This doesn’t mean kill yourself by working 50+ hour weeks, but it does mean you need to learn how to cut through the slack in a workday and eliminate procrastination and distraction. Learn how to get down to work and focus efficiency. It’s a learnable skill.

Thanks, that's a great point and reminds me of the 'quantity trumps quality' parable[1]. I've found necessarily more of my time goes in reviewing code, reviewing design, setting up collaboration and proposing work (vs being able to ship more code). Any good ideas on dealing with this change in the nature of work that has to be done at more senior levels? I've also found, as I ship features fast, I don't always retain all the new stuff I learn until I've touched it a couple more times. Maybe this has something to do with how I've been learning, but any suggestions there?

[1] https://blog.codinghorror.com/quantity-always-trumps-quality...

> that's a great point and reminds me of the 'quantity trumps quality'

Quality versus quantity is a false dichotomy. You actually need both quality and quantity (or more accurately: speed of delivery). Don’t ship bad code for the sake of shipping code because this will damage your reputation.

The actual tradeoff most people are up against is speed versus complexity. If you find yourself in endless process meetings or talking about new architecture changes over and over again but it takes forever to actually write code, then you’re stuck on the complexity end of the spectrum. Complexity feels good in the moment because it feels like you’re doing the right thing and reducing risk, but the more complexity you add the less product you actually ship.

Learning how to simplify to the bare minimum, reduce process overhead, and ship quality, minimal code quickly is the art you want to learn.

When you are trying to become a good junior engineer, or a good senior engineer, it feels more like "checking off the boxes on the checklist". There are many skills that are expected of you, and you should gain them.

But you are kind of past that point. There are a lot of different "archetypes" for being a really good staff engineer. You can become very skilled at a particular technology, or very knowledgeable about a particular product, or very politically effective within a particular organization. For different people, the best strategy here is different. It depends what you're good at, and what you really want to be good at.

For your question - if you joined a new team, what skills would you really need? The answer is, it depends on what the team needs.

I think you might be focusing on the wrong details. The first step in taking your career further is to find a team that is a good fit for you. That means two things.

1. The team's mission is big enough that it needs a staff engineer (or another staff engineer)

2. The skills needed for the role of staff engineer on this team are skills that you have or can acquire

On some teams it will just be impossible for you to become a staff engineer, because there just isn't enough opportunity or importance, as perceived by the management. On some teams it will be impossible for you to become a staff engineer, because it will require a skill that just isn't you. So go find the right team, and then it will be far more obvious what you need to do to become a ridiculously good backend staff engineer.

Feel free to email me if you'd like to chat more, I have been the engineering manager for a number of people who have gone through the senior -> staff period in their careers....

Thanks for the reply, this reframing of the question is insightful. I'm definitely more in the camp of generalist backend with some political savviness, because I tend to enjoy collabs and getting things out more than a technical deep dive. This leaning does make me question the utility of my skills vis a vis the more technical depth that a lot of folks acquire working in focused areas of interest. I guess I hadn't thought of politically / generally effective as an archetype for higher level eng positions, especially since that seems harder to translate when moving to new teams and companies. Folks have described the 'T' engineer as the most desirable shape for eng, and I assume the depth in that T is some particular tech niche or stack that you should be building out to grow.

> Feel free to email me if you'd like to chat more

Appreciate this.

The depth in the T does not have to be technical. The word "political" is often not specific enough and people think "well I'm not political so that isn't me" so maybe I can give some more detailed example scenario of how someone could give a staff engineer performance without being a narrow technical expert.

Let's say... an important piece of business functionality is divided between five backend services, each of which has its own engineering team. For the past couple years there have frequently been either performance problems or system-wide outages and nobody seems able to resolve it. Engineer X is on one of these five teams. They organize some processes, find some key people on each team to work with them, and convince the relevant managers to allocate resources to their project to improve reliability. The initial project is a success. For the next couple years, usage of these backend systems goes way up, since they are important to the business and now much more reliable. As usage grows, naturally reliability problems keep popping up, but Engineer X keeps their cross-team process operating and engineers participating in the process rapidly fix the reliability problems.

You could call these skills "political" but it might also feel just more like "being organized and communicating with the wide variety of stakeholders".

Thanks for the concrete example, I can relate to this much more. It seems much harder to know, when interviewing new teams to join, whether this particular kind of skill set would be useful or valued, as opposed to being an expert in some technical niche.

When interviewing, tell the team's manager that you're really interested in working somewhere that you can have a big impact. Ask about the most important projects they have upcoming and about the biggest problems the team is facing, and politely dig into it. Once you are aware what the biggest problems are, you can usually theorize without too much trouble what skill sets are needed.

Thanks for these insightful posts. I’m in a role that is exactly what you described above. (Perhaps prior to the “initial success” phase, due to political challenges.) It feels like I’m tilting at windmills in a management vacuum. Any general thoughts on how to detect in advance if these kind of projects are worth pushing through or if one should deftly sidestep them? Seems to be an unfortunate but necessary corporate skill.

Yeah it's an interesting balance of asking per permission and pushing forward & begging for forgiveness. There's no true answer and it really depends on trust + culture + teams you're interacting with.

I've often error-d on the side of side stepping once I get a feeling that people have enough context on what I'm attempting and won't hate me for it :) even if not green-lit by upper management..

Get to be better than average at picking projects and tasks that have identifiable, meaningful deliverables. A history of highly evident impact is a huge part of a personal brand of success. Learn to avoid projects where the best outcome is "fail less hard." Avoid being brought in as life support. Don't be left holding the bag.

Obviously, green-field projects hold the promise of easy prominence. But you can learn to spot other opportunities that aren't as obviously sexy, yet get you name recognition.

Yes, I had the same conclusion about careful project picking and greenfield projects did seem to hold more promise than others. I've found that its kinda hard to get on prominent greenfield projects as one of the founding members though, without some 'in'. Many of the ones I've looked at, by the time they start staffing publicly, have picked a bunch of the staff level folks via networks. My experience might be narrow on this though.

I meant in your place of work. It's still generally the case that you won't be picked for flagship projects, but you can get better at finding remaining opportunities than others are.

My personal experience - after nearly 3 years at my current employer working mostly on features and bug fixes for existing big projects, I've found myself at the centre of several greenfield projects that fit my expertise perfectly. I am now one of "the guys" to go to about these exciting long term projects, which is exactly what I was looking for.

Unless being life support if what you want to be known for: the superhero that can work with shitty code bases and get results while improving them. Guess what most software company have: plenty of shitty but mission critical code!

There's nothing as uniquely shitty as getting placed on a floundering project, being told it can't be allowed to fail, and then getting absolutely no staff or resources to prevent that from happening.

Take time to reflect. A lot of folks focus so much on doing that they don't take the time to take a step back and see what's working and what isn't. You might be shipping a lot of good stuff, but maybe you're making assumptions too often, or making the same mistakes over and again, or simply not taking the time to just _think_.

Take time to share and listen. Teaching, mentoring, coaching, etc. others, especially those with less experience, is good, but is often done as a one-way transfer of information. Instead, find people who ask good questions (those with the least experience often ask the best ones) and be open to exploring those questions together. It's hard, because you will base your answers on your experience, which is by definition limited (it is for everyone), but it may not be the right answer in all situations.

Do you have a good way to identify and question your assumptions? You're right in that there's learned patterns that have worked well in situations which I'll keep applying moving forwards, although I do try to enumerate the trade-offs. There's probably blind spots there.

> Instead, find people who ask good questions (those with the least experience often ask the best ones) and be open to exploring those questions together. It's hard

Great call out. Tangentially, if I'm being honest, not knowing answers to basic questions makes my insecurities surface, definitely a character flaw to work through.

> A bit of a click-baity title, apologies.

On the contrary, your attitude of "I want to become ridiculously good at this" will carry you 50% of the way there. Good luck!

I haven't been around the block a lot nor do I consider myself as "ridiculously good" but from my experience shipping is king, every time.

It's very easy for us to focus on the tech and think that more knowledge, more performance or whatnot is the next level. While we have some superheros (John Carmack, Dennis Ritchie, Dan Abramov) that are famous for incredibly focus and deep technical skills, for the rest of us, we are evaluated on getting stuff done, reliably and consistently.

There are, of course, situations in which "shipping" has to do with performance (I once had to rewrite and algo from python to rust to gain about 100x performance), but in those cases it's often explicitly defined as a business requirement, i.e. more revenue or lower expenses.

Another tool in my toolbox has been understanding better the "business" of coding, that is, how does software engineering work fits in the "supply chain" of a company, from idea to the sales guy making cold-calls. Two books come to mind, "The Unicorn Project" and "The Phoenix Project".

At any rate, the desire to keep improving yourself, growing and learning it laudable!

I found "The Phoenix Project" a great read, although the learnings there were things I had picked up from places I had worked at. Haven't read "The Unicorn Project" yet but it's on the list.

If you pay deliberate attention, you will find that some problems affect projects dramatically more often or more severely than others. If those problems were easy to solve, they might not exist. If you learn to solve them, you will be a lot more valuable.

However, since this isn’t about listing trendy techs on your resume or l33tcoding, I don’t know if it will make you more in demand or not

Don't make simple things complicated. Simplify complex things. backend developers generally like to build complex crap.

In addition to the good advice already posted, some links here that could perhaps be useful for you: https://news.ycombinator.com/item?id=28007862 (mainly the first two links in the list)

Thanks! I really like Julia's blog, I hadn't seen those entries before!

My pleasure! Her blog is really great.

Internalize DDIA and go deep on some topic (databases, event sourcing, kubernetes, etc) that interests you.

1. automation. Write code that removes humans effort.

2. do hard things. Solve problems other people are incapable or unwilling to solve.

3. practice. Rinse and repeat until you get it right. Be insanely persistent.

4. humility. There is no secret sauce. There is no magic toolset, framework, or assistant to do your job for you or cheer for you. Toil in silence and repeatedly fail until you get it correct. That pinnacle of success is the expectation not an achievement.

read this https://staffeng.com/book and decide which type of staff path you want to follow.

the author Will Larson https://lethain.com/ goes into detail about all types of staff engineers

also follow him on twitter

The non-tech stuff seems to be most important at this level. One thing I notice some senior SWEs lacking is instinctive understanding of how prioritize and which tradeoffs to make to maximize their value. Beyond that it seems to be almost entirely political / luck of the draw. Figure out how to play the game.

Learn databases.

Reply on news.ycombinator.com