Hacker News Reader

I worked on a project that was paying the bills while another team was getting to work with new cool stuff.

What I learned from that project is that being indispensable sucks. That lesson has recurred in my career either for me, or when I got wiser, for other people.

It's not enough to write reasonable code that 'works' or document it as it is now, there's a whole unmeasured and thus under-encouraged aspect around maintaining a legible edit history. I often am trying to get into the head of the bug author to figure out why they thought this was a feature in order to preserve functionality while fixing bugs. They aren't always around, and they aren't always fun to talk to. If you have the history, the person is free to work on other things. To gift them to people they mentor, or otherwise re-home them. If you don't have that, if it's all in their head, they have to leave to get away from the project.

If I can figure out what you're doing I can figure out how to change one or two conditional blocks to fix a problem. Your code stays intact. If there's no history anyway, and the code is sketchy, I'm far more likely to take the code away from you by rewriting it. We can't have black boxes on the team. That's a ball-hog, and ball-hogs get benched.

If you don't show your work, then you're going to find you have fewer and fewer interesting responsibilities. Because "how we got here" is also a "why" and we ask Why all the time.

I agree, but am often outnumbered by people who enable "squash commits on merge" and that simultaneously add many process barriers against actually merging pull requests.

Everyone wants to be indispensable until they don’t. Getting people to plant two years ahead is an eternal struggle.

I want a meaningful commit history, that actually enables me to find the useful commit message. My coworkers often create PRs where the branch itself contains merges and a lot of bad commits that are not centered around a single piece of functionality or activity (like formatting, refactoring).

I try to have them clean it up but often the best thing I can get is a squash.

I defend atomic PRs and squashing merge commits.

It's simple, makes reviewing easier and works for 90% of the cases.

Maybe I'm missing something, but I find it painful to rebase into a neat history, and my colleagues would probably hate me to police them around to do it as well.

PRs should be small units of change. PRs should be the edit history. If your process involves one PR per feature this makes them large and unwieldy, hard to review, hard to test, and more likely hard to rollback.

I commend your attempts to do CI and PRs at the same time, but most places have too high PR friction to make that work.

So it becomes a fight between the two.

> How does this interact with individual or career growth? I don't want to work for the manager who insists I stay on the first few projects I worked on at the company forever...

Grow a garden, not a tree. Let's disambiguate the word "project" from the word "system" (not "product" -- that's sales). Let's call a project a time- and scope- limited effort to create or modify some system(s) in your suite (your garden). I think it's a common mistake to mix these two up. If projects just drag on forever, then yes, employees can feel stuck. By this definition, I certainly would not want to work on one project forever.

What about working on one system forever? The problem there is that to stay busy, you're forcing attention on a system that might not need attention. At the end of a project, a system should be fulfilling its purpose, and it should do so indefinitely, forever. That purpose may expire, but the system shouldn't. There should be no urgency to keep messing with a working system.

So let's work in one garden forever. Systems will grow; we'll prune and re-plant them; we'll give the healthy ones the barest maintenance it needs; we'll give the dying ones the sweet release of death; we'll shape and braid them together. All your institutional knowledge about how System A works will be great when you're writing System F that depends on System A, or when we're splitting System A into Systems A1 and A2. I would absolutely love to work at a company where I can work in one garden for the rest of my life. That's my dream job.

Sidebar: do not make the mistake of assuming "system" means "microservice"! That's only one of many ways to define system boundaries.

I agree with this - while at a single less-than-a-few-thousand-people-company I'd generally call that a single "garden" - and think it's a good extension on the original article. The author, though, focuses very narrowly on the "code" level instead of the "components of the system" level.

If you can grow people into being the ones who know how the components of the system work together at a layer above how the code for any given component works, you can have a retention plan and keep institutional knowledge. (Or even then move people into a layer above that, as the system grows, etc).

You still won't retain everyone, some people want to jump between gardens, but you will be able to retain a LOT of people, and it's much better than telling people they have to own all their individual lines of code forever.

> At the end of a project, a system should be fulfilling its purpose, and it should do so indefinitely, forever. That purpose may expire, but the system shouldn’t. There should be no urgency to keep messing with a working system.

Wonderfully stated! Losing sight of this is I think where projects start to run into trouble with scope-creep and the ”fix” vs “enhancement” issue. It also shifts the perspective of what is a satisfactory state for a finished project vis-a-vis maintenance plans, dependencies on aging systems, etc. Your garden metaphor is also beautiful and IME accurate.

I don’t wholesale agree with the writing but it does mention working with someone who worked on the project and can share the theory. In that case, a new dev is hired, and you instill all the knowledge of the system in the new dev. Once you’ve trained them enough, you can move on. New devs will adapt and create new theory and solutions. Rinse and repeat.

That's that "second gen" vs "first gen" thing but it actually leans in hard to the "no movement" right after introducing that concept - the "Each team needs to be composed of a majority of first-generation developers" quote is from that paragraph after the distinction.

If you try to fix the theory knowledge/documentation/practice problems purely through insisting the "first gen" stays where they are, they're going to leave you for more interesting work, and then you're in exactly the situation you wanted to avoid. The only way out is "more effective practice of theory and knowledge transfer" but all the author seems to have to say about that is "This is a team management issue that can only be addressed by a manager who understands the fundamental nature of software development. There is no magic development process that will prevent these problems. It takes skill."

E.g. that's the real hard part yet they completely punt on trying to dig into it.

That kind of misses what is being said in this article. The first generation programmers support a constrained population of second generation developers, and over time the second generation developers have written enough code that they become the first generation.

A big component of what this article is saying is basically that knowledge transfer in the way you're describing doesn't work, and if you dig down to the theory this article is based on, new devs adapt and create their own theory and solutions was specifically called out as a problem. In case 1 of this paper, it's described as second generation programmers submitting patches to the first generation programmers that don't make use of the structure of the program, and add things that are trivially done by the existing systems. Then, a third, disconnected generation of programmers takes over the maintenance from the second generation, and they've added so much cruft to the system that the original powerful architecture can still be seen, but isn't in use because of all the additions.

[0] https://pages.cs.wisc.edu/~remzi/Naur.pdf

> A big component of what this article is saying is basically that knowledge transfer in the way you're describing doesn't work

Knowledge transfer doesn't work at all :) It's not a thing that's possible. The best we can do is for someone with knowledge to attempt to explain it, hope they do a good and complete job of it, and then hope that the explainee is able to internalise it and come to the same/similar conceptual understanding that the explainer had in their mind.

It doesn't tend to be done very well at corporations. It's a skill that good teachers have developed after much practice, which isn't something we tend to hire for.

And it requires favourable conditions, and one of those conditions is not "it's my last day tomorrow and the boss asked me to bring you up to speed on what I've been doing these last six months. You only had 30 minutes free in your calendar, so... let's go"

In the underlying theory it's described as actually worse than that. A fully documented system explained by the first generation developers wasn't enough to transfer knowledge, the documentation only made a difference once the programmers had built a working mental model of how the software worked, and then they still submitted patches that would duplicate functionality because they didn't understand the system.

But this doesn't boil down to "you have to replace the impossible problem of knowledge transfer with infinite employee retention"?

Is that an improvement?

(I'd also argue that I've worked on systems where I've developed a better mental model of the problem than the first-gen people I inherited it from, because I as a maintainer was under less pressure to get anything that worked out the door initially.)

EDIT: and looking at the paper you linked there, https://pages.cs.wisc.edu/~remzi/Naur.pdf , I'm in complete agreement with "the designer's job is not to pass along "the design" but to pass along "the theories" driving the design" - it was always easier as that maintainer to take over a project where people could tell my why not just what.

Yeah, I was going to comment something like this : on a complex project even "first-generation" developers start out something more like """third-generation""" : after all the goal is to solve some problem, and the clients themselves might not even actually know what it is !

> they still submitted patches that would duplicate functionality because they didn't understand the system.

A lot of the time they don’t want to understand the system. Maintaining some old legacy system is no fun. Programmers want to do new things their own way. And if there’s no one around to guide them then they will.

This is a management and cultural problem. Enterprise software resembles failed open source projects - no one wants to work on them so they die, get forked, or re-written and replaced.

Contrast this with successful open source projects. A vibrant project will integrate new developers into the culture (or show them the door), and conversely developers who want to change things will have to prove themselves.

The Naur paper seems to take as fact that even if motivated to learn, taught by the authors, and given good documentation… new developers just don’t pick up a complex system. The paper used a compiler for some language Y onto a target system X as an example.

Personally, I don’t our ability to learn is that bad. Given time to come up to speed and motivation to read the code and understand it, and some good (consistent) coaching from people who fully understand the code, I believe new developers can be integrated into a complex codebase that they haven’t written.

But this really draws a stark relief from the way software engineering on existing code is treated by every manager I’ve ever met, which is likened to reading a book once in a language you already know. One read through is enough to grok it. If we land somewhere closer to this source material on the spectrum I think that’s a win.

I also look at my own thoughts on existing code. I’ve had the strong urge to refactor out old code I didn’t like… and that is interesting to me. Did I not like the style or just not understand it and want to make my own mental model my way?

The idea that we’re just replacing engineers and they are just re-writing the same code with the same outcomes because we think of them as Lego blocks is interesting as well.

> How does this interact with individual or career growth? I don't want to work for the manager who insists I stay on the first few projects I worked on at the company forever...

To me this is the a huge incentive for building strong teams. Nobody should be indispensable; it's a disservice both to the company and the person. Pair programming, collective code ownership, intentional cross-training, good automated tests, continuous deployment, and good automated monitoring have all been very helpful to me there. And vacations! Everybody should take long vacations where they are out of contact. By managerial force if necessary.

+1 on long vacations. My team is in Europe with 38 days of PTO and while it offends my American capitalistic sensibilities, it's created so much resilience because everyone is always on vacation, we've built our company ops to reflect this reality.

Like Netflix's chaos monkey but for people; unplug anyone from the system and the system should still work.

At least one CTO has actually tried exactly that: https://danlebrero.com/2021/06/30/cto-dairy-lucky-lotto-chao...

Is having longer PTO in your business a detriment or neutral/net gain to overall productivity in your experience?

Kind of funny that being allowed to live life outside of work is offensive to American sensibilities.

He speaks for himself

It doesn't "offend" "capitalistic sensibilities". It's just a result of optimizing the wrong metric.

Decision makers for American companies tend to use "Time spent at work" as a proxy for "Productivity". That used to work just fine when the job was pulling levers on an assembly line but turns out to not make sense when you are talking about software development.

I think the phrase "American capitalistic sensibilities" is well used here. American capitalism has a long history of optimizing for that particular wrong metric. It's culturally distinct from, say, Mittelstand capitalism: https://en.wikipedia.org/wiki/Mittelstand

> My team is in Europe with 38 days of PTO and while it offends my American capitalistic sensibilities

Do you actually think of yourself as a capitalist though?

Lead moves on to something else, everyone moves up a role, add a new junior. Repeat as often as necessary.

Alternatively: just promote and pay your people as they develop, even if they're in the "same role".

Because when you don't, your folks write reviews like this: https://www.glassdoor.com/Reviews/Employee-Review-Inspire-RV...

> How does this interact with individual or career growth? I don't want to work for the manager who insists I stay on the first few projects I worked on at the company forever...

The first project is maybe a bad example but if we say the "nth" project instead then:

- not always changing members doesn't mean never changing, companies should just not want you to jump between projects all the time (except maybe as junior programmer) and you shouldn't want this either as it's hard to build up proper deep expertise this way (instead of a lot of shallow expertise)

- if you are still in the same company and the company is healthy then it should be fine if you get asked questions about your previous project for some time after you switched (through not arbitrary, just as an "emergency" thing)

The problem is that the concept of carrier and seniority is seriously messed up in the software industry. Often shallow knowledge about a lot of things is more valued then actual expertise in the things needed. Similar you will find a surprisingly lot of people believing that a senior engineer which doesn't try to get into team lead/people managing positions is doing so because they "surly are incompetent and shouldn't be hired" (IMHO completely rediculus but I have still seen it often enough, just most times in a less directly phrased way). But in the end senior software development skills and team lead skills are distinct skill sets. Implicitly pressuring someone who is good at solving technical problems into a team lead position (or the other way around) is just a pretty stupid decision from a human resource utilization POV. (This depends a bit on the definition of team lead and yes team leads also need to have technical skills but most important they need to have the skill to listen to their team and not get stuck on their opinions, etc.).

I've worked in incident/crisis management for several big, complicated, grown-via-acquisition companies with a tangle of internal systems that were never quite fully integrated.

When TSHTF and we need some expertise from the original builders or operators of a system, it's a lot nicer to have them on another team than at another company. If they still work for us, somebody can make the case to them (or their boss) that borrowing them for a few hours to fix it is in everyone's interest. A lot of times they are in some kind of management position now and they kind of smile and say "Wow, I haven't thought about that system in a long time. Sure, I can take a look." and it's even kind of a fun diversion from whatever budget meeting they were in.

If they work for somebody else, the answer is often "sorry, I'm busy" or just a hangup when you explain who you are and why you are calling.

An employer shouldn't want that for you either. I think what this ends up being is an argument for good documentation and onboarding so you can transmit the "theory" to other people.

Reasonable amounts of churn are manageable if you do a good job of acculturating new people into the team. This has actually been one of my concerns about the switchover to WFH. I know a lot of that acculturation used to happen by osmosis and people working next to each other. I think my team doesn't do as good of a job maintaining it when we're all working separately. We're a lot more efficient now, but I think we haven't figured out a good way to share and transmit context like we used to and I worry what happens if we hit a period of flux in our company. I worry our team culture is a little more fragile now.

> I know a lot of that acculturation used to happen by osmosis and people working next to each other.

I used to think this too, but I think we need to be cautious of magical mechanisms for knowledge transference. I think the real mechanisms are code reviews, discussions of issues+code+project, consults / shared problem solving, design reviews. Just sitting next to someone doesn't work, there needs to be a shared task or problem for knowledge transfer to happen.

I think all of these shared tasks happen with remote workers just as effectively, as long as you have good team comm hygiene.

The biggest difference between remote and in-person is that many of those things can happen more easily without being scheduled. The organizational management skill bar you need to hit as a team is lower when the communication friction is much lower.

Talking to your teammate who sits nearby in person is both lower friction and higher bandwidth than sending them a chat method.

"Adhoc conversations" don't magically lead to knowledge transfer, the probability of any adhoc convo being long-term-meaningful is low, but they greatly increase the cumulative probability.

> How does this interact with individual or career growth? I don't want to work for the manager who insists I stay on the first few projects I worked on at the company forever...

By documenting your work and/or mentoring a junior employee to pass on your tribal knowledge.