Hacker News Reader

Comment by willvarfar | Hacker News Reader

Its interesting because, although I don't know the linkedin codebases nor worked at linkedin, I've seen a few codebases that sound scarily similar and am familiar with a few companies that sound organisationally and politically scarily similar.

And I generally advocate 'finger gun' approaches.

Finger gun rewrites can be implemented in a good way. Often when you have n clients trying to do the same thing, one of those can be used as a basis for serving the other platforms. Or if you start over, you can do it in a good clean quick concise way.

The general success ingredient is putting a small team of veterans on making the new system. Success always comes from a small team of engineers who are both domain experts and technical experts rolled into one. (Controversially, I think that generalises; all success comes from this, even for mundane keep-the-lights-on problems. And everyone else is just slowing them down.)

The big problem most tech management everywhere keeps repeating is that the next big system will be built the least-experienced.

Would be really nice to hear the symmetrical interview from a finger-gunner.

View on HN

>The big problem most tech management everywhere keeps repeating is that the next big system will be built the least-experienced.

Most places I've worked at, this ends up being the case because the most experienced are needed to fire fight and to keep things running.

Stop describing my workplace

Stop being the least experienced!!! (by tenure alone)

> (...) because the most experienced are needed to fire fight and to keep things running.

By that account the most experienced started out as the least experienced who built a mess, accrued more technical debt, and built up their experience keeping their ball of mud from crumbling.

They are also more resistent to change because this underlines all the crap they did. It's their baby, and their reputation is pinned on how successful it is.

New is not necessarily better, but old and stable also doesn't mean fine engineering.

>started out as the least experienced who built a mess, accrued more technical debt

THIS

"experienced doesnt always mean good. This is why you build teams.

The CFO is your enemy as an engineer.

>The general success ingredient is putting a small team of veterans on making the new system.

I don't disagree with this approach. But one thing I find is:

Sometimes those veteran teams are heavily invested in existing concepts, tools, and approaches. It makes sense, they have seen these things work and they're proven to some extent.

But it's always not the best choice.... and while the veteran teams often start these projects they often aren't the people to "finish" them as they get pulled away. And it is oh so easy to start projects and make decisions if you never see the outcome / have to deal with the fallout.

> Sometimes those veteran teams are heavily invested in existing concepts, tools, and approaches. It makes sense, they have seen these things work and they're proven to some extent.

This is very closely related to why these veteran teams end up successful. Existing concepts, tools and approaches should be the default; if your company uses Python with MySQL you better have a really, really good reason for your new project to use Rust and CockroachDB.

> while the veteran teams often start these projects they often aren't the people to "finish" them as they get pulled away.

I disagree with this. The veterans are veterans because they've already spent time maintaining the existing systems. They've handled oncall, major bugs, and seen all kinds of stupid shit that happens with software. They'll get pulled onto other projects eventually, sure, but they have a higher chance of getting a system into production in front of users, IMO.

Some changes are better than others. If your product works in python/MySQL then rust probably isn't a good fit. Maybe Ruby or PostgresSQL would be better fits.

If your main product is C or C++ then you should be learning about things like rust. Maybe you decide modern C++ is good enough, but if so you should be working with the C++ committee to push safe C++ (as a C++ developer I'm hoping this comes in C++26... I'm looking at how I can put rust into my code for now).

The point is you should know what the good next steps to investigate are and what are not. Rust and Python are both great tools, but it is never okay to choose between the two on any project - the choice is obvious before you get to choosing. Both tools have other competitors that you perhaps should be looking at though (Rust - ada, C++. Python: ruby, go)

> Some changes are better than others. If your product works in python/MySQL then rust probably isn't a good fit. Maybe Ruby or PostgresSQL would be better fits. [...] Rust and Python are both great tools, but it is never okay to choose between the two on any project - the choice is obvious before you get to choosing.

I actually really disagree with this. Python and Ruby are extremely similar in the problem they solve. If you're using MySQL as a basic OLTP database, and you need a basic OLTP database, you shouldn't use Postgres.

I'm talking organizationally here, not just on a single project. You gain nothing by introducing Ruby to your company that's filled with Python experts. Instead, you introduce a bunch of friction and overhead as developers have to relearn things like syntax, how to set up their development environment, how to do dependency management, testing frameworks, etc...

The time to introduce new tech is when it is fundamentally different enough that it could matter. Maybe you have a bunch of Python code, but there's a certain area that's very performance sensitive: you could look at using Rust, Go or C++. Maybe you have MySQL, but you need a database that's used for complex OLAP queries rather than low-latency OLTP work. There you could look at Redshift, Snowflake or the various Postgres OLAP configurations.

Where experience comes in is knowing when these new tools are worth the overhead they cause. Less experienced engineers vastly underestimate the overhead that comes with introducing new tooling. This even applies internally; for example, if you've used Flask for every project you've done at your company for the past 5 years is it really worth switching to Django?

> The time to introduce new tech is when it is fundamentally different enough that it could matter.

I think you agree more than you realize. GP is essentially saying there are few situations where you will be deciding between Python and Rust on account of organization and technical reasons. However, sometimes you should consider introducing new technologies, but only when you identify that the problem being addressed is much better solved by a new technology, fully aware that this will cause various sorts of friction within your organization.

I disagree. There's no magic to being on call that means you understand something well enough. That's simply not automatic.

Following through with something from start to "finish" is far far more valuable than fixing one off bugs and then doing something else, and then something else.

If anything I think it can lead to a lot of poor assumptions based on very skewed experience.

Depends on the nature of your oncall. If your oncall is just following documented restart procedures when your app gets stuck, you won't learn much. If your oncall involves deep investigation into your system and how it interacts with other components (services, databases) etc you'll learn a lot.

per some sibling comments though, years of experience != veteran, to be clear. There's certainly people who have been working for 10+ years, sometimes even at the same company, and don't really understand what they're working on.

I think the implication is they’ve seen all problems before and can engineer their new system to solve them.

Most software is never finished so I have no idea what arbitrary point you’re deciding they need to see it through before qualifying here.

> But it's always not the best choice....

I think it comes down to the personality of the veteran. If they're the "i've seen some shit, so I'll 'architect' the fuck out of this to handle any type of change" type veterans, then it's likely to fail. But if they take their knowledge of how the system has grown over time and architect to handle the types of changes they've seen, then there is likely a greater chance of success.

Personally, I think a having some junior and mid-level engineers on the rewrite is good because it's a good way to see how the system is all put together (as opposed to just one area) and they'll be able to have conversations with the veterans about how the system has evolved as it grew and that is valuable information.

Meh. I’ve met brilliant juniors and worthless veterans. I think it’s more important to recognize brilliance in people… That spark of hard work combined with vision and creativity.

Yeap, good point, tenure != expertise.

If I could edit the post I'd insert a qualifying "expert" before veteran, where the expert veteran meets my definition of the 10x developer :D

Any veterans who aren't gonna get the project done quickly _and_ properly aren't the kind of veterans I have in mind.

There’s definitely a middle ground between ‘we need to know how this plan will overcome every obstacle we will encounter before we even begin’ and ‘there won’t be any obstacles what do you mean?’

I always want to have a sense that the plan gives us realistic options for dealing with the obstacles. And that doesn’t just mean technical options - we’ll need to have the people to tackle it too. Eg it’s not great if the plan involves a bunch of teams running servers, and while technically there are obviously ways for all those teams to operate and scale those servers, if they don’t have the time or skills to do so then that option of having the teams run stuff is not really realistic.

But on the other hand planning a careful course that navigates round every obstacle is just as bad because

1) the obstacles will likely have moved by the time you get there (eg we spend ages figuring out how teams on legacy JS will be affected by the project only to find by the time we get there everyone has already moved to typescript and all that prep was YAGNI).

And 2) there are probably obstacles right in the path you’ve carefully chosen that you don’t know about yet and when you hit them your plan will grind to a halt because there was only one way planned and this was it.

We’re reading a cartoon podcast description of a complex architectural debate here though, so who knows which of these straw men was actually close to what was happening in LinkedIn.

This is a great summary. Obviously I bring my own perspective to this, because, well… it happened to me, and I “lost”. But my read was that the folks I disagreed with and ultimately parted ways with were falling into the bucket you label ‘there won’t be any obstacles what do you mean?’.

The tradeoff around “do we migrate to TS?” is a great example here, by the way. A huge part of our decision-making last year around the big app at LinkedIn was on exactly that question: If you are going to stop and throw the entire thing away, you should waste exactly zero time migrating any of it. On the other hand, if the thing is going to be migrated incrementally, you should accelerate the TS effort because it will make it so much easier to code-mod the code base.

(I don’t agree that it’s a “cartoon” podcast description, but it is definitely extremely compressed and Adam and I chose to focus on the personal dynamics over the technical details, because diving into the latter would have made for a 5-hour episode. Also: it’s really tricky to do any of this kind of public discussion in a way that doesn’t just end up reading as a one-sided self-defense!)

> putting a small team of veterans on making the new system

I talked to someone who was leaving his job because he did this in order to meet a critical date for a product launch. The fact that his group of 3-4 people did what the company could not in a year ruffled so many feathers he said the board told the entirety of engineering that under no circumstance would anyone be allowed to dictate any detail of future projects.

Yeah, I've been these "cool Skunkworks" teams and I literally made no more money, maybe got a $5k spot bonus (super rare), and burned myself out over a 6mo launch. Spent my entire thanksgiving week on-call putting out fires, literally every hour, while everyone else at the 800 person company was on vacation. Not a single executive or product manager involved in launching was there or checked in on things. Some moron (I hope he reads this, my nicks not anon) let us launch without connecting our credit card fraud system so we (ops, me) got slammed for an entire week with fraud server launches. Jason, you're a moron, and that's an understatement.

It's like being paid in exposure. We'll pay you in cool tech and learning!

Then you see the 20 go devs around you making the same money writing grpc services 8 hours a day home at 5 relaxing.

I'm not really sure how this relates. The story I was told had the guy working as the lead with several less experienced but still competent programmers for around 2 months. There was no crunch, burnout, or anything of the like.

You're being downvoted because most devs are blue collar and hate the idea of passionate talented devs doing something amazing quickly and effectively. Any anecdotes of small elite team success must be crushed.

If you are working more than 8 hours a day you have nobody to blame but yourself I think. Especially in a situation like that.

To me it seems crazy that a board would be getting their hands dirty on engineering "specifics". I would think you would have engineering level leadership making those calls and the board would either trust that person or should move on.

Anyway:

I will see that I always worry about super teams because super teams tend to get free rein, so of course they can be faster and more successful. But sometimes that's about the bureaucracy and not even the team members.

I've started projects with a team and due to time factors decided "I'll just take all of this." and it worked out great. That's not because I'm amazing... it's just the old adage about "What one programmer can do in one month, two programmers can do in two months".

Unfortunately, I've seen places where these super teams / people are setup and endlessly praised and it goes on and on and it becomes very clear it isn't them, it's the structure.

let me tell you a simple story

1. team A developed some really nice looking graphs from customer data we already collected

2. a member of team A pointed out he is colorblind and the palette did not work for him

3. product manager took the time to research this and figure out how we might make the palette configurable for those who are colorblind

4. CEO got wind of this and called everyone involved into a meeting. The decision of the CEO was no one would ever use colors for any data presentation. The only allowable color was a shade of gray

I'd like to point out that using only a shade of gray for "colors" effectively makes the entire planet colorblind

As someone who is colorblind I find shades of gray better than the colors I often see. OTOH, I can see most colors just fine, it is just that some colors most people think are very different look the same to me. There are a handful of people on earth who cannot see colors at all.

The article is not very specific so I can't really tell one way or other

5 year plan is sounding really bad, though

Yeap! We can't predict the future but we can all be very surprised if whatever tech is picked today is still a good fit, or even still exists, in 5 years. Especially in the javascript space.

As a sibling mentioned, this isn't really true in the JS space anymore. The JS frameworks and react/vue have pretty much won, so now you one of those UI kits and whatever build system/ancillary tooling your framework says you use. There might be churn under the covers, but the framework abstracts that away for you (though occasional migration issues are bound to come up with big changes).

it's always quiet before a storm :) It'll be fun to fast forward five years and see whether there wasn't some big disruptive change and everyone is now using rust++ for client dev or something? :D

This is becoming less true than it once was. React has been ‘this month’s JS framework’ for 10 years now.

Anecdotally React was not in serious contention (for corporate use) until about 2016 or 2017.

And React itself has not remained a stable API (rise and fall of hooks).

This is a reasonable take, and I mostly share it. None of us were especially happy about that plan! However, and I alluded to this on the episode, it wasn’t actually a five year plan to get to a very specific endpoint. It was, instead a 3–5-year plan to make steady incremental changes to our entire system in ways that meant that if we realized our end destination needed to change (e.g. from React to Svelte or something not even built yet) 2 years along, that was totally fine.

It's not clear to me what ended up happening. Did they start a rewrite from scratch in React?

They started a rewrite-from-scratch of all the apps, including mobile, using a custom in-house server-driven UI framework with a Kotlin DSL backing it, without so much as doing a proof of concept to find the tricky parts first. And by “started” I mean “committed to it” before anyone had so much as written a line of code to spike it out. Which, YMMV, but to me that seemed then, and still seems now, just wild.

Yeah. I hope they're at least doing it gradually. As in, rewrite a small component and have it run side by side with the rest of the original app, then rewrite another one and so on.

Oh.

Well, that sounds horrible, and will definitely take more years than rewriting it to React.

It's more FUN though!

Most of the time, you have to deal with codebase. And when you finally have chance to work on a greenfield project, you will have to compromise between your opinions and wishes with those of everyone else who is involved.

> when you finally have chance to work on a greenfield project

the person that comes along next is going to moan about how much of a mess of compromises and hacks your, now, legacy codebase is

I have never heard of a finger gun rewrite - what is that?

From the podcast:

> It was completely what another colleague of mine once described as being in finger guns mode, meaning like, yeah, yeah, yeah, this is going to be awesome, man, kind of finger gunning at each other without answering any of the kinds of questions about what does it look to like to operate this when we’re trying to support hundreds and hundreds of engineers…

Gotcha thanks.

> the next big system will be built the least-experienced.

This sounds plausible and is consistent with my experience, but I'm curious if you have a reason for why this is? Is it just that the experienced folks are already working on the established systems, and we don't know what the next big one will be? Or is management doing something intentionally that causes this? Or something else?

Perhaps the experienced folk are "difficult" and have "opinions" about the best way to write the new system which is somewhat at odds with what management think they they need. Difficult opinions may include "no, it doesn't need to be 'AI powered' or include a blockchain or be a combinatorial explosion of configurability".

The inexperienced folk will be more "cool yeah let's do it". You can't blame them, it's how to become experienced.

Tbh this was kind of my conclusion as someone new to the field (23). In technical consulting, I feel like this phenomenon is even more pronounced since A. most technical consultants don’t get the time to become domain experts in their client’s business, B. most technical consultants aren’t entirely technical experts (consultants first, developers second), and C. consulting management has a vested interested in keeping things this way because they don’t have the tools to make things different. Maybe this doesn’t happen as much in boutique consulting firms but I wouldn’t know (hire me if that’s the case).

It’s given me an idea for a two-way portal where on one side, technical developers join the platform, self-organize into independent, transparent teams (not just developers but any role in development e.g. UX designer, project manager, etc) and on the other, companies join, post their potentially cross-functional work requests which are then distributed to the relevant teams. Another important concept is that these instances can be federated(?) so existing consulting firms can integrate this new kind of work allocation into their pipelines (only their consultants/their invited companies). There are a bunch other concepts I’ve thought through but that’s where I’m at now.

I feel that a system like this provides better value to a company vs. traditional consulting. First, the responsibility of team formation, networking, and allocation is moved to the service. These three things are hard for humans to do themselves and are responsible for so much consulting admin bloat/inefficiency and companies have no choice but to play. Second, it in theory improves the quality of work teams. With this focus on smaller, self-organized teams and a UX that allows for transparency I.e. companies can view teams, their members, and track record, companies know what they’ll get and not need to roll the dice to see what available resource gets assigned to them. Moreover, I’ve read that consultancies have a tendency to throw certain resources on a large work item where they exist to just to cash in budget and write bad code. This problem is completely mitigated with this approach. Finally, it’s a decentralized work environment (I think). Besides OSS, I don’t know what other platform allows for developers to have so much autonomy and self-determination. You could argue sites like Upwork or Freelancer but these platforms seem more focused on small, individual jobs rather than larger work items which justify having larger, more competent teams.

Without negating your idea, I’d add that as a (non-IT) engineer of some 15 years, I’ve found that my first 8-10 years was building my technical capabilities, however beyond that it was really about learning how to navigate (at increasingly more complex levels) the business systems and fit the technical to the business.

Being a principal engineer, my major value is not in my technical capacity, but rather my ability to shape the technical solution to the best fit for the business - be it to do with existing tooling, processes, systems, etc. Though I can do the technical solutions (probably more efficiently, too), my value is much stronger when I steer multiple junior engineers doing the technical work whilst setting the guideposts so they align the solution to what best fits the business.

I would therefore suggest that one major hurdle would be that these self-organising teams might not be able to bring to bear their full capacity insofar as they are not familiar with the businesses to which they pair. Though you can sometimes see parallels between company processes, understanding the nuance and detail that makes them different is where so many consultancy projects fall over. Think about the canonical meme of a team of consultants coming in and spending millions developing a solution that turns out to be a lemon. It’s not for their lack of capability, it’s that it’s devilishly hard to really, truly fully understand a business’s processes, systems and cruft in a sufficient level of detail to truly shape a solution such that it remains maintainable and fit for purpose in a long-term sense.

That’s the real reason why veteran employees are worth their weight in gold. It’s that they are the custodians of the institutional memory and can bring that forward when it is needed.

This of course has the important counterpoint of the necessity of seeding new ideas into a business, and that’s an entirely different essay in itself, however suffice it to say that new ideas need to be integrated in a sustainable and controlled fashion - protected and nurtured from being driven out by the existing ways, but also moulded to integration where it’s necessary. Failure to do either is a speedrun to pain.

Edit to add: If you think that IT is somehow immune from this relative to traditional meatspace engineering disciplines, I’d say two things: 1) tech still goes through this, just sometimes operates on different timescales, and 2) go see if a COBOL greybeard collecting FU consultant wages to maintain old OT systems agrees with you.

For what it’s worth I don’t think what you’re describing here is the same thing I meant by “finger guns” at all! What you’re describing looks to me like good incremental development.

With my “fingers guns mode” phrase, I really had in mind a blithe disregard for any of the real-world engineering problems and constraints that we knew we would hit (because they just inhered in what we knew the solution needed to do). You don’t have to solve those right up front, but you do need to (a) acknowledge that they exist and (b) have some notion of what your approach to them will be.

I’m extremely in favor of taking a small team of veterans (who are open to doing something novel!) on a new system, as you say—let them leverage all their expertise, while being careful to avoid Second System Syndrome, to build something by using all of the knowledge about those pitfalls/problems/etc. to avoid them. (The risk there is that you can end up overfitting on previous experience! That’s real!)

>The general success ingredient is putting a small team of veterans on making the new system. Success always comes from a small team of engineers who are both domain experts and technical experts rolled into one. (Controversially, I think that generalises; all success comes from this, even for mundane keep-the-lights-on problems. And everyone else is just slowing them down.)

Fully agree with this for all scopes of projects. Larger projects are compositions of this principle with functional communications between focus areas.

There's no financial incentives for Maintenance though, which means "leaders" have less and less incentive to fix what they have versus building from scratch. In my experience in these cases it comes down to (like most things) whomever the least flexible person in a power position decides to do.

Fundamentally the problem with all large organizations is that desires and incentives become diffuse the more people are a part of it and the "alignment" of what should happen drifts, until eventually nobody cares to maintain the original purpose.

You can argue this is good or bad or inevitable, but either way it's a structural issue between the broader world changing, and a organization that seeks consistency

I'm also generally a finger-gunner. I don't identify with the stuff OP was criticizing though; in particular I want people to poke holes in the plan: those are usually requirements we need to meet or at least talk about why we don't need to meet them. But yeah, other than that, I generally think it's easier to rewrite things than it is to understand the intricacies of a gazillion-line software project. That probably strikes a lot of HNers as problematic (this is why we keep getting new JS frameworks every day! this is why we keep reinventing email! this is exactly why we keep building gazillion-line monsters!), but I don't really mean it like that, I mean it more as a manifestation of Gall's Law [0].

To me, it's all about energy. If the energy is behind "fuck it, rewrite it", then that's what you should do. If the energy is behind, "preserve the valuable IP in the existing code" then that's what you should do. Actually I don't think it's even "should"; I think it's "will".

> Would be really nice to hear the symmetrical interview from a finger-gunner.

I found the initial framing of like "expediency vs. the right thing" not quite right. My read was that Chris just found fault with the finger-gunners' bad engineering. Here's a really good excerpt:

---

Why doesn’t Code Review just solve this? That was a direct quote from Jim. And my answer was, well it didn’t, and it won’t again in the future because people are going to make mistakes and just be better. Again, it’s not an answer, because what happens when it’s some junior on the rotation who thinks, yeah, that seems reasonable, and this PR was made by a very senior SRE?

Why am I going to question whether that value is reasonable? Like, yeah, there’s a bunch of boxes. That seems fine. Those kinds of things happen. And to that question of engineering systems, one of the things I think about a lot is, does our system only work or does this process only work? Or does this tool only succeed?

If I’m acting like a senior engineer, on his or her very best day or, or does it work if I’m a super junior engineer who’s having a bad day and we really want our systems to be workable for the latter case. And that helps all of us because sometimes, even though I’m a very senior engineer, sometimes I have bad days.

Sometimes my brain doesn’t feel like it’s working at all. Does the system still support me on those days or does it punish me? I really would like to not be punished.

---

I'm gonna try to be polite here, but Jim's "why doesn't Code Review just solve this" take here is shockingly naive to me. If he doesn't know that defects slip by even the best engineers with the best code review processes, I have to question his experience and judgement. Chris' response of "well it didn't" is all that needs to be said: we empirically can't rely on this system. Failing to recognize that is malpractice. The length Chris goes to to explain why it's malpractice speaks to his professionalism and patience, but yeah it's clear he was in a situation where someone pretty incompetent was now calling the shots.

[0]: https://en.wikipedia.org/wiki/John_Gall_(author)#Gall's_law

This might be No True Scotsman, but it doesn't sound like you are a "finger gunner" after all.

After (fairly) long time in the industry, it seems to come down to:

- understanding what one's clients need (not what they ask for)

- understanding what's in the realm of the possible (technically, politically, organizationally)

- understanding what are the possible futures (good, great, and abject disaster)

If, after answering all those questions, one still advocating for a rewrite, you've shown the maturity needed to undertake the effort.

"Finger-gunning" is usually related to skipping those steps, not the ultimate decision.

(edit for formatting)

Fair! Yeah I mean, I will say I don't need a complete plan, but unknowns make me nervous, and maybe a true finger-gunner never doubts their ability to figure it out.

Reply on news.ycombinator.com