Ask HN: Where to learn about driving engineering change in large organizations? | Hacker News Reader

Ask HN: Where to learn about driving engineering change in large organizations?

8 pointsgreymalik4 comments

I’ve helped establish best practices and healthy engineering cultures in greenfield scenarios, when I was the first technical hire in an early stage environment. I’ve also come into a small company with significant technical debt and introduced changes to the engineering practice in ways that were embraced by the legacy team. But I’ve never had the chance to reform engineering practices and culture at a very large organization, I’m considering it as a path for professional growth, and I’m looking for ways to learn more about doing it before I dive into the deep end.

I’m talking about a software company with dozens or even hundreds of engineers, with large amounts of technical debt, that has a giant, monolithic codebase that’s highly-coupled and poorly tested, a release process that is slow and brittle. And in this scenario, I either lack the authority to dictate global change or I think that an authoritarian approach would be inappropriate and ineffective, or I have no authority at all and have to drive change locally, from the bottom up.

If I didn’t know a lot about what best practices were, I’d be reading things like Clean Code. If I didn’t know a lot about strategies for approaching technical debt, I’d be reading things like Working Effective with Legacy Code. If I didn’t know a lot about leading a small team in general, I’d be reading things like Radical Candor. But none of those quite capture the problem I’m describing.

Can HN recommend any books, blogs, videos, podcasts, personalities I should be looking at?

> I’m talking about a software company with dozens or even hundreds of engineers, with large amounts of technical debt, that has a giant, monolithic codebase that’s highly-coupled and poorly tested, a release process that is slow and brittle.

I think you may have a flawed view of large software organizations. While I'm sure some companies exist that match the above description, I've more typically seen numerous siloed teams, with their own codebases, with both loose and tight coupling to other teams, with a variety of processes, and therefore a variety of quality around the teams.

So it is not as simple as just picking a universal pain point and transitioning to a new process. It is about understanding the drivers and history that got each team to where they are, picking apart which of those drivers is still relevant, and looking for shared pain points between teams that are now open to change due to changes in the business since things were set up.

Because of this, I'd focus less on the technical side, and more on communications and business, as those are the rabbit holes you'll need to go down to build consensus from multiple teams to mutually move to a better place.

For personalities try Jason Friedman. Martin Fowler's Refactoring is a nice read too.

But personally, I'd like to ask what business goal are we looking to solve? Because regardless of how murky a codebase can be, if that codebase keeps hitting a business goal, then we really can't drive a change in practices.

So, we first present business goals. Here's a few examples:

If the business goal is to give better estimates for fulfilling custom features faster, then testing would be the first in the list. Not even the release process. Tests lessens cognitive load for developers when they're asked "how long can we make this feature?". They'll have an idea that it'll probably break a bunch of services, etc. a.) If you're in leadership, ask for tests when people make a PR. b.) If you're not in leadership, look for the smallest part of the monolith. Write 1 integration test, add a CI integration, put it in a PR. Co-workers would probably like that.

If the business goal is to give better SLA's (turn around time for addressing breaking bugs, scaling better), then the release process would come first.

Regarding technical debt: One way to address technical debt is to slowly address it by folding in the effort of cleaning and refactoring inside a feature request. ex: making a new form about 2FA takes 3 days, but you can say 7 and explain that we'll setup a CI test build, then hit and address some debt regarding User Authentication along the way.

It'll introduce as a "Clean As You Go" culture in the legacy engineering team, which will pay off in the long-run instead of having a huge amount of time just dedicated to addressing technical debt.

The danger of separating "address technical debt" as a separate effort is it sometimes results into some unnecessary abstractions and over-refactoring: i.e. trying to make certain things too generic, bordering on building a framework.

But then again, if the business goal was to make a framework for a business goal (maybe something the bizdev can sell), then it can definitely be made as a separate effort. Otherwise, it'll be seen in the bizdev as someone just "playing" around with tech.

I'd suggest you start with identifying and quantifying the areas of improvement. Tools to analyze code quality, unit test coverage and defect leakage could give a sense of what the problem areas are. From there, you can build a consensus on the to-be state. This apprach would shine light on the relevant problems and would reduce resistance.

This isn't quite what you're looking for, but you may find some inspiration in The Phoenix Project (Gene Kim, George Spafford, Kevin Behr).

Reply on news.ycombinator.com