Comment by mjr00 | Hacker News Reader

Comment by mjr00 | Hacker News Reader

Parent Full thread

> The other reason is that you cannot split up a large established codebase without first understanding it. I have seen large codebases successfully split up, but I have never seen that done by a team that wasn’t already fluent at shipping features inside the large codebase. You simply cannot redesign any non-trivial project (i.e. a project that makes real money) from first-principles.

This resonates. At one former company, there was a clear divide between the people working on the "legacy monolith" in PHP and the "scalable microservices" in Scala/Go. One new Scala team was tasked with extracting permissions management from the monolith into a separate service. Was estimated to take 6-9 months. 18 months later, project was canned without delivering anything. The team was starting from scratch and had no experience working with the current monolith permissions model and could not get it successfully integrated. Every time an integration was attempted they found a new edge case that was totally incompatible with the nice, "clean" model they had created with the new service.

I worked at a company that had a Rails monolith that underwent similar scenario. A new director of engineering brought in a half dozen or of his friends from his previous employer to write Scala. They formed up a clique and decide Things Were Going to Change. Some 18 months and 3 projects later, nothing they worked on was in production. Meanwhile the developer that was quietly doing ongoing maintenance on the monolith had gradually broken out some key performance-critical elements into Scala and migrated away from the Ruby code for those features. Not only had it gone into production, it made maintenance far easier.

> Meanwhile the developer that was quietly doing ongoing maintenance on the monolith had gradually broken out some key performance-critical elements into Scala and migrated away from the Ruby code for those features.

Yep and that's what I've seen be successful: someone who really knows the existing code inside and out, warts and all, needs to be a key leader for the part being broken out into a separate system. The hard part isn't building the new system in these projects, it's the integration into the existing system, which always requires a ton of refactoring work.

> needs to be a key leader for the part being broken out into a separate system

Indeed, the developer was one of the best programmers I've known and absolutely the key person on the system. The New Guys Clique were the sort of developers, you might know some, who come in, look at the existing systems, decide it's all wrong and terrible, and set out to Do It Right.

I've seen almost this exact scenario play out, although in my case it was just one person as opposed to a clique. He had just come from a much larger company in the same business, and almost right away he proposed that we should rearchitect a significant portion of our software to match the way things were done at his previous employer.

His proposed architecture wasn't without elegance, but it was also more complex and, more importantly, it didn't solve any problems that we actually had. So in the end it was more of an ideological thing.

He seemed to take it personally that we didn't take him up on his proposal, and he left a few months later (and went back to his previous employer, though to a different group). Don't think he was around for even a year. He wasn't young either; he was pretty experienced.

What was the language and proposed architecture? Nothing is quite as grating as being hired for work on an Erlang/Elixir code base and seeing basically no correct usage of the platform and it'd definitely result in an instant suggestion to start correctly leveraging Erlang, piece by piece. It's not even a question of architecture as much as it is completely ignoring what makes the platform good, IMO.

It was all c++. The existing architecture consisted of a number of different executables, which were mostly statically linked (except for stuff like the C and C++ runtime libs). His proposal was to factor out the business logic from each executable into a shared library, and then (if memory serves) we would only need a single executable that could work with any of these new shared libraries. Like a plugin architecture.

This would mean that when rolling out a new version, usually only the shared library would need to be replaced. So.. significantly more complexity, due to now having a more arms-length interface between the executable and the business logic in the shared libraries. And the payoff was.. when deploying, we would upgrade a shared library instead of an executable? Still doesn't seem worthwhile.

Even if you’re generally suspicious of so called best practice/ design patterns / Martin Fowlerisms.. this is a time for the strangler approach. (Parent and siblings are already talking about the same idea without naming it.)

Rewrites from scratch never work with sufficiently large systems, and anyone that’s been involved with these things should be savvy enough to recognize this. The only question is around the exact definition of sufficiently large for a given context.

https://en.m.wikipedia.org/wiki/Strangler_fig_pattern

A similar, more concrete approach is parallel implementations, as written about by John Carmack[0]. I suppose the main difference is that parallel implementation has you very explicitly and intentionally leave the "old stuff" around until you're ready to flip the switch. I've used this approach in large-scale refactorings very successfully.

One of the benefits is that you get to compare the new vs old implementation quickly and easily, and it lets you raise questions about the old implementation; every time I've done this I've found real, production-impacting bugs because the old system was exhibiting behaviors that didn't match the new system, and it turned out they weren't intentional!

[0] http://sevangelatos.com/john-carmack-on-parallel-implementat...

I second this approach. I've utilized it successfully in an ongoing migration. I also second the need to have engineering knowledge from the previous system available. I was lead for 5 years on a previous system before being tasked as solution architect of new total rewrite to cloud native. The hardest part of such a migration is maintaining sponsor buy-in while you build the parallel run evaluation strangler fig integration with the old system and get some existing flow polished enough to take live. If you happen to have a rule or scripting system in place piggy pack off of it so you can do an incremental migration.

Also an issue is that the director attempted a full rewrite as a separate project.

You can do successful rewrites but your rewrite has to be usable in production within like a month.

If you don’t know how to achieve that, don’t even try.

The quiet developer was able to get their own rewrite done because they understood that.

Looks like the director of engineering showed some classic inexperience. You can tell when someone has done something before and when it’s their first time.

> a full rewrite as a separate project.

And it was never constrained to rewriting the existing system. The rewrite plan was motivated by the entirely reasonable desire to make further improvements possible, an additional mistake was the attempt to add major improvements as part of the rewrite. The new guys made their disdain for the existing system obvious, to the extent that their intent for the rewrite ballooned into a ground-up rebuild of everything.

Things You Should Never Do, Part I: https://www.joelonsoftware.com/2000/04/06/things-you-should-...

> You can do successful rewrites but your rewrite has to be usable in production within like a month.

I strongly disagree with this, and it reminds me of one of the worse Agile memes: "With every commit, the product must be production-ready.". [0]

The rewrite has to be generally not behind schedule. Whatever that schedule is is up to the folks doing the work and the managers who approve doing the work.

[0] I've worked for an Agile shop for a long time, so please don't tell me that I'm "Doing Agile Wrong" or that I misunderstand Agile. "No True Scotsman" conversations about Agile are pretty boring and pointless, given Agile's nebulous definition.

OK so you're actually right, but the actual criteria of "whether you can do this" depends on a lot of factors from the project to the people.

But there's no way to really describe it. It's like explaining to somehow how to parallel park or do a kickflip... you can only explain it so much.

I like to say "it should be usable in production soon" because it's generally a good approximation that takes into account what you might have to work with. It's an upgrade from advice like Joel's who just say "IT NEVER WORKS"

> It's an upgrade from advice like Joel's who just say "IT NEVER WORKS"

What you originally said ("It must be usable in production within a month") is equivalent to "Just don't do it, because IT NEVER WORKS" for all but the smallest, simplest projects out there in Professional Programmer land. [0]

> But there's no way to really describe it.

There really is a way to describe it:

"The rewrite has to be generally not behind schedule. Whatever that schedule is is up to the folks doing the work and the managers who approve doing the work."

Establishing that schedule is the same sort of cost/benefit and expected-level-of-difficulty analysis that should be done before planning any nontrivial work in Professional Programmer land. All but the most green or most sheltered-from-Process programmers are at least aware of this analysis. Many of those who are aware of it have participated in it.

[0] Or the most well-designed projects, which have small, easily understood pieces with easily-comprehensible interactions with the rest of the system, that can be quickly and easily replaced with new pieces. There's not much out there like that... and I'd imagine the task of "making major changes to how many of those pieces interact with each other" wouldn't be usable in production in a month for most of those systems.

If the schedule is three years, and in the meantime the product being rewritten isn't getting maintenance, the company might as well go ahead and fold and save everyone pain and disappointment. https://www.joelonsoftware.com/2000/04/06/things-you-should-...

If the product is not getting needed maintenance, then yeah, maybe.

But, man, sometimes software is fit-for-purpose and can really be just be left alone for extended periods. Other times, the users of that software upgrade on a hemi-annual or annual schedule (or even LESS frequently), so they'd never notice a three month delay in new releases.

> sometimes software is fit-for-purpose and can really be just be left alone for extended periods

I suppose if the software maker is a functional monopoly in that space and the customers are cursed withe vendor lock-in, sure. I can think of Major Software companies that barely maintain their products yet companies stick with the garbage because what choice do they have?

> I suppose if the software maker is a functional monopoly in that space and the customers are cursed withe vendor lock-in, sure.

Given this statement, it might surprise you to learn that there are folks who aren't keen on software development for its own sake, and really rather dislike having to re-learn how to use their tools when those tools get significantly changed with no clear benefit to those users.

See also: the folks who are perfectly happily using "ancient" versions of -say- Photoshop, or WordPerfect, or Word or...

literally what I wanted to do as an opinionated junior

This is the way. You absolutely can turn shit codebases into something nicer and faster, and this is best done by people who know the system, and maybe even more important, knows the operational context the system exists in.

I once came into an old codebase like this as a junior, thinking I can start again. And I was gently but firmly told by my boss that it wouldn't work, this software is crucial to operations that support all our revenue, and while it can be improved it has to keep working. And we improved the hell out of it.

Am I naive for thinking that nothing like that should take as long as 6-9 months in the happy case and that it's absurd for it to not succeed at all?

You know so little about the team, the organisation, the codebase, other concurrent obligations (e.g. prod support), and the way the project is run. The only way I can imagine one having confidence in a statement like "nothing should take that long" is naïveté.

Then maybe 18 months wasn't too long and they should have been given more time. But seriously?

It really depends. Honestly 6-9 months would have been an optimistic estimate even if it were 2-4 devs intimately familiar with the existing codebase. Permissions is a very cross-cutting concern, as you might imagine, and touched a huge amount of the monolith code. A big problem was that permissions checks weren't done in a consistent layer, instead scattered all over the place, and the team responsible for the rewrite, being new to the code, was starting from scratch and finding these landmines as they went. Scoping was also unclear and kept changing as the project went along, at first to pull in more scope that was peripherally related, then to push stuff out of scope as the project went off track. And during all these changes you have to keep the existing auth system working with zero downtime.

The devs were good developers, too! Two people on the team went off to Google after, so it's not like this was due to total incompetence or anything; more just overambition and lack of familiarity with working on legacy code.

Maybe. There's a lot of dragons hidden inside enterprise code. Only if you know all of them can you really succeed the first time around.

At a large enterprise, 6-9 months is blazingly fast.

Everything takes longer than you think and this sounds like it involves at least 2 teams (the php team and the scalar team). Every team you include increases time line factorially in the best case.

It takes a lot of time to have meetings with a dozen managers to argue over priority and whatever. Especially since their schedules are full of other arguments already

authorization and access control is an awfully difficult problem, as soon as you derive from user defined ACLs on all persisted objects and user/groups based granting data. Each access can have an arbitrary rule that must be evaluated with all dependant data, that will end up being anything in the persisted data. How to you make this long rule list maintainable, without redundancy, ensuring that changing re-used rules won't introduce regressions on all call sites ?

> Am I naive for thinking that nothing like that should take as long as 6-9 months in the happy case and that it's absurd for it to not succeed at all?

Bluntly, yes. And so is every other reply to you that says "no this isn't naive", or "there's no reason this project shouldn't have finished". All that means is that you've not seen a truly "enterprise" codebase that may be bringing in tons of business value, but whose internals are a true human centipede of bad practices and organic tendrils of doing things the wrong way.

> whose internals are a true human centipede of bad practices and organic tendrils of doing things the wrong way

Currently there. On one hand: lot of old code which looks horrible (the "just put comments there in case we need it later" pattern is everywhere). Hidden scripts and ETL tasks on forgotten servers, "API" (or more often files sent to some FTP) used by one or two clients but it's been working for more than a decade so no changing that. On the other: it feels like doing archeology, learning why things are how they are (politics, priority changes over the years). And when you finally ship something helping the business with an easier to use UI you know the effort was not for nothing.

If you find me any resources to build access control on arbitrary (I mean it, arbitrary) rules the right way, I would be very very (very) glad.

I think that's just called "code".

Authz can make the most otherwise trivial features into a depressing journey in the valley of edge cases.

No, you’re not naive. If it’s done by one-two people that know what they’re doing, it should be done much faster.

If it’s a big new team that doesn’t know what they’re doing, working separately from existing codebase, with lots of meetings… I see no reason why it would finish at all.

Only in the sense that you seem to not understand how terrible big companies, big teams and big code bases are for efficiency and productivity. It pushes the bar for reaching desired results way up, and the time it takes to get there even more. No one should ever want to be part of organizations, teams or code bases like this, for their own sake.

>Am I naive for thinking

Yes.

Reply on news.ycombinator.com