Rewrite, refactor, or reinvent? Lessons from 6 software rewrite stories (2019)
herbcaudill.com
herbcaudill.com
The system was huge and poorly understood, and a lot of people used it to do most of their work. Extremely critical system for the very fabric of society.
There had been several previous attempts at replacing it. They'd sort of fizzled out along the way, so they were stuck with several half-finished iterations of replacements that were in used in parallel with the old system.
This was it though. Hundreds of people. Micro services. Modern JS frameworks. C4 architecture and hexagonal design and cloud native and every goddamn buzzword that's appeared in a programming blog the last 10 years. Just get it all in there! Everything the cool start-up kids were using needed to be apart of this. It wasn't clear exactly how it was going to better, but it sure was gonna be modern.
I didn't see the project through but I could see where it was heading clear as day. It was only a year in and maintenance was already becoming a problem. Shit was falling apart. It was a nightmare to debug. Nobody knew how the heck all the services quite communicated.
I'll eat my shoe if they don't spend the next 25 years fixing leaky pipes with this microservice mess bolted onto the previous iterations as an even more grotesque programming-cronenberg.
If you're going to replace a creaky old legacy system, never ever do it with the latest technology. Do it with tried and boring technology.
Otherwise soon enough you will have two problems.
That, and everyone with any sort of deciding power were technology-illiterate bureaucrats who waste no opportunity to try to make us somehow use AI to somehow enhance this system in some fashion, largely so they could announce to the world that their system use AI somehow.
Have they chose golang, or (God forbid) Rust, they will most likely be fine.
There are choices, which are very dangerous, as they increase complexity dramatically, like micro-services, or "event-driven" architectures, or Kubernetes. Add all three and good luck lifting project off the ground.
The whole thing was rebuilt using new tech; Java for the most part, cloud-ish hosting (our "own" bare metal servers and OpenShift via a Norwegian company), and this stack was never a problem. In fact, not much was a problem, because:
The old system was _riddled_ with unit tests that could easily be replicated in any language. All of the bugs and/or minor issues over the last 30 years had been documented and had unit tests covering them. This was what saved the rewrite. Our rewrite could rely a lot on TDD based on those tests, which made it possible for us to trap problems early on.
Without those tests - written by programmers who probably are long dead - the rewrite would never have succeeded, or at least not in the time frame given, and at least not on budget.
Since then, whenever our consulting company is involved in anything that has to do with existing software, we ask the client about how their test coverage is;
* None, but you will give us $1 million per hour? Nah. Doesn't matter. Find someone else.
* Lots, and you'll pay us $1,000 per hour? Tell us more.
So: create (unit) tests for _everything_. Fixed a bug? Create a test for it. Changed some behaviour? Create a test for it. Removed a feature? Create a test for it.
EDIT: Final stack looked like this:
* Self-ish-hosted servers (non-Norwegian cloud providers was not an alternative because of privacy issues).
* Java as main programming language on the backend, but many services was written in C#.
* Frontend didn't have to be fancy, so some simple JS-stuff there; think auto-complete, no fancy DOM-handling or anything like that. Application as a whole was very server-centric.
From what I've heard, they are now trying to get away from Java, and move more and more over to C#. I just hope they keep the unit tests. :)
The tests were only ever used once for that rewrite (the customer terminated the contract shortly after the rewrite was complete, and the entire codebase was thrown into the rubbish bin) but I still think it was worth it. The tests didn't even run that many times because most of them passed at the first attempt but I suspect the effort of enumerating cases had a strong impact on the successful design of the rewrite.
Spend the first four hours sharpening the axe, and so on.
Great analogy. I'll remember that one.
Also, after some DNS-digging just now, it seems like they have changed to a different (Norwegian) provider.
This is the crux of it. The hard part of a rewrite or major refactor is not building out the new stack from scratch using all the latest tech. The hard part is getting the initial system in a place where it can be rewritten. This includes -
- Adding robust testing, both unit and integration/system/functional.
- Working, functional CI/CD pipelines that execute these tests. I have encountered so many legacy projects without CI pipelines, some of which even had tests but were just expected to fail. Not marked in the code as expected to fail, mind you, just expected to fail by developers, who shrugged and only ran tests they thought were relevant to their code change.
- Code cleanup, particularly if you're doing a slow migration by functionality (which you should be). If you're making a "user microservice," you need to structure the code such that all user-related functionality is isolated and written in a way that works as a microservice, before you write a single line of the microservice itself.
- Infrastructure to do parallel implementation[0], including comparison and reporting of differences in results.
The main problem with all of this is that it's boring. Rewrites are almost always, of course, advocated by developers who have to deal with the code on a regular basis. If you tell developers that they can do a rewrite, but first they have to spend 6 months writing an integration testing framework and fix all of the broken unit tests, most won't go for it. They want to jump directly into using Kubernetes on the cloud with Erlang.
The problem compounds when you look at the wider industry, because honestly, it's often a great career move for the developer who starts up a rewrite with Kubernetes and Erlang. They can stick around for 6-18 months, put "Rewrote legacy system using $HYPEDTECH" on their resume, and start shopping around. Doesn't matter if the rewrite ends up being a dumpster fire, which it normally is. That's a much better story for your career, unfortunately, than talking about how you deeply understood the domain and the quirks of the classic ASP code powering your current company's most critical software.
[0] http://sevangelatos.com/john-carmack-on-parallel-implementat...
It clicked for me when I started thinking of them as DSL's, where the "domain" is "business stuff" ( or equivalent, government data processing etc ) Of course they have limitations - but they're also often part of a full platform and very opinionated as to app and data model, and within those constraints programmers can be crazy productive.
So many of these rewrites fail, and I think a big part of it is people don't fully realise the ramifications of the fact that they're replacing both a platform and a system built on that platform. Sure, there's probably a wizzbang new architecture, which well intentioned smart people will be able to prove improves on that platform in every way - with reams of documentation about the choices made and the way things work... but they're still choices, and sometime they'll get misunderstood, or ignored, or subverted by people who think they know better or who just need a short cut - and it's often big enough project that staff turnover is actually a significant factor too, so opinions may shift. But many of the architecture/data model decisions are just built in to those older languages. The architecture isn't subject to the same drift pressures because they're not choices, and that makes the system less complicated overall, and especially over time.
Of course limitations of the various language implementations, as well as the very human distaste of working within a constrained system is probably a big part of the reason that these legacy languages effectively died.
But I think we lost something valuable, without many of us ever realising.
I think the hardest conversation circles around language choice. One problem with choosing innovative languages is that you can end up on a dead end. Zipcar, for example, used Tcl. While I am a fan of Tcl, it became obvious around 2012ish (IIRC) that this was not the future. There was a first attempt at a migration using rails, but that ended up being ditched for a JVM stack. (I don't know if most of it is still in groovy, but there were also pieces in ruby, kotlin, and java.) We also had a large number of performance and billing problems we needed to solve, and as we were guaranteeing compatibility and interoperability (a big bang switchover was too risky for many reasons), we had a lot of performance and usability issues for years.
It cost far more than management expected in the outset for many reasons, but at the same time, staying on Tcl was unsustainable for developer recruitment and retention purposes. It's something that biases me toward blub languages now as a manager unless you have a very good reason to innovate there.
I agree that this is a real phenomenon, and I find fascinating because it illustrates the role of fashion in software development. TCL may have been quirky compared to other languages, but actual working software was successfully developed in it by more-or-less typical developers. Yet it failed anyway because it wasn’t “cool”.
Not being trendy meant that fewer brains were thinking about it, evolving it by improving tooling and writing libraries, producing training materials and evangelizing it. Ultimately it just stagnates.
I think it does speak to the amount of labor that goes into making a language viable as an option, though. Java had paid labor. Tcl didn't have enough unpaid labor to succeed while python did.
First time I’ve heard of tcl being used outside of an FPGA or EDA toolchain. Far out.
I'm not sure I get this. Any competent programmer can learn Tcl. So find one versed in scripting languages like Python or Ruby, and they can learn how Tcl works in a week and learn how the program is structured over a few weeks and start making small changes. I'm skeptical that this training overhead would really cost more than replacing the whole system if it really was working well.
You're looking at it soley from the PoV of the company.
> Any competent programmer can learn Tcl. So find one versed in scripting languages like Python or Ruby, and they can learn how Tcl works in a week and learn how the program is structured over a few weeks and start making small changes.
Sure, but why would they want to? They'd be wasting time on a skill that adds exactly $0 to their market value.
The longer they stay with you, spending time getting experienced with Tcl (and not with some other popular language), the less they are worth.
> I'm skeptical that this training overhead would really cost more than replacing the whole system if it really was working well.
For the company, no. For the developer, most developers tend to shy away from dead-end tech stacks. If you get someone close to retirement (10 years away), they'd be happy to use your dead-end tech stack, because to them their current market value gets more irrelevant with each passing day.
No, I'm not. I personally am not at all concerned about "dead-end tech stacks" because the kind of companies that care that you have specific skills rather than overall engineering competence are not the ones I want to work for. I'm sure I'm not the only one.
And as people left, the theory of the program was being lost, turning it into "legacy code." There were other problems as well - in particular, the initial data structures that were coupled to billing made it difficult to try other business models, and the industry was starting to innovate with point to point in other markets. Those were solvable in Tcl, though. The real problem was people.
Right, it's always something like that. Thanks for clearing that up.
I get the impression (having never used it) that Fogbugz's users liked it but that Fog Creek couldn't win new larger customers once Jira arrived on the scene. And since it was a shrink-wrapped one-time purchase FC couldn't fund a paydown of their tech debt.
Netscape will forever be remember though for this:
"Less than two years after it was released, the company’s $3-billion IPO launched the dot-com era."
In hollywood the flagship products were Alias PowerAnimator, Wavefront Visualiser, Thomson Digital Exploer, Softimage 3D for animation, Sidefx Prisms for particles/effects. PowerAnimator had the best modeling tools like NURBS. Softimage had best animation tools. None of these ran on Windows and all were running on SGI's own IRIX. Microsoft bought Softimage in 94 and ported it to Windows NT. Suddenly Silicon Graphics saw Wintel as serious threat. So they bought and merged Alias,Wavefront, TDI to form Alias|Wavefront and began to write Maya which would combine the technologies from all three. Softimage also started to write its new one codenamed at that time Sumatra which would eventually release as XSI. The race was on, and the two big giants SGI and Microsoft were having a proxy war now. Everyone was keen on seeing who is going to come out first and best product.
Meanwhile Sidefx was also writing something new from scratc which they eventually released as Houdini in Dec 96. Sidefx has always been a smaller niche player and has stayed true to that till today.
Maya 1.0 came out in Feb 98 and it was immense success. A lot of hollywood studios migrated to Maya because Sumatra was nowhwere near release. Eventually XSI came out around 2000 I think and it was too late, but not too little. XSI still had more powerful animation capabilities but they had lost a lot of ground to Maya interms of mindshare and customers.
Eventually Microsoft sold Softimage/XSI to Avid. In the mean time 3D Studio from Autodesk was not a serious player when it came to hollywood stuff. So Autodesk bought Softimage from Avid. Unfortunately SGI also had real financial trouble due to Intel finally catching up interms of speed. They also sold Alias|Wavefront to Autodesk. So now Autodesk had the two major 3D graphics packages used in hollywood. Autodesk eventually killed XSI and continued Maya at their high end, and 3D Studio Max as their lower/medium end product.
No 'e'. Silicon is for computers, Silicone[1] is for breast enlargements.
[1] Insert jokes about Silicone Valley here.
Vesting.
I've come to realise that realistically, platforms -do- change, and eventually you will need to either commit to a rewrite (even if you choose to do it by spinning up a whole new company like Freshbooks did).
As an example, a SaaS app from 2010 might have been barely responsive, might have been put together with jQuery, and the backend might've been written in a framework that died off in the meantime. And while you can do a lot of little improvements and refactors here and there, ultimately the foundation on which you've built your system gets to a point where it slows you down more than it benefits you, and there's no real way forward.
The stories in the article make me think that the old adage of "never rewrite" is really more like "eventually you will either rewrite or die".
You can bring up a larger team for the rebuild but that introduces its own risk, or you can leave customers out in the cold for years, like netscape did.
Continuous rewrites are the only way to handle that kind of system, just like in a large enough city just to keep up with natural decay it becomes necessary to always have construction and road works projects going on.
Basically the high touch (from a developer point) areas of the app are always the most modern and the first place to get whatever treatment we transition to (React, typescript etc) and there’s some really untouched areas that might remain prehistoric for years, and that’s fine.
Its worked pretty well for us, and I can’t see why it wouldn’t continue that way.
I'd say, historically, you were also lucky in the timing. The last 4 years have been absolutely rock solid for frontend development, compared to the time before. Still way more fluid than I personally think it should be, but I'm willing to ack that things have gotten a lot more stable.
Oddly, the backend has gotten worse in these years. The amount of service related churn that has happened in the past decade has been impressive. And mostly not in a good way. I can hope that we are going to be a bit more stable now.
We used react a lot externally to the main product as early as 2014, but didn’t introduce it into the main product until we were comfortable.
Using things outside the critical path is definitely the right choice. Making things to throw away is also great.
I do get worried, sometimes, that it is also correct to let the new coders pick the styles and frameworks. Letting them choose is a huge boon to their productivity. And letting them advance on a bet like this can payoff really well. Just expect losses along the way.
Your software is not understandable to others, so it's impossible to adapt.
Your software needs high quality documentation about the mental models used to create and think of the software. But too many developers think the code is the documentation, meaning their code is legacy as written.
I keep emphasizing it, looking at this project is like looking at a complied code, you can understand what this portion does by reading the program but You can't understand what the dev/architect had in mind when they did it, architectural diagrams + proper high level docs can do wonders. I can understand what those modules are doing, but it will take me years before I can grasp the entire system to know where to look for every fix/new increment.
As an example, let’s use Discord; what if in 10 years the Discord codebase is seen as obsolete and requires a re-write? You can’t really make a “Discord 2”, because now you split up your userbase between Discord 1 and Discord 2.
In this re-write you would have to make sure that you aren’t splitting up the community that you rely on, since it is part of the product offering (it is basically a social medium).
When you should rewrite software from scratch and what should be kept in mind doing it.
[0]: https://www.joelonsoftware.com/2000/04/06/things-you-should-...
That is a bit harsh on the new team that just inherited someone else's mess and the One Guy who understood it is not there any more.
Worst case was where decision was made, resources had been invested and then it was an ongoing theme for years to come (this particular company I have in mind is still talking about next version 6 years after initial resources were invested, I was the one to build foundation system which works to date even though I left 4 years ago)
Best case was in the company I was contracting for, they had huge and almost 20 years old pearl codebase which they decided to replace with python ecosystem. In 6 months the company was already actively switching to the new system, exposing bugs early etc.
In my view each time management is where it can result in good or bad outcome.
I'm convinced that the way forward is not to refactor or rewrite it, but to produce something new and different that draws on the 30 years of experience learned during the development of the original product. And, like Basecamp, run them side-by-side and allow users to chose what they want.
Are the original authors still there or do you have insanely good documentation?
Of course happy they didn't because if they did Fogbugz would've had an incremental upgrade path on PHP all the way into modernity and would have been an acceptable thing to work on, preventing Trello from ever existing most likely.
If you do something like build a facade in front of the old service, and gradually move functionality into the new layer, this can be seamless for users.
But of course, this doesn’t give you the ability to revisit product functionality, like the Basecamp rewrites. I think those are really great examples of how to continue to innovate and experiment, while keeping your old customers happy, and the key point there would be “rewrites can be done safely if you commit to keeping older versions around”. Basically treating the versions as distinct products, instead of the rewrite replacing the old version.