How To Survive a Ground-Up Rewrite

On the other hand, if you're saying to yourself "This code really sucks, they wrote their own complex CSV parser, I can rewrite this to use the CSV library in the standard lib" and things like that, no matter how big the task may seem, it'll work out in the end.

Another danger i ran across were moving goal posts. Long rewrites will typically be interrupted to extend the old system with new features, which increase the effort to rewrite.

Here's some ways to do a rewrite well:

Set up the rewrite as an alternate, competing product and let the two compete in the marketplace. This requires having the resources to support developing two such products at the same time using separate teams for each, which is not something many companies can afford. The advantage to doing it this way is that it gives the rewrite time to mature before it's fully replaced the old system. What most people tend to not realize is that mature software with fundamental flaws can often be superior to immature software on a better foundation. Real-world use is orders of magnitude more effective as a QA process than any sort of in-house testing, and always well be, products that have been tried by fire and made the cut should always be respected no matter how horrible their underlying code is.

If you look at a lot of the major switchovers in software you'll see this sort of trend fairly often. For example, consumer Windows took several generations and almost an entire decade to fully switch over to the NT core.

Another good way is to rewrite piece-wise, in place (refactoring). There are lots of different ways to do this regardless of architecture, ultimately you're essentially creating a bubble of new code that grows and grows until it completely replaces all the old code. The advantage of this is that it takes much less effort than a full rewrite and you can do it incrementally and slowly. Often it's easier to mature each new component in isolation than to try to mature an entire new product.

And, of course, there's the hybrid approach. Work component-wise in an "add + deprecate" fashion. Create the new component, have it work in parallel along with the old component, and then deprecate the old component and work toward moving client code to the new component.

Competent programmers can maintain other people's code, those who cannot want to re-write everything. Train wreck in progress. Run from these people, or fire them if you're in a position to do so.

Immediately brought to mind the first large shop I worked in out of school (about 1992) where they had a big chart on one wall titled "Punch Card Elimination" with a downward trending line. I was actually shocked that a large, multinational manufacturer was using punch cards for anything, particularly when my project involved building cutting-edge (at the time) client/server systems that included wireless handheld devices on the client side. My introduction to the counterintuitive world of enterprise IT: you will often see opposite ends of the technology spectrum in use, simply because the value proposition of eliminating the old was never high enough before.

What about when what you're rewriting is basically an intensely-overgrown prototype that was never meant to run in production in the first place?

You're supposed to completely rewrite those, aren't you?

I am wondering what sorts of info I need to go looking for to get this finished. I have thought about it before, contemplated asking around, and realized I didn't even really know what questions I needed to ask.

I also found this, which struck me as helpful but not quite what I wanted to read about: https://www.linkedin.com/today/post/article/20121226171356-6...

In the past, we used subsets of data to test our migration tool. Always. ALWAYS, there was something bizarre in the the excluded subset that caused the migration to break.

When you have a PHP project where there's a mix of procedural function (the code is at the top and the html at the bottom, so it's easy to split right ? Hum and those includes, what do they do exactly ?), ... classes (We used classes to put our functions in so we do OOP, right ? Err... no) and parts of a heavily customized Wordpress. Plus two-three half started in-code reorg/rewrite that went nowhere because the devs jumped the ship.

You're told you have to I18N the whole spaghetti meatball soup. You try a few weeks and then see you're going nowhere and start to realize that you're doomed to a rewrite. Oh, and you're a team of two. Sometimes you really have to. And migrating data .... such a pain in the ass ! Test encoding I'm looking at you ! At of hand int code (would not call that enum) I'm looking at you too.

But. It's not always the case. And right now I'm starting the reorganize and refactoring a project that was already in a descent shape. And it's much more interesting, less straining and business owner have more visibility on the work done and are happier.

So yes, whenever you can, incremental rewrite is the way to go. But never say never. There are contexts where doing a rewrite is just plain necessary because the code is in such a bad shape you just can't add any feature anymore.

The analogy used is of a ship at sea. Replacing the ship in a single step, while it is underway, is impossible. Replacing too many planks at once is foolhardy. (I thought the analogy was Oakeshott's but I can't find the original source).

So instead you replace the planks one at a time. Yes, this is tremendously slow, wasteful and frustrating. But when the ship is a system on which you utterly rely, there is no other safe path.

Hayek expanded on the problems of pure rationalism more generally; it's a trap we as a profession tend to fall into. We substitute our preferences (for a tidy, orderly system) for an appraisal of actual reality. Real systems have accumulated complexity that cannot be wished away. Worse: the complexity is often essential and not merely accidental.

What the author describes sounds like a complete nightmare.

On the projects I've worked on, I would like to think that a rewrite will not be needed.

If it does, it will be because we've now understood the problem space so well that we can start from scratch and avoid all the 'pivots' that happened in the original code base.

If so, do you rewrite those tests? Or do you keep them, no matter how ugly they are, because new code still has to pass all the old tests?

Get your data source in order, then refactor in small achievable projects from there. Better for motivation and better for end users.

edit: unflagged after modifications

Ground-up rewrites are occasionally necessary, always difficult, and often hideously political. There's no binary right answer on this question. Sometimes it's the only answer, but that's a terrible place to be.

My shortest job tenure ever (108 days in the winter of 2011-12) was in a company where I was brought in to fix the cultural and social collapse that had followed a badly-thought-out "rearchitecture" led by a 25-year-old CTO's protege who (a) was on his first real job, having burned out of college and landed in retail, and (b) didn't know what the fuck he was doing. I'm still deciding whether I should share that story, in full, with the world.