Lessons learned from the big rewrite
anton-pirker.at
anton-pirker.at
Maybe they could have started by refactoring the schema in-place (That is, within mysql and webtek). Then change from mysql to pgsql - still behind webtek. Maybe separating the change of webtek -> django from the interface change is a bit harder, but you could still try to restrain your self as much as possible and then make a second iteration of the interface after the initial change.
I've taken this approach before, when migrating a big product and it worked quite well. It seems as a lot of extra work at first, but I'm pretty sure that is not actually case.
Having done this a few times, when switching from old to new, the new framework would have a DB agnostic ORM. What works nicely is building the models, duplicate templating, getting a mirror working. Shut down the old site, and then it's possible to go on with normal development.
So, it is important to consider rewrite and redesign as separate goals.
This isn't even a question to me, but to the blog author. My. Point is about order of steps in the rewrite. It is not harder to rewrite first, redesign second, it is easier. I've done it multiple times in c++/c#, perl/python, php/python, r/python.
However messy it may be, there's usually a lot of domain knowledge to be learned from such a legacy codebase.
I had the same experiences last year. There wasn't even a single person who knew the 'old system' on the whole. A receipt for many detours and much extra work.
If you have a written detailed guidelines of what you are doing and why, which design decisions were made and why, which data representation is used and why and what corresponding algorithms are and why, then one could re-write parts of the system in different languages/frameworks just for fun or to explore and compare.
Coding or rewriting is second to deep, hierarchical understanding of hows and whys. This is how the most of programming exercises are designed - once you have understood how it should be, which results each procedure should return (wishful thinking and tests before code, to define domain and range of a procedure - ideas as old as SICP) then coding is the most trivial part.
Look at antirez - he is a rare example of a guy of deep understanding before code.
I wholeheartedly agree with this! The big challenge is that you have the patience (and the patience of the management) to understand the old system thoroughly before starting to write new code.
If there was some catastrophic, quirky failure of backup, and you could only save either (a) all your production code, sans test, or (b) only your tests, which would you rather save?
Assuming you've written thorough tests, they win out over code every time. The tests remove all ambiguity.
I'm reminded of a story that illustrates the point:
A young woman always cuts the ends off of her pot roasts before cooking them, following her mother's example. One day, she asks her mother why they do that. The mother doesn't recall, so she asks the grandmother. The grandmother looks at them both, shrugs, and says, "When I was your age, we lived in an apartment with a very small oven. We had to cut the pot roast down to make it fit!"
Tests alone provide little value to anyone. At least if you had the production code you could keep it running in its current state while you added back those important tests bit by bit.
Most of the tests wouldn't be needed as good code tends to be churn-free. Finally if you wrote the code TDD style, then writing tests for even the hairy bits would be "easy" compared to rewriting the whole system.
A huge portion of my day-to-day agony at my current job stems from a lack of tests. The developers on my team (all lifelong employees, where I'm a contract hire) are terrified of making more than the absolutely necessary changes to code. I can't replace literally copy-pasted code with a shared method without push-back. To even speak of refactoring is verboten, at least in front of our internal customers. Testing small changes can take days, as it must go through QA and the customer, and surprising bugs still slip through. As someone who went through Fowler's Refactoring as bedside reading, this process and turnover rates drives me batty.
Here's the thing: We have the production code up and running, and I'm already adding tests where I can, but we constantly run into implicit requirements. Even the senior developer, who's worked on this code base for years, keeps discovering things about it. We continue to discover genuine bugs, too. The production code empirically lacks enough information to reproduce the tests.
And "good code" begs the question. In fact, it begs two questions: (a) How do you know code is "good" if it isn't tested empirically, and (b) how can code ever be churn-free when the requirements themselves keep changing?
Even in the face of changing requirements I've found that there are substrates of code that do their job without (showing?) major bugs for years. They never needing fixing. That's your good code. If you had unit tests, you'd never run them anyway because you never touch that code. They are the "select()"s of this world, but specific to whatever you're writing.
Now if the whole system had to be rewritten, the old tests would be useless anyway. You mention requirements that have changed, so those tests you had would no longer apply. You'd have to write a new system and new tests ;-)
FWIW I've found that peer code review and static analysis tools give you a lot of what unit tests would anyway, without the overhead of having to write the tests and update the dead code when you rewrite the affected part of the system.
I mean... what's the point of code anyway? Is it to be "perfect"? Or is it to do a job, earn you money, and be more or less maintainable? Constantly refactoring to stay in the same place is worse than copy-pasting code, from a business owner's POV. Sometimes it is better to have 2 copy pasted methods that are deployed and the customers are happy, than trying to have 1 perfect method sat in development. If those 2 methods never need updating again, you wasted time refactoring. If you would need a third copied method later on, that's the time to refactor.
Maybe I'm being a bit devil's advocatey. Unit tests and refactoring are good tools, but they are not "free" (since you have to test the real system anyway at some point) and Uncle Bob is a bit of a snake oil salesman anyway :-P
I agree. Typical Uncle Bob BS
With the source code you have the product. You lose nothing, maybe some information on how it works in some corner cases (which is derivable from the source code anyway)
A good rule of thumb is that reading code takes at least as long time as writing code.
The second one is "the second system effect" from Mythical Manmonth, which says that when people re-design a system (or a language) they tend to make a kitchen sink from it, throwing in everything that comes to their minds (hello, Rust?)
There is also a common sense notion that a sequel for the sake of a new version is usually much worse than the original.
What is common of them all? It seems that still the best writing is rewriting, but only if your understanding of the details has been grown and now you have less clutter in your mind. But if you do a rewrite without much thinking or just to get shit done, there is a high probability that you will fail.
Lots of solid advice and patterns for approaching thing sort of thing.
- Mixing refactoring and new feature.
- Introducing new technology
- Changing behavior of old code that were not directly concerned by the rewrite
...
Many things that should have been done after that. But, all I can say is if the company can handle this amount of work, it will always be a very good way to improve the quality and durability of the website.
Because if you simply rewrite code, there is a big chance that your company will assign to you, new task that are not related at all and you will never be able to introduce these feature.
NB I've only had one project go badly wrong that way and I guess it was a useful learning experience :-)
Especially try to avoid "big bang" switchovers.
I meant identifying N particular tasks that need replacing, for example user accounts, content management, front-end, etc. Then for each one, isolating that part of the old system, eg. using a proxying layer. A replacement can then be built alongside, with the proxy sending all requests to both. Results can be compared, and differences treated as test failures. When the replacement is working, the proxy can be switched to serve the new system's responses, and once everything's settled down, the old system can be turned off.
Once all of the new system is in place and replicating the behaviour of the old system, the rewrite is finished. New features and improvements can be worked on as normal, as a separate phase.
What's more, we only need to feature-freeze the component that's being migrated, rather than the whole thing (as long as coding standards for new features enforce our newly-imposed compoentn isolation).
Technically, never reached market until 5 months ago. I have of course, launched after each time the product was finished but would give up due to being burnt out etc. An entire year spent starting from scratch which would be my 7th attempt.
3 month in after launch I was going to rewrite it again, I decided, this would've been my last attempt in the software business. I decided that enough was enough, this is it. I ALMOST rewrote it again but having done that 7 times already, instead of doing that, I just ripped out a lot of code, made architectural changes and reused what I could, whatever was tested. 8 times a charm....
My lesson learned from past 6 years pursuing insanity, restarting from scratch means you will have to solve the same problems again and again. A blank canvas is NOT a solution. Don't do that, it's insane, I was insane but not anymore.
I was just really crazy and stupid to pursue my own vision of what the solution would be when in reality I lacked the experience (I started with barely knowing how to code).