Ask HN: Have you ever inherited a codebase nobody on the team could understand?
How did you deal with it? (Reverse engineer requirements + rewrite, convincing higher-ups to cut ties with the code, something else?)
But sometimes it is good old fashioned workplace politics. It is risky to take on an unfamiliar codebase as any problems are now your problems in the eyes of management. And it is, practically speaking, impossible to account for all edge cases and surprises that may exist in a legacy codebase. Therefore, framing the codebase as terribly written and a total disaster achieves two things politically. It helps set up blame for any issues in the future on the previous developers and their "terrible codebase" and opens the door for a much more enjoyable and lest risky greenfield rewrite.
My number 1 red flag of working with a developer, unless they are very early in their career, is hearing them describe a codebase as awful. Most really are not that bad and are usually just using unfamiliar and less than ideal design patterns and coding practices.
I’ve worked on several codebases over the years where a mgr said “here’s a special case, inputs of (foo, bar) should give baz2 not baz1. Coder whips up a hacky workaround to that very specific ask in the simplest place possible, not the right place. A few dozen requests come in like this over the months. Overrides and exceptions are now scattered all over the code base, without any semblance of order, functions no longer do what they advertise because data is modified downstream, etc.
Sometimes the time to redo it right is worth the investment in terms of future maintenance. Other times it’s too foregone and you have to bite the bullet and stick with the mess.
That hacky workaround you're talking about? Well, my task depends on it. The workaround too depends on another hack. With the amount of time I have to work on it, it's a no brainer.
Earlier in my career, I tried to fix them, but they started to break other "hacks" and down the rabbit hole I go. Sometimes it's just not worth it.
Maybe you can propose to hack the code in an accumulative +1x time from the previous hacks (thus making it clear that next hacks will take more and more time), or design it into the system with a nice and constant 2x. That could be a strong indication for management that the hackish way does have a long term cost.
What are we usefully communicating with the word, if 90% of codebases qualify?
Maybe software development is hard, sometimes unpleasant enough, to justify the crazy salaries, and the endless complaining about every codebase more than 2 years old is not quite justified..
As a collective group, software devs are really prone to self-flagellating while they go about delivering billions of dollars in value.
is this not what awful is?
People pick the best way they can come up with at a time and place, and everything is hunky-dory for one fleeting moment, and then the moment is gone. Any code that's good enough to survive will see the world change around it, until you've got a system that was developed for one niche and then spent years evolving into a new one as business and technical requirements changed, and will have plenty of quirks to show for it.
A good programmer can deal with that.
Comically bad security holes, actual or implicit (framework-created) SQL queries in a for loop for no good reason (well, because the developer had no idea how to use SQL, and hadn't developed an appropriate allergy to unnecessary network communication, are probably the reasons), hilariously misguided attempts to fix the wrong thing to improve performance ("we'll move it to jruby!" well sure but your actual problem is you chose an inappropriate database and are using it poorly, but at least you made your build pipeline worse for marginal benefits, so there's that). Ruby in your node project just to run a very basic task queue (!?). Et c., et c., c-beams off the shoulder of Orion, tears in the rain, et c.
Inheriting something even half-decent is really, really unusual. I wouldn't bad-mouth half-decent. I just rarely see it.
[EDIT] this probably varies a great deal by platform. I imagine it's a bit less common to have a total train-wreck of an iOS app, to pick another platform I've worked on, than something server-side.
Hahahah Oh man, that's great. BTW, we had people who did this. Our performance is 2x better with half the servers after we undid this "optimization" at our org. I suggest benchmarking first.
I harbor no ill will toward these folks. Hell, selfishly, I'm glad there are so many. Having half an idea what you're doing is guilt-inducingly easy, pays amazingly well—especially after the confidence-boost and resulting swagger and negotiating attitude that comes with seeing this kind of thing over, and over, and over, for years on end—and it's disturbingly easy to be a or the "smart one" in the room when you're kind of a dummy, in fact.
[0] Dear backend devs: if you don't simply love backend work and/or if you aren't very well appreciated compensation-wise, and especially if you have long-term career aspirations that involve shifting more toward the biz/architect/management side for the extra social status and higher late-career pay (in most of the industry outside the huge West-coast tech companies, anyway), consider moving to more high-visibility pastures—though ideally not web frontend, as, incredibly, it's still a trash fire and comp is so-so, mostly. Unless you just like trash fires, which some people do.
I'm dying.
- untyped languages that run through A LOT of layers of wrappers/indirections to get things done; How do you keep track? (with typed languages it is easier to figure where the next step is going to; though code in java likes to do this trick - it then obscures the details by using some container mechanism, like spring, etc) Do you manage to reduce said numbers of indirection without breaking things?
- microservices like to do that a lot (going through n levels of services just to get to the target server - with different languages and RPC protocols involved). I had once spent a week to add a frigging attribute. Any tips here?
- gui code often turns into a mess (like when it takes it short of a week to add a frigging detail); how do you deal with that?
- how do you deal with the problem of missing context? (when trying to read through the source and there is nobody there to explain any rational of what is going on?) (for me knowledge of other systems often helps to bridge the gap, are there any more empirical approaches?)
For me these things are hard, though i manage (sort of). How do you do?
- Stepping through with a debugger. Static analysis call graphs in stricter languages[0]. Runtime call traces and visualization in dynamic languages. Often you can find some tool for it[0]. If it's that bad you're probably not gonna be able keep your head wrapped around it long term, so you'll be falling back on those tools often until/unless you refactor.
- Similar. Find a way to trace requests (you are really, really gonna want such a tool when things go wrong anyway if you're microservice-heavy) and record what happens, if it's so confusing you can't figure it out otherwise.
- Not much to do but isolate and replace piece by piece or screen by screen as the opportunity presents itself. Hardest part of this is that your new stuff may well look and work better than the old, which can lead to UI inconsistency, so it ends up being an exercise in inconsistency tolerance or management.
- Just gotta leverage tools to tell you more about the codebase than you might get from just looking at it (as in point 1) and do a lot of probably-boring and hard-to-measure difficult work. If it's still under development that means someone ought to be able to tell you what each largish part (feature, screen, section) should do, at least from the perspective of the end user, which may be useful. If no one can tell you that then why are you maintaining it, right?
Jumping off that fourth point: a great preventative measure for ending up in this position (or putting someone else there) is to make sure your code is written such that tools can more easily tell the reader facts about the project. Typescript over raw JS, that kind of thing. Static types are communication. They're communication that can be verified by a machine to be correct, more or less. They're great. Tests fail or pass. They tell you what the test writer expected to happen, and whether that's happening. They are communication. If all your comments get stripped and no-one updates the docs for two years and you get hit by a bus your types and tests still communicate. Even outdated test suites aren't totally useless. And static types pretty much can't go stale like docs or tests can.
[0] https://github.com/TrueFurby/go-callvis
[1] https://github.com/jamesmoriarty/call-graph (no endorsement, haven't used it, just an example.
As a result you end up with every function using different nomenclature, different approaches to encapsulation, differing levels of compliance to Demeter, bizarre side effects because they made sense in the one day that developer was allowed for that feature, and so on.
So “less than ideal patterns and practices” does indeed count as “awful” code where the maintainer has to learn a new coding style for every file in the codebase.
Having said that, I much prefer working with code that is very simple and redundant with a verbose API, than with a codebase that is overly refactored and abstracted.
Finding the balance of the two extremes would be my definition of an ideal pattern.
So, what if in reality, the codebase IS awful, and the product itself suffers from delays and setbacks even on seemingly simple tasks? Still a red flag if a developer says the sky is blue? Maybe you're the red flag in that case?
Unfortunately people often aren't incentivized to discuss the actual color of the sky,but rather the color as either they would like to see it or that would suit their agenda the best.
Code isn’t produced in a vacuum and there are always external pressures for why this was hacked in rather than done correctly. It just kind of makes you come off as tone deaf and inexperienced. “My code is perfect and everyone else’s is garbage.” (It isn’t).
There are always concrete things that can be done to mitigate some of the awfulness and improve a codebase. A good developer will identify the root causes and fix them. Discuss and identify actual problems and solutions rather than your considerable (and understandable) angst.
Everybody has reasons for everything they do. Even if those reasons were made up after the fact to match the evidence (everybody wants to be sane even - no, especially - when they subconsciously do crazy stuff).
Anyone attempting self improvement has to first contend with the fact that their reasons are somewhere on a spectrum from wrong to excuses.
From a business perspective if you can tell me why it’s helping the business that’s the only reason that matters.
Also a very important life lesson to realize that the reason we did something isn’t a reason to keep it around. Sort of a “what have you don’t for me lately”. Hacking something four years ago to land a contract that saved the company isn’t a good reason to keep that code now. We did it because we did it, but we don’t have to keep it that way.
When I was a more junior engineer, I thought I would be appreciated as helpful when I demonstrated that I could figure out and become productive in a new and umfamiliar codebase quickly. This was indeed the case! For a while, at least.
Over time, exactly what you predict set in. I was expected to account for all possible surprises in legacy code. In addition to being handed more tasks in legacy code, I was also expected to account for all edge cases and surprises in there. Being able to blame the previous developers, believed by management to be excellent and highly competent, would have saved me a lot of grief.
Perhaps unsurprisingly, this did not exactly thrill me and lead to wonderful performance on the job.
However, there are degrees, and probably thresholds. If your code to manage web site users is using string theory, despite the assumedly concise, elegant and refined code, it would make it awful to all programmers, but the two in the world who master both programming, web site user management and string theory.
Otherwise, bad design patterns and bad coding practices definitely characterize awful code bases.
But the worst, is when it’s so bad that you can hardly refactorize it.
I complain/observe about codebases all the time, especially on projects with an SLA. It’s bit like fire code. When the shit hits the fan you [don’t] want people wandering around trying to figure stuff out.
Hallmark of a bad team (project management, really) is if the same three people are required for every emergency.
The interesting thing about complaining is that nobody wants to talk to the optimist about problems. It makes them feel small and stupid.
I find myself in the role of troubleshooter often, because I’m good at it, I acknowledge that the struggle is real, and at the end I try to summarize the experience, and diagnose where they went off the rails.
When you see 20% of the team have the same stumble, the problem isn’t with them. If the stumble was expensive then it needs to be addressed rather than blaming the victim. Nobody wants to talk to someone who shames them, either.
But I don’t believe in the classic rewrite. I believe in prevention and station keeping (don’t let it get that bad). The human brain can only deal with so much. If you want to keep adding to the project you have to keep cleaning old code and improving your practices. Slowly you’ll have a very different codebase, and a team where everyone senior can fix most problems.
Edit: a word
In any case there’s never been something that I’ve run into that I’ve not figured out. It takes time, and the hard part usually is not figuring out what it does but the weird edge case of the moon aligning with Venus and then the output suddenly changes. This is why understanding the requirements is more important than the code. I don’t care if the code is bad if I can more or less write a test case against it and make sure it does that.
That said complete rewrites never happen. It usually is only rewriting portions when that is cheaper than fixing. It is the if it isn’t broke don’t fix it adage.
The only time ever I’ve been stuck was when I saw a proprietary software implantation on top of a custom software package ontop of Solr (technically ontop of the JVM) create a large object heap issue ontop of a proprietary OS (Windows). It wasn’t code related I diagnosed that it was a GC issue, but it wasn’t in any code I had source to. A Windows update ended up fixing it. And this is why working with enterprise software is hard.
My process looks like this:
Step one: Identify sources of reflection, this is the triskyest. Hopefully the only dependencies are open source, so you generally know what they do and grep can usually find the rest.
Step Two: go code spelunking. Find your entry points. Find your main() or framework equivalent. Find callsites for rest endpoints, rpc, jmx, etc.
Step three: find other "external request processing" endpoints. Do you have timer threads? Reading a Kafka stream and acting per record? Etc.
Once you understand those, you can interpret where most any stacktrace is from. Good old Intellij or Eclipse can give you all the callsites for a functions as you root around. You should slowly get a feel for which part of the code things get called from.
Now start asking questions like: what data is shared between these entry points? What's mutable? Is it all done safely?
Hopefully this wasn't too narrow an example. I'd imagine it'll apply to any services.
Other tricks---run the code through linters or other stylistic nit-pickers and fix those too. I haven't yet used a code reformatter, but that's a quick way to get code into a consistent style.
It's not the most sexy work in the early stages but very rewarding when you help turn a failing situation around.
Sometimes it's bugs and bad algorithms or data structures. Sometimes it's misunderstood requirements and the fix has been surgical rewrites. Most often a mix.
And most often that struggling but working product still brings money and real value to people or business, not like most of greenfield shiny start-up..[cough!].. throw-away projects.
That was the key actually. At first I diligently tried to understand the code, refactor, rename, move things around... but at some point I crossed a bridge where I suddenly understood, this code is not an asset to be protected and cared for; it's a liability that sucks my time into it and creates not just low, but negative, productivity. And if I don't kill it now, it may suck other victims in. By that time I understood the business processes better and had better implementation ideas anyway, so I began "deleting with extreme prejudice" and rewriting. Even so, I still had to read and understand the code I was deleting or replacing, and was constantly thinking like this: https://www.youtube.com/watch?v=vbr9akNELdc (Yep that's Airplane! and you probably won't recognize the actor who 28 years later would play hitman/bodyguard Mike Ehrmantraut on Breaking Bad.)
Eventually I re-did the important parts and let the rest fall by the wayside. I didn't attempt to duplicate all the functionality embodied in that mess. It took 2 years but I remember the day I finally deleted the last piece of shitcode. Management was behind me all the way, because they were aware of some of the issues with my predecessor, plus over time they've become content for some reason to kind of just take my advice about this and other things almost implicitly. So I was lucky in that way; I was free to tackle it how I saw fit. They didn't really have much choice anwyay though, for the kind of money I was making at the time! (not great)
Your example is an extreme case, and I think that even in non-extreme cases, code that has inertia is bad. The inertia is just not AS bad as the inertia in your story.
If you can't, (as an example,) wrap your mind around a codebase that implements a bit of business logic, in a day, that code is too complex. It should be replaced with something that can be understood quickly, and changed quickly, when the person who maintains it today, dies, retires, or goes on vacation.
(In the above, substitute "day" for whatever period is suitable for your situation, and "person" for whatever employee-unit is suitable for your situation; "lead dev", "team", whatever.)
Software has more inertia than hardware, and that is INSANE. There is software at my employer that is OLDER THAN THE FUCKING X86 ARCHITECTURE, and has gone untouched for much of that time.
"If it works, why touch it?"
A business needs to understand what is running (for legal and other reasons) and needs to be able to fix it, change it, or replace it very quickly, because if it is implementing a business process, it is important for business, and if it is important for business, then it needs to be able to adapt as the business adapts.
Some things change very little, true; that doesn't mean everyone who worked on it should be allowed to retire, then rehired as contractors 20 years later when the mainframe hardware finally goes out of support.
That's a more literal example of not being able to understand the codebase I guess.
It's been 10 years, I wonder if I could crack it now.
I don't know what the solution should be in that case? Keep domain-specific terms in French?
On the other hand, translating domain-specific terms can also be very confusing. For example at my previous position we built a rental contract system that was very specific to the Dutch rental system/laws. We translated everything to English, but a lot of stuff is just funky because it's a specific Dutch term without a real English translation (for example names of specific laws/procedures).
My advice is to give up, destroy your code base, and become a sheep herder. You're screwed no matter what.
(He also didn't understand how functions worked, but that's another matter...)
I contracted for a company that had no software team, and had outsourced the development of their embedded product to the lowest bidder. The original firm had delivered code that met most requirements, but were not willing/able to resolve issues with random crashes or implement additional features. The product manager reached out to my employer at the time for help.
When I took over the code base, my initial attempts to modify the function of the code rendered the device completely non functional, so I focused on restructuring the code without changing its behavior. I moved code into functions, functions into libraries. I added parameters to existing functions so that global variables would have to be injected, rather than accessed directly (this helped make it clear what the inputs and outputs to the function were.)
Eventually, I modeled state external to the system with state machines so that it would be clear when code was trying to manipulate a resource that hadn't been initialized yet. (This helped make some bugs stand out like a sore thumb.)
Through incremental changes and testing at every step, this refactoring made the structure and flow of the program much easier to understand. After only 2 weeks of refactoring, I was able to identify and fix the bugs that had been causing the random crashes. I was also able to add new functionality to the well structured program in a fraction of the time it had taken to do the initial refactoring.
The best part about restructuring/refactoring code is that even after totally reorganizing the entire codebase, I still only had a high-level understanding of how it all worked; I didn't have to personally grok every requirement or fine detail as I would have needed to do if I'd rewritten the code from the ground up. Refactoring was slow going at first, but it really saved the day.
There were some smart cookies on the team, but we were all in fear of this code. The person who wrote it spent her days sitting in the cafe downstairs, reading novels. She'd check some logs, and occasionally come and yell at us for doing something wrong, which she would never explain. It turns out that the system had objects, but they were all embodied by consecutive spans of entries in those arrays. We went on a trek through the bowels of this corporation, asking around for documentation of the 3rd party software that used those sequences in those arrays, but no one had it, and that company no longer existed. If you tried to explain to her that Object Oriented code would have instance methods, she'd always bring up her PhD in math.
A coworker of mine spent a week charting out one of those methods, and managed to rewrite it in 1/6th the line count, with no errors. However, that didn't really help, as it still implemented the same weird merging algorithm.
1) Complain early and loudly about past mistakes from other devs, so that management know delays are not your fault. Once you've made enough noise, attempt step 2.
2) Reduce the scope of changes significantly. Management want X, but you explain only 10% of X can be done in the available time given the current code base. If they accept, great - you're touching less existing code, but may have to dig around a bit. If you can convince management that the reduced scope offers little business value, try step 3.
3) Push for a rewrite. Get budget, get resources and eventually deliver. If you're a good dev/dev manager, you may even get to be the hero that delivered something that works amazingly (aside: a lot of devs take on too much in this phase and often burn out). If you do deliver, you'll be worshipped as the authority on the system for months/years to come! Happy days!
Eventually, however, even your beautiful rewrite will decay into festering spaghetti, as random requirements get incorporated. You may even, deliberately, introduce complexity into your code to justify/protect your own job. At some point, the very thought of diving into your own code may fill you with dread, and you'll start searching job boards.
The cycle then repeats.
But I realized the more I understood, the more dependencies needed to be changed. While it's still doable, things started to break and I ended up learning way more about the project than what was required to finish the task. In the end, it took me longer to finish it.
The fact that most of my coworkers would write a hacky solution in a short time, it made me feel incompetent. Managers without a technical background don't look at code, they look at results. So how do you even do #1 when the "other devs" still work there and can get stuff done much quicker?
Then eventually I decided I wanted to learn some newer tech, so I started playing with Linq-to-SQL and ASP.NET MVC and Razor and Bootstrap, and over the course of three or four weekends and evenings I did a ground-up rewrite of the whole thing for fun. After a a bit more time flailing away with the old mess, I showed my side-version of it off to my boss, and it wasn't that hard a sell, being much prettier and less buggy.
It helped that there was nobody around who was invested in the old code base.
Generally, I've found it is a lot easier to effect this kind of change if you just do it stealthily and present it as a fait-accompli, because otherwise people get so bogged down in debate and fear that any impetus to actually take a risk and do something evaporates.
I do agree with your assessment that it is much easier to drop a working thing in people's laps and get buy-in than ask for permission. This is also what causes pork barrelling in projects.
Either that or significant chunks of it were retained and your improvements were mostly cosmetic.
I still don't know how the preceding mess got to the point of being such a baroque cobbled-together monstrosity, but it was enough copy-pasta to keep an Olive Garden supplied for a year, and a raft of incomprehensible stored procedures that did SELECT * FROM table WHERE xyz in bizarrely complicated ways involving multiple casts and nested temp results.
IME these projects are usually more complicated than all but the most wild of business logic.
The company itself was in biotech (cancer diagnostics) and was relatively new, spun off from a rather well known research lab. They quickly realized that their system was incapable of scaling (or being maintained properly...) to the needs of a business.
The code itself was written primarily by one man. He was quite bright, but not really a software engineer by trade. Usual stuff; no source control, mish-mash of technologies, spaghetti code, and home grown algorithms and hardware to solve well understood problems with available solutions. New management wanted me to rewrite everything in C# (against my protests. Not because I dislike C#, only because the code dealt primarily with image analysis and hardware control/robotics.)
I began by doing exactly as you proposed; I reverse engineered every bit of code. I took extensive notes (it was hard to follow) and walked through each step from sample prep to image acquisition to analysis to result. I started writing each sub-system only after I understood how everything pieced together.
The real bitch of it was that the original developer relied on automating ImageJ for nearly all of the image analysis. As my original requirement was to not alter the results in any way, I literally rewrote large swaths of ImageJ in C#. Bugs and all.
Well, turns out ImageJ (Java) is compiled with /strictfp. C#/.NET does not support this, so my floating point results were oh so close, but not identical. This was initially a problem for management... until the CEO was replaced, along with my boss, and the new team thought the entire project was a dumb waste of money and had me build a new system from the ground up.
That system was released (successfully) early this year. I began work on it nearly five years ago now, with many detours along the way. I now work elsewhere.
The worst projects are those when the company doesn't really want to upgrade, so the only requirement is "make it exactly like the old system"
In my first job out of college, I upgraded an approval workflow engine from VB3 to C#. It was written by someone who had never heard of state machines, so it had a weird ad-hoc design that would e.g. get confused if two documents were in the same state at the same time.
I demonstrated the bugs and suggested an alternative approach that would be simple and robust, but management wouldn't have it. They reminded me that the job was to make it exactly like the old system.
At what part of the project did you leave? I can't imagine anyone staying in the same company doing the same rewrite for 5 years.
Honestly, having worked in medical devices for more than a decade, a five year dev cycle isn't crazy. All said and done it was about three years of dev due to the aforementioned detours (shiny object chasing by management.)
In the beginning there were two other devs on the team, but they were web guys and primarily concerned with CRUD stuff. They were laid off and ~1 year later we hired three more, but again, primarily concerned with the web side and third party integrations. I was all hardware, image processing, image viewing, and image analysis, only helping them when needed. These images are multiple gigapixel (~20GB uncompressed), so just managing and viewing them is a lot of work.
You probably know this already, bit look for assay dev labs with existing software groups. Best of both worlds. Otherwise you run the risk of developing software in a culture which inderstands nothing of software dev. Not fun.
Preaching to the choir. My thesis was a lot of that, haha.
Thanks a TON for the recommendations!
Large insurance business paid a previous contractor to write up a simple web app to consolidate public rates.
So this business could go and answer questions like "how much is my competition charging for XYZ?"
This "simple web app" turned into the previous contractor writing his own insane web framework from scratch in Python, because I guess Django or something was not good enough...?
Anyway the result was something that was almost impossible to read, had who knows how many security vulnerabilities, and was an awful experience for the insurance company.
Lots of times implementing a new feature meant changing the web framework so you could actually implement it.
Company already spent who knows how much on the previous contractor, so after a small cost working with me to evaluate what else was needed to complete the project they decided to go another direction.
I planned out how we could migrate the app slowly over to Django (it was a SPA as well, but he did at least use React there not something crazy he wrote himself) but they didn't have the budget.
Unfortunate. Could have been a really cool tool for not a lot of money, and I would love for businesses to be more eager about developing such products. The concept was the perfect example of a business-specific use of software to give the corporation an edge.
You might not have access to the data sources, you might not have the context to interpret them in a meaningful, and then the company you are targeting is probably your only potential customer (or you have to cold sell to every competitor, so now you have a sales job, right?)
This is the job, just shut up and get on with it.
Don't do rewrites. Morph the code towards "North." Old code has a reason for existing and being correct: It's there. From an evolutionary and survival of the fittest model, things that are out there already have a lot going for them. The ugly warts are battle scars of nasty edge cases and bugs.
This is one of the main reasons I strongly value the skill of reading code in engineers. It's rare and so important.
This is a big issue I run in to. I've spent the last ... 10 years working as an independent consultant, and am often brought in to projects which are... a mess. Almost by definition - if they were good and you could hire cheap people to make stuff 'work' you wouldn't be calling me in.
This assumption that "it's correct" is probably the biggest sticking point I hit, in various incarnations. Was just talking to a colleague this morning about rewrites vs refactoring. Whenever I go for a rewrite, there's issues, but the refactoring is almost always way way way way way underestimated.
Recently, came in to a project 3 months ago that is 5 years old, built with 2 different web frameworks merged together (as in 2 different set of session handler clobbering each other at seemingly random times), a mix of raw JS, jquery, angular and vue, 0 unit tests, 0 docs, things breaking, and the client/owners generally insist that things used to work like XYZ". No, they didn't. In fact, I can point out that XYZ never worked. You think it did, but it didn't.
There's this assumption in "never rebuild" that "don't do it... you're no smarter than the people who came before you, you'll make the same mistakes, or worse, those 'ugly parts' are there for a reason, etc". And yet... you' ALWAYS hear people say "don't ever roll your own crypto!". OK... well... at some point you have a conflict here, and you can argue for just 'refactor' away the crypto, but you may come to realize that every major tenant of secure/modern practices has been seriously violated.
I'm not only fighting code, I'm wrestling with a client who think things were working just fine, or "hey, we might need a little tweak on ABC here". No... digging in everywhere, I've found "oh, that report about 'this number changed' last month? - the numbers are calculated wrong, and have been for 4 years, but this is the first person to report it". Having one numeric field be treated as both 'immutable' by some section of the code, and 'live/updateable' by others...
If you have a full team of people who are dedicated who live/breath project X, and you're all on the same page, working fulltime towards a singular resolution... yes. Refactor. Agree on refactoring, schedules, priorities, etc.
> I strongly value the skill of reading code in engineers
If what you're reading ends up being the equivalent of "see dick run" level of code that's in charge of PII, and you're seeing spelling and grammar mistakes in what you're reading, there comes a point when you say "fixing this is not worth the time involved".
If this was a physical building or structure, and you determined that the foundations were so fundamentally bad and were constantly failing, or found that the materials used were misrepresented, you'd probably face some legal issues if you covered that up and kept building anyway.
If you're coming in to something that is clear the original team was in way over their head, you're facing multiple security issues, logic/data issues, have no tests, sample data, repeatable builds, and the project will take weeks to get to a rebuildable state, and conflicting/contradictory input from stakeholders... a fresh rebuild will likely make more sense. Take segments from previous code if you can extract them, but only when it makes sense.
What I'm wrestling with now is currently owning 4 years of technical debt on my own. "Person A before you used to get this done in just a few days... Person B always worked much faster". Well... yes, because they didn't test, didn't ever fix anything correctly, and I'm not plugging 15 holes at the same time, and every change uncovers 4 more critical data integrity issues. I could also work fast if I was ignorant about what I was doing, or just lied about what the impact would be. And hey - let's not have any tests to run at all, so there's never a quick way to validate if I'm telling the truth when I say "this won't impact anything".
Sounds pretty typical to me.
There's also an assumption that the reason for it existing is still valid and necessary. MANY TIMES that's not the case. Without reviewing if things are still necessary, you're needlessly supporting stuff that is a net negative (and often leaves a wider security attack surface).
None of that is relevant any more.
And source code that was demonstrably older than the binary.
It was written by a contractor, who was blackmailing us for the up-to-date source code.
In an FDA-regulated industry.
How did we deal with it?
We pretending nothing was wrong, and prayed that no show-stoppers would happen. We begged for permission to rewrite, but it was deemed to be too expensive.
And I left that company as soon as I could.
Many years ago, I became the designated maintainer for a legacy inventory system that was used for internal audit purposes. The system ran on a PDP-11/70, and was written in a combination of Basic-Plus (this was a long time ago) and COBOL. I only had a passing familiarity with either of those languages. I find the challenges of learning a code base usually boil down to a few recurring problem areas:
* Problem domain knowledge: The system was used to track leased telecommunication facilities, and had a lot of obscure business logic built into it. At least half the challenge was reverse engineering business logic from the code.
* Coding style: I always find it challenging to get comfortable with another developer's style. Mismatches in assumptions/preferred approaches can make it really hard to get comfortable with someone else's code. This particular system had some truly weird programming choices, including a "screen driver" (similar to curses) written in COBOL.
* Code complexity: I was pretty lucky that the code was not very complicated. With all the other challenges, if the code had been complex it would have probably been an impossible challenge.
* Language knowledge: the original developers had used some features of Basic-Plus and COBOL that were a little obscure, which made understanding the code base that much harder.
Luckily I work in a tech company where the fact that nobody knew this code and it took years to effectively ramp up on was argument enough that it should go away.
Actual removal was a much messier story. It had tangled deeply into adjacent systems so you couldn't "just replace it". We are in the later phases of something like the Strangler Pattern (https://docs.microsoft.com/en-us/azure/architecture/patterns...) where we built higher-level interfaces over the top and gradually re-implemented the underlying functionality without using any of these custom frameworks.
That said, it's a long term project that is easy to lose steam on. It's been very important to regularly revisit our goals and how we're attacking them...AWS has released services that fundamentally changed our approach (for the better) in the years since this effort started and we've probably cut off at least a year from the overall effort by adopting those instead of continuing on the original course.
I wrote up some of these ideas about accomplishing big projects that span years at https://medium.com/@scott_triglia/ask-a-tech-lead-i-have-to-.... The parts about regularly re-evaluating the next steps in your course of action were directly inspired by this project I just described.
During development a number of requirements changed and they had already burned through multiple other devs before hiring me. Eventually Xcode updated and it was required to use the new version to deploy against the latest version of iOS, this version of Xcode was not able to process his pile of macros in the same way as the old version. The QA team had already updated all of their test devices leaving us with no way to test the existing code.
This, combined with the CEO's son's unwillingness to sit down and walk anyone through the code, led to me sitting in a face to face with the CEO alone in which I explained all of this to the best of my abilities in layman's terms. He asked me for a solution and I said have your son fix it as he is unwilling/unable to walk anyone through it and being the 5th dev they had hired trying to figure it out, I put in my resignation. It was a fun 3 weeks.
I later found out through a friend who did media work for the company they were selling this product to, that they were never able to deliver and ended up getting sued for breach of contract.
- Big picture:
try to identify the integration points with other systems or entry/exit points into the code.
See if the code is logically (and hopefully actually) divided into separate smaller parts. If it is, try to work out the main purpose of each part, its integration points, and if there are any obvious side-effects.
- Detail:
Is it building, clean-building, testing etc? That will make it a lot safer to explore and experiment.
For a single source file people have different approaches to "reading" it. Some people add notes as comments as they go through the file, they don't have to be permanent well formatted "Comments", rather just things to reduce the memory strain. Other people remove blank lines, comments and extra whitespace to try to compact as much actual code into a single screen to look at the code paths.
- Repository:
Is the code checked into a source control system with a log history? If so look in there for clues as to WHY things were changed, this gives a good indicator of changes to requirements and also can explain why some parts of code may "feel" different to others (they may have had to shoe-horn in a new change to an existing codebase).
- Pragmatic:
The previous people (just like you) probably never had a chance to refactor or clean up any tech debt.
It probably still mostly works for what we want to use it for, so a rewrite is simply out of bounds. You just plant your face in the dirt, and start plowing ahead. You learn enough about it to get done what is needed, and get out of it as fast as you can.
Strange code isn't all that bad if you're the only one in it. And since you didn't write it in the first place, you can always blame anything that goes wrong on it being awful and brittle. And you can even get that module slander done preemptively, so that when you finally get something working, you're the conquering hero, returning home from battle with the monster. And if you break it beyond repair, you finally get to rewrite it. You can't really lose, except for the torture you undergo while you are actually wrestling with it.
Aside from terminal breakage, if it wasn't worth rewriting any year in the last 20 years, this is probably not the year, either. But sometimes you do the reverse engineering, and find that you can replace the whole crufty thing with 3 lines and a library function call somewhere in your regular code base, and now the execution step that used to take 3 hours takes 100 ms. That feels pretty good, in the moment. Less so when management just gives you a little pat on the head and says, "Well done. Run along, now."
So, first off, try to make a realistic estimate for your higher-ups of how many hours it will take to refactor this, and phrase it as "at least...[x]...and perhaps much more". It is quite possible that you will get back the answer, "it's not worth that". Then, you are in the uncomfortable position of being the Bearer of Bad News.
Depending on the ability of your upper management to accept bad news, you then either: 1) gently and politely insist to them that the situation really is this bad, or 2) start looking for an exit
But, before setting yourself up as the person who brings bad news, get a gut check from some teammates as to whether your estimates of how much would be required, are more or less on target. It takes a while for organizations to accept bad news, and you may need to let people who say "it won't be that bad" win the argument for a while, and then circle back to it in a month or so.
I can't figure out how a company with that kind of culture could stay in business, but it did.
I think the only thing to do in that kind of situation is your number 2 option.
The solution almost always involved a certain amount of reverse engineering, and damn sure I always tried to advocate getting away from software we couldn't maintain effectively. The third leg of that tripod is isolation. Reduce the number of things that depend on the code, and the number of ways that they use it. If nothing else, that will reduce your exposure. It will also tell you what code paths are important to understand and which are not. Finally, it can help guide implementation of tests, or of a replacement.
If you're really stuck with such a piece of code, some of the advice in a blog post I wrote about a similar challenge might apply.
http://obdurodon.silvrback.com/navigating-a-large-codebase
That's about learning a codebase that's notable mainly for being large, even if the original developers are as available and helpful as could be, but looking at it now I see quite a bit that applies to this case as well.
From my experience(currently on a 10MM line of code, 15 year old codebase) in software the code is the business so it's important to understand it.
First trap: "Oh, I'll just document everything."
I could say just do it, but do it for yourself. I could say that it won't be maintained over time and it'll rot and may do more harm than good when someone goes to reference it and thinks it's still an up-to-date understanding of the system.
But I'd rather just tell you to not waste your time doing it in the first place and focus on not falling into the second trap.
Second trap: Rewrite.
I could say it's easier to write new code than it is to understand it. I could tell you to be ambitious and stay up nights and weekends rewriting some view layer logic bullshit.
But I'd rather tell you that you're not as smart as you think, your solution may be more complex than the currently impenetrably complex behemoth before you, and that you should instead focus on not falling for trap 3.
Third trap: Replacement.
I could tell you there's cheaper off-the-shelf solutions available that solve the same problem; it's simply a matter of spending money and reading documentation.
But I'd rather encourage you to really, truly embrace the final realization.
Final realization: You are dumb and will never, ever understand all the complexities of this system as there are too many interdependent moving parts and other similarly complex subsystems.
And that's okay!
This is a system built over many years that consists of hundreds of thousands of tiny decisions made by hundreds of different "very smart" individuals just like you solving complex business problems giving the climate of the time it was built in.
So approach it like you would approach a beast in the wild, with caution and grace.
Heroes are at the morgue, and just make sure the thing doesn't go down.
Our team was handed the project to rewrite it. It was a secret project and we were very careful. We didn’t want to spook the original developer.
It was a lot of hair pulling and tracking of the code. Lots of gruelling work.
Still have nightmares about it now.
The project eventually went from horrible to interesting once all of the legacy was dealt with.
https://news.ycombinator.com/item?id=18442637
(But the current question is different enough that it has seeded a different kind of thread.)
Most often, these clients initially just want us to maintain the codebase, making minor changes and updates. In this case, we will simply familiarise ourselves with the code, working within its limitations to do the required work. Over time we might refactor parts of the code as we do this maintenance work because it makes maintenance easier for us.
Eventually it gets to the point where the client wants major changes (And sometimes it starts here). If we are comfortable with making these changes within the codebase, typically if we have been maintaining it for some time, we will refactor what makes sense to achieve the changes, and continue to work within it. If we don't have that familiarity with the codebase and the project is at a scale where it makes sense, we will rewrite the code at this point.
In very few cases, the project is too large and we are too unfamiliar with the code, and we have to tell the client we're not able to do the work within their time/budget requirements. At this point the client will either leave us, or have us continue maintenance. About half the clients that leave shop around, try out another agency or two, and come back in the end with a greater understanding of the scope of the project.
How I have dealt with it:
1. Never complain. Never bad mouth any of my predecessors. Whatever they did wrong, I probably did somewhere else just as badly. We all have.
2. Never be bashful about what is wrong. Be objective. Be specific. Keep asking questions until there are no more answers.
3. Become the new expert. Don't depend on "higher-ups" for too much judgement. Convincing them should be about as hard as convincing a mother that her bleeding child needs a band-aid. If they need convinced, they are part of the problem.
4. Long conference room tables, paper, scotch tape, and multiple colored highlighters are your friend. Others thought I was crazy, but I have had to paper table and walls with technical debt to a) understand better and b) have everyone visualize what we're up against.
5. Don't be afraid to rewrite anything with a couple of caveats: a) Do it in less that a week. b) Do it only to understand it. Plan to throw away what you rewrite. c) If you can use what you wrote instead of what's already there, that's gravy.
6. Priority of what to learn: a) The data base. b) The code. I have even written utilities to scan and label data to understand what the hard disk looks like before ever venturing into the code mess.
7. Priority of things to refactor/rewrite: a) Rename variables EXACTLY what they are. This can be very difficult but will probably give you the biggest bang for the buck. This step often opens the flood gates for everything else. b) Remove duplicate code and modularize. c) Reduce long conditionals. 800 line case statements suck. d) Remove early exits. The idea is not to improve the performance of the code, it's to understand what it's doing. Multiple exits can be very confusing. e) Fix white space, maybe more, maybe less. The process of doing these things almost always provides better learning that just reading it. Sometimes I have had to rewrite something and then throw it away just to understand what we have.
8. By the time you reach this step, you probably know more about what we've got, what our problems are, and what the speed bumps will be in the future. You probably won't have to convince anyone to do anything except come to start seeing you as an excellent resource.
9. <sarcasm> Bitch about all of the above at home at night. You may ruin your marriage, but at least you'll still have a job. </sarcasm>
I have this coworker who has this terrible habit of giving the most meaningless names imaginable. Local variables are often just named "tmp". Or maybe tmpNum if it is a number. We are writing a program to implement various tasks as background threads, and the classes have names like "Process01", "Process02", etc. At one place, he declared five or six constants with SQL queries, named sql1, sql2, ...
The worst thing, in a way, is that I managed to convince him to let me rename the SQL constants, but he insists that naming a local variable "tmp" is actually a Good Thing so you can immediately tell it is a local variable. I kid you not. (For Great Cthulhu's sake, I wish I was kidding!)
But there still is a difference between trying to come up with good names and failing on the one hand, and naming a variable "tmp" to indicate that it is a local variable, because I totally cannot see that when looking at the code...
[1] I do have to admit that I totally love foreach-loops, when I was still coding in C, I ran into / caused my fair share of off-by-one errors, they were nightmares to figure out.
It is good to keep in touch with a bunch of fellow nerds as a support group to vent out.
The product was essentially made up of 5 code bases in a single repo that built and deployed 5 different executables that worked together. That itself wasn't so bad, it was mostly modern C# and I found it a decent code base to work on.
What was bad was that one of the modules that we were expected to support was written entirely in VB6. The last stable release of VB6 was in 1998. I couldn't just download something and install it and work with the VB6. It turns out that the accepted solution according to stackoverflow, and this is the conclusion I came to independently too, was to go on EBay and try to buy a copy of software/compiler/IDE/whatever and then to even get it to install and work you need to do all kinds of things like turning off certain keys in the registry before installation and enabling them again afterwards then doing a bunch of other modifications to make sure that it actually works when run in compatibly mode with Windows XP.
Our official line was we went from supporting 5/5 modules to 4/5 modules.
Fun times.
I was the one who actually set out to build the VM image.
Not a trivial task at all to get the VB6 side of it running. Which is the exact reason we are supposed to do things like making VMs for development environments.
It would be nice to be at all places in all points of time with perfect knowledge in order to stop bullshit from happening in the first place but I can only be here and now.
I was part of the third team that left in as many years. Just another lesson of a single individual being the only decision maker across all departments.
It was kind of sad everyone came in trying their best to improve the software and environment, but they were quickly shot down by their boss. If you didn't copy paste his code you'd get a snarky email a few days later about not following the guidelines, that were never written down mind you.
My lesson for rewriting a legacy application, have the right management in place. If there is an old dev that does not want to change, the project is a waste of time and money, he'll have more sway over the executive team and often get his way.
I legitimately can't work out if such an intolerable boss is more painful or the constant need to scroll the IDE and having to do it without the mousewheel.
I feel for you. I truly do.
You must study the code and document what you can. Reverse engineering it is sometimes the only option in your quest to understand it. Then you’ll need to make decisions about how to move forward given the constraints of the business whether it be time, resources, other priorities. There is no one size fits all answer but don’t be shy about vocalizing the risks and making sure the business knows the pain points. Sometimes you have to sell the problem you now have and get buy-in to really do something to fix the mess.
One thing I’ve realized is that when people don’t take the time to document and write clean code sometimes it boils down to their idea of job security.
Other times it turns out the problem is there is no single owner of this code base and it’s been hacked to death.
Lastly when something is ugly, enough and messy enough...then you can just call it proprietary technology...I kid I kid.
Once I've got it to a point where I can read it with minimal cognitive load (I like condensed code with little to no white space and no orphan brackets) I'll make sure it still works and pick a spot or feature in the finished application and try to find its code. Work backwards until I've figured out what makes it tick.
In the process of doing that I usually see how a lot of other things tick and get a sense of how and where the rube goldberg machine starts.
The hardest part is understanding the rationale of the developer. Many don't impart such details in the comments. For that I'll try rewriting sections with less code than the original and see what the adverse effects are.
I can asure you that in every step allong the way, the code (-base) I created horrified me 5 years later.
My point being: I appriciate the advice "throw it away and do it proper" but I can asure you, given enough time, the next person will not understand your solution.
Being it old languages, being it old paradigms, being it olt skool tricks of the trade: code stales. The best advice on this subject I read here (a couple of times) is to try to understand the requirements and take it from there. If that is not to your liking, you proberbly are in the wrong line of work and should try to do only greenfield stuff.
We considered a full rewrite, but this was too time consuming and did nothing to solve the immediate issues. It was also risky, we had a "working" production app.
We ended up writing an extremely thorough integration test suite. Making changes is still painful but we know for sure we aren't breaking anything. If we have the time/drive to rewrite the test suite could be re-utilized.
the first one was a mumps project, with cache. it was written by a competent developer, but he was using it as a way to learn mumps to forward his career in the health care industry. the issues were more specific to mumps itself, while trying to maintain and add features (there are thousands of articles online about issues in mumps, if you want to fully understand the struggle). it was eventually rewritten, with tests, and supported by a small team.
the second was for the same company, but a very different developer. this developer despised version control, and considered foxpro the "one true language", even after Microsoft itself had abandoned it. the codebase was riddled with bugs, fixed in various versions deployed for various customers, so there were a ton of misc bugs and "features" strewn throughout 20-30 "codebases", but no comments, short variable names, and poor practices. from what I could tell, the developer had been drunk for most of the development and any changes, and thus the original was used as a template for features and discarded as quickly as a simple web app could be written and tested.
otherwise, the codebases that I have have inherited have been at least understandable, but sometimes best practices weren't used, or too "clever" of solutions were chosen instead of making larger needed codebase changes which meant much more difficult code to maintain, but nothing has come close to those two.
I liken maintaining large, old, arcane code to the care and feeding of a dinosaur. Maybe a Brotosaurus. It's sitting there cropping the treetops happily, farting occasionally.
You want it to do something else, you have to poke it, prod it, yell at it and it slowly gets up and takes three steps. Then it sits down again.
You'll never get very far that way, and it'll never do much more than it does. If that's cool, ok. If not, then its time to consider another approach.
I'm speaking from experience, I inherited a project based on TaxiAnytime, aka "Uber App Clone Source Code". What a complete clusterfuck, obviously written in as incomprehensible a style as possible to create attachment sales for customization.
In the PHP/Laravel code three widely used patterns stand out, the "single return statement pattern", which obviously creates the pyramid from hell. In every method. Add to that, the 700+ line methods. Everywhere. And the icing on the cake, I'm going to invent the term "WET" here to describe it. That means the opposite of DRY. Did I say three? WET has the knock-on effect of anti-encapsulation.
I set out to reverse those patterns where I needed to make changes. In about 2 months as a single man team, I had a handle on it and was extending the app. A team of 5 should be able to make short work of it if they can get over the NIH and YUK factors. And yes, beware the rewrite. Coding always seems like it is easy. Until you need a resilient functioning system.
"[Working Effectively with Legacy Code]" by Michael Feathers really helped me get through a situation like this.
My recommendation is not to try to understand the code per se, but understand the business that the code was being used in/by.
From there, over time, just start writing really high level end-to-end tests to represent what the business expects the codebase to do (i.e. starting at the top of the [test pyramid]). This ends up acting as your safety net (your "test harness").
Then it's less a matter of trying to understand what the code does, and becomes a question of what the code should do. You can iterate level by level into the test pyramid, documenting the code with tests and refactoring/improving the code as you go.
It's a long process (I'm about 4.5 years into it and still going strong), but it allowed us to move fast while developing new features with a by-product of continually improving the code base as we went.
[test pyramid]: https://martinfowler.com/bliki/TestPyramid.html [Working Effectively with Legacy Code]: https://www.amazon.com/FEATHERS-WORK-EFFECT-LEG-CODE/dp/0131...
I strongly agree with this. I've done at least 4 or 5 successful complete rewrites of old code bases, and I have found, rather than even 'business' the word for this might be 'context'.
If you can contextualize a piece of software, it's functionality and operations, you can have a much better understanding of an existing codebase.
I have to assume someone is using the software therefore there is some tribal knowledge of what it does? Otherwise this is maybe SAAS software that users use and some functionality is exposed that would allow you to begin decomposing backwards toward expected input/output. You're almost black-boxing at that point.
I will admit that I have, on very rare occasion, scream tested a piece of software running on a server that nobody would claim ownership or knowledge of either on the eng. team or within the org.
Who is, in that case, using the software? They obviously understand the context by which the software is at least going to work, otherwise, why is the software being rewritten?
Who is requesting the rewrite? Do they know what it is supposed to do? Is there an executable build of it that exists somewhere?
Rewrite is being pushed by certain parties because we're unable to meet feature requests quickly with the existing system, and it's being assumed that a rewrite will fix that problem. The team is barely functional though (from the top down). I've seen a few failed projects now and I don't think the rewrite will ever be accomplished. If we manage to rewrite, it's far from certain that we'll do a better job than the last guys did.
Approval Tests (http://approvaltests.com) can be a huge timesaver when you're getting that initial black box characterization put together.
Besides being an important part of getting your bearings, talking to everyone who relies on the software to get a better understanding of how they interact with it can be a great time saver, too. It's amazing how quickly you can clean up legacy code with the delete key, provided you can confirm nobody's using it anymore.
The wholesale rewrite is a will-o-the-wisp. Very, very attractive, yes. But usually when people chase after it, they end up drowning in a quagmire. That isn't to say that you shouldn't strive to get rid of all the bad code, but do it as a long-term, component-wise, in-place rewrite.
You're absolutely right, but the problem comes when the code itself is the only authoritative documentation of what the code does, and in a lot of cases, the only authoritative documentation (or even the only documentation, period) of what the code is supposed to do!
The result is likely some big ball of mud, partially understood by the (now gone) original developers. One that is hacked at and refactored at the edges by those who inherit it.
Consider an actuarial risk calculation model, a payroll system, an air traffic control system, or perhaps something simpler like a model of a double-entry accounting general ledger. Could you wholistically understand the code base of a double-entry accounting solution?
Such large complex systems cannot normally be rewritten economically prima facie. Instead if possible, typically small parts are carved off, rewritten and delegated to, until the economics and inability to add new features absolutely force some attempt at a rewrite.
I quit six months in.
I lost most of my hair six months in.
What typically happens is that it is mostly left alone. As the years go by, small changes are be made, only making the situation worse. Developers occasionally offer to management to take the time to refactor it, but management refuses to prioritize that work.
Eventually, new requirements are drafted that the existing code simply cannot meet without first refactoring it. At this point, the development team has the unfortunate responsibility to inform management that making the existing code meet the new requirements will actually take more time than rewriting the whole thing from scratch. The technical debt is now due.
The development team then hopes that the situation provided a learning experience for management regarding code maintenance.
So it's a matter of 1 vs 2 refactors, and the management chose one by delaying as long as possible.
One, adopting necessary libraries that are lacking features, have way too many features, or questionable code quality. Examples have been an animated slideshow carousel, Python and PHP OAuth, and a WordPress theme. I thoroughly read the code for each of those, deleted the parts I didn’t need, and literally rearranged everything.
Then there are the times I go back to my code after a few years and have no idea what is going on. If possible, I just leave it alone if it’s working. Sometimes I need to use a new language or have developed a better coding style. In those cases I’ve rewritten it with a close eye to whatever edge cases I seem to be handling in the old code.
(In this case, technically one person on the team - our most sophisticated developer - was able to decipher it. But every time he needed to touch it it would take him 2 full days just to understand the code again. Just wasn’t worth keeping when it was so pointless to have and so easy to replace.)
For a more complex codebase, the equation might have been different.
The custom algorithm was a suboptimal choice as it was prone to passing badly corrupted blocks as correct. Worse, the first stage bootloader was in masked ROM so a true fix wasn't possible, only workarounds.
Also, when I see a codebase I don't understand, if I'm paid to work with it, I spend my time learning the stuff. Debugging tools help: traditional general-purpose debuggers, graphics debuggers like pix/renderdoc, network tools like wireshark/fiddler, OS-based tools like strace/procmon. For different projects different ones are the most useful, sometimes even custom built tools are most useful.
In then end, just like eating an elephant... one bite at a time. Eventually you'll have everything broken enough into separate functions that you then understand the whole better and can cleanly rewrite the whole thing.
It requires going back to first principles: digging into the docs on the systems it interacted with, interviewing users to see what they did and what they expected, in short a nightmare
I took over a suite of flash video players/recorders for a company years ago. The variable names were exclusively 2 and 3 letter acronyms. No one on the team even knew Actionscript. Up until my hire, the various builds were all considered immutable since no one had any idea of where to start. Given their integral nature to the business, this effectively stagnated entire areas of product and business development.
What you put in our question was precisely my approach. I did my best to reverse engineer requirements by code review and sale people interviews. Once i had a confident list, i got approval and killed off what i can only assume were swaths of unused features no one knew about. Rewrote + rearchitected the entire thing from the ground up.
The release was worse than you would have expected. Turned out there were lots of these players linked outside of our website. These versions of the app were ones we had considered deprecated and were unknown to everyone in the company including the cofounding CTO. This required immediate rollbacks and hastened development to re-add features that were being used by a subset of our longest lasting clients.
While the releases themselves were rocky, the entire effort was an unquestionable success. The effort consolidated all the features into a single reusable player and opened up years of feature dev that enabled millions in new sales.
https://www.joelonsoftware.com/2000/04/06/things-you-should-...
I know that everyone thinks their predecessor was an idiot. And I agree that full rewrites are usually a bad idea. But sometimes, the predecessor genuinely was an idiot, and the code really is that bad. If the use cases and inputs and desired outputs are well documented enough, a full rewrite can be the right choice.
It depends on the size of the system, too. A full rewrite of something as complex as Netscape is asking for trouble. If it's something that can be rewritten in a day, or even a few weeks, it might okay to go ahead and do it.
Of course, by the time you've got enough experience to accurately estimate how long a rewrite will take, you've probably got enough experience to just refactor the old code without going insane.
It is a very popular fake money gambling app. Most of the code is very old and no one knows how it works. The only work actually done is fixing bugs that the previous bug fix caused.
The code had to be destroyed and started from scratch. It was absolutely unsalvageable.
PostgreSQL allows C extensions. A function which used to work was now segfaulting occasionally so needed to be fixed. Original author was gone, nobody really understood the PostgreSQL extension system (which involves a ton of macros.) No version control, no useful comments, no spec. I read the code (the core of the logic was straight-forward C, it was just the interface to PostgreSQL that was hard to understand) and wrote a new version in Python (another language for which PostgreSQL supports extension functions.) That version was too slow, by two orders of magnitude. Went back to the C implementation. Did a little light refactoring to separate the core logic from the PostgreSQL interface. Then wrote a test harness program in C to drive call only the inner function. Wrote unit tests until I reproduced the segfault. Fixed the C code and tested it with my test harness. The PostgreSQL wrapper just called the (now correct) inner function so it now worked too. Checked the fixed code and unit tests into version control.
A PhD (no longer with the company) had fit some simple neural net models in R. He'd written his own code for this because, according to him, the standard packages in R didn't support a few of the bells and whistles he'd wanted like ReLU activation. Not only was the code in-house, but the models themselves were saved as serialized R objects. No specifications for any of this stuff. Apparently the company had been using this code to score medium size databases for several years. When we wanted to scale up to a much larger database (approximately 30 billion rows) the problems with the R implementation became apparent. Fairly slow, high memory usage, worse yet a slow memory leak on large jobs, and worst of all occasional silent crashes where it would simply stop and exit with a successful status of 0 and no error message. This time I took the approach of reading the code and de-serialized R objects. I re-wrote the implementation in Python using numpy arrays and wrote a small R program to read the serialized models in the .RData format and emit a cleaned up JSON object that could be easily read from Python. Luckily I didn't have to port any of the cross-validation/training/optimization stuff; just the prediction part. That meant 80% of the code could be safely ignored. However, the devil was in all the one-off special cases in the serialized R objects, many of which had behavior different from default. I would test by comparing the predictions by the two programs on small batches of a million. These predictions were floating point numbers between 0 and 1 so when both programs agreed to within 1e-5 for all million I knew it was correct. It took a week to track down all the special cases, though, and the special cases made it impossible to just use a standard neuralnet library like Keras. (We already had several Keras models in production and a set of tools to manage them; that would have been easy for us.) Proprietary code begets proprietary code I guess. At least the new implementation was much faster and didn't leak memory so could deal with the whole database in a single long running process. I pitched the idea of re-training all models from scratch in Keras using our more modern tools but management wanted 100% backwards compatibility and to preserve the value-add of the PhD.
Improving a codebase is a well-known subject, so I'm not going to comment on it further.
If you decide to ultimately abandon it, you need to understand it won't happen tomorrow, and perhaps not before a few years (for example, I'm 2 years in with a codebase I decided to abandon, and it's probably going to be at least 1 more year in production). Stakeholders hate when you spend time just rewriting it for the sake of it (from their perspective).
What you want to do instead is use a strangler pattern: your new codebase should "strangle" the old one, and deliver value VERY quickly, which will convince the stakeholders it was the right choice.
First, all new features are in the new codebase. If possible, start with easy features that have as few dependencies as possible with old codebase. Any call between the two should be in a special wrapper in your new codebase, so you can start having a sense for what code will need to be rewritten, at some point.
Then, start to "strangle" the old codebase: wrap ALL calls so they go through the new codebase first. That will allow you, somewhere in the future, to cut off the old codebase part by part, and avoid the full rewrite effect, as well as quickly revert to old codebase if bugs are uncovered.
Once you have that, you can more easily identify which parts should be replaced first: performance issues, too many bugs, new features needed...
When you reach the end, it's a matter of convincing stakeholders you absolutely need to go to the last mile, with good arguments. If you can't find good arguments, you probably don't need to go to the last mile yet.
Subtle bugs had crept in over the years, usually relating to the different sort orders or rounding rules of the different platforms. Some minor bugs had become features (the users didn’t want the bugs fixed because they had become reliant on the incorrect version and didn’t want to have to restate their results with the correct version - the idea being that having to explain and account for the change was more trouble than accepting the minor defect).
The codebase had been altered regularly with minor changes over the years, by a succession of contractors whose names appeared like biblical king lists in program headers. The changes were usually minor, and the approach had always been to graft on some new functionality or extra edge case, rather than redesign anything. Many of the contractors had been actuaries rather than professional programmers, so there were hair-raising sections of code that achieved the required result in extremely obtuse ways. Real outsider art.
There were huge vestigial sections of code and redundant outputs that nobody ever used, but because it was part of a bigger end-to-end process that was also poorly understood and onerous to test, those sections and outputs were always kept just in case they were significant.
In a way, this was a relief. It meant you didn’t need to understand a lot of the code, as long as it kept producing the outputs everyone expected it to.
I was part of a project to migrate all this code onto yet another new platform. Did I take the opportunity to do a grand refactoring? Heck no. The project was already overdue when I arrived, and I had two other projects to work on at the same time. So I did what all those contractors had done before me. I lifted-and-shifted with the least invasive changes possible, ran just barely enough tests to convince the users it was good, and moved on.
I still work for that company. The codebase is now more than 30 years old. It’s had another platform migration since, as well as the same old stream of minor change requests. Bolt-ons on top of bolt-ons on top of tactical kludges.
The thing is ugly and horrifying. Ramshackle and arcane. Congealed, not designed. And yet... it’s managed to carry on producing the outputs that this business needs it to. So is it really all bad?
Bring all the code into a set of org-mode files each file has one code block, which emits back the original code. You can verify this using diff or (better) putting the initial repo into git if it's not already and seeing if your emitted code causes any unstaged changes to appear. Start dividing the blocks into more logical chunks (includes, declarations, definitions, one block per function implementation, etc.). Use a hierarchy like:
* foo.c
#+BEGIN_SOURCE c :tangle foo.c :noweb tangle
<<foo-includes>>
<<foo-file-variables>
<<foo-functions>>
#+END_SOURCE
** includes
#+NAME: foo-includes
#+BEGIN_SOURCE c
#include<stdio.h>
#include "bar.h"
#include "foo.h"
#+END_SOURCE
** file variable declarations
#+NAME: foo-file-variables
#+BEGIN_SOURCE c
int x, y, z;
some_struct a, b, c;
#+END_SOURCE
** Function bodies
#+NAME: foo-functions
#+BEGIN_SOURCE c :noweb tangle
<<foo-baz>>
<<foo-quux>>
#+END_SOURCE
*** baz: int->int
#+NAME: foo-baz
#+BEGIN_SOURCE c
int baz (int i) {
// some logic
return some_val;
}
#+END_SOURCE
A benefit of org-mode here is that sometimes you want to test some functionality (and have no unit tests yet). So you try to test baz from above. But once you build that source file and a second foo_test.c file you find out that foo has a function, quux, which has dependencies outside of just the header files and foo.c itself. To build this thing you have to build bar.c and maybe even the whole Linux kernel. "Shit!", you say, "How do I handle this?" Well, org-mode to the rescue:
* Testing Foo
** Testing baz
#+BEGIN_SOURCE c :tangle foo_baz_test.c :noweb tangle
<<foo-includes>>
<<foo-file-variables>>
<<foo-baz>>
// maybe some other things like the header references.
// some test code that is able to focus in on just the baz function
#+END_SOURCE
Even if you don't care about testing your code, the above process (sans the testing part) will allow you to perform a disection on your codebase and get it documented and specced out properly.
NB: I rather like Perl, though I haven't used it much at work since leaving this company a few years ago. The ...things that follow are not an indictment against Perl. The problem wasn't the language.
Even used properly, Perl is a pretty dense language, so a million lines of Perl in one place is quite impressive.
The company's code started in the mid 90s and continued to grow. When I arrived, there were a couple of guys on the operations tools team with me that had been there 7-9 years I believe. They understood the code better than anybody, and they were both very bright guys, but things came up, at least a couple of times a week, that would surprise and sometimes mystify them. After some digging, debugging and sniffing around, they'd usually approximately get somewhat near a root understanding.
That's fairly amazing to me. I've helped create some enormous code bases my decades long career, in one case, even larger than a million lines. And I'd certainly find myself surprised from time to time, but never for very long.
So these millions of lines of code had, over the prior 17 years, grown organically, and had, essentially, never been refactored.
The code itself was, for the most part, quite tidy. And the underlying concepts and structures were pretty simple and elegant. They represented approaches I mostly agreed with.
But...almost nothing had ever been removed. No refactors, over millions of lines of code, over most of 20 years.
There were side effects everywhere. Many of the deep, underlying methods had had so many arguments added that perhaps one third them would be useful or used in any given invocation.
And, of course, there were virtually no tests.
After somewhat getting up to speed on this system, I declared it DOA, and started to push for a complete rewrite. I've been around the block enough times to understand and comprehend the hazards of that approach, and I didn't mentally pull that trigger lightly or quickly.
But....the organization would not have it. The two senior guys on my team were fine with the idea, but many other long-timers were not.
That code is, to this day, with no doubt another 100k lines of Perl added to it, (poorly) powering fundamental and important pieces of a huge company you have all heard of.
So management basically threw bodies at it. Lots of bodies. And, even though the operational quality of the products were objectively fairly poor, the particular market niche didn't need or demand better, so the company made and continues to make a ton of money.
Some quick advice: Rewrites are almost always a bad idea. The requirements are almost always at least as difficult to discover as they were in the first place. You will also miss things or incorrectly decide that something isn't important now. These can often kill your project before you get a chance to replace the old system.
But the most important reason for not rewriting is because there are almost always business reasons for extending the existing application during the rewrite period. This gives you a moving target for the rewrite. Additionally, you will find that the "legacy team" who is adding code to the existing system will be seen in a better light than the "rewrite team" because they are actively solving business problems. The "rewrite team" will be seen to have no value until they ship something. As more and more features are added to the legacy application, more and more resources will be added to it until someday someone will say, "Why are we rewriting this again?" and cancel the rewrite. It doesn't happen every time, but in my career I think I've seen it in about 90% of the rewrites.
So you need to get comfortable with the legacy code. The first thing to do is to make the build and deploy process as painless as possible. You probably can't get time allocated to do it, so with every piece of work you do, steal some time for that. If you are on a project where they have "build teams" and it's actually impossible to build the application yourself, fix that as a matter of priority.
Once you can reasonably work on the code, you need to start introducing tests. The best advice I can give is to read Michael Feathers's book "Working Effectively with Legacy Code". This is a must read. I think there may be a newer version of it, but even though the old version is very dated technology wise, the techniques are still rock solid.
Fight the urge to refactor/rewrite large portions of the application. Instead, pay attention to the code that you touch the most. Ensure that this code has good tests and once it does, fence it off from the rest of the code base and start improving it. Code that you never touch can be the crappiest in the world. Code that you touch once only has a one time cost, so don't fret over it. Code that you touch every single day needs to be amazing. Concentrate your efforts there.
The last piece of advice I have is to look at the kinds of requests you get. If you get a lot of similar requests for functionality (for example lots of reports), then make that part of the system easy to work with. What you want to do is match the ability to work with the code with the expectations of the customer. If they intuitively think, "This should be easy", then work hard to make it easy. Say things to your stakeholders like, "You/Users asking for feature X expect this task to be easy for me to do. It's not. I need time to make it easier." Usually they will see the sense in that. If they expect everything to be easy, use that back on them. "I can't rewrite the whole application without stalling our business plans. I can make some parts of this easier than others though. Which parts are the most important? Note if you say X is important to be easy, then I have to spend time up front to make it easy. We have to be careful about our budget". That's the kind of language that business people can understand.
Finally, have fun with the legacy code. You aren't likely to make it (much) worse. Use the opportunity to experiment with new ideas. However, I caution you to avoid the temptation to transition to newer technologies (you'll never get it finished -- just like a rewrite). Instead, think about the techniques in the newer technologies and start introducing them in your old code base. IMHO, this is always more fun that simply using something off the shelf anyway. Ironically, I find that working on legacy code is the most liberating thing I can do on a professional team. You can always say, "Well, this is crap. Anybody mind if I replace it?" and almost always people will welcome it.
- Pick the program with the most egregious errors. This was part of the suite that would upload tracking information to the purchaser, and cost us large fines when something went wrong.
- Find the user(s) of the software and pay them a visit. Solicit buy-in (there was some concern that the new IT director might have started this initiative in an effort to automate people out of their jobs) by explaining that I'm planning on making the software easier to use (easy, since it was awful and everyone hated it), and then have them walk me through exactly how they used it. This turned out to be a terribly inefficient process involving lots of paperwork shuffling, but I ignored that temporarily in favor of just finding out how it was supposed to work now, and what sort of ways it went wrong. Get a list of likely low-hanging-fruit bugs.
- Track down the source code to the program, and put it under version control. Fortunately I had a copy of the previous developer's computer, which had an up-to-date version once I found it. However I did have to test that it did everything it was supposed to by running it in production, which was a bit nerve wracking.
- Set up an automated updater (I used Microsoft's ClickOnce installer, which checks for updates on a shared SMB drive), and replace all the copies of the program I could find with auto-updating ones. (This required asking people to pass around word of a replacement by word-of-mouth as they heard other people were using it, since nobody had a list of all the users.)
- Buy ReSharper, and start doing mechanical refactorings on the codebase to fix the obvious and easy code smells. What the changes are doesn't really matter much; the point of this exercise is to start to get a feel of where everything is in the code. Since you're just using the ReSharper commands, there's no risk of breaking anything by doing this.
- Fix a few easy bugs, and push an update out to users. I started with making a list view sortable (literally a one-checkbox change that saved 30 minutes a day) and a few similar small issues. This immediately showed a previously unprecedented level of interest in the users' problems and also got them used to using the auto-updater before any more major changes came along.
- Continue with more major refactorings and bug fixes, pushing out a release every few weeks (faster if you can focus on just the one project). I usually tried to include at least a few user-facing changes in with the internal stuff, but occasionally the release notes were just "better performance" or "major internal improvements, so I can do feature X next week".
The really important part of this process is understanding not only how the software works (and how it's supposed to work)--which can probably only be done by refactoring instead of rewriting--but also getting to know how the users use it and what their actual needs are, so you can suggest improvements that wouldn't necessarily be obvious to someone who doesn't understand the entire system.
I love ReSharper, and it has been worth every penny that I've ever paid for updates, but you still have to be very careful. I've walked into a few very heinous bugs where simple refactorings have broken things badly. Mostly this was because of people doing evil things with reflection and dependency injection, that should never have been done, or because of arcane config-file based development, where even ReSharper's excellent "Find Usages" and code analysis engine cannot fully understand what is going on.
I've had a few more-or-less realtime multithreaded projects that I've worked on where you can't really put a debugger on a system and halt it, without breaking things in interesting and misleading ways, and the only good option is to fall back to ye olde printf debugging.