Hacker News Reader

> Let's say you start to write tests and start to see issues crop up. Now what? How do you fix those things?

I assumed the tests wouldn't be for correctness, but for compatibility. If issues crop up, you reproduce the issues exactly in the rewrite until you can prove no one depends on them (Chesterton's fence and all).

The backwards-compatibility-at-all-costs approach makes sense if the product has downstream integrations that depend on the current interface. If your product is self-contained, then you're free to take the clean slate approach.

> I assumed the tests wouldn't be for correctness, but for compatibility.

You're assuming that the people coming in to write these tests can even make that distinction. How do you even know what the compatibility should be without really diving deep into the code itself? Given how screwed up the codebase already is, it could be multiple layers of things work against each other. OP mentioned multiple versions of jquery on the same page as an example.

Writing tests for something like that is really a waste of time. Better to just figure out what's correct and rewrite correct code. Then write tests for that correct code... that's what moves things forward.

> How do you even know what the compatibility should be without really diving deep into the code itself?

You can pretty much black-box the code and only deep dive when there are differences. Here's what I've done in the past for a rewrite of an over-the-network service:

1. Grab happy-path results from prod (wireshark pcap, HTTP Archive, etc), write end-to-end tests based on these to enable development-time tests that would catch the most blatant of regressions.

2. Add a lot of logging to the old system and in corresponding places in the new system. If you have sufficient volumes, you can compare statistical anomalies between the 2 systems

3. Get a production traffic from a port mirror, compare the response of your rewritten service against the old service one route at a time. Log any discrepancies and fix them before going live, this is how you catch hard-to-test compat issues

4. Optionally perform phased roll out, with option to roll back

5. Monitor roll out for an acceptable period, if successful, delete old code/route and move to the next one.

The above makes sense when backwards compatibility is absolutely necessary. however, the upsides is once you've set up the tooling and the processes, subsequent changes are faster.

All of that, while technically correct and possible, is vastly more complicated and time intensive than a rewrite of what the OPs description of the codebase is.

Yes, it absolutely is - but the trade off is a far lower risk of introducing breaking changes. Depending on the industry/market/clients - it may be the right tradeoff

In my eyes, a rewrite won't be introducing breaking changes. It would be to figure out what functionality makes money, then replicate that functionality as best as possible so that the company can continue to make money as well as build upon the product to make even more money.

We're talking about a webapp here, not rocket science.

The biggest problem isn't even the codebase in this situation.

When you keep finding bugs like that while refactoring and making things better, it will demoralise you. The productivity will stop when that happens.

It also require above average engineers to fix the mess and own it for which there is not much benefit.

Your refactoring broke things? Now it's your turn to fix it and also ship your deliverables which you were originally hired for. Get paged for things that weren't your problem.

If I was a manager and assigned this kind of refactoring work, I will attach a significant bonus otherwise I know my engineers will start thinking of switching to other places unless we pay big tech salaries.

People keep quoting Joel's post about why refactoring is better than rewrite but if your refactor is essentially a rewrite and your team is small or inexperienced - it's not clear which is better.

Parallel construction and slowly replacing things is a lot of unpaid work. Just the sheer complexity of doing it bit by bit for each piece is untenable for a 3 person team where most likely other two might not want to get into it.

> It also require above average engineers to fix the mess and own it for which there is not much benefit.

That's not true, it doesn't require above average engineers. It requires a tech lead that has the desire and backing to make a change, and engineers willing to listen and change. It doesn't require a 10x engineer to start using version control, or to tell their team to start using version control for example..

Source control seems like a straightforward first step, regardless of what approach is going to be taken going forward

One would think, but how do you go from source control to deployment on the production server though? If they were editing files on the server directly, there could be a whole mess of symlinks and whatever else on there. Even worse, how do you even test things to see if you break anything?

It is a can of worms.

Yeah, lot of worms...and if while refactoring things break. You are on the hook for scanning through that complex monster at 3 am and finding the issue and fixing it for no additional pay in most cases.

Doesn't Git support symlinks? Empty directories could be trouble though. One would have to put a .GITKEEP into every directory before checkin, and a step at deployment time to remove them again.

Just start somewhere. These guys are making changes, actual functional changes and bug fixes in that environment meaning they already have all the problems you imagine are going to get in the way of fixing this mess. So stop fretting and just start small with one tiny thing. It doesn't really matter with what. You don't even need automated tests necessarily. It's a small simple flow that needs 10 minutes to run the same test steps manually for? Write them down and do it manually, I don't care. Just Do it.

Been there, done that. Slightly differently where they had a test server and prod server. So already better except one day I made a change and copied to prod. Yes it was manual. Just scp the files over to prod. And stuff broke. Turned out someone had fixed a bug directly in prod but never made the change on the test server.

First thing I did was to introduce version control and create a script to do make deployment automatic meaning it was just a version control update on Prod (also scripting languages here). Magically we never had an issue with bugs reappearing after that.

Pretty simple change and you can go from there.

The above code base was over 20 years old and made use of various different scripting languages and technologies including some of the business logic being in stored procedures. Zero test coverage anywhere. You just 'hide' small incremental changes to make things better in everything you do. Gotta touch this part because they want a change? Well it could break anyhow so make it better and if it breaks, it breaks and you fix it. It needs judgment though. Don't rewrite an entire module when the ask was adding a field somewhere. Make it proportional to the change you need to make and sometimes it's not going to be worth it to make something better. Just leave it.

Not sure the little hammer will fix much. And making folks use a method in new code pisses them off. "You say important I do this your way this time, even though there are 1000 examples of doing it the other way. I feel persecuted and your way is pointless, because it doesn't fix everything anyway. And its slowing me down and making me look bad."

Not rational but folks don't have to explain their feelings. You will be hated.

The little hammer definitely fixes. It does it in the same way as water cut the grand canyon. The beauty is that it works over time.

Now as for how to get the other devs on board, I agree with you that you can't just barge in and tell them everything they are doing is wrong etc. I never said to do that and I'm replying to a specific comment in the thread not the original Ask HN.

I.e. when I write about what I've done in the past, I got buyin from my boss and my colleagues on what I was going to do. But I didn't just sit there and kept doing what they had done over the past years. I changed lots of other little things too in the same manner.

So if we do want to talk about the original Ask HN and how to get the existing employees not to hate you, you can start by letting them tell you about what they think the problems are. What are their pain points. They might just not know what to do about them but actually see them as problems too. Maybe they've tried things already but failed or got shot down by others in the company. Maybe they did try to introduce version control but their non tech boss shot them down.

Of course it may not work out. Some people really are just stupid and won't listen even if you try to help them and make them part of the solution.

Startups have runway and can die when big-company processes forced up on them. It can sink them.

I'm not sure where you're pulling that from. There's no mention of startup here. Neither in the original (actually the opposite I'd say, 12 years and just a business unit).

None of what I said is a big-company process in any way. If in your book using source control is a big company process that will sink a startup then be my guest and I will just hope we never have to work together. Source control is a no-brainer that I even use just for myself, have used in teams of two and teams dozens to hundres. The amount of process around is what scale with the kind of company. Source control is useful by itself in every single size of company.

Source control is necessary and simple, yes.

Code review, coding standards, required tests for everything, multiple stages of deployment - are not simple and can stall development. Done wrong they can sink a company.

It's easy to read the worst possible construction on what other people write here. It's never a good idea.

Btw I worked at a startup for 8 years. It was still a startup, depending on new investment to meet the monthly. In any case the described dev group was behaving in a way that used to be typical of startups. And even business units in larger organizations have runway.

They can literally copy the whole directory from their local machine to production as a first step for all I care.

How do they test things on production? If there’s a bug how do they revert to the previous version? There are way more issues without source control than with.

"Github actions!? They don't even have source control to begin with."

Right: no point in adding any tests until you've got source control in place. Hence my suggestion for a shortcut to doing that here: https://news.ycombinator.com/item?id=32884305