Hacker News Reader

Comment by somewhereoutth | Hacker News Reader

The most foolproof way to avoid regressions is to leave the code alone.

The best testing is battle testing - if it is working well after plenty of contact with the customer then leave the code alone.

Bitter experience has taught me this.

---

I'm going to expand on my comment a bit - we need to be clearer on what (production) code actually is. We believe it is the definition of the runtime behavior of a given system (and of course it is exactly that).

However, for any reasonably complex system, a complete understanding of the definition/behavior is beyond any single person. Thus code is in fact a recording of the social and technical process of producing the system, and as such embeds decisions and solutions that may not be evident to anyone just reading the code.

When viewed in this light, it is easy to see how dangerous refactoring can be.

View on HN

That just leaves the code in a state where it's not clear what's wrong with it, because it works most of the time. There are just two types of code: obviously correct, or not obviously wrong.

If you write some code and you don't know how it works, it's most likely not correct in some edge cases.

Code that is written to be so simple it's obviously doing the right thing is better than battle tested mess of if statements.

I've given interviews of fizzbuzz and even though developers usually recognize they need a separate case for 15, they don't always do the else ifs right (maybe test for the 15 case last when you've already hit the divisible by 3 case first, or similar silly mistakes). In that case, doing the string accumulation refactor makes it clear that the code has no mistakes because it for sure adds a Fizz first and a Buzz second in the 15 case. It's also a good refactor for extending to Bazz (divisible by 7) as well, so it's something you will want to do when you want to add that feature

"If you write some code and you don't know how it works, it's most likely not correct in some edge cases.

Code that is written to be so simple it's obviously doing the right thing is better than battle tested mess of if statements."

Only if it actually do the right thing in ALL cases. In a lot of refactors I have seen (and done myself) that has not been true.

Yes, but then the refactor value is dubious. To write simple code you actually need to understand the difficulty of the problem. Why is certain code easier to talk about than other type of code? Why is it better written this way and not another way?

If you don't understand the problem enough you may hack at it in another wrong way.

If that's the typical performance on fizzbuzz, I can't even imagine what the performance would be like on more advanced questions.

Humans are very bad on whiteboard interviews. I like giving people a laptop better now

The point is to make folks answer without the help of external tools, such as an IDE, on the basis that they will be able to reliable accomplish more advanced tasks with such tools in practice. Whether or not this theory is mostly reliable or barely, or not at all, depends on the viewpoint.

If a more advanced question is posed, then the assumption is that the requirement is for the prospective employee to accomplish even more advanced tasks in practice.

Would you rather get someone who can do fizzbuzz correctly on a board the first time in 7 minutes or someone who can write it in 1 minute on a laptop incorrectly, test it, and fix it in 30 seconds?

I see no value in the "gotcha" of you made a logic mistake since we all make them and there's a reason why we test code

If they make 1 'logic mistake' on fizzbuzz then it's likely they'll make 10x, or more, 'logic mistakes' on their actual work. Even with a laptop and all the tooling that makes them 10x better, because the real work will be at least 100x more complex.

I made a mistake in fizzbuzz the first time I did it on a whiteboard because I wrote it in "pseudocode"

I wrote if (n % 15) print "fizzbuzz" ...

I got away with it because my interviewer assumed it was some kind of shell which means 0 is true

I should have written n % 15 == 0 to be absolutely clear what I'm doing

note that in Rust this won't even compile:

    3 |     if n % 15 {
      |        ^^^^^^ expected `bool`, found integer

But I don't have the type checker to find mistakes for me when writing on a whiteboard which is not realistic. I can't commit this code because CI wouldn't merge a build failure anyway

Huh? Did you accidentally share private notes?

No, I'm saying I've made a logic mistake in Fizzbuzz in an interview because I forgot to put == 0 in the condition. If you think that makes me a bad programmer, that's your problem, I just don't think whiteboard interviews are representative of the pull requests I do after working for several hours on something

What? It seems like you have the problem?

I don't think I've ever interviewed anyone like this, so far?

The comment is bit too bizarre to puzzle out.

> The most foolproof way to avoid regressions is to leave the code alone.

The most foolproof way to avoid car accidents is to avoid ever being around a car.

Surely that's not the only objective, right?

That’s the opposite analogy. The OP is saying drive the car as much as possible and as long as it doesn’t have problems don’t mess around under the hood.

Some people tinker with adding mods and things to their cars which increase the probability of the problem.

Using the engine analogy, a plane is a better example than a car. Especially as the mental picture of not surviving a crash is more vivid.

Would you rather fly on a plane that has been used daily for the same flight for the past 10 years? Or fly on a plane that’s either never flown, just got out of the shop, or hasn’t been flown in years?

Oh, planes are a great analogy. If you insist on flying them without changing anything, you can even face some criminal charges.

But you also can't go and redesign the entire thing.

If _nothing_ has ever been changed on the plane, I'll take the new one, thank you. I don't know how it lasted that long, but I won't take any chances.

Refactoring is analogous to changing the composition of the plane, rather than repairing newly broken parts with identical ones.

I'll fly on the 20 year old 737-700 and you can fly on the spanking new 737-MAX.

Please, at a minimum, change the oil according to manufacturer specs. Or eventually blow up your engine.

> The OP is saying drive the car as much as possible and as long as it doesn’t have problems don’t mess around under the hood.

If that's what you and the OP are saying, then you're conflating the codebase with the end product (program). Yes, the end user (at least in an ideal world) shouldn't need to modify the end product. That wasn't ever disputed, though.

The point (if I have to use your car-modding analogy instead) is the manufacturer's engineering team very much does need to be able to evolve the design (read: source code) for future versions of the car.

Testing is a form of battle testing. Code that has been thoroughly tested is akin to a running system for a year. On the other hand, modify that legacy system without test, that "battle test counter" resets to zero and you get to go through the process again.

Classically speaking, refactoring is only for code that has thorough coverage. Refactoring code that is not under test might be better worded as rewriting.

Obviously, rewriting critical and complex code that has evolved over time with a variety of fixes and random features bolted on with random conditionals - obviously that is risky

> Code that has been thoroughly tested is akin to a running system for a year

Except that isn't even remotely true. Perhaps if you system has a purely machine interface, with a limited 'dictionary' of interactions, then a test suite could replicate deployment to a very large degree, but otherwise at best it can give only an indication of how use-worthy the system is.

Take a (reasonably complex, with human interface) production system, implement as much testing as you like, and I can guarantee any significant refactoring will introduce a regression, and quite possibly a serious one.

The problem with testing is that you need to know what to test. So unless you have some method of capturing every possible way a system is actually used, you will always run into the 'unknown unknowns' that you didn't realize even needed testing.

> and I can guarantee any significant refactoring will introduce a regression

Refactoring suffers from having almost no meaning. I would interpret what you are saying as re-architecture and/or update of system design. That is what most people mean by refactoring today. The article laments that people skip the part of "make it clean" in the "make it work, make it clean, make it fast, in that order"

In theory refactoring is a safe activity to do because you are only updating something that has a defined API which is tested.

> Except that isn't even remotely true.

This is hyperbole. How long before a system is actually fully and truly battle tested? 1 month, 1 year, 5 years? I'm making up numbers; but you could think of it in terms of iterations and code paths. How many executions of a block of logic before you start to trust it more? The passage of time adds more certainty because conditions vary over time. Without any testing, that production release is going to be iteration number one in the journey of battle testing that updated (which can be called re-written) code. All I'm saying is running that code through a battery of tests fast-forwards (some) of that battle testing time.

As an analogy, car tires - testing is like running a car tire for a few thousand miles before giving it to the customer. When getting a new one (ie: any update to the code at all), the customer is not getting a new tire with zero miles on it, where perhaps the rubber is bad and that would have been discovered easily after about 10 miles. That car tire might still have issues after 20k miles that are abnormal, but nonetheless the testing released a car tire that had more confidence compared to one with no testing at all.

I think my favorite analogy would be more RPG & DnD like games. Testing is like having your resurrected characters start at level 3 instead of level 1. Are they level 9? No, but at the same time it's far better than starting at scratch every time. Every time any modification is done to the code, you don't have to start back at level 1 on your journey to get back to battle tested.

Alternative response:

>> Code that has been thoroughly tested is akin to a running system for a year

> Except that isn't even remotely true. Perhaps if you system has a purely machine interface, with a limited 'dictionary' of interactions, then a test suite could replicate deployment to a very large degree, but otherwise at best it can give only an indication of how use-worthy the system is

The last sentence there, I would state the exact same about battle testing. Both testing and 'battle testing' is just executing code, no real difference from that perspective. The difference comes in the form and variety of scenarios.

I'm making up the duration amount for the sake of examples, and clearly that can vary based on the system and the efficacy of the testing.

My assertion is that testing is akin to running code in production for 'n' units of time. Perhaps it's exactly the same as running that code for one second in production. Does one second in production give much more than an indication? Not really, but testing & battle testing are in the same boat.

> The problem with testing is that you need to know what to test. So unless you have some method of capturing every possible way a system is actually used, you will always run into the 'unknown unknowns' that you didn't realize even needed testing.

This is just a problem with building systems. This line of reasoning can become a bit nihilistic (why do any testing at all, manual or automated?)

Let's say a system has 10 major permutations, 2 of them are unknown unknowns. Let's say further that it takes 10 seconds running in production to be aware of half of those cases. Applying tests that cover those 5 scenarios is then akin to running the system for 10 seconds. If i have 100 modules, I change one, and there is a test suite that gives me the confidence of having run all 100 for at least 10 seconds (in varying combinations of together, in isolation, in shadow environments, etc).. then I will have effectively gained 10 seconds of battle testing before having released anything.

> The difference comes in the form and variety of scenarios.

The difference is that one is real, whereas the other is simulated. It is very hard indeed often impossible to perfectly simulate the actual use of a system. Thus the simulation will be incomplete, and quite possibly in fundamental ways that aren't appreciated until problems occur. It doesn't matter how many tests there are or long they are run for - they can only be at best an indication of how the system will actually behave when confronted by its 'battle' environment.

I agree. Rocket science is a good analogy here, there is no way to simulate space, on the other hand there are a number of smart checks you can do when a rocket is updated before putting it back on the launchpad.

Sure, but a lot of poorly written spaghetti code that works is sometimes impossible to extend or add new features to without a major refactor or complete rewrite.

Currently dealing with some 13 year old objective C code for a mobile app that’s gonna require a major rewrite cause it looks like it was written by someone who should have never been writing code in the first place

There is a mathematical technique called simulated annealing.

https://en.wikipedia.org/wiki/Simulated_annealing

The idea is code goes through development cycles. You have the most leeway to make changes in the first few development cycles. But less and less as time goes on. And the beginning of a development cycle you can make larger changes than at the end. Between cycles the code is frozen and you never touch it unless you can prove a change is needed. And development cycles are triggered by the need to for gross functional changes or interface changes.

Good analogy. Where I am we try to follow this process for each release cycle - so big changes early, then becoming increasingly conservative as to what gets altered/added as the release date approaches.

Leaving the code alone, even one that works reasonably well, is not an option.

Lehman's laws of software evolution say so.

https://en.m.wikipedia.org/wiki/Lehman%27s_laws_of_software_...

You can only ever leave your code alone if it's modular and independent enough so that new changes on your system do not depend on changing it.

I mean, yes and no, and aside from that - the article isn't even about regressions.

> embeds decisions and solutions that may not be evident to anyone just reading the code.

Strongly agree! Good code communicates this to other programmers (including yourself in 3-6 months). If the code doesn't do a good job of this, that alone can not only be worth a refactor, and when you refactor while "what it took to understand what this does" is fresh in your mind, you can write the new version that much more effective at communicating those learnings.

However that requires your learnings to be complete (in respect of the code you are rewriting). My point is that for any reasonably complex code base of some age, that may not actually be possible even with the best of intentions. There will be a corner case you'll miss, or a necessary optimization you'll leave out. Of course this is over a spectrum, so refactoring code that only you wrote quite recently is probably ok and indeed a good thing!

It’s fantasy, but if each battle was distilled in to a test and added to the test suite then it would be possible to touch production code without breaking business functionality.

Indeed it is a fantasy, however a fantasy that is all too easily and often indulged - which is how regressions end up back with the customer.

I don't think it's as simple as that.

I agree that one should not take something like this lightly. Far, far too many "refactorings" really just boil down to hasty rage-coding at some predecessor who had differing stylistic opinions, or keeners who've never heard of Chesterton's Fence getting frustrated with some annoying design element they don't understand.

But also, sometimes the code really is tangled and difficult to maintain, such that it's difficult to make any modifications without introducing regression defects.

Concrete example: I recently did a refactoring of a piece of code my team maintained that was so extensive that, by the final iteration, it was arguably a rewrite. The old code was sprawling and complex, could basically only be comprehended by the staff engineers, and tended to be ground zero for defects. The new code is much smaller and simpler, and easier to modify, and every single member of the team is comfortable reading it.

- but -

I only did this after months of careful observation, quietly building a case for the change and a plan for how to do it. This process included interviewing everyone I thought might know anything about this module, including folks who had moved on to different teams, to make sure I understood as much as possible of what was there and why it was built that way. I discussed my plans with them and sought (and, ultimately always got) their blessing. And then I spent more time bolstering the test coverage, including hunting for edge and corner cases that the existing test suite had missed, when necessary, adding additional tests that did not depend on this module's implementation details so that they would not be broken by the refactor. And then, when I finally did do the actual refactor, I didn't do it all at once. I broke the work into many small, carefully controlled iterations.

Definitely agree! I suppose this is where the 'art' of software engineering comes into play, to know what can and should be refactored, and how to go about it in a safe way.

True of fixing bugs, too. The odds of introducing a bug when fixing a bug is between 20 and 50%. (Per one study from decades ago. If you can produce better numbers, please do.)

But the problem is, even if code is battle-tested, someone always wants it to do something more. To get it to do more means changing it. And when you've changed it, they'll want another change. And then another.

So you either don't make changes (even bug fixes!) or you do. If you do, you either refactor the code as you change it, or you don't. If you don't, the code becomes more and more brittle, that is, harder and harder to change without breaking something.

So I think you are drawing wrong conclusions from true observations. You are correct that you can break code when you change it, even by refactoring it. Keep it in a state where the probability of that is minimized - where it's clear what the code is doing, and how, and why. To do that, you need to keep refactoring.

Refactoring for no reason is not useful. But what happens in practice is you have a new requirement and you realize your hacked together “battle tested” system can’t possibly support it without breaking everything. So then you need to refactor first.

And if you are actively developing then refactoring along the way is generally better than if you wait until the very end.

it works when the requirements never change. If somebody demands change, you screwed.

Or if you suddenly run into a bug that causes the entire thing to collapse for unknown reasons.

Reply on news.ycombinator.com