Hacker News Reader

Comment by turnsout | Hacker News Reader

I used to have the same attitude toward unit tests for my solo projects… But a few years ago I was working on library that had to do a lot of vector math, and I was doing SO much manual checking after every change. I wrote a bunch of tests so that I could see what was breaking when.

What I didn't expect is that the tests gave me so much confidence to improve, extend and refactor the code. Before it felt so fragile, like the whole thing could crumble if I don't quadruple check my work. With the tests in place, I changed the entire underlying data type from a struct to a native simd vector in an hour. Made the changes, fixed a couple of bugs, merged the branch, boom.

I don't strive for anything close to 100% coverage, but the handful of tests that cover my critical code have saved me so much time.

View on HN

"Confidence" is a good word for it.

When teams don't have good test suites (meaning mostly automated, if not entirely, and reasonably comprehensive), then you can watch as over time you get either increasing error rates or a massive slow down in team performance. Even for small changes, having no tests or primarily manual end-to-end/integration style tests means that you have little chance, unless it's a particularly good code base, of determining the actual effects statically (by analysis) or dynamically (because your tests will take too long and are likely mostly ad hoc so you will miss edge cases).

Teams that don't care (or are pressured into not caring) will deliver crappier and crappier software over time. Teams that do care will eventually stop delivering software at all, or at least no more meaningful updates.

All for lack of confidence.

Unit tests can never give you "confidence".

> you can watch as over time you get either increasing error rates or a massive slow down in team performance

I hear this story repeated over and over and yet I consistently witness the opposite of what you're describing: teams start investing in unit testing and they noticeably slow down and ship more bugs at the same time.

What can give you confidence is simplicity. Being able to fit the entire system in your head. Being able to say what are the side effects of your changes. If you feel like you're losing those, it's time to take a step back and rethink your architecture. But of course, nobody has the balls to take a step back and think, to admit that you failed at something. It's so much easier to just slap a bunch of unit tests on your big ball of mud and hope for the best.

I didn't mention unit tests, so maybe you meant to respond to someone else? Or you're just shouting into the wind about something that wasn't said for fun?

I was writing about tests generally, in case you were wondering.

The comment you responded to says unit tests give confidence, you seem to agree.

"Tests generally" and unit tests are pretty much the opposite camps in testing world.

I agreed about the word "confidence" and the importance of testing for confidence. You read something into my comment that I did not write nor did I intend. Good for you, I guess.

If you can fit the entire system in your head, it's probably a relatively simple codebase.

I've written a Photoshop file format parser twice, and trust me—between byte alignment changing constantly throughout the spec, at least three different string encodings, sub-formats and errors in the spec—it does not "fit in your head." Certainly not enough to confidently add features without worrying that you've broken the blend mode parser.

And of course, if you can and do fit a complex system in your head when no one else on the team can, congratulations: you've made yourself into a bottleneck and flight risk.

> If you can fit the entire system in your head, it's probably a relatively simple codebase.

Or it has good boundaries and I never have the need to load the whole system in my head.

You're not loading the whole OS kernel in your head every time you call fopen(), do you?

> Photoshop file format parser

Would you say that PSD format is an example of good design? Is complexity inherent to the problem domain, or is it accidental?

You are forced to replicate complexity created by other engineers. Woe is you.

My bet is that Adobe also has a lot of unit tests, and contrary to what unit-testing advocates claim, it resulted in a terrible design you are dealing with.

My friend, half of software engineering is handling complexity created by other developers. If you're implementing a Mastodon client, you don't get to just refuse to deal with OAuth because its design offends you on a philosophical level.

But I get to refuse to turn OAuth into a cross-cutting concern in my application. You won't see refresh tokens referenced in the UI code. And I get to use a 3rd-party library that solves 90% of the pain for me.

I strongly disagree on the ratio. If you're building Mastodon client, most of your effort should be spent on features and UX, not on OAuth and the like. If half of your time is spent on those, either you're at the beginning of the project, or you're at the beginning of your career. Or maybe you just care about engineering itself more than the end product.

If you peek at that third party OAuth library, you will discover… it's complex. This is what I mean about pushing the complexity around on your plate. You can make it someone else's problem (and become dependent on them), but the complexity is still there. There is a minimum level of complexity that any OAuth system needs to have in order to function. QED

OAuth library is a perfect example of lousy engineers inventing complexity.

Here's a talk where Jack Diederich mentions it: https://youtu.be/o9pEzgHorH0?t=870

It is indeed complex: 200+ classes. Plenty of unit tests and mocks.

And then there's his implementation:

- 0% unit test coverage.

- Lightweight at less than 200 lines, including blanks and docstrings.

QED.

This is great example of unit tests, but I don’t think it’s comparable with OP’s scenario. Math laws are ‘fixed’ and well known, you can just write fixed tests to find the discrepancy between the laws and your implentation. Where OP’s point is about ‘startup business logics’, the ‘laws’ here are constantly changing all the time, which means you will constantly throw out most of your tests along the way.

I agree with the sibling comment that says confidence is a good word, because it is.

I've done three or four major refactors on a FOSS project of mine. The refactors went without any hitch because of an almost excessive test suite.

Not having tests for complex things make such constructs feel fragile.

If you're building simple things, tests can be overkill.

If you're building complex things, tests pay for themselves rather quickly.

If you're building complex things... most likely you should be building simple things.

Most software in the world implements business rules defined by people who are way less smart than your average engineer. And yet our software is more complicated than ever.

There's a difference between accidental and necessary complexity.

Not everything can be simple.

Mathematical algorithms are an example where there's necessary complexity and unit tests backstop defects when you refactor code.

^ THIS, 100%. Some SW engineers truly believe that complexity is the enemy, and that it can always be eliminated. Sometimes it can, but other times, you can watch developers push complexity around on their plate to avoid eating it.

They move logic to a different library, break it into 10 libraries, introduce layers of abstraction, try increasingly complicated View/Model architectures… And ultimately the code is no easier to understand or manage than when they started.

But complexity is the enemy. Where it can't be eliminated, it can be contained. That's what software engineering is about.

It is nearly impossible to find a complex domain that doesn't naturally break into smaller subdomains.

That's because "domain" is human concept. The ones that are too complex and don't have natural boundaries within them just don't stick.

There are two main reasons software becomes complicated:

1. It is over-engineered at the beginning of it's lifecycle.

2. It is under-engineered in the middle of it's lifecycle.

Interestingly, unit tests contribute to both of these.

For an engineer it's always hard to admit that your system is complex because you failed at engineering. It's much easier to claim that you're working in a complex domain: you get to shift blame and boost your ego at the same time.

I get it, and this is why a lot of SW engineers think they could solve any systemic issue in the world by breaking it down into sub-problems. Unfortunately it isn't true.

I'll give you a concrete example: AML (anti money laundering). If you build a fintech with zero dependencies, you will need to deal with this. It's not manufactured complexity—it's an authentically complex problem which stems from 1) the actual law and 2) the risk profile the business is willing to assume. Now, you can come up with clever ways to make the rules of your AML policy composable and easy to change, but the reality is that you will immediately hit an ocean of edge cases that require ever more nuanced mitigations. You can make keep subdividing and decomposing the problem, but you will end up with an architecture diagram which is (wait for it) complex. So yeah, you shouldn't over-engineer before you launch, but eventually, your AML system will take on a certain amount of unavoidable complexity.

Now try to do all of the above WITHOUT unit tests, and see how far you get.

I fail to see how unit tests are going to help. If rules are independent, overall complexity is low, even if there are thousands of rules. If rules are dependent, unit tests are useless.

> introduce layers of abstraction, try increasingly complicated View/Model architectures

> you can come up with clever ways to make the rules of your AML policy composable and easy to change

These are the exact design decisions that are typically accompanied by unit tests.

Those are the design decisions accompanied by developers who think complexity is the enemy and seek to "contain" it at all costs—even when it adds hierarchical/organizational complexity.

And yes, if we're being pedantic, an AML system would need unit and integration tests—but the question is whether or not tests as a whole are useful. So unless you're arguing that an AML system can be made so simple that no tests are required and it "fits in your head," let's agree that testing has its place.

> Those are the design decisions accompanied by developers who think complexity is the enemy and seek to "contain" it at all costs—even when it adds hierarchical/organizational complexity.

Not in my experience. Overengineering and unit-testing always come hand in hand, likely because both are the "enterprise" way of doing things.

> the question is whether or not tests as a whole are useful

Huh? Nobody ever asked this question.

Questions I'm always debating are:

1. Whether unit tests are useful (no in 99% of cases)

2. Whether more tests = better (no)

3. Whether you should invest in automated testing before manual becomes painful (no)

> unless you're arguing that an AML system can be made so simple that no tests are required and it "fits in your head,"

Some tests will be required. But yes, it should fit in your head, otherwise you cannot reason about it. Building AML system that you can't reason about is a road to failure.

It doesn't mean that you need to load implementation of every single rule in your head before you make any change. But you need to fit, for example, the idea of rule engine and enforce constraints on rules working together. If you build rules that can do anything with no constraints (depend on each other, override each other), then you're screwed. You're going to be playing whack-a-mole with your tests, and while it will feel productive, it's a death sentence.

And that's why I said "most likely".

That said, mathematical algorithms:

1. Rarely need refactoring. If you often refactor your math, you probably mixed it with something else (unless your business is math).

2. Rarely have state. You're always dealing with values, not entities.

3. Rarely have dependencies. There's nothing to mock, so the boundary between unit tests and integration tests is very blurry.

4. Usually have a formal oracle of correctness.

So yeah, math is a place where unit tests might be ok (if you truly have a lot of complex math).

Reply on news.ycombinator.com