Comment by hinkley | Hacker News Reader

Comment by hinkley | Hacker News Reader

Parent Full thread

Very, very few people know how to make cheap tests.

I have one guy who writes fixtures that couples all of his tests. When I need to add functionality, first I have to replace his fixtures with basic stubs, just so I can add another bit of functionality.

And we have two people who copy his coding style. Probably to fit in.

I think, based on my own experiences and working with people learning to write tests, that 'simple' tests often feel more like the pejorative meaning of the word. They get uncomfortable, like you're debasing yourself to write such boring code. Surely I can write more interesting tests.

Yes, you can write more interesting tests if your goal is for nobody to ever touch your code. Mine, mine, mine. The best tests, I know where my new code went wrong simply by reading the test output (not the test). If I refactor I should be able to maintain that coverage even if the order or structure of things gets completely switched around.

Complex tests begat complex tests. If your old test is 2-3 lines of code, I just delete it and write a new one that does the same thing.

I never got to understand testing. It's a thing that every developer is supposed to know and do well somehow.

I always end up writing tests for my programs, but with all the stubs and fixtures and weird stuff I get into hairy problems that I don't understand. Tests are supposed to be helpful, but writing good tests is ridiculously hard.

Or maybe it's just because I'm a perfectionist, and I should be happy with crappy tests that at least give me a little bit of information about regressions

It's one of those things where people who really believe in it are very vocal and will shout over anyone who dares to think there are any problems. Not everyone loves them.

But, as I'm a test-lover, I have to say they're pretty amazing if you can get everyone on-board and writing good tests.

As for how hard they are... It depends on how the code is structured. If the code is designed to be testable, it's pretty easy to write simple tests. Integrations get a little tougher, but as long as you know what the other side should return, it's pretty easy to stub them out.

Testing existing code that was not designed to be testable is a lot harder. It typically will introduce a lot of side effects and won't be properly separated. It can be tested, but IMO just isn't worth the hassle. I've found that you end up spending as much time writing tests as you would have fixing the bugs that arise... And code doesn't get used forever. Modern dev culture insists that someone call for a rewrite eventually, no matter how good the code is, unless you're working in a system that already has decades of legacy code.

My advice would be to start a small project of your own (or wait until you get a new project at work that's from scratch) and write tests from the start. It'll teach you a ton about testing and help you get the system off the ground quickly and solidly. I did this for the last big server-side system I wrote and the tests helped tremendously. The bugs that QA found were generally things that I didn't consider or that they designed wrong. The tests helped me get everything working and keep it working while they changed all kinds of things.

Sadly, once that project ended up in the hands of other devs, the tests slowly died and I was the only one trying to fix them. I eventually went to another project for a while and came back to find the tests basically didn't work at all, and they died completely at that point. But they did their job at the start, and were totally worth it.

I've seen a lot of people say not to design code to be testable. That instead tests should work around whatever shape the code should be in "naturally". Writing this out this seems stupid, it's been pretty influencial on how I code (for the past year or so, I'm a complete beginner).

To me, the shape the code should be is largely the same as easily-testable code anyhow.

You want functions that do small, self-contained things so that you can reason about them efficiently. That also makes them easier to test.

Yes, sometimes you end up with a monstrous function because of requirements... But then, that's how you have to test it anyhow.

That said, some of the more intense testing things, like dependency injection, definitely change the shape of the code into abnormal forms. So I can agree there that it's adding complexity to the code that it shouldn't, just for testing. Some projects may warrant the increased complexity to trade for stability. I don't think most do.

Small functions, sure.

The most common case of this I've run into is should I use dependency injection just for testing. My function creates a Frob and interacts with it. By default Frobs persist changes to a DB and sync over the network. I don't want to do that in a test. Now I need to pass a frobFactory of (FrobConfig) -> Frob to my function.

Sure in theory I could use my new configurability for all sorts of things, but in practice I only end up using the alternative path in tests.

When do you not end up needing something like this?

> I have to say they're pretty amazing if you can get everyone on-board and writing good tests

This is my main complaint about code coverage metrics. I want you to be good at test writing before you write the bulk of your tests, not after. It’s hard enough to get people to write good ones, let alone go back and fix the messy ones they wrote badly (recall I said good tests are easier to replace than bad ones).

If you write a bad enough test it can take longer to fix it than throw it out and start over. But since you didn’t clearly capture the constraints the first time, do you dare delete it? What aren’t you testing after?

I kept telling people for years that I'm awful at writing tests. You're coming to me for advice because I'm less awful than you are.

Nobody really seemed to get what I was trying to tell them. It parallels my experiences with Scrum "Well if it's not working you're doing it wrong". I love tests, and I hate tests. It's a very codependent relationship.

I did run into a string of people who had either very good value statements or very good questions, which helped.

But I still find tests I've written that violate the guidelines I try to set for the team. There are probably few moments in the week when I have, to be completely blunt, my head so far up my own ass as when I'm writing tests. The only time I am juggling more info at once is when I'm debugging, and in debugging I don't have to worry about the future reader. I doubt I am unique in either regard.

Unless I go out of my way to make the tests as agonizingly boring as I possibly can.

Pain is information, and an encouragement to change what you’re doing or how you’re doing it.

A little reorganization of your code can make an order of magnitude difference in how fiddly the scaffolding is. Does that mean your code structure is informed by tests? Yes, yes it does. But classic car designs are informed by access for repairs, and some vehicles are cherished for their ease, others become a topic of complaint and camaraderie. Really any industrial design has an element of this.

Oh absolutely, keeping things simple should never be restricted to just production code. Testing code should be simple too.

The idea that tests inhibits refactoring I find problematic though. Good tests enable refactoring. If the tests are too complex for anyone to change the code, the the code is too complex. Any complex system should be composed of simple components. Now this applies to monoliths as well as microservices.

I agree it is easy to fall into the trap of thinking that tests are boring though as devs become more experienced there should be less of that thinking, hopefully.

>> If the tests are too complex for anyone to change the code, the the code is too complex

In enterprise software, the pattern i see is specifically coupling rather than garden variety complexity. There’s a really pervasive idea that unit testing means 1:1 mapping of test class to production code class. To put it another way, there’s this really common but terrible idea that tests should map to your current implementation rather than the behaviour you want.

Worse, some check in test cases that should only exist locally during development, they do things like directly test private implementations. A related issue is no one ever deletes tests.

Testing done well, is utterly transformational. I took a piece of enterprise software, it wasn’t badly written per se but it was hard to release. 80%+ of releases were rolled back due to issues found in production. For each release, 1 of 7 “experts” were required to review your change to this component (to make it worse it changed fairly frequently). The problem was only 2 of those 7 had a track record of only approving good changes. I’m ashamed to say i was one of the other 5 - i reviewed and approved a couple of changes that had to be rolled back from prod.

So, i thought sod this and i grabbed the stingy nettles. The code was not written to be testable but fast forward 16,000 (i had a small team) test cases which execute in around 16 minutes and over the past years that software continues to be released as frequently but has NEVER had a prod rollback since. The 7 experts idea is disbanded, it’s just a normal pull request these days. Lead time on changes is about 1/3 what it was - all the faffing about exploring model output in excel for hours if not days is gone, it’s just all captured as tests.

It wasn’t cheap to do this but back of the envelope calculations, it repaid within 3 months just on lack of prod outages.

> A related issue is no one ever deletes tests.

I think this is a huge problem. I’m not sure why it exists.

“If we have more tests it will be harder to change the code.” is an argument people make sincerely. Then... just delete the tests. If your argument against having these tests is you can imagine a point where they become a burden, just keep in mind that it’s really, really fast to delete code. “Untested” is a state we can get back to at any moment.

Maybe the tests will provide a lot of value until then. Maybe if we do a big refactor we’ll delete the old tests that are too low level and make new ones. That’s fine.

I joke that we should make the team watch one of those Hoarder TV shows, but it's a Tears of a Clown sort of situation.

Try telling people that you're deleting a flaky test and they'll stop working on their current story to try to stop you. It's as predictable as the phases of the moon.

> To put it another way, there’s this really common but terrible idea that tests should map to your current implementation rather than the behaviour you want.

I think if you were prioritizing, then the most bang for the buck comes from making your white box tests as stupid-simple as you possibly can. Since those demand a particular implementation.

> If the tests are too complex for anyone to change the code, the the code is too complex.

No, more than that, it's a sign that the tests are not just testing for how code behaves, but also its implementation details.

With well-written tests, you can completely re-structure some code and, if it still does the same thing, the tests for that code will still pass.

If the code being changed has several components with their own tests, then either those components have no interface change, in which case the previous paragraph applies to them, or the do, in which case their tests should be easy to adapt.

> Good tests enable refactoring.

I don't disagree. So when I do find tests that inhibit refactoring, what do you suppose I infer about those tests?

Reply on news.ycombinator.com