Hacker News Reader

Comment by msoad | Hacker News Reader

In my experience unit tests don't catch many bugs. On other hands end-to-end tests always surprise me with the bugs they catch. Based on that, I'm not sure advising for more stubbing is a good advise. Maybe I'm wrong but that was my experience building UI apps.

View on HN

The distribution of when bugs are caught, and what bugs are caught, by different kinds of tests is very different. Units tests catch bugs while they are being written and when the units they test are being refactored. The bugs they catch are trivial, silly, 'would have been caught anyway'. Outside of those activities, they rarely catch bugs. On the other hand, integration tests more consistently catch bugs and catch bugs you didn't foresee, have to investigate before understanding.

You don't remember the last time a unit test caught a bug, because on average it is much longer ago and if it was a test for a unit you wrote this morning, you usually blame it on 'work in progress' and think you would have caught that trivial bug anyway.

I think the nature of units tests make it easy to underappreciate them.

This is too true. A team I was on started seeing way more value in integration tests and ended up writing way more integration tests than unit tests.

What ended up happening is that the code quality ended up worse because they stopped running tests very often because there were too many integration tests and they took too long to run. Sad part is, a lot of those tests would have made a lot more sense as unit tests.

In my experience building backend apps, unit tests are way more effective at catching logic bugs than end to end tests.

End to end tests are great for catching connectivity bugs though.

UIs tend to have a ton of connectivity and little logic/computation.

An added benefit of using unit tests over integration tests is that unit tests shows you what went wrong very clearly. If an integration test fails I have literally no idea where to start hunting for the bug. It could be located at any level in the stack.

I highly recommend this brilliant talk by J.B Rainsberger on how integration tests are a scam: https://vimeo.com/80533536

If you don't have that much time to write tests I'd recommend integration tests. Then at least you have some tests. Treat integration tests as smoke tests. If there is smoke then you can start looking for the fire. Integration tests are always better then no tests at all. At least they tell you there is a fire even if it is vague in telling you where it is.

As someone who always writes unit tests before any production code, I don't know how I would have the time to not write tests. It drives the design and reduces time debugging by a ridiculous amount. I work with information systems though, might be different for other fields.

Since I've switched to a language with a richer type system (Scala) I've found a lot of what I used to do with unit tests can go into the type level instead (see e.g. http://spin.atomicobject.com/2014/12/09/typed-language-tdd-p... - but in Scala the techniques are a lot less verbose to apply).

I am most familiar with Javascript & Python, but I think you're onto something having fiddled with Java for 6 months or so. Some logic obviously can't be replaced with only a rich type system, but a lot can.

It depends a bit on the problem you are trying to solve and the structure of your code. I have projects where only end-to-end tests find bugs and others where unit tests are helpful. What is never helpful is imposing some sort of semi-religious "thou shalt do x" in every case.

That's why you don't stop with unit tests. You make sure to automate testing of ever layer of the application. It does require a bit more effort, but it's it's a savior when the UI breaks, and it's not something obvious yet breaks the user experience.

I prefer to write end-to-end tests first, then fill in integration and unit-tests on a as-needed basis. If I am finding it complicated to write, or if experience from production says it is a problematic area.

After all, the important thing is whether your application/service/library does what it promises to its users/consumers. Whether some function/class deep inside works or not is mostly irrelevant (can help in fixing the issue faster though).

Unit-testing-first has a tendency to make the tests very fine-grained, often need changes when you refactor internals (how do you know test did not break?), and needlessly keeping rigid internal details that one should have the freedom to change. Have seen plenty of code being kept around because it had lots of tests - when it was not actually used for anything at all!

Note: Above approach is coupled with a pretty strict adherence to keeping projects small and independent, preferably <10kLOC.

For me, the value of writing unit tests is in the writing of them more than the running of them.

Alternately, the reading of them as well.

Reading the unit tests can help you understand an unfamiliar project by acting as documentation, telling you what is expected behavior.

Yeah...for me, unit tests are the gift that you give to the developer who will be maintaining and improving the code 6 months from the time you write it. Beyond documentation, it gives him/her the ability to refactor the code without fear of it breaking horribly and tells that developer which behavior of the application is intentional and which was accidental. And before you say, "Screw 'em...I've got a code freeze to meet," consider that the person maintaining the code 6 months from now is probably you.

IMO, this all relies on having good unit tests.

If you have bad application code that's hard to read, what's to say that the tests will be any better?

How would you have good unit tests if you have bad application code? It's writing the tests that forces you to improve the application code, because otherwise you won't find it straightforward to write the tests.

Admittedly not in node, but I've found that writing end-to-end tests but substituting different 'units' for in-memory or not depending on the test marries the best of both worlds. In-memory tests tend to focus on behaviour.

In the rush to shoe-horn tests into a taxonomy, or 'integration' vs 'acceptances' vs 'unit' we often loose sight of the actual value proposition. Assisting design and building confidence to make changes.

Though you also have to ask how many additional bugs you'd have if there were no unit tests.

> In my experience unit tests don't catch many bugs. On other hands end-to-end tests always surprise me with the bugs they catch.

Obviously if you've walked the walk with unit-tests and done proper TDD and all that, you've caught the bugs before submitting your code.

Ofcourse the remaining bugs (which there will be plenty of!) will be found in your end-to-end testing where all the bits are tied together.

This is only natural and does is no way suggest unit-testing was a wasted effort.

"Obviously if you've walked the walk with unit-tests and done proper TDD and all that, you've caught the bugs before submitting your code."

Unit testing and TDD are quite separate things. Perhaps you should try writing unit tests after you write your code and then you can come back and add value to the conversation.

> Perhaps you should try writing unit tests after you write your code and then you can come back and add value to the conversation.

Nice snark there. Feel free to keep it to yourself.

To add actual value to the conversation (as opposed to your contribution), I can very much recommend the book "Working Effectively with Legacy Code"[1] for how to handle unit-testing in the scenario of existing "legacy" code-bases.

It's full of useful tips and methods to get testing in place "anywhere" and has a pragmatical (as opposed religious) approach to getting it done.

To spark some interest: The book defines "legacy code" as any code not covered by unit-tests.

It may be seem dated (from 2004 and all), but it's been the most useful book I've read on unit-testing by far.

[1] http://www.amazon.com/gp/product/0131177052/ref=as_li_tl?ie=...

You came along and dismissed the parents views without any real thought.

If you write the tests afterwards you can better gauge how well they catch bugs. It sounds like the parent has tried this and has found them pretty useless.

I've personally found that unit tests are not worthwhile for many components, and yet are critical for others.

I've been meaning to buy that book for a long time, thanks for the refresher :)

I've found that there are way more books for "greenfield" software development and not so many books for what 60% of people actually do (maintaining other people's projects, legacy or otherwise).

my experience is like yours that unit test developed in development are rarely useful.

however, having a framework in place helps a lot when working at user submitted issues: those are the issue one should strive to replicate trough unit test. In a six month of production, you get a fairly robust suites of test on all corner cases undiscovered in development.

Try to add fuzz-testing into your unit tests.

You can use both things...

Doesn't sound like you're unit testing well then. I'd take good unit tests over integration tests any day. Integration tests are inherently flaky.

Your sentiment is strange to me - I've only ever seen unit tests as absolutely worthless. The reason integration tests are flaky is exactly the same reason they are valuable.

Someone who checks in broken code will also take flaky tests as an excuse not to look at the build results, or to convince themselves that the failure is caused by flakiness and not their changes.

If I had a dollar for every time I asked someone why they weren't fixing their shit, and they answered "Oh, I thought the build was failing randomly again." I'd have a lot of dollars.

The fact that your tests are flaky at all means that the integration tests have revealed a bug that you need to fix. The tests have done their job. The fact that people aren't looking at the results speaks to a larger cultural issue.

Not if it's a timing error in the test harness. That's usually what people mean when they say flaky tests. Selenium, especially three or four years ago, would simply fail occasionally no matter how good your tests are.

Yes, there are bugs that look like bad luck, but are really flaws in your logic. I have fixed a lot of bugs that look Random, but eventually you're left with, for instance, selenium just refusing to click a button.

Unit testing well takes 3-4x as long as integration testing well. Sometimes it pays off; other times it doesn't.

Reply on news.ycombinator.com