Hacker News Reader

System level integration tests are also proof that the work we did is correct and we can move on. And it is better proof than unit tests, because we aren't mocking everything. And they are far more likely to survive a large and risky refactor than unit tests.

Don't confusing testing in general with unit testing. Just use the right tool for the job. If your unit tests aren't catching a material number of bugs compared the the effort spent, compared to other testing methods, then don't do them. Unit tests have benefits such as quicker execution time, etc. - but that has to be weighed against cost.

System level integration tests also tend to be more flaky. Both unit and integration tests are useful.

Unit tests are also pretty quick to write once you have the mocking setup.

That's one of the dirty little secrets with end to end tests that almost no one talks about. You will probably spend more time running after ghosts in the machine than finding actual bugs.

I've had that experience too. But also the opposite.

I've worked both on code where writing the tests was more effort than the code, and on code where writing the tests was easy, quick and helpful. The latter makes sense, after all a good test is straightline code, zero ifs, zero loops. But the former?

I think the key is that mocking should be used sparingly, but without hesitation.

Plenty of people talk about it:

http://googletesting.blogspot.co.uk/2015/04/just-say-no-to-m...

The dirty little secret that nobody talks about is the cause of the ghosts: poorly engineered tests.

They usually tend to be quick to run too; integration tests are rarely fast enough to run during development.

>System level integration tests also tend to be more flaky.

That's usually a sign that they've been engineered poorly or you have bugs in your code.

System level integration tests need appropriate environmental isolation and solid asynchronous & multithreaded code. Nobody can be bothered to write these properly for tests, hence the flakiness ("ooh let's just insert a sleep here" / "eh, does it really matter which version of postgres we run?").

The biggest benefit of unit test isn't catching bugs, but making aggressive refactorings possible.

Huh. Here was me thinking that was what static type systems were about. ;-)

static type systems != unit tests

Static typing does not allow you to refactor code with confidence that you didn't break the application logic. Only automated tests can do that, regardless of type system.

Sure. But an awful lot of failures in refactoring in dynamically typed languages would be avoided by a static type system.

As it happens, I am a Rust fanboy with all that entails, so I (like everyone else in such discussions) am clearly biased.

My point was in jest, as the “;-)” (because HN simply discarded a U+1F609) indicated.

That's what strong type systems are about. A strong explicit dynamic type system with appropriate tests can track down bugs more easily than a statically typed weak type system.

One could also say that bad unit tests make aggressive refactoring impossible. Both sides of the coin don't necessarily make for a persuasive argument.

units are likely to be aggressively refactored, no?

My experience is that more often than not, unit tests get thrown away in aggressive refactorings, and system level tests survive. If you have a system that depends on units A, B and C, and refactor it to use a totally different structure divided into units D, E and F, then the unit tests for A B and C get thrown away, as they don't really have an equivalent in D, E and F. But the system level test stays, as the system as a whole does the same thing. So, the system level test is what guarantees the safety of the refactor.

That is not always the case, but that happens in most of the large refactorings i've been involved in.

This is true of any code you throw out. If you throw out the code, you should definitely throw out the corresponding tests.

It doesn't invalidate the fact that unit tests increase your confidence in your own code.

The point is, as your tests get more and more fine grained, and the units themselves get more and more fine grained (in my experience these go hand-in-hand with a TDD approach), the changes that the units themselves maintain intact approaches zero, because otherwise the only refactorings possible are completely trivial.

You can achieve this with non unit tests as well.

Sure. I should have just said "test".

Unit tests can also prevent aggressive refactoring by being heavily coupled to the implementation due to the nature of mocking essentially private details of the code under tests. You then have to aggressively refactor your tests along with your code and all the safety that the test would give is gone.

"Unit tests are [...] proof that the work that you just did is correct and you can move on."

Unit tests only can proof your software to be buggy. Even for the extremely simple "two, a function that, given no arguments, returns the number 2", your unit test can't verify that to be _always_ true. It may fail to return on Tuesdays (https://bugs.launchpad.net/ubuntu/+source/file/+bug/248619), internally use a web service that doesn't work on leap days (http://www.wired.com/insights/2012/02/leap-day-azure-outage/), other calls may overwrite the 'constant' it returns (http://programmers.stackexchange.com/questions/254799/ever-c...), etc.

You can verify it to always be true as long as the dependencies (which you've mocked) hold. If they don't... if that's a possibility then you might need some error recovery somewhere, not necessarily in this particular unit.

> Even for the extremely simple "two, a function that, given no arguments, returns the number 2", your unit test can't verify that to be _always_ true

That is true, I don't think anyone argues against that point. What unit tests do, is verify that under the code works under specific conditions (as defined in the test's set up). When unit tests fail under those specific conditions, then you know there is a problem.

Another thing I like about unit tests is that it is much easier (and faster) to test different combinations of conditions

In functional programming languages, the type of "two" would indicate whether it has side effects. If not, then "two() == 2" is conclusive proof.

And they are also "executable documentation". If you as a developer want to know how to use something, unit tests can be of help.

Then, writing tests for the sake of code coverage, is like judging take a driving license exam and being judged on the streets you visited or how much gas you used. Tests need to be written with logic verification in mind.

I am deeply sad this article got upvoted. People should have a sense of responsibility when giving visibility to stuff like this.

Have you seen people reading tests much in practice?

I've heard the "executable documentation" line but my general impression is that people usually read the production code. They'll only look at the tests when they fail or they need to modify them.

That's assuming the code even makes sense. Have you ever been looking at some code, having no precise idea what it does, with no commit history to give an explanation? That's when unit tests would be useful documentation.

Sorry if this is what you were specifying when saying "executable documentation", but have you seen the way Rust handles documentation tests [1]? The idea is to provide documentation via comments before a function definition, and inside this documentation you can provide example uses. The test-runner automatically extracts these examples and verifies them. In this way, your documentation and tests are always in agreement with each other, and a developer can be aware of expected behavior by reading either the code comments or the auto-generated html docs.

I'm rather fond of this system, personally.

1: https://doc.rust-lang.org/book/testing.html#documentation-te...

Thanks, you got me testing rustlang testing facilities. Very enjoyable. Makes you want to do low level programming 10x more than with c/cpp.

If I start working on a new project and they have tests, I always start there. Heck, if I resume working on an old project of mine, I start with the tests.

> I am deeply sad this article got upvoted. People should have a sense of responsibility when giving visibility to stuff like this.

Did you actually read it? Because what you just wrote, is almost exactly the same as what the article author wrote.

His point, as I read it, was that unit tests can have purpose when you don't blindly make them for code coverage.

His case for having unit tests was:

> Keep unit tests that test key algorithms for which there is a broad, formal, independent oracle of correctness, and for which there is ascribable business value

But I don't think he would disagree with the case of providing some "executable documentation" as well. A few lines of code that demonstrate how to use a class or library, along with expected results, is definitely valueable. And those are exactly the kind of interfaces you don't expect to change even when refactoring.

> Unit tests are not albatrosses around the neck of your code All code is a liability. The less code usually the better. KISS principle.

> writing your code in a why that makes it easy to test generally also improves the API and design of the entire system You can get the same result applying SOLID principles, without the cost of maintaining extra code. The presence of unit testing doesn't guarantees easy to test code either (and this is the worse case scenenario, bad designed code with a lot of test coverage that is broken in each refactor).

> is it possible that tests get in the way of completing a project in a timely manner? Are you dismissing this as a point? Time is a resource and writing unit tests adds to that cost.

Unit testing needs to:

* Make you faster because there are less bugs in the long run.

and/or

* Prevent money loss caused by bugs compared to the money lost caused by the opportunity cost of writing unit tests instead of new features.

Also when writing unit tests developers are less prone to refactoring when requirements change as it will require to refactor also all the unit test affected.

You can agree or disagree with the points int he article, but it is NOT just "bullshit".

Tests are not proofs, types provide proofs. Tests demonstrate that the exact conditions being tested actually work. That's not a bad thing but when working with static typing and robust type systems the need for unit testing goes down tremendously compared to dynamic languages.

While I agree with you to an extent, I'm a huge Rust fan, longtime C/C++ and Java engineer, I disagree that a strongly typed language allows you to not write unit tests.

Unit tests verify logic, the compilers only guarantee syntax. Which yes, handles a huge set of issues that in a dynamic language would require 100% test coverage, you still need to check the logic of the code. Thus, unit tests are still a huge benefit.

One reason I love Rust is the build in #[test] feature. Builtin tests from the ground up, awesome.

I didn't say not write tests but that it reduces the volume of tests needed for largely the reasons you stated

Types provide semantic information as well so checking verifies semantics as well (extent depends on the strength of the type system).

Saying you're a Rust fan and then saying compilers only guarantee syntax when the entire point of that language is to have a constrained subset in which types guarantee no data races/leaks at compile time doesn't make sense to me.

I didn't mean to imply that rust doesn't provide more than just syntax. You are correct that rust provides a huge number of guarantees around semantics, but honestly, I think it's not a traditional compiler, it provides a ton of features that you don't find in C compilers for example.

Rust is the greatest thing since sliced bread, my comment was only about statically typed and compiled languages in general. I think you read more into my comment than I intended.

Of course it's a compiler. A compiler verifies that its input conforms to the syntax and semantics of its own language, and generates machine code (or some other language) equivalent to the input.

The only difference is that Rust's semantics is waaayyy more strict than C's.

I'd counter that with dynamic language, easy modularization and the ability to create single function modules that are composable, and written as such, that code is more easily understood and doesn't require the same type of testing as more complex systems in static language, that lend to layers of indirection and complexity for the sake of testability.

I'm not saying that testing isn't needed in either place, but a lot of the times "enterprise" practices that leak into scripted environments make code far more complicated and difficult to understand than it needs to be.

Why do you assume that modules in a statically-typed language are necessarily more complex? (Except for the type annotations, of course.)

Not every statically-typed language is Java. ;)

True, it does depend on the language... C#, for example is limited to Namespaces and Classes, you can't write a simple function without a lot of wrapper... partial & static classes help, but it's not the same.

Though, I will say writing tests against and with a dynamic language tends to be far easier when things are written with modularity in mind.

It gets voted up because there's a grain of truth. Some (usually inexperienced) programmers really do write a lot of mock-heavy, boilerplate tests that don't verify any behavior of importance.

Maybe you haven't seen it. But keep in mind that consultants get called in when something has gone wrong. They tend not to see clean codebases by teams that are working well.

Actually, unit testing only ensures two things. 1. Tha you've at least looked at and/or considered every piece of code at least twice. Two, it adds friction when writing complex code that isn't modular or a functional composition.

It is not a guarantee of anything beyond that. However, having something close to automated deployment means that you should have very close to 100% coverage if only to ensure the above two things.

One thing I've rallied against at several points, more so in scripted environments is DI/IoC, it's often not needed, and there are often simpler solutions than what is generally used. The benefits are being able to have multiple targets for an interface, and being able to more cleanly unit test some systems in a strongly typed compiled language.

All of that said, I don't always go for thorough testing, but I try to write code in such a way that it's more modular, and would be easier to test, should the need arise.

> The benefits are being able to have multiple targets for an interface, and being able to more cleanly unit test some systems in a strongly typed compiled language.

The benefit for DI/IOC for me is mainly not having to worry about how to compose the system.

If some new module I'm writing needs a particular dependency, then I just ask for that in my constructor.

I neither need to know nor care how to set it up or initialise it. That's taken care of by the container - either automatically (convention based registration) or by the maintainer of FooService.

This makes it dead easy to create smaller integration/test harnesses: run the normal container build step, and then resolve an instance of my class, out it pops with all the dependencies handled.

> they are proof that the work that you just did is correct and you can move on

Well... they should be. The number of Tests I've seen that are functionally useless is far too high.

Because what you test is much more important. I know I'm not imparting new wisdom here but if your test can't survive a refactor it's probably a) far too fragile b) poorly written and c) testing the wrong thing.

I actually would not be surprised if a large percentage of Unit Tests are useless, other than coverage stats, but I agree with you that this article is very full of hyperbole to the point of ruining any point it might have had.

Yes, I totally agree with this. I've worked in large orgs, where there were QA engineers that needed work, would come and ask, what tests they could write. My response was always, "well, I think I handled all the positive use cases, so you can add negative test cases", which I've never really seen a huge point in (though that depends on the interface contract).

> given that this was written by a consultant, is it possible that tests get in the way of completing a project in a timely manner, thus causing a conflict of interest in terms of testing?

That's not really what conflict of interest means, but it's certainly possible that his being in that position gives him a different set of priorities than, say, a project manager, and he's working backwards to come up with an argument that justifies his opinion.

Perhaps it would have been better to phrase that as 'competing priorities', but you got the point.

>they are proof that the work that you just did is correct

Nope. They may be evidence, but they aren't proof. Proof comes from formal systems like type systems.

I've also found unit testing to be a fairly good way of documenting code. A unit test will often times shine light on an angle traditional documentation doesn't always cover.

> I have a serious question, given that this was written by a consultant, is it possible that tests get in the way of completing a project in a timely manner, thus causing a conflict of interest in terms of testing?

Doubt so, he works for a software testing consultancy. They've been around since 1994 apparently. http://rbcs-us.com/

I don't belive that you read the document. You are arguing against points which have not been made in the article or which are more nuanced than one would believe by reading posts such as yours.

The article is very reasonable, logical and a good counterpoint to the countless TDD anecdotes which obsessively focus on unit-tests to the detriment of any other types of testing or design practices.

It's not a proof that your work is correct. It's a proof that a few cases, out of potentially millions, or billions, work.

Yeah, anyone that invokes Turing machines to claim unittests don't work has completely missed the point of testing their code. Perhaps if the article was talking about "formal verifications of code" it would bear some resemblance to good advice.

> If it's unit testable, then it has decent separation of concerns

But then the separation of concepts is driven by the testability, not the problem per se. It means that there's possibly a lot of complexity just for supporting testing.

I don't ever unconditionally trust code, but my faith rises:

* a little, if it passes a decent unit and/or integration test suite;

* a fair bit more, if it survives a round with humans who didn't write it but did set out to break it;

* quite a bit more, once it's been in production for a while without complaints tracing back to it;

* gradually approaching, but never reaching, perfect trust the longer it survives exposure to the world.

A lot of bugs will make it past step 1--even through tests with 100% code coverage. Hopefully no one takes them as anything near proof of correctness.

I'm not a strict TDD follower, yet a byproduct of my development process leaves me with unittests. They give me confidence that the code I just wrote behaves the way I want it, and deleting them after that just ensure that refactoring isn't on the table without a significant time investment.

The drawback of this, is that the test suit grow fast, and need to me groomed pretty often, but it's a fair price to pay and in the end it's faster for me, and produce far less bugs.

>Unit tests are not albotrosses around the neck of your code, they are proof that the work that you just did is correct and you can move on.

IMHO, this is a failing of the type system. If you have functional purity, you don't have a need for unit tests, as they are blended into the code you write. Functional purity gives you the power to formally verify the correctness of your program.

> but to say they are useless is just sloppy

or link bait

Yes, that.