Hacker News Reader

Comment by avip | Hacker News Reader

Ok let's stop writing UT and see what happens. Wait... we've already tried that, and we know the result pretty well:

1. In dynamic languages, simple type mismatches, wrong variable names etc. are now caught in "top level system level test". Yes these are bugs that should have been caught by a compiler had we had one.

2. There's no documentation as to how something should work, or what functionality a module is trying to express.

3. No one dares to refactor anything ==> Code rottens ==> maintenance hell.

4. Bugs are caught by costly human beings (often used to execute "system level tests") instead of pieces of code.

5. When something does break in those "top level system tests", no one has a clue where to look, as all the building blocks of our system are now considered equally broken.

6. It's scary to reuse existing modules, as no one knows if they work or not, outside the scope of the specific system with which they were previously tested. Hence re-invention, code duplication, and yet another maintenance hell.

Did I fail to mention something?

UT cannot assert the correctness of your code. But it will constructively assert its incorrectness.

View on HN

> 3. No one dares to refactor anything ==> Code rottens ==> maintenance hell.

In my experience, with enough poorly written unit tests and a hard line policy on unit test coverage (which often results in poorly written tests), the unit tests often turn out to be as much a barrier to code maintenance as the regular code is.

I'm an advocate of judicious unit testing, but taking it too far is worse than not doing it at all if you have integration or system / functional tests IMHO.

Sounds like my life these days. Been given some smelly code (2k loc) that I cannot fully comprehend, so I just went out and naively refactored it.

Code was covered without assertions, so I insisted spending some time into writing at least a few while everyone else just asked me to test. And of course once in production tons of bugs started to appear. I still believe I can write system tests though.

This is the real trick. Unit tests encourage you to maintain abstractions and layers. If your abstractions are good it works well. But if you need to change abstractions it now becomes a much harder problem.

One nice part of more system level tests than unit tests. Hi ahead ans refactor a bunch. If the same inputs still give the same outputs you should be golden still.

> If the same inputs still give the same outputs you should be golden still.

That's dangerously subtly wrong. It only holds if your inputs include both positive and negative cases. If you only push positive cases through your code and it still gives you the same outputs then it might be broken in subtle but non-obvious ways, aka happy-path testing.

Better be very careful with this.

You can refactor anything if you are conscious of backwards compatibility.

I think there's some arguing past each other in discussion of this. You've seen some terrible programmers who don't write unit tests, and I (and others) have seen terrible programmers that do write (terrible) unit tests. Tests that mock everything and test nothing. Tests that assert bad implementation details but not important results. Huge volumes of terrible tests that are a huge burden.

I agree, having some good tests is very good. Obviously.

But I disagree with the hypothetical statement "someone writing a lot of tests is always doing a better thing than someone writing a lot fewer tests".

No silver bullet and all that. Also annoyed by "agile methodologies" and "best practices" in general. Good sense and taste really can't be turned into a list of rules you can blindly apply anywhere. I suggest reading http://alistair.cockburn.us/Characterizing+people+as+non-lin...

You are correct that writing lots of tests doesn't imply good programming. But lets consider the logical converse: does good programming imply lots of testing (not necessarily unit tests)?

I say yes. Of course this is a big, complicated, world; it probably contains good programs without tests. But in my usual experience, the moment a programmer - even a good one - neglects testing, bad stuff starts creeping into the code.

In OCaml (or any similar language) 95% of what you "unit test" for in Javascript just goes away.

If people cared about software quality, they would use strongly typed languages before even thinking about "unit tests". What people actually want is a) to hack it out quickly and b) to have a lot of busywork to do as a form of job security. Hence 50,000 lines of "unit tests" per 10,000 lines of application...

I have never met anyone who doesn't care about code quality yet writes unit tests...

then you have never worked in anything but a small focussed team.

The majority of development is performed by people in corporate environments, 9->5. They simply know that they "have to" write unit tests. No reasoning is performed, simply churn out the code to satisfy the boss and walk out at 5pm.

Sorry, you are correct; what I wrote was simply stupid. I have, in fact, encountered one method with a cyclomatic complexity of 20+, which had a single test (it verified that the result was not None).

I just tried to put that code behind me and yes, those "tests" were there because they were mandatory.

Blech.

Here is a list of basic errors in Python. Only builtin ones, I'm not talking about ones from other libs. And I'm talking of error categories, as we all know one exception can be used in a lot of different contexts:

  BaseException
   +-- SystemExit
   +-- KeyboardInterrupt
   +-- GeneratorExit
   +-- Exception
        +-- StopIteration
        +-- StopAsyncIteration
        +-- ArithmeticError
        |    +-- FloatingPointError
        |    +-- OverflowError
        |    +-- ZeroDivisionError
        +-- AssertionError
        +-- AttributeError
        +-- BufferError
        +-- EOFError
        +-- ImportError
        +-- LookupError
        |    +-- IndexError
        |    +-- KeyError
        +-- MemoryError
        +-- NameError
        |    +-- UnboundLocalError
        +-- OSError
        |    +-- BlockingIOError
        |    +-- ChildProcessError
        |    +-- ConnectionError
        |    |    +-- BrokenPipeError
        |    |    +-- ConnectionAbortedError
        |    |    +-- ConnectionRefusedError
        |    |    +-- ConnectionResetError
        |    +-- FileExistsError
        |    +-- FileNotFoundError
        |    +-- InterruptedError
        |    +-- IsADirectoryError
        |    +-- NotADirectoryError
        |    +-- PermissionError
        |    +-- ProcessLookupError
        |    +-- TimeoutError
        +-- ReferenceError
        +-- RuntimeError
        |    +-- NotImplementedError
        |    +-- RecursionError
        +-- SyntaxError
        |    +-- IndentationError
        |         +-- TabError
        +-- SystemError
        +-- TypeError
        +-- ValueError
        |    +-- UnicodeError
        |         +-- UnicodeDecodeError
        |         +-- UnicodeEncodeError
        |         +-- UnicodeTranslateError

52 exceptions.

Now, on this list, how many can be linked to language typing ?

        +-- TypeError
        +-- ValueError
        |    +-- UnicodeError
        |         +-- UnicodeDecodeError
        |         +-- UnicodeEncodeError
        |         +-- UnicodeTranslateError
        +-- AttributeError

And ALL those errors can also be triggered with something a type system would no catch : runtime class generation, unexpexted user input, corrupted database, wrong data header, etc)

So basically, relying only on typing will cover about 10% of the error types, most of which are caught with linters such as flake8 and a code intelligence tools such as Jedi. You are making a very weak case.

I'd also exclude:

        +-- StopIteration
        +-- StopAsyncIterat

which are used for mere communication rather than describing real error cases.

Also, you should exclude:

   +-- SystemExit
   +-- KeyboardInterrupt
   +-- GeneratorExit

        +-- ImportError
        +-- MemoryError
        +-- SyntaxError

        |    +-- IndentationError
        |         +-- TabError

because unit tests won't help you to catch these, either (because they are either thrown directly on program start, or depend of specific system circumstances rather than bugs in your code).

Finally, I would argue that most instances of this exception are caught by a type system:

        |    +-- KeyError

Why? Because most of the time the key is not user input, but a hard coded string, and accessing non-existing properties of an object/struct is caught by most type systems on compile time.

    > I'd also exclude:
    > 
    > +-- StopIteration
    > +-- StopAsyncIterat
    > 
    > which are used for mere communication rather than describing real error cases.

Absolutly not. If you implement very low level logic, you can use those. I have to catch them several times in my life.

    > Also, you should exclude:
    > 
    >    +-- SystemExit
    >    +-- KeyboardInterrupt
    >    +-- GeneratorExit
    >

Certainly not. You should DEFINITLY unit test for those. I do it all the time:

    def test_ready_keyboard_interrupt(aioloop, app):
        beacon = Mock()
        real_stop = app._finish

        def beacon_stop():
            beacon()
            real_stop()

        app._finish = beacon_stop

        @app.on('running')
        def stahp():
            beacon()
            raise KeyboardInterrupt()

        @app.on('stop')
        def stop():
            beacon()

        app.ready()

        assert beacon.call_count == 3

You must check if you app behavre well when killed.

    >         +-- ImportError

That a type system can't check for, but a good linter can. I never have import errors because I got proper tooling.

    >         +-- MemoryError

That you should test for of course. Last week I worked on a split/merge algo on big files. I had to test that.

    >         +-- SyntaxError
    > 
    >         |    +-- IndentationError
    >         |         +-- TabError

Well, doh :) Type system or unit tests won't catch that. Tooling, tooling, tooling.

    > Finally, I would argue that most instances of this exception are caught by a type system:
    > 
    >         |    +-- KeyError

No, no no !

KeyError should be super mega tested:

  - keys can be any mutable in Python, not string;
  - keys are very often generated on the fly, not constants;
  - dict are mutable, you can add or remove keys.

>> [SyntaxError, IndentationError, TabError]

> Well, doh :) Type system or unit tests won't catch that. Tooling, tooling, tooling.

You need neither unit tests or a type system to catch those. And you also don't need any tooling. The Python interpreter will throw these exceptions as soon as your program starts.

(Unless, of course, you rely on stuff like heavy dynamic importing at runtime, but that's really rare. I usually see this only for web servers in debug mode, where they auto-reload the app after a changed source file. But then my above comment applies: The faulty program crashes right away, you can't miss that.)

    > - keys can be any mutable in Python, not string;

You mean immutable.

Well, in many typed languages, a lookup on a map will return an option type. If the key is in the map, it returns Some(value), but if it isn't, it returns None.

By encoding the possibility of a missing key in the return type, you force the programmer to deal with it in the program.

So in fact, having a good type system can help deal with those kinds of errors as well.

"relying only on typing will cover about 10% of the error types". It's not about the percentage of error types. Its about the percentage of errors. A very high percentage of unit tests in a dynamic language are written to catch type errors. That work is a waste of time when a type system could check it for you. The reason for this is that type errors are involved in all code. A FileExistsError only applies if I am working with files. (Plus with types you get the added benefit of inline documentation and better tooling.)

It's also about the compiler guaranteeing to enumerate every code path in and out of every function. A human writing tests is limited by what they think will happen, but the compiler knows.

> A very high percentage of unit tests in a dynamic language are written to catch type errors.

Absolutly not.

If you do it you are doing it wrong.

Check this out: https://github.com/Tygs/tygs/blob/master/tests/test_componen...

The type checks are "isinstance". It's 10% of the checks, and most of the time I don't even have those, my colleague insisted.

The rest are the most important tests: they test behavior, not types.

Do not assert a dynamic language means testing types, it's just not true. Bad testers test types.

"isInstance" doesn't cover the full range of type errors. It simply covers whether something is, say, an Int or a String.

Types can encode much more. They can encode behavior.

> A very high percentage of unit tests in a dynamic language are written to catch type errors.

I'm not sure where you got that information. You don't actually write tests to catch types. You write tests to check behavior. While it is possible that actual type errors come out of the woodwork while doing that, that is not the reason why the test was written in the first place; and after fixing such a type error, the test itself is still valid (again because it is making sure that the code exhibits the desired behavior).

I don't think I have EVER written a test just to check for types (in Python). Neither do I pepper my code with 'assert isinstance(x, some_class)' because it just goes against the grain of a dynamically typed language.

In strongly typed languages you embed constraints about your programs behavior in the type system.

Really? What language allows you to encode business logic into its type system?

I'm talking about things like, what happens if you press this button; is this calculation correct; does this lookup give me the desired results; does X get stored in the database correctly (or retrieved from it); etc. I don't know of any type system that lets you check these things.

(Also, I assume s/strongly/statically; Python is a strongly typed language. Granted, not everybody agrees about the terminology...)

You're moving the goalposts now; those are integration not unit tests. But the type system certainly can validate/enforce that the column you select from the DB into an integer, really is an integer in the DB too. OCaml does this with Postgres. It can also make sure the end result of a calculation is of the correct dimensions, that you're not adding cms to inches, etc etc.

A lot can be said with types beyond simply checking things like "is this a String or an Int?".

Depending on the power of your type checker, you can say (with varying degrees of convenience) things like "this integer will always be positive", "this is always going to be a non-empty list", "these two functions can be composed", "I've exhausted all possible values for this variable; I know it because the compiler told me so", "it is safe to call this function, and I know it won't touch the database or do any kind of I/O", and many more.

Whenever you called a function and it wrote to a file when you didn't want that, that's also a type error!

How do unit tests protect you from a corrupted database?

You are making a very weak case.

The ones you write: this month I'm working on medical data, we got tests where we scramble the database to test the reaction of our software of course. Error reporting, cleaning, logging. You mean you don't do that ?

Most of these would be caught at compile time in a language like Elm or (to a lesser extent) Haskell. A common pattern in some of these languages with actually strong type systems is to encode these failure possibility points in the type of the associated functions.

Also, in Elm as an example, there is no runtime class generation, the type of user input that is possible is checked at compile time, and you're forced to explicitly handle many of these potential failures. If you want to crash the program if you get unexpected user input, you literally need to type "Debug.crash" and then you pretty much deserve what you get.

If you're actually interested in maintainable code you should concentrate on reducing the state space of your program wherever possible, then using exhaustive (if possible) or property testing to fill in most of the rest of the gaps. Unit tests can help too, but I really think of them as less useful than all of the above.

You can not catch a keyerror at run time. You can not check for user input with a type system. Those have nothing to do with types. Input and output are out of your control, they need check sanitizing, and dynamic language are actually great at that.

You are wrong. You can very much catch "key errors" at compile time.

The bugs are way more subtle than that. You're lucky if your bad programming is caught somewhere at runtime. In most cases, Python and other weakly-typed languages will do everything they can to fit the data you're providing with what the code is expecting. This means converting strings to numbers and the other way around, plus a lot of other oddities you would not always expect (see PHP's intval for a crash course in bad language design). All that results in bugs that are far from obvious, and that don't always raise neat exceptions.

EDIT: I stand corrected, I don't know enough about Python in particular. The argument is still valid for other languages with weak types.

You clearly haven't done much Python. It doesn't do implicit convertion. Byte are separated from strings and you need to decode explicitly. You can't add '1' and 1.

Python is dynamically type. It is NOT weakly typed.

This is actually something Python doesn't do... generally, if you have a type error in Python, you get an exception at runtime.

I certainly don't disagree with what I guess will be the overall direction of your argument.

Python is not "weakly typed", nor does it ever automatically convert strings to numbers or the other way around.

Heck with Python3 it won't even convert bytes to String.

> In most cases, Python and other weakly-typed languages will do everything they can to fit the data you're providing with what the code is expecting. This means converting strings to numbers ...

Python is strongly typed, as well as dynamically typed.

  In [11]: 1 + '1'
  ---------------------------------------------------------------------------
  TypeError                                 Traceback (most recent call last)
  <ipython-input-11-861a99da769e> in <module>()
  ----> 1 1 + '1'
  
  TypeError: unsupported operand type(s) for +: 'int' and 'str'

> And ALL those errors can also be triggered with something a type system would no catch : runtime class generation, unexpexted user input, corrupted database, wrong data header, etc)

Runtime class generation and unexpected user input can actually be handled in types. Any language with Type:Type, generative modules, first-class modules, and other variations all handle various forms of runtime type generation. Something like lightweight static capabilities [1] can ensure runtime input conforms to expectations encoded in types.

Furthermore, you are way underselling types with that restricted list. NameError, SyntaxError, ImportError wouldn't occur in any typed languages at runtime. LookupError, AttributeError, AssertionError, ArithmeticError wouldn't occur in some of them.

Finally, your breakdown doesn't cover how frequent any of these errors occur. For example, even if typing were to solve only TypeError, if TypeError consists of 90% of runtime errors, that would be a huge win.

[1] http://okmij.org/ftp/papers/lightweight-static-capabilities....

> Runtime class generation and unexpected user input can actually be handled in types. Any language with Type:Type, generative modules, first-class modules, and other variations all handle various forms of runtime type generation. Something like lightweight static capabilities [1] can ensure runtime input conforms to expectations encoded in types.

How would you if you can't deternmine type in advance, check for the type it will be ? I don't understand.

> Furthermore, you are way underselling types with that restricted list. NameError, SyntaxError, ImportError wouldn't occur in any typed languages at runtime.

Absolutly not. Those are not linked to typing in any way, and any decent Python editor catch them.

> LookupError, AttributeError, AssertionError, ArithmeticError wouldn't occur in some of them.

AssertionError ? Serioulsy ? Do you even know what it does in Python ?

And LookupError, unless you got constant sized containers, which has nothing to do with types, you can't check that.

ArithmeticError ? Come on ! Are you numbers constants ? You need to check the inputs so that they belong to the domain of your problem, at that's it. Nothing to do with types.

> Finally, your breakdown doesn't cover how frequent any of these errors occur. For example, even if typing were to solve only TypeError, if TypeError consists of 90% of runtime errors, that would be a huge win.

Yes but it's not. My last week have 90% keyerror, and empty values. The input is usually the source of errors.

Types are useful, and they come at a cost. Whether your want to use them or not is a technical choice to make. But selling types the way it's been done on this thread is dishonest. Or you are all here working with algo problems and very little user input. Which are the easy program to code. The hard part would be the algo, not the code. If you are dealing with a user app, a web site, a video game, typing will not save you from the 90% of the bugs, and you DO need unit tests.

> How would you if you can't deternmine type in advance, check for the type it will be ? I don't understand.

Compilers do this all the time. Just consider all those programs that dynamically generate code based on objects that they haven't seen in advance, like object-relational mappers. Those can all be statically typed and ensure that they generate correct code [1].

> Absolutly not. Those are not linked to typing in any way, and any decent Python editor catch them.

It's all part of compilation, and furthermore, we can type check code generation so that names errors don't even occur in runtime generated code. Sorry, but all of these errors are related to type checking.

> AssertionError ? Serioulsy ? Do you even know what it does in Python ?

http://research.microsoft.com/apps/pubs/default.aspx?id=1386...

Just google "static contract checking" to find plenty of work, including compilers already available for Haskell. Heck, code contracts have been widely deployed on .NET for years now.

> And LookupError, unless you got constant sized containers, which has nothing to do with types, you can't check that.

Please read the lightweight static capabilities paper I already provided. It demonstrates using the Haskell and OCaml type systems to statically assure that all array bounds are in range, even for dynamically sized structures. So yes, you can check that, which doesn't even go into dependent typing where checking such dynamic properties is the whole point.

> ArithmeticError ? Come on ! Are you numbers constants ? You need to check the inputs so that they belong to the domain of your problem, at that's it. Nothing to do with types.

You don't seem to realize that types are simply logical propositions about a program. They can be ANY proposition about a program, including that indexing is within bounds, that a concurrent program has no race conditions or deadlocks [2], that programs terminate, that HTML forms correctly serialize/deserialize values from/to structures [1], and more.

Like most dynamic typing enthusiasts, you don't have a proper appreciation for the true scope of static typing. You are correct that typing has its costs that aren't always warranted, but you are incorrect about where you draw that line because I don't think you fully understand how powerful static typing truly is.

[1] http://www.impredicative.com/ur/ [2] http://www.kb.ecei.tohoku.ac.jp/~koba/typical/

Just a minor note that contracts are available in Python too, via PyContracts.[1] They are very useful.

[1] - https://andreacensi.github.io/contracts/

I think there's a reasonable confusion here - people always talk about "static type checking" when what they really mean is a superset that might be more clearly referred to as "static checking" or maybe even "static analysis". It's true that type systems often improve static analysis capabilities, but I don't see why some useful analyses (like catching SyntaxError and probably NameError) couldn't be done for a language like Python. After all, the editors have basically already implemented it.

Your view of what static type checking is or isn't seems to be limited to your understanding of types (and what can be expressed with them) in Python. There's a world of languages outside Python, even for languages with dynamic typing!

You absolutely can handle runtime errors via the type system just using a Maybe.

This is very patronizing, you're not better than some other developer because you fell in love with OCaml or similar strongly-typed languages.

I don't think this kind of criticism is healthy, you're trying to live on top of your ivory tower calling out everyone else who isn't like you as lazy and looking for job security, you really do think there's no other reason for people using languages that you aren't so fond of?

Honest question, because this kind of message is just unnecessary aggression.

It really depends on what kind of software you build.

There are certain kinds of software for which strongly-typed languages (in the sense of ML, not C/Java) are heavily underused. For those cases, parent's rant is quite appropriate and mostly justified.

For example, we really, really want to have our browsers written in a language like Rust, rather than C or C++. Same with web servers, and public-facing web applications.

Exactly, I can agree completely with this kind of statement and I'm also one expecting to see more ML-family languages being used for what they are good at.

Well, the ML-family languages are general purpose languages, so I'd say they are good at most things.

It isn't patronising to point out the failings of weakly typed languages

Only he doesn't - he just makes claims. You happen to agree so you are satisfied - but the test is, would you like that post if it stated something you don't already agree with?

> he just makes claims.

When was the last time you asserted a variable's type in a statically typed language in a unit test where you weren't testing reflection?

I never did that in a dynamically typed language, so what's the difference?

The whole point of dynamic/duck typing is that you don't care about the specific type, only that it works as you expect it to.

But I'm certain that you have written tests to assert that an object implements a method or that a null isn't propagated in some case or another (or if you haven't written those, you likely have some bugs somewhere).

I totally agree with the sentiment up thread that it's super condescending to think you're a better programmer if you use a language with a good static type system, but from a personal perspective, it's so much more pleasant to let the compiler check this trivial stuff so that I can focus on testing actual logic.

You wouldn't write a unit test to assert an object implements a method. You would test if the method works as expected. This will of course also fail if the method is not implemented at all.

You test for functionality, not types.

Here's a concrete example: say you are writing a test for a method that takes an object and asks that object some question, eg. `object.is_something()`. You can either explicitly assert that `object` implements `is_something` or just let it throw the normal error if it doesn't. Either way, you should test the functionality of the method when it receives an object that does not implement that method, because the behavior in that case is visible to callers, may come to be relied upon, and may cause a regression if changed.

I find that lots of tests end up being of this sort, that I'm uncomfortable if I don't write them, that they clutter up test files and obscure the more interesting tests, and that they are exactly what type checking catches automatically.

To some extent verbose UT is the tax I'm paying for the joy of not using statically-typed language.

I'm happy to pay that, but by no means suggest it's the right trade-off for every scenario.

Sorry to be pedantic, but I just have to point this out, since the mistake is made over and over again in this thread:

It's not about strongly vs weakly typed languages, but statically vs dynamically typed languages.

OCaml is statically typed. Javascript and Python are dynamically typed.

What you are talking about are benefits of statically typed languages.

For more information, please see:

http://blogs.perl.org/users/ovid/2010/08/what-to-know-before...

It is good to maintain an understanding about the difference.

On the other hand, many of the most egregious problems are due to weak typing and not as static advocates, well, advocate, static vs. dynamic.

Having a really good type inference engine helps make statically typed languages much more productive - but lisps and the like have real advantages too.

Probably more accurate to say strong AND static typing. There are plenty of languages that are statically typed but have very weak type enforcement in the compiler and where idiomatic usage tends to use Any types and reflection and non-hygienic macros and many other type shenanigans, allowing tons of bugs to seep through. OCaml is not one of these languages.

What are some examples of such languages? I can only think of two...

Go for instance, with the use of «interface» everywhere you need generics.

Well, if we're really going to be pedantic, then it's about typed vs untyped languages[1].

[1] http://stackoverflow.com/a/9166519

I don't remember when was the last time I've made a type related screw up. In my experience almost all bugs in my code (and there is plenty of them of course) are either something completely stupid like a typo, or some edge-cases not covered properly by code logic (which only proper testing and QA can catch). Back in the days of working in C/C++ things were different, I'd quite often forget to do some required casting. Of course, compiler would warn me about it, but it would still kill my flow, so I honestly never regretted moving to the land of dynamically typed languages. To me it significantly reduced the number of errors since you take one layer of complexity out and thus have less of dull errors you can possibly make. Not the other way around, as you imply.

> I don't remember when was the last time I've made a type related screw up

Maybe the languages you are using aren't capable of encoding the kind of screw ups you do make as type errors? Typos are trivially covered by most static type checkers, but like you say that's not a very interesting kind of error (except, of course, with languages that silently introduce fresh variables. Those can introduce hard to find errors...). More interestingly, they can also handle other problems like dereferencing nulls, not considering all possible patterns of a value (think "case" conditions), calling a function on the wrong kind of value, etc.

I am not saying you are wrong, and I think the OP of this thread overstated the benefits of static typing, but here are a couple of counterpoints:

> either something completely stupid like a typo...

Typos are a bigger problem in languages that silently instantiate a new variable in response. If it's on the rhs, would you prefer your IDE to warn you about using an unititialized variable, or let it go? Your coding flow is paid for by more failures (some of which will be WTFs) in testing (and testing is part of coding these days, right?) That may be the right trade-off, in which case you may be able to disable warnings in your IDE.

>...or some edge-cases not covered properly by code logic (which only proper testing and QA can catch).

The 'only' suggests that they are inevitable, and they are in a statistical sense, but whenever my code logic fails to cover an edge case, I ask myself why I overlooked it. Sometimes it is not something I could have anticipated even if I had thought more about what I was doing, but often it is.

Just as you said, statistically speaking being stupid from time to time is an inevitable part of the process :) When I manage to anticipate every angle correctly then usually there is no bugs, so it's not that relevant for this discussion. I was talking about those "other" moments, and usually, yes you are completely right, it's just me not being focused or not looking from the right angle on the problem. Thing that really sucks in programming is that computers almost never make mistakes on their own, it's always us who screw up the code :P

When you are writing unittests for JavaScript, does 95% of your asserts check that values are of the correct type?

I've been using flow type which has made the transition from F# tolerable, so none of my assertions are type checks but still a lot of null checks because of no language support of option types.

But if you develop without flowtype, do 95% of your JavaScript unittests consist of type checks?

In a proper type system a la Haskell, ML, Rust, etc. you can encode significantly more into the type system than you can with JavaScript's rather anemic offerings, even if it were statically typed. It feels almost like contact programming in that you must handle everything out else the code won't compile.

I'd agree that Haskell/OCaml/... will get rid of some unit tests that are to be written in other languages, but you still need to test if the logic and semantics of your code are correct.

Sure, but removing entire classes of things you have to check is a really good win. Combine that with the power of fuzzers derived from legal values of the type, you can knock out a lot of bugs from your programs.

Sure, but just because a type system can't catch all bugs doesn't mean it isn't valuable to use one to completely eliminate certain classes of bugs.

Unit tests as documentation in my experience is only true in a very minor, almost trivial sense, and for reasons that are very much in line with the subject of the original article.

For the most part, unit tests aren't organized around what a unit is supposed to do from a use case perspective, but more about how it happens to be implemented today. The article refers to this with the inability of most developers to indicate which business requirement would fail if the test failed.

When you're attempting to understand or refactor something where the testing for the most part tests the implementation of this particular unit rather than it's purpose, the tests themselves offer little in either documentation or refactoring confidence than the implementation itself.

What's worse, making your code "more testable" often compounds this effect. As code is taken to an extreme level of modularity, and unit isolation is enforced rigidly in tests, the tests often do nothing other than specifically test every line of the implementation of the unit (i.e. method 'foo' on class 'Bar' calls method 'a' on class 'B' and passes it's results to method 'x' on class 'Y', returning that result)

IMHO "unit tests as documentation" isn't quite the right description. Its more like "guaranteed to work example code and tutorials".

One example is where there might be preconditions for calling a function. This type of information is necessary to know but can be hard to work out even with rich documentation. Without documentation, you are going to be reading a lot of code to try and reverse-engineer the internals.

With unit tests, you can do a quick "find all references", see how the method is called within different tests cases and be confident that the setup actions in the given test fixtures will work for you.

If a method has preconditions, say X cannot be null, Y must be positive, and X > Y, you could test this in a unit test. But honestly make it the top 3 lines of the method with Asserts. Living with the code is going to be generally better than living off in some test. You can read it and see it very easily.

Sure. I don't mean simple local pre-conditions like arg validation but more system level issues that are inappropriate for asserts e.g. maybe you need to hook an output sink to an event trigger before you invoke a function to see any results. Not having done this or doing it in the wrong order is not strictly a mistake from the system's perspective but would you leave you stuck as a coder.

Integration test should catch if for a given input your entire app does nothing, right?

The issue isn't how to test pre-conditions, its how to know they exist and how to meet them correctly. An integration test just gives a big ball of mud of every production concern whereas a unit test provides simple example code for a single concern that is maintained and tested.

It feels like discussing this without real code is likely to cause confusion. The type of systems I have in mind are those that invert control through e.g. pub/sub events, plugins, strategies, dependency injection etc. Unit tests help document how to use components in a system that has traded clarity of control flow in exchange for extensibility.

I have found these kind of systems that put a ton of effort into unit testing are wasting time. And when it comes to refactor, you end up creating weeks of extra work to make the new unit tests pass.

How a system is wired together is not that exciting and not worth unit tests. If you screw up loading a config file on startup o bet your first integration test fails. Why would you unit test this?

Assertions are most of the time completely useless and just a waste of time. If you put these assertions in one method it will break only at runtime. Usually it is a not very used method and the code that caused the assertion to fail is shipped in production and you realise only too late that it's broken. If you had a proper unit test instead of an assertion you would have caught the bug as soon as you pushed the code and the continuous integration kicked in.

Why not put an assertion, then add a test that deliberately checks if it's triggered?

The comment that I replied to was about using assertions instead of tests. You can certainly write an assertion and test it after, but unless it is absolutely necessary I would prefer to avoid polluting the code with "documentation". The right place for that is in the unit tests, not in the middle of production code.

I disagree, the right place is in the code where the functionality is defined. And the comment you replied to i didn't read as "assertions instead of tests". That's absurd. You should have tests that exercise the code that contains the assertions. It's the best of both worlds. You get code automatically tested and you get inline documentation that can't lie to you.

I say code the happy path, and then have assertions.

Saves writing 12 unit tests that show passing null breaks the code. No one cares about that.

And the comment train I was on was using unit test for documentation on valid inputs and outputs.

Asserts are documentations on valid inputs and outputs that very cleanly break when violated.

There is a difference between unit tests and integration/ system/ component tests. I think the author is talking about pure unit tests, i.e. the ones that test only one specific piece of code and mock everything else away.

Too fine grained unit tests actually make refactoring more difficult, because a refactor typically touches so much code that they have to be rewritten. But component tests do provide the safety net that you had in mind when you wrote your reaction.

What.

> Ok let's stop writing UT and see what happens. Wait... we've already tried that, and we know the result pretty well:

Stuff still doesn't work and doesn't get delivered on time. That's all we know, and unit tests didn't change that.

> Yes these are bugs that should have been caught by a compiler had we had one.

WHAT? What kind of dynamic languages are you working with?

> 2. There's no documentation as to how something should work, or what functionality a module is trying to express.

Unit tests - at least those made in TDD style - tend to obfuscate that; they usually reflect the implementation instead of the interface anyway.

> 3. No one dares to refactor anything ==> Code rottens ==> maintenance hell.

You seem to know very timid programmers. Or the kind who think tests are their safety net, so they don't bother to actually read the code they're changing and understand what it does.

> 5. When something does break in those "top level system tests", no one has a clue where to look, as all the building blocks of our system are now considered equally broken.

Look at the crash log. Or think for just a second instead of saying "it's broken" and giving up.

> 6. It's scary to reuse existing modules, as no one knows if they work or not, outside the scope of the specific system with which they were previously tested. Hence re-invention, code duplication, and yet another maintenance hell.

Re-invention, code duplication, etc. have nothing to do with tests (hell, most unit tests I've seen have quite a lot of code duplication in them). It's not scary to reuse existing modules without tests, it just requires to... use them.

Pretty much all the points you listed are examples of people shutting down their brains and hoping someone else's unit tests will think for them. This is a very bad approach to programming. It's the wrong kind of laziness.

> You seem to know very timid programmers. Or the kind who think tests are their safety net, so they don't bother to actually read the code they're changing and understand what it does.

Muhahahaha! Sometimes it's almost impossible to understand a piece of code, especially if that code is old and has had comb-overs from a lot of developers. Unit tests at least might give some hints to what's happening.

Pretty much all your answers show that you never worked in a big enough project and that you don't understand unit tests. When working in a codebase with adequate test coverage your bugs are more likely to be caught. Please, don't tell me that you don't need tests because you don't introduce bugs. I heard this argument multiple times in the last 10 years, and it was always proved untrue.

The point number 2 simply proves that you don't know what a proper unit test looks like if you say that a unit test obfuscates the documentation.

The point number 3 just confirms what I thought, that you believe that you don't need tests because you don't introduce bugs. The most interesting part is that you think that other people that write unit tests are lazy. Very interesting position, I would say that whoever tries to refactor a big chunk of legacy code without adding an appropriate coverage is certainly lazy and worse overconfident and probably never did anything similar in the past.

The point number 5 proves that you never, ever, worked in a big project if you think that it's enough to look at the logs to find a bug. It's common knowledge nowadays that earlier a bug is caught the less effort it takes to fix it.

Finally I think that should be clear that the very bad and toxic approach to programming is not the one that involves writing unit tests. This approach tries to formalise the requirements in a set of scenarios, describing each condition that must be satisfied for a given event/input. Your approach seems focused on just writing code, without a proper design phase, without understanding completely the requirements with a close feedback cycle with the users and with a complete disregard of existing functionality that you think you can write better because you are a better programmer given that you don't need unit tests.

In my opinion not writing tests is laziness.

Unit tests are mostly useless. If one wants their tests to be anything more than a happy path and sad path that the developer thought up (combined with an accretion of sad paths that were found later as bugs), then one would do themselves a tremendous favor by adopting property-based testing and mutation testing.

Doing so forces you to define several constraint-based models for your program, and then layering mutation testing over the top helps you figure out how good your tests actually are.

>> Unit tests are mostly useless

The speed difference though, integration tests just take too long to be able to completely replace unit tests.

It depends. Both on project size and how your tests parallelize. If you can integration test an entire project in 30 seconds, that is fast enough.

Fast enough to have it in a loop for by the book TDD? No but I'm not sure by the book TDD adds any value. You mash test once before committing and if all is good you go ahead.

With a decent hierarchical decomposition, integration testing at component/module/subsystem boundary achieves both good coverage and low latency test results.

I worked on a big ETL database migration tool in Python and it had a top level integration test with the smallest possible subset of data, like 500k end to end. Then each subsystem (it was architected like a nanopass compiler) had a more complex set of integration tests that only checked that pass but more rigorously. The whole project (10k lines of code) had under 10 unit tests that double checked tricky low level primitives. Every test was independent and could be run in parallel, my longest test was just under a minute, my shortest was about 1 sec. The shortest tests were run on module import ensuring both an easy affordance to testing and a tight feedback loop with no unit tests to impede refactoring.

I see a uncredible difference in the architecture quality in projects with unit tests. People test forces people to think about interface, forces them to use their code, torture it, break it, knowing it in and out and see the limits. Just for the intellectual tool UT are, they are valuable.

Now they have a cost, and sometimes they are not worth it, but they are certainly not useless.

Writing property-based tests will incentivize you to think about those things too, and it they will actually exercise your code more strenuously across a wide spectrum of inputs and corner cases.

Unit tests seem mostly worthwhile as documentation or tutorial for how to use something or for what something is "supposed to" do. But as validation of completeness, correctness, or coverage of your design and implementation... they fall extremely short and leave a lot to be desired.

In either case, if software quality and reliability actually matters to your problem domain, then mutation testing should be considered an absolute prerequisite to any kind of testing method (unit, property-based, integration, etc.) otherwise you have almost no way to measure the quality of your tests.

There's one insidious thing about no tests when dealing with dynamic languages: when you have no tests, no one knows what data looks like (typical code: `bigJson = getBigJsonFromServer()`). Is 'foobar' an array or string? What kind of string is it exactly? Can it be null, or it will be "unknown"? Put two years of cowboy coding on top and you're screwed.

Of course you can create unit tests with fake data, though IMO it's a bad practice. I think it's best to have unit tests working on real production-like data (in addition to have edge-casey-data tested as well).

>2. There's no documentation as to how something should work, or what functionality a module is trying to express.

Unit tests of complex code is often more complex and obfuscated than the code itself, hence its usefulness as documentation is very limited.

>3. No one dares to refactor anything ==> Code rottens ==> maintenance hell. >4. Bugs are caught by costly human beings (often used to execute "system level tests") instead of pieces of code.

Lack of unit tests doesn't necessarily means lack of automated tests. Functional tests when applicable, are far more useful in my opinion than unit tests.

> Unit tests of complex code is often more complex and obfuscated than the code itself, hence its usefulness as documentation is very limited.

I've found that expressing unit tests in the language of the problem domain and never stressing implementation details wherever possible is the ONLY way to prevent this from happening. Avoid mocking and test doubles where possible, but sometimes they're the right tool. The tests should be treated as production code that gets boyscout rule'd as people go into it for whatever reason.

>"Avoid mocking and test doubles where possible, but sometimes they're the right tool. The tests should be treated as production code that gets boyscout rule'd as people go into it for whatever reason."

I agree, but a lot of people (at least in my environment) would play the "that's not an unit test" card, which is fine to me, we should care about the least intrusive way to automate code tests regardless of the philosophy behind it.

Only bad unit tests are obfuscated. In the good ones even the users can easily understand what the test is doing and can even add new test cases.

>Did I fail to mention something?

Yep. Integration tests under "top level system test" and above "tightly coupled unit tests" are not only excellent at catching bugs in integration code, they don't require hours of tedious mocking and don't have to be completely rewritten when you refactor the code base.

>No one dares to refactor anything

Is a problem on heavily unit tested code. Unit tested = tests that are tightly coupled to the code = refactoring turns the tests red whether or not anything was broken = people prefer leave the code alone.

Don't agree. Refactoring does not mean that you need to refactor the tests, but that you will be testing other pieces.

That is, when you refactor the old pieces are still there (in the libraries), and their tests still work, but you are maybe not using them anymore higher up in the hierarchy.

4. Bugs are caught by costly human beings (often used to execute "system level tests") instead of pieces of code.

The author went to some length to point out (correctly) that test automation is an orthogonal issue. Reading beyond the title, we see the issue is that an obsession with unit tests, and especially an obsession that is focused entirely on a measure as simplistic as code coverage, tends to result in a sub-optimal use of development resources (which are always limited.) A redundant or otherwise unhelpful test (of any sort) does not gain value by being run automatically.

Speaking of (3): I have a bunch of newbies I'm mentoring. Every time I explain about unit tests, I show them a class I wrote that had 37 lines altogether. After I finished working on that project, a new team took over. They didn't write unit tests (or maintain the existing ones). A month later, that class had a method with 1500 lines.

It's been around a year since development has basically frozen on that project.

What has the number of lines got to do with the unit tests?

That functionality had to go somewhere, no matter how badly written it was.

Unit tests don't magically save you from code bloat.

Also I work on 3 code bases written by other people with zero unit tests, I refactor them all the time. Other developers have refactored them. Not having unit test is not a big barrier to refactoring.

> Unit tests don't magically save you from code bloat.

No, but the idea is that such a method would be hard to test, considering it (presumably) does so much. So in order to test it properly, you are encouraged to break it up into smaller methods. (Whether somebody actually does that is a different story, of course...)

Are you saying the requirement to unit tests is the main reason why those developers don’t write 1500-lines methods?

I'm saying that they would be prevented from writing those things if they also had to test them. Since they're not required to test their methods, they're free to write code that's pretty much impossible to reason about.

I think people who write such methods without a good reason (like inner loop of very performance critical C++, or code in any language that’s generated automatically from something else) aren’t terribly good programmers.

Demanding those unit tests might indeed force them to write shorter methods. It doesn’t forces them to write good code. By “good” I mean fit for particular project and requirements: I don’t believe there’re universal criteria for code quality.

The only way to fix that and make them better programmers — teach them, not force them to waste their time writing those unit tests.

Besides, writing and running system tests teaches the developers about other system and OS components — and they better know and understand (to some extent) your complete technology stack, not just the layer they’re currently working on. Writing and running unit tests teaches them nothing interesting; it’s boring and involves a lot of copy-pasting.

To add to that - if the only thing keeping programmers from writing 1500-line monstrosities is the requirement to test their code, that's even more worrying, as anyone who's willing to write garbage like that is going to do a terrifically bad job of breaking out abstractions and modules, and now you have 1500 lines that are impossible to reason about split into 20 different classes.

I don't know... I find it much easier to reason about smaller methods. They didn't write just THAT one, there's plenty of code in that project... it's just a couple of methods that are horrible to work in.

It's absolutely true that all else being equal, it's easier to reason about smaller methods than larger ones. Most of the time though, what you're trying to reason about is an entire process rather than a method in isolation, and how small methods interact with each other is an important part of that.

It's absolutely true that there's almost never a time when a 1500 line description of a business process is the optimal approach. Conversely though, if abstractions are done poorly, those same 1500 lines split into a maze of different classes that don't cleanly abstract a particular, human-comprehensible part of the process can be just as hard to deal with.

Oh god, you are describing my previous project. Especially 3. and 5., which makes the most tiny bit of changes take months and endless amount of money and manpower.

Agree!

Reply on news.ycombinator.com