Hacker News Reader

Comment by lordnacho | Hacker News Reader

I'll put the question to the other readers:

How often do you find, despite having written tests, that there is some bug in your software? And how many of those times did you think that you should have considered it beforehand, rather than that it would be impossible to foresee?

In my experience the most useful tests are the ones that came from some unforeseen bug, which was then fixed and a test case built around it, so that it wouldn't get "unfixed".

The least useful tests are the ones for cases you know not to invoke, because they are obvious. Like how you know when you divide by a variable, you know it can't be zero. So you make sure it can't be zero, making the test case a bit moot.

View on HN

Perhaps I'm just an outlier, but I think every time I've gone in to write tests for my code (usually post-testing rather than red/green testing), it's rather quickly invalidated some assumptions I made when writing the tests.

"Oh yeah, that's going to need to be thread safe."

"Oh yeah, that might be nil. Rather frequently."

In my experience, testing has been much more beneficial as an exercise of my mental model of the code and its interaction than for refactoring. But to that end, I can think of quite a few times that I've been very grateful for a unit test suite while I did a large refactoring.

I've had a very similar experience.

It's worth it's weight in gold when you can flesh out edge cases and logic issues before going all-in on an architecture, and being able to refactor without fear is a great bonus, but it's not always applicable.

Unless the majority of the tests are "external" (i.e. testing an API by calling it's endpoints and testing the return), a major refactor is going to require updated tests, and it's all too common that I see people take the "easy" way out and fix the tests to conform to the way it's working after the refactor rather than the way it should work.

I agree, tread safety and race conditions I got hit a lot, other than that test coverage provided a lot I guess

  > tread safety and race conditions

I feel like this comment wouldn't be out of place in an F1 thread.

>Perhaps I'm just an outlier

If you are, then I am as well, because this is where a huge part of the value of tests come from - validating assumptions early and quickly and cheaply.

I've found that this depends heavily on the software in question.

For most of what I do (basic line-of-business type webapps & servers) the vast majority of the code works on the first try and is obvious enough that I never break it; during development there tends to be a few sections that break repeatedly while I'm changing other things, and those are the sections I write tests for. (Generally this is a few percent of the entire codebase.)

On the other hand, more 'computer-sciencey' projects tend to need a correspondingly larger proportion of test cases. The most dramatic example of this was when I built a Lisp interpreter as a side project: this was the only software I've ever written fully test-first, and I don't think I would have been able to get it working at all without a full test suite that hit every line of code at least twice.

As a follow-up question, did you (or could you have) develop that lisp interpreter on the purest TDD approach of:

- Write a few tests that fail

- Write code until your tests stop failing

- Repeat until the program is done?

In my experience, problems that require writing the tests first normally require writing all the tests first. If you start solving only a subset of them, you will have to rewrite most of the code once you start looking for the other tests.

> did you (or could you have) develop that lisp interpreter on the purest TDD approach

Yes, that's what I meant by "fully test-first". I was following along with Paul Graham's The Roots of Lisp[1], so I had a convenient set of pre-made test cases. I would start by copying his examples into a test case:

    it('exec atom on a symbol is true', () => {
        var symbol = Symbols.get_symbol_named('foo')
        expect(exec(empty_scope, [natives.atom, [natives.quote, symbol]])).to.equal(true)
    })
    it('exec atom on a non-empty list is false', () => {
        expect(exec(empty_scope, [natives.atom, [natives.quote, ['a']]])).to.equal(false)
    })
    it('exec atom on an empty list is true', () => {
        expect(exec(empty_scope, [natives.atom, [natives.quote, []]])).to.equal(true)
    })

And then write my code:

    natives.atom = new Native(function (item) {
        var i = exec(this, item)
        return (i instanceof Array && i.length == 0) || Symbols.is_symbol(i)
    })

Once I got through the paper that way, I had a complete Lisp interpreter working, and wrote a few small programs in it with no trouble at all.

> In my experience, problems that require writing the tests first normally require writing all the tests first. If you start solving only a subset of them, you will have to rewrite most of the code once you start looking for the other tests.

This has been my experience as well. As I say, this is the only program I've ever written fully test-first; I think it was only possible because I already knew exactly what the inputs and outputs were going to be (having them well-specified by virtue of being a Lisp) and which ones I needed to implement to be able to get later ones working (with the help of the paper).

Generally I find it's not possible to work this way; my usual approach for solving problems where the code isn't immediately obvious is to write (and generally rewrite) in parallel:

- A set of usage examples and/or documentation, to clarify what exactly I'm trying to do and make sure the interface makes sense.

- The actual implementation (or, in early stages, some pseudocode).

- The tests (if any), to check that what I've written actually works. Often these are the same as the usage examples.

Frequently questions raised while writing one of these will influcence the others, often significantly; often I'll only think of an edge case while I'm writing the code and then have to go back and add it to the tests, or realize that the code could be simplified by changing the way the API works.

[1]: http://www.paulgraham.com/rootsoflisp.html

I'd wait until the Lisp interpreter works significantly and then write reasonable code in Lisp like:

  (it (atom 'foo) t)

If it fails, it just says something like:

  test failed: (atom 'foo) returned nil; expected t.

So no need to have a string there. Save the "blub-level" testing for things that are not testable through Lisp. (Or, really, things you are justified in not wanting to expose such that they are testable.)

(Sure, it could be the case that both the interpreter are wrong, and the atom function are wrong such that the test passes. That level of breakage isn't likely going to pass much of a significantly detailed test suite.)

If I'd waited until the interpreter to work before I wrote tests, I'd never have gotten a working interpreter. For atom the implementation is obvious, but the more complex functions were riddled with bugs on the first implementation, and the lexical scope handling took about three complete rewrites before I got it right.

If this had gone any further than the three days of off-hours time I spent, I certainly would have rewritten the test suite in Lisp; however it was primarily an academic exercise to really 'get' how Lisps worked and push my comfort level; writing all of my tests in the language I was trying to write would have pushed things a bit to far.

In fact, I never got around to writing a parser, so even if I'd written them in Lisp it would have looked like this:

    var Symbols = require('../Symbols')
    var natives = require('../natives')

    var foo = Symbols.get_symbol_named('foo')
    module.exports = [
        natives.tests,
        [natives.it, [natives.atom, [natives.quote, foo]], true],
        [natives.it, [natives.atom, [natives.quote, []]], true],
    ]

I did this with brainfuck.

There should be a law banning people from asserting the constants have the expected value in tests.

True ;-)

Do you mean test explicitly and fail graciously? So even test over false inputs, but check that the code fails with the right exit?

I mean if you have a method whose definition is literally just "return 'abc'" and then you have a test assert that that method returns 'abc', knock it off.

I call those types of test, "testing that the compiler works." Not really something you should be testing! It also applies to testing code that is just a wrapper around third-party libraries.

I was taught that there should be tests against things like class constants so that you have a failing build in the event that someone changes a const that's part of the specification.

This was from some TDD course that my employer brought in where they emphasized that consts that are part of a specification or requirement should be tested against to prevent them from being accidentally changed or changed without full consideration of side effects/review against the program spec.

Is this what you're referring to?

I guess so but I disagree with that teaching. All it does is make me change it in two places.

That's sorta the point - to make you think before you change. A constant without a test is part of the implementation, and you can refactor as necessary to make the implementation perform better. A constant with a test is part of the interface, and if you change it, client behavior has changed and the tests should change with it.

Yes, OK, but what I'm telling you is I think it's just as mindless. If anything, I look less at the tests when I know I'm going to break fifty of them with any real change at all. "Yeah, yeah, whatever" and copying and pasting the new values is the more likely behavior in this scenario.

This is true only for a very particular kind of constants, like the ones defining API error codes (eg. define PAGE_NOT_FOUND 404). And those should be covered by integration tests, not unit tests.

I see bugs in my code very rarely nowadays, and they're always things I should have seen beforehand. In my experience if you're seeing edge cases at all they're a sign that your model is wrong - the business processes we're replacing with software are always simple, there's really no reason for code to ever be complex. The problems we get are those we introduce for ourselves by overcomplicating things in code.

I do think of myself as using TDD, but I use the tests to drive the model and often end up not needing the test - a bit like https://spin.atomicobject.com/2014/12/09/typed-language-tdd-... . With experience, more and more I'll skip past the writing a test and deleting it after, and go straight to encoding the properties that I actually need in my types.

Testing for me is never primarily about getting it right the first time. Although writing tests in parallel with coverage can definitely help to validate your assumptions, particularly with negative testing that you might not test manually.

The greater value of the tests is later on, when you need to make a change, and want to know whether you broke some existing functionality. Or, once you have a framework of tests around something, being able to quickly write a test that fails due to a discovered bug, and knowing that you fixed it by passing the test.

From the receiving end (long history of "ops" work), I've seen that a lot of the code itself tends to come out really well and the developers who write the code are a lot happier.

That said, many of the feature requests from the ops' side end up taking an extremely long time to produce, if ever. Very basic requests that would save the team dozens of hours per week go on and on longer than they should. In these sorts of shops, the ops side ends up pretty miserable and builds up a lot of frustration for the development team, which drives the two groups apart.

More recently, my team's been asked the ops team is being asked to write tests for ops' prod bugfix automation. Categorically speaking, we're not developers, so it tends to take us longer to write tests than the devs who write them daily. Issues pile up and span across months, rather than days. The reasoning behind the tests is understandable since it helps build assurance that our scripts won't break the world. Though, the ops team is battle hardened enough to know that if we break it, we have to fix it - it's in our best interest to write code that's reliable as possible. I don't think any of us would be in our roles if we didn't understand it. Frustrating stuff.

Yes, and a special case of this, which is rarely correct in "I don't get paid for tests" code, is tests that make sure errors are reported or handled correctly. It's rarely a good idea to ship a NullPointerException just to see if you correctly turn it into an error code or something.

I TDD a fair bit, and both kinds of tests are useful. I do really love the tests that I write to replicate a regression bug. The other types of tests are more like living documentation of my internal APIs for future developers, and are still useful.

you're describing two different types of tests.

one kind is the acceptance test, where you write a test that specifies correct behavior under normal circumstances (including expected error scenarios).

the other kind is a regression test, where you reproduce a discovered bug in a test environment, then fix it, and make sure that the test covers the fixed code to confirm that the bug is no longer occurring.

both types of tests are important and for different reasons. they aren't mutually exclusive and they aren't addressing the same problem.

Similar question can be, which kind of tests failing most frequently, this is a good insight on what kind of tests you should focus on your team

>>The least useful tests are the ones for cases you know not to invoke, because they are obvious.

Maybe you know not to invoke them because you wrote those features, but what about the rest of the team? Or the poor schmuck who takes over your codebase after you leave?

Test cases help future-proof your code against the unknowns and the unexpected. I think of them as guard rails on a path. When the path is on straight terrain, they aren't necessary. When it is crossing the side of a cliff though, you're glad they're there.

Reply on news.ycombinator.com