Hacker News Reader

Would you fly an airline that did 24 hours of verification before every flight thus making flights cost 10x as much? (and same argument for all your other examples...)

Our job as professionals is to verify that the software we write does what we expect. Our job is also to manage costs appropriate to the application (not just up-front but over the long haul). For me, that means that we want our tests to maximize value per effort.

I am not saying a) write fewer tests b) tests are a waste of time or c) tests are too expensive. I'm saying: a) write better tests, b) some tests are a waste of time and c) let's write tests that do more work for us.

[disclaimer: I work at https://circleci.com, so I do have a vested interest in disagreeing with you ;) ]

I think is a strawman. If running a huge suite of automated verification was really cheap, then airlines would run it before each flight.

Hey Paul, I totally agree that the cost of running big test suites can be mitigated and CircleCI is doing awesome stuff there.

That does nothing to address the problem of maintaining a large suite of tests tightly coupled to your code. That is, the cost of updating many example-based tests when you change the code.

I guess that depends on how much of your suite is in flux. We have largely example-driven tests, and our suite is pretty big: about 16kLOC. I can't think of a case where a significant amount of time was spent fixing the test suite (and in those cases, it was generally useful to think about why).

Also, I haven't seen anything that works better. Generative tests are nice in theory, but I haven't seem the be that useful for anything other than short, pure, functions.

I've built generative tests for an entire betting exchange. We run the database in-memory, throw generated events in one end and make assertions about the state when processing is finished. The tests range from very specific, narrow pseudo-unit-tests to coarse properties like 'if we process 1000 random orders from 80 different customers in 3 different currencies, the total amount of money in the system remains constant'. I can't imagine reaching the same level of stability in the core exchange if we had to write all the test cases by hand.

Really, what it comes down to is that with a language like Clojure, that has very isolated side effects and works primarily with pure functions, getting to high test coverage requires a lot less testing code.

My problem is just how adverse to utilizing a proper QA department many developers are. Sure, unit tests and other automated systems can and should be used to help out in creation of a system. However, realize that there are folks who have the sole job of testing a system. Unsurprisingly, many of them are actually very good at it. Instead of spending a lot of time trying to make a more convoluted model just so you can test it, use the real tools at your disposal to accomplish the same goal.

I fully grant that the line between "convoluted model" and "adequately testable" system is a tough one to spot. I am open to some good rules in distinguishing them.

A QA department doesn't locate the bugs for you. They report bugs, and then developers have to repeat the procedure and investigate. This is a costly round-trip.

Automated testing is intended to be a much cheaper way to stop bugs from propagating up to QA, and it usually provides much better and more immediate information about the precise nature and location of bugs.

But an automated testing suite does not write itself. Not even in a "statically" typed language. In those worlds, the staffing costs of having a dedicated QA team can be far cheaper than engineering your system to be perfectly modelled for testing. This is especially true in areas that do not already have established models. (And is why early attempts at such endeavors as bridge building focused on prototypes and iterative solutions, and the rigorous mathematical models came later.)

If you remove the word 'perfectly', you're just saying that a dedicated QA team is cheaper than writing tests.

I doubt that very much.

You'd be surprised. Consider the effort many game studios go through to get the testing done on a game before release. It is a pipe dream to think that everything can have a full automated test suite written for it. I suspect that the only things this is true for, are things which have been written so many times as to be almost trivial. (Not that we don't make enough mistakes on those systems...)

Though, maybe you were just questioning the "perfectly" portion. I mean "perfectly" in the sense of "validates all of the user's requirements." Not in the "covers all paths" or "anticipates all possible scenarios."

Consider, crash test dummies. This is a form of testing that is far far far more expensive than having a complete model of the system. Yet, we still do them. We do not rely solely on them, you say? Of course not. Apologies if it sounded like that was my point.

Edit: Also, it is also possible to use traditional QA tools without having a fully dedicated team. That is to say, unit tests are not a QA tool. Yet, we constantly try to replace the QA team with them.

You are proposing QA as an alternative to writing tests. That makes no sense. They serve different purposes.

Nonsense. I am proposing QA teams and tools in addition to all of the other tools that a development team uses. If you, as a developer, are having a hard time testing something, ask QA how they would test it. Make sure they will be able to test it.

If after that interaction (or before) there is an easy way to construct your model such that this can be fully automated, do so. But, essentially, make sure you are working with QA.

This would be akin to a football team ignoring the special teams and not considering them in any decisions. Sure, if you have a perfect offence this will work. Most of us do not.

Edit: So, not sure why I began with "nonsense." Apologies for the tone on that.

So your comments have nothing to do with testing in clojure or even code level automated testing in general, and this is just you telling 'most developers' that you think they should work with QA more?

Well... I jumped in at the discussion pivot regarding the concern over "writing fewer tests" being difficult. My point was simply if you want some good leads on which tests you need to focus on, coordinate with QA. Development shops tend to work in isolation to QA. Often, there is a sense of hostility between the groups. This makes zero sense.

So, my point is to emphasis that all of the tools my direct pivot discussed were not necessarily created by the pilots/doctors/etc. Instead of the emphasis being on "having smart programmers that write lots of tests," lets move it to "having programmers that focus tests on problematic areas with insite gained from other teams."

(That make sense?)

I agree that testing getters/setters...is a waste of time, but...

But it isn't necessarily. Say I have multiple implementations of a class, e.g.:

    trait TimeSeries {
        def datapoints: Seq[DataPoint]
    }

    case class SequenceTimeSeries(datapoints: Seq[DataPoint]) extends TimeSeries

    class ArrayTimeSeries(times: Array[Long], values: Array[Double]) extends TimeSeries

It's pretty straightforward and nearly pointless to test that `sequenceTimeSeries.datapoints` does what I expect. On the other hand, applying the same test to `arrayTimeSeries.datapoints` exposed an off by one error in my implementation.

Out of curiosity—I'm not so familiar with Scala—how does an off-by-1 arise here? Aren't you just going to zip (times, values)?

I did this a while back, so I don't recall the exact off-by-one error. But I don't want to to times.zip(values) - that would involve constructing explicitly a List[DataPoint] which is precisely what I'm trying to avoid with the ArrayTimeSeries. Rather, I defined a custom Seq[DataPoint], which had the apply(i: Int) method construct datapoints on the fly from the arrays.

(I admit that the error was entirely stupidity on my part and not the result of anything complicated. If I was as smart as Rich Hickey, I probably wouldn't need so many tests.)

Ohh, I see, array-of-structures/structure-of-arrays trouble.

So test that one. Please, I encourage you to engage your brain in the process of choosing what does and does not need to be tested. That is the whole point.