Clojure and Testing

We don't "ship" a product. We're a humanoid robotics research laboratory, we have a fairly large and old codebase for what we do (~1MLOC of Java, plus a whole big whack of C++ for JNI), and since we neither have a customer nor an OSS community then we make no guarantees about our API nor do we frequently generate stable, frozen binary releases. As such, the code is constantly being refactored and evolving; especially our numerical based stuff; things like physics, controls, etc. (we use an in-house simulation environment that one of our PI's created while he was in grad school, so even stuff like the 3D engine or the simulator data viz GUI's can change at any given moment).

I realize that this is entirely anecdotal and highly specific to what we do, but without high coverage and solid CI (We use a self-hosted Bamboo instance with nightly and every-commit builds), our code would be in complete and utter disarray. The build routinely breaks a dozen times a day when one of the control theory guys checks in a parameter tweak to our state estimator or introduces a new component to one of our modular controllers (we have test suites that actually run headless versions of our sims and mathematically verify the walking behavior).

I guess my point is that, as is the case with anything in programming, choose the right tool for the job. Evaluate what you're doing, what your goals are, the nature of your code-base, and decide (as is brushed over in the source article) whether or not the opportunity cost is worth it. For us, we'd be up shit creek if we didn't have high coverage.

If you write clojure, you use immutable structure (in most of the case, if you don't this comment doesn't apply), and so you end up writting only functions that take some data in, and return some data out.

If you are smart enough to work bottom up and you can create different layer of abstraction once a layer is finished and once you are sure that it does what it is suppose to do, then you can really stop to test that layer. (I hope is clear what I mean...)

The only test I can see having a real sense in clojure is when you use impure functions (functions that don't only get something in and spit something out, but that also do some work, writting in a DB maybe...) or when you think that your function/abstraction will be re-factor someday-somehow.

However in the case of the refactoring, the test need to be written still in a clever way: you have such functions tunnel A->B->C->D where you know that you will refactor C so there is no point in test A, what we really need to test is just C and its direct "neigbourd", or even better making sure that B returns what C needs and that C returns what D needs.

Or even better, since a clojure functions is way too long if it is more than 10 lines of code, well maybe all this testing is already too much, maybe the pre-post clojure's condition are enough. (This if we are talking about small functions, if A, B, C and D are abstraction layer you can't just keep all in your mind, so tests really makes sense here, still however doesn't makes sense to test A at all, nor the whole B, nor the whole D)

You know, the "pragmatic" approach that many programmers take - "writing tests is hard and time consuming, so let's write fewer tests" is a recipe for disaster and frankly the idea that if you just think about the problem more, that you're make fewer mistakes is very much along those same lines.

Here is the real problem - humans are fallible and we make mistakes. This is why pilots have flight checklists, why manufacturing has quality checks, why engineers have building codes, and why at hospitals checklists and procedures save lives

If pilots and mechanics just "thought more about the problem" they would still make mistakes. You might be a brilliant surgeon, but thinking you're too smart or too good to go through a checklist could cost someone their life because of a small oversight.

Would you feel good going under the knife if you knew that your life was in the hands of someone who thought they were too smart to scrub in?

We are a young profession with very few actual quality standards. Luckily, most of the software that we build isn't critical to people's safety, but I'd be terrified if the software powering medical robots, life support machines, and airplane autopilot systems were built by people who were "too smart" to write tests.

When a software bug costs people their lives because some programmer thought writing those tests was "too expensive" or "a waste of time" how smart is that programmer?

A: Pasting changed code into repl

B: Re-typing code

C: Hitting "previous command" and "return"

The overhead of automating this process (by making a test suite) seems high, but I haven't ever regretted it.

If you don't yet know the expectations/abstraction you have for some code then you're still deep in the design phase whether or not you like it. Good tests—like good static types or property checks—can be useful to smoke out the properties your abstraction will eventually fulfill. If they slow you down, don't use them.

But once you know your abstraction and start depending on it then please test. Most likely a good test will, right there, show you that you don't actually know your abstraction very well.

[1]: http://steve-yegge.blogspot.com/2007/01/pinocchio-problem.ht...

The value is testing is that it saves time in the long run. If you don't have a decent test suite, then as your product matures, you become unable to do anything because each new feature breaks a dozen old things.

Alex is right about the opportunity cost (see my own post "Testing is bullshit": http://blog.circleci.com/testing-is-bullshit/). But not testing is also an opportunity cost.

Basically you can spend T minutes writing tests, or D minutes debugging problems that tests would have found. If you don't test at all, then D will rapidly (within weeks possibly) become so high that you can't get anything done. By contrast, T will rarely go above 50% of time spent on a project (and if it is, you need to rethink how you test, unless you're building rockets or medical equipment).

My favorite poor-man's test-feedback-loop in Clojure is to use the built-in core.assert function to put tests at the bottom of the source file in combination with my editor's eval-buffer command.

Test DSLs, test directory/file boilerplate, and setting up an autorun cycle are the three things that constitute enough of a barrier to prevent me from testing, so those are the three things this example eludes.

Here's a file of a `to-slug` function that normalizes drug names into URL slugs.

    ;; Production code

    ...

    (defn to-slug [s]
      (hyphenize (simplify-name s)))

    ;; Tests

    (defn assert-slug [a b] (assert (= (to-slug a) b)))
    (assert-slug "ocusert pilo-20" "ocusert-pilo")
    (assert-slug "ogen .625" "ogen")
    (assert-slug "ogen 1.25" "ogen")
    (assert-slug "hydralazine hydrochloride, hydrochlorothiazide and reserpine"
                 "hydralazine-hydrochloride")
    (assert-slug "hydralazine hydrochloride w/ hydrochlorothiazide 100/50"
                 "hydralazine-hydrochloride")

    ;; Sandbox

    (to-slug "00-d-amphetamine")

And since eval-buffer always returns the result of the last expression evaluated, the above "Sandbox" section represents whatever case I'm currently trying to extend `to-slug` support for. The result of which is returned on every eval-buffer.

In this case, `to-slug` doesn't correctly handle "00-d-amphetamine" yet, so as I modify the production code to handle it, I can mash eval-buffer after every change to (1) see if my change introduces regressions and (2) visualize my progress on handling "00-d-amphetamine". Once it works, I add an "00-d-amphetamine" assert-slug and move on to the next case.

As to-slug matures, I can then move the asserts to a dedicated test directory if I want to and replace the slime with minimal case tests. -- Things that otherwise slow down my exploration phase.

----

I think core.assert fills the chasm of a function's/file's lifecycle between "I'm not even sure if I want this function or what its input/output will look like" and "my source code definitely needs this and this is its API".

In Ruby, I like https://github.com/sconover/wrong.

"Not all who wander are lost."