Hacker News Reader

I don’t have much to say beyond what I already did re “testable” code; forcing code to be “testable” can be bad, but in reality making code more testable probably mostly amounts to breaking it down into simpler, functionally pure bits.

Not everything unit tests terribly well. I haven’t seen it work very well for things like React or Angular components. It does work well for small bits of apps. Like for example, lets say you have a component where a good deal of what it does is extracts information out of a URL. You might have tests that exercise the entire UI, entering URLs and checking the HTML output.

The “more testable” version of this, imo, would be separating the URL parsing and extraction bit to a single free-standing routine that outputs some data given an input URL. You can then table test that bit. Then testing that this ties into the UI correctly could be done in the integration or end to end testing.

In case of table based unit testing, I think it often works great, as it can act as a running log of regressions and newly discovered edge cases that can even serve as a sort of document of expectations for other developers, and while it cannot be used to test all sorts of code, it has wide applicability and you can see it in webapps, Go servers, the Wine and libinput sources, etc.

It’s easy to get stuck on a single strategy to rule them all, but I think that often is a bit presumptuous and maybe dogmatic. Tests are a toolbox. Not every problem is a nail.

>in reality making code more testable probably mostly amounts to breaking it down into simpler, functionally pure bits.

I know. The problem is:

* This process often introduces bugs. How are you going to catch those bugs? Not with your tests, you're changing this code precisely so you can write tests. It's a catch 22.

* Sometimes people do this only to discover that the simpler "functionally" pure code is pointless to test because it's so trivially simple. Somebody literally did that today on the code base I work on. The code as a whole still has bugs but those tests won't ever catch one. They'll just break when the code changes. Plus that "refactoring" probably introduced bugs. This I think is what the concept of "unit test induced design damage" was getting at.

This isn't a problem with tests as a whole. Or TDD. It is partly a problem with people who use the terms "unit test" and "test" interchangeably (this engenders entirely the wrong kind of thinking). It's mostly a problem with unit testing as a concept (i.e. not the specific frameworks themselves).

Having nice clean code interfaces is also often conflated with unit testing - this is a mistake. One does not necessarily lead to the other.

Michael Feather's "Working Effectively with Legacy Code" goes into some detail on this. When you have bad, untested code, you don't start refactoring. You start writing high level "characterisation tests", then you refactor. After that you can still write the unit tests for the better components.

I've followed this process a few times and then noticed that the last step didn't really add much value.

It felt good at the time doing it because that's what I was "supposed" to do. I'd achieved the supposed "testable code Nirvana" and... meh.

The first step was life (or at least, career) changing though. Bringing a piece of shit code base under control with integration tests was a process that blew my mind.

That's what led me to start questioning the efficacy of jamming architectural changes into code in order to sacrifice at the altar of the unit testing gods and that maybe, just maybe, unit tests' steep demands and limited value means that they suck.

You need a combination of unit tests and integration tests. Even trivial tests can help prevent someone ‘fixing’ your code in a refactor and breaking it, by defining the properties of what the code should do through tests. But it depends on the project - sometimes it just doesn’t matter, sometimes you need more formal V&V.

You need to think beyond the concept of unit tests and integration tests.

The future will have higher level and lower level executable specifications. The future won't have "unit tests" and "integration tests".

I work in a formal methods environment with proof checkers, property based testing, Isabelle/Coq/TLA+, and we still have unit tests and integration tests as concepts. You need a common language and these work well.

The distinction is easy. A unit test test whether a single unit works as expected. Generally, a failed unit test comes with a good pointer to what is wrong, and failed unit tests are rare.

Meanwhile integration tests test whether multiple units are integrated properly to produce expected results. When an integration test fails, you don't really know what is wrong. Instead you look at or write unit tests for the smaller units to see whether they fail, or whether the way you tied them together has a flaw.

The distinction is easy, yes. It's still important to think beyond the concept of testing.

For instance, there is no split - no separation of concerns between specification and execution in a unit test.

I don't quite get what you mean in the final sentence.

Are you saying your specification and your unit tests should be seen holistically?

I'm essentially saying that we should write specifications and a translation/implementation layer that "tests" the specification against reality. Maintain a separation of concerns.

For instance.

This is done in theory by cucumber/gherkin but it's done pretty badly. Good concept, bad implementation.

My broader point, though, is that unit tests are a bad concept and bad abstraction for this among many reasons but it's so embedded in programming culture (to the point that when people say "test" they automatically mean "unit test") that other approaches almost become "unthinkable".

I think you mean something like system testing, but backed by BDD-style specs?

TBH, I personally think you're being a bit too hard on unit tests - the number of times they reveal bugs in our our code bases (and in my own code - the horror!) is ridiculous. I do find them really valuable.

That said, testable OO code (I'm primarily a C# guy) has certain constraints, and sometimes making it testable results in a high level of abstraction - so much so that individual tests almost don't seem to actually test anything meaningful, and it becomes difficult to see where the logic and behaviour is, without digging through 42 layers of abstract classes, interfaces and factories.

Recently I've come to favour a kind of inverted test pyramid, where instead of unit tests forming the most substantial foundation, system tests do instead, followed by integration tests, and finally by unit tests at the tip. I find this leads to a "sensible" level of abstraction, where unit tests are used where they are most valuable, and system tests keep tests meaningful. Depending on your code, it might be quite tricky to setup systems tests, and it might need quite a large time investment, but IMO it's worth it. If you're able to dockerise your entire solution, then it's much easier to do.

I’m bearish on unit tests as well. Integration and e2e tests get you closer to the spec and are easier to change when the spec changes or your product inevitably evolves. I reach for unit tests when something is foundational or hairy, but IMO they should be a last resort. They fill your application like sand in gears otherwise.

I find I often have an integration test that is failing, and then I start adding unit tests until I find the error. Especially when the failure is not amenable to investigation by a debugger.

Have you had a chance to check out the Screenplay Pattern at all? I have had great success using Cucumber to stitch together common abilities/actions/question types. My step definitions usually consistent of a single line of words grouped by a few parentheses and commas. Groovy/Kotlin make this particularly nice.

You seem to be describing property based testing?

What do you see as good integration test tooling?

Something where the integration tests:

* Are clear and easy to read and form a specification of sorts.

* Which are cheap to build.

* Where the interactions the code has to the "outside world" are easy and cheap to mock/test/keep under control - e. e.g. time, database, browsers.

* Has good debugging tools such that it's easy to track down the source of the bugs.

You're talking about things like dependency injection. I'm not a huge fan of testing but "testable code" when written in a certain way (without dependency injection) actually has many external benefits that don't have to do with testing per se.

Really the way you need to structure your program is to divide functions in your code between things that can be unit tested and can't (IO calls).

This can easily be done without dependency injection which is likely what you're complaining about.

For example Don't do this:

    #unit testable with mock (Bad!)
    function add_one(key: str, database: Database):
         return database.get(key) + 1

Or.. even worse:

    #Unit testable with mock (Bad!)
    class Adder
       Adder(database) -> None {
          this.database = database
       }

       add_one(key: str) -> int {
          return this.database.get(key) + 1
       }

Do this instead:

      #IO function, Not unit testable
      def get_database_value(key, database) -> None:
          return database.get(key)

      #Unit testable without mock! Good!
      def add_one(value: int) -> int:
          return value + 1

      #IO function, not unit testable. 
      def composition(key: str) -> int:
          return add_one(get_database_value("name"))

Unit tests are great, for testing unities.

What self-contained unities are there on your code? If you are doing low level system programming, I bet there are a lot of them. If you are doing high-level CRUD, I bet there is none, all of them you import from third parties.

Writing unit tests for non-self-contained code is crazy, and leads to all those problems people identify. In my experience, the problem is that the most vocal evangelist believers of unit test are all on places that work on the high level stuff.