The Cucumber Test Trap

Why is it so hard for people to realise that Cucumber's main value proposition is not testing, but to establish a shared understanding between various roles on a software project?

For example, in Ruby you can have a method with 2 parameters of X and Y. Maybe X should be a string and maybe Y should be a float. You can unit test the method to make sure it does the right things, but as soon as something else calls that method there is no guarantee they will call it correctly or pass in the right type of arguments. Thus, there are tons of potential integration problems all over ruby code.

I think this lack of strong interfaces has trained ruby developers not to trust unit tests and so they go down the path of integration and end to end testing. Due to the weaker type and interface guarantees of the language, you will write more tests to get the same level of confidence in your code. By compensating for this language weakness, you are falling into "The Cucumber Test Trap" or more generically you aren't following the testing pyramid sensibly.

I like writing tests though my experience with software development has been limited to school projects, personal projects and intern projects. Nothing major. Nothing at the scale of ecommerce or Google scale.

But one thing I learned from writing tests (unit tests, integration tests and functional tests) is that do what is straightforward and don't worry about the strict TDD cycle. I worry about logging and error messages.

I've spent a whole week learning Lettuce. At that time the research project I was working on lacked tests (the project was using Django). I was told to look up Lettuce because writing Lettuce tests is like writing stories.

My awful experience was trying to wrap my head around the framework, trying to do many things that would take many steps and many tricks to complete. The table was awful. I didn't like it.

I gave up. I turned back to the built-in unittests module. I wrote some tests and from that point the only testing framework I have to add is the mock library. When I need an isolated unit tests I would call mock. When I need to test integration between modules without involving database calls or HTTP requests I will use mock.

Testing should be straightforward and should remain simple. Maybe Cucumber works nicely with Ruby's RSpec (I heard a lot about rspec and how rspec is so awesome and helpful for testing, not sure if that's true since I am a Python developer).

I also don't think writing a million test cases will work. In the end, while I think unit tests and integration tests are useful and awesome, I like to catch up with my functional tests. Most of my tests are done on the server side (so just use requests library to make HTTP calls). Depending on what kind of application you are testing, the testing tools can vary.

In the end, keep your tests simple. If something breaks, you better fix it quickly.

Logging , incidents collection and error messages are the most important thing in deployment and testing. If I have to spend hours to trace down the bug (the process died half way) or user is confused why the service is not working properly, I will waste more cycles to build my product.

My solution was to abstract away the things that made it slow and swap in faster components for the tests. Network connections were replaced with in-memory buffers, external programs were pulled into the same executable, and timers were simulated. Bam. The tests compiled in minutes and ran in seconds. People actually ran and listened to the tests because they were fast and reliable.

Of course, I was then left with no real end-to-end automated tests. The 'almost end-to-end' tests I had covered most of the software, though, and manual testing was used to catch the rare bug that would slip through. My conclusion was similar to that of the article: if you were going to do real end-to-end testing, you couldn't write very many tests.

Instead we need to keep focusing on the conversations. Find the scenarios that matter, that are important to document, that are worth automating and push everything else down into lower level, isolated tests.

How then do you prioritise which tests are actually important to document, or better still edit out the ones that aren't? It seems like it would be difficult to be consistent.

I find it interesting, if not surprising, that the Cucumber community is discovering exactly the same issues that we did with Fit: namely, that it encourages brittle integration tests, and that people don't use it for its intended purpose of collobaration.

I've come to believe that these problems are unsolvable.

I worked on and promoted Fit for several years. Eventually, after some deep soul searching, I concluded that Fit (and by extension, Cucumber) is solving the wrong problem. The value of Fit (and Cucumber) comes from discussions with domain experts. The tools encourage a certain amount of rigor, which is good, but you can get that rigor just as easily by discussing concrete examples at a whiteboard.

The actual automation of those examples, which is where Fit and Cucumber focus your attention, has minimal value. In some cases, the tools have negative value. Their tests are fundamentally more complex and less refactorable than ordinary xUnit tests. If you want to automate the examples, you're better off doing it in a programmer tool.

Some people got a lot of value out of Fit, and I'm sure the same is true for Cucumber. They got that value by using it for collaboration and focusing on domain rules rather than end-to-end scenarios. My experience, though, was that the vast majority used it poorly. When a tool is used so badly, so widely, you have to start questioning whether the problem is in the tool itself.

Ward and I ended up retiring Fit [1]. I've written about my reasons for abandoning it before [2] [3].

[1] http://www.hanselminutes.com/151/fit-is-dead-long-live-fitne...

[2] http://www.jamesshore.com/Blog/The-Problems-With-Acceptance-...

[3] http://www.jamesshore.com/Blog/Acceptance-Testing-Revisited....

It is absurd to complain about cucumber, if cucumber is used for things it wasn't intended to do.