A Case Against Cucumber

I've been working in Rails since 1.2, and the vast majority of developers I've met have been of the opinion that cucumber sucks. There's a really vocal minority on the internet though, so you'd get the impression that it's pretty popular.

That said, almost every rails project I've worked on has had the cucumber gem included at some point, just not actively used.

IMHO the vocal Rails community puts too much emphasis on testing. I write tests for my code, but not as many as some would like. The RSpec book in particular is a gigantic load of crap. And I say this as a fan of comprehensive tests! The ridiculous lengths that book goes through in testing that codebreaker game are so detached from reality it makes my head hurt.

My impression (as a static typing bigot) is that Cucumber style testing is like the crap shoot of dynamic testing (what should I type? What are valid commands? What params do they accept?) made an order of magnitude worse by using regular expressions.

...who thought this was a good idea? (Sorry, being harsh.)

That said, I think the idea is conceptually fine, and, as usual, I think could actually be fairly pleasant with the right (static) tooling applied, e.g. ThoughtWorks's Twist:

http://www.thoughtworks.com/products/twist-agile-testing/

Which is a GUI that provides all the nicities of "what commands are allowed", "what params do they take", "oh, refactor this command name from abc to xyz", etc.

I think PM/testing types would love this sort of setup. Granted, it would still take dev investment to get the fixtures/commands/etc. setup. But if you have a huge line-of-business app that will drive a business for the next 5-10 years, I think that's a good investment.

Disclaimer: I've never actually used Twist because it's a commercial product. Yes, I know I suck.

Cucumber has been a great success for our team. We've trained the BA's how to write cucumber scripts, which in turn has done a lot to train the BA's in how to think about the stories and features that they are requesting. The dev will sit down with this feature definition and the BA's, and the feature is groomed into a viable and feasible feature that is testable. We strap these to a web automation (we use specflow to watin), and get a fairly significant return on investment.

Cucumber is like any tool; if used correctly, it's great, and if it's used as a de facto tool, then you're going to struggle with it.

For the most part, in my experience Cucumber isn't worth the effort (one place called it "Encumber"). If you have product managers or BAs who can read code, then it's easier to capture the requirements with more developer-friendly testing tools. And if the product managers or BAs do not read code, Cucumber doesn't usually help much.

Having said that, I did work a gig where the tester would sit with the subject matter experts, and as they would talk, he would capture what they were saying as Cucumber tests, and he would then echo the tests back to them to see if they were right. Afterwards, he would then add whatever other tests he needed (checking weird edge cases and such), and code up any additional fixtures he needed. Then it was just a matter of me getting all the tests to pass. It was a really nice way to work - I knew unambiguously when I was done with something. (It helped that the tester sat right next to me.)

I'm a huge fan of the gherkin format, it let's me sketch out a feature and then work through it methodically, testing as I go. But cucumber has a critical flaw, it forces you to think about code while you're writing. Because step definitions are global you can't just write any old thing, you might fire off a step you wrote previously. This is only a side issue though, compared to the biggest problem, step reuse.

    Given a person "fred" exists
    And a person "ethel" exists
    And a fatherhood exists with parent: person "fred", child: person "ethel"

Did someone mention cucumber steps are written in english? What happened to:

    Given Ethel has a father called Fred

I moved over to Spinach[1] about 18 months ago and I've been much happier. Steps are specific to a feature so I can go to town with shakespearean english. Reuse isn't an issue either, it just happens in the step definitions, not in the gherkin files. Using more natural english tends to lead to fewer larger steps anyway, so it feels a lot less like call-by-regex.

[1] https://github.com/codegram/spinach

TL;DR stop bitching about an acceptance testing tool when you're not using for acceptance testing.

The problem isn't with Cucumber. The problem is your cukes suck. This is probably also a problem in your Rspec/Minitest/whatever tests. If you're using cucumber the way 90% of the cucumber tests I've ever seen (indeed, a great many i've written myself) are written, then you're writing integration tests, and probably crappy ones at that.

  Given a user
  And I go to some really cool page
  And I click on some button

is NOT a domain test. Domains, for the most part, don't have 'users' (unless maybe your domain is drug addiction). Or 'pages' (unless you're in publishing). And they're definitely not about 'buttons' (unless you're a tailor.) [Yeah, you can probably find some domains where those are actual concepts. They're probably not all concepts in the same domain.]

Write actual domain tests for your acceptance tests. Do this regardless of your framework—there's absolutely nothing preventing you from doing it rspec or anything else. Fuck step reuse (gherkin alternatives like spinach are great, specifically because the steps are isolated to the test.)

If you don't want to write acceptance tests, and you feel comfortable with that, no biggie. Lots of people don't. But don't create a straw man out of Cucumber just because you don't understand layers of testing.

This goes for Cucumber fanboys as well. Don't push cucumber into layers it doesn't belong. Its a domain testing tool. Not an integration test. definitely not a unit test. Don't push it on a team that doesn't want it. You can write the same kinds of tests in rspec. Disclaimer: I like cucumber. I am not using it in my current job because it wasn't a good fit for the company. I will probably use it again one day. I still write AT's.

This:

  Given I hate cucumber
  And I post a scree against it
  Then the haters will rejoice

is pretty much the same as this:

  describe "hating on cucumber" do
    it "produces a response" do
      given_i_hate_cucumber
      when_i_post_a_scree
      then_haters_rejoice
    end
    def given_i_hate_cucumber
    end
    #.... 
  end

I've been using Cucumber for almost 3 years now, but if I were to start a new project today I'd pick Capybara with MiniTest or RSpec.

When I first started using Cucumber I thought it was cool that "anyone could read the test and understand it"

But that one advantage doesn't really matter that much when everybody in your project knows Ruby and doesn't need the natural language side of it.

I think you're perhaps missing the point in one or two places.

Cucumber helps techies (devs, testers, etc.) and non-techies (product owners, scrum masters etc.) work collaboratively. I don't think the authors of Cucumber ever said that it enables non-technical people to 'read and understand the underlying code'.

Ultimately though, I agree with your conclusion but for slightly different reasons. I use both RSpec and Cucumber daily; they are both awesome and provide a similar end result. I enjoy writing Cucumber features but in certain circumstances it doesn't scale for large/complex applications (in my experience). I think that this isn't a problem for many people though, unless they're doing something wrong 'under the hood'.

I couldn't agree more. The pipe dream is that you have BAs or other non-technical team members write the Gherkin specs and that saves the developers time because "the tests practically write themselves!" (Assuming you're maintaing your giant library of awkward cucumber regex matchers...). In reality it takes much longer to sit down with someone and teach them how to write in Gherkin, all for the goal of having a human readable spec that doubles as test steps. It's much better if they write the specs however they want, in normal english, then you can translate those into normal rspec or test unit tests.

It's a laudable goal, but the abstraction is so leaky in reality it just ends up creating more work for everyone.

Cucumber, gherkins, capybara, handlebars, sinatra... do these goofy names bother anyone else? I'm writing code, not hosting a children's show.

It completely depends how you use it. There's so much misuse of it out there I can understand why people hate it so much. But done right it's an extremely valuable collaboration tool.

I'd recommend the Cucumber Book if you want to read about how to use it well:

http://pragprog.com/book/hwcuc/the-cucumber-book

I've also blogged a lot about how to use Cucumber well here:

http://chrismdp.com/tag/cucumber

Cucumber is part of our workflow and, although I don't have a control for this comparison, I think it makes us more efficient. Our product owners write in Gherkin (non-technical != illiterate btw) and can essentially throw the feature files over the wall and any dev can pick it up and get to work. Now, there is conversation around every feature but sometimes only at estimation time.

Our POs also use git (non-technical != vcs illiterate) so we have a history of who wrote/modified what spec when.

Also, we have an additional simple layer of abstraction between feature steps and the actual tests. Most of our step definitions are three line methods that make calls to a ui driver that knows the details of our ui; or to a "given" driver that knows the details of our database/models. This ensures that there is only one place where anyone will implement, say, a user logging in.

  Given(/^I am a logged in user$/) do
    given.a_logged_in_user
  end

while elsewhere there might be

  Given(/^I have signed in$/) do
     given.a_logged_in_user
  end

If the log in process changes, we just edit the given driver.

We generally only write golden path features. Edge cases are handled with lower level testing.

Our feature files stick to specifying business value rather than ui implementation. For example, we don't do this:

  Given I visit the homepage
  When I enter "children's bikes" in the search field
  And I hit the submit button
  Then I am on the results page
  And I see a fieldset with a legend "Children's Bikes"
  And I see a table containing "Huffy Sprite"

We would write it like:

  Given I search for a specific product category
  Then I see the search term I used
  And the results of the search

When a dev needs to work on an app they haven't touched in 6 months, it's nice for them to read feature files like this to get up to speed on what the app is actually supposed to do. Then can then dig into the ui and given drivers to see how it does it.

There is additional infrastructure and PO training involved, but overall, I'd say cucumber improves clarity and communication on our team.

While there is initial overhead in hooking in the tests, the power comes when you have modularized test steps that you can reuse for multiple tests.

At my last company we had QA write all the cucumber tests and someone would hook in the new statements, which is the overhead you suggest. Now if you write 3 different gherkin statements that do the same thing, then that is not optimal.

Like many problems, I don't think the tool is at fault. It does what it claims. Provides human readable syntax for test cases and lets you hook that in however you want.

I think it's a valid point to think that extra layer is unnecessary, but I wouldn't go as far as discounting it and saying that no company has derived value from it.

Like all code, the messes usually stem from how the code is implemented, not the language itself.

Cucumber was most developers first exposure to capybara (IIRC, capybara was developed by the cucumber folks).

Since capybara is so awesome, it gave a lot of people a nice impression of cucumber.

But unless you actually have a customer in the loop writing cucumber tests, just use the capybara directly.

Not a fan of cucumber. I started it, but the extra layer of abstraction was too much complexity for too little benefit. Instead, I think you can make almost as easy to read and understand acceptance tests with plain old rspec.

I just TDD'ly implemented a "user signs up for Direct Deposit" feature, and the acceptance tests look like this:

  describe 'Direct Deposit' do
    before do
      @workflow = DirectDepositWorkflow.new(create(:user))
    end
  
    describe 'User sets up direct deposit with correct info.', :js do
      it 'user sees confirmation page - individual' do
        @workflow.sign_in_and_visit_direct_deposit_form
        @workflow.submit_correct_form
  
        expect(@workflow).to have_confirmation_message
      end

      ...
    end
  end

Then you can make the workflow spec helper class:

  class DirectDepositWorkflow < Struct.new(:user)
    include Capybara::DSL
    include UtilitiesHelper
    include Rails.application.routes.url_helpers

    def sign_in_and_visit_direct_deposit_form
      ...
    end

    def submit_correct_form
      fill_in 'name', with: 'John'
      click_button 'Submit'
    end

    def has_confirmation_message?
      page.has_css? '.success_alert', I18n.t(...)
    end
  end

To me, this is the perfect blend of easy to read, high level specs (serving the rule of cucumber/gherkin), with easy to understand, rspec syntax and ordinary method definitions.

"When I write a Cucumber feature, I have to write the Gherkin that describes the acceptance criteria, and the Ruby code that implements the step definitions."

As someone who recently started using cucumber, this was definitely a frustration at first. But after a couple weeks of writing step definitions, and focusing on making them reusable (using some of the tips here: http://coryschires.com/ten-tips-for-writing-better-cucumber-... ), I ended up with a pretty good bank of general steps that I could cobble together into new features without much modification. The result was something much more readable and reusable than if I wasn't using Gherkin on top of the step definitions.

I've posted a response here: http://mattvanhorn.com/2013/09/19/a-case-for-cucumber/

The underlying assumption of this article is that it's easier to read code than it is to read english.

Well even for a developer of 15 years even the most obtuse English can be more easily absorbed than the cleanest code.

The speed of parsing and understanding code is directly related to the amount of time spent writing it and how recently it was read.

English can be read and understood at the same speed every time.

There is also a non-linear relationship between your feature file size and the underlying code they represent, especially if they represent integration testing. A single feature could test code from 10 different files.

For a single developer in an codebase they know and understand well and have recently worked on, Cucumber can act as friction on development.

For a single developer who hasn't worked on a codebase before, or who is returning to a codebase after some time Cuke steps will definitely improve their ability to understand what the code is doing and by providing a regression test suite improve the quality and probably speed of their output.

Code is a language of instruction - not communication, documentation can help, but it is easily forgotten and fails at communicating integration.

Assuming you're focused on writing reusable steps and building up a solid test architecture I don't see cucumber costing you time. You also really shouldn't need to go to code that often to debug a test failure. Write better test frameworks, write tests for your tests, and self report issues in your test framework when they come up. E.g. unable to create test fixture -> this->skipped("reason");

In my experience the ability to just plug and play steps and quickly add additional coverage for edge cases is a huge bonus for test automation.

But again you have to actually engineer things and write reusable steps and intelligent enough test frameworks that you don't need to often dig through the test code.

I think this issue is somewhat similar to how when writing unit tests you'll want to build up a library of reusable custom asserts targeting your domain so you can quickly exercise your sut and validate things. It's definitely an upfront costs to build things up like that but in the long run it pays off very nicely when you avoid taking on too much test debt.

Disclaimer: I'm not for or against the use of cucumber.

The second claim against cucumber / gherkin is not it's fault, it's the toolset's fault. Using grep to find your step definition is not very effective.

If you use a tool like Rubymine, you can jump directly to the applicable step definition through one key combination.

> Non-technical people don’t read code, no matter how easy it is to read.

There is something deep and surprising about this fact.

I love cucumber. When you join the development of a project it's a great place to start and learn what the project is about.

The whole 'customer writes features' thing never worked for me either, but just having it as automated specifications for my projects makes it very valuable for me.

I drank the Cucumber Kool-Aid and had an app with basically 100% Cucumber coverage. If I wanted to change something, I would first write a cuke for it, write specs as necessary, and only as necessary write actual code.

Of course I got into the same spot that almost everyone gets into doing that: too many "and then I click..." scenarios. Great coverage, but then the requirements change substantially and the test base is just a massive snarl of interdependent assumptions. At one point I had effectively written, in cucumber, a natural language interface to my app.

But then at some point I needed to move and I needed to move fast and I just let the whole infrastructure rot. Now I'm doing almost everything on the client side anyway, so it's all a little moot.

All of that being said, I'm still a Cucumber fan, but I approach it differently. Here's what I think:

You should have cukes for the 5-10 things that if they don't work make your product completely pointless.

On Gmail, there would be cukes like:

    When I receive the email from Fred about dinner
    And I open it
    Then I should see the message
    When I reply
    Then Fred should get my reply

And those cukes basically don't need to have wildcards:

    When /^I reply$/ do
      click "reply_button"
      fill_in "message", :with => "I'll be there!"
      click "Send"
    end

Essentially:

* Use cucumber as a way to force you to keep track of the handful of absolutely critical integrated experiences

* Don't use fancy grammars... Think of steps as human-readable function names.

* Don't do anything fancy in the steps. They should just be a list of actions that users would have to do.

And yeah, there's no reason you couldn't do all of this in Steak. I think Cucumber is just nice in that it encourages you to write it out in human terms.

Towards the end, he says "...but I do enjoy the Gherkin syntax. Not for testing, but for gathering feature requirements..." and this is where cucumber or fit or FitNesse is useful - you gather requirements, then because these requirements can be run as tests you start running these requirements as tests and now you have a living document of your product and product owners can read the tests (which are the requirements). If you look at such tools as mere test runners, then they do add an extra layer of complexity, but these tools were not meant to be just test runners... so you are using it for the wrong thing... :)

We started using Cucumber 6 months ago and I hate it. My biggest issue is that it's unreliable. 90% of the test failures work if you do it exact same steps manually.

It seems like it always times out waiting for something on the page to load.

Damnit, I was hoping for an article about the health drawbacks of cucumber (I'm boring that way), and it's about another Framework.

It'd be awesome if people started tagging these as [technical] or [computers] or [vegetable-afficionados].

Rant out.

"I don’t prefer to use Cucumber for any of my testing, but I do enjoy the Gherkin syntax. Not for testing, but for gathering feature requirements. It provides a very clear and concise way of explaining a feature, without confusion. But that is where the line is drawn."

That is also where I draw the line. The whole argument that other non-technical folks can read Gherkin may be true, but I can't imagine them enjoying it.

It's like reading computer-speak. Why can't features just be described in English...?

Yes. And he's not the first one to say this. There was a post saying the exact same thing about a year ago, maybe even on HN?

But, yes, just yes.

(I'd go even further and say "why rspec when you can test::unit/minitest? especially for Rails since the minitest path is officially supported by rails." Again, the extra layer of stuff is just more stuff to understand and troubleshoot when your tests aren't working as you'd like -- for unclear benefit.)

The thread seems to be overrun by a bunch of whiny crybabies.

No one forces people to use them, yet they seem to have a great time crying about it without trying to defend their position.

Cucumber and RSpec are among a set of testing tools which has really changed the testing landscape. It would be really wonderful if people spent half this time and energy on trying to create a better tool or joining the open source groups of Cucumber and RSpec and trying to make it better.

I'm a .net dev so I've been using SpecFlow (which uses Gherkin) and frankly, I love it. I write a happy path and a sad path for my scenarios, make sure my step definitions are well parameterized and my QA can go off and create as many new scenarios to test all the edge cases he wants. It works wonderfully.

As a product/program manager... I can tell you that writing RSpec tests are just as easy to write as Cucumber tests. And, with less overhead.

Really, I'm not a Ruby guy, so I thought it's some vegetable story... cause I love cucumbers (& tomatoes)... Ah...

Figured the bottom voted comment would be about the virtues of fresh cucumber.