The problem with test suites is that, as they grow and age, it becomes harder and harder to understand why tests are testing what they test. When a years old test fails because 5 is not 7, you cannot really fix it unless you understand why it was supposed to equal 7 in the first place.
Cucumber is a way to describe requirements at a higher level of abstraction. They provide rationale on top of mere assertions of behavior. This is worth the small overhead of writing step definitions.
I guess you can get the same by heavily commenting your tests. But you cannot enforce this easily in a team.
Our biz people can't write useful requirements so I doubt they can write meaningful Cucumber scripts either.
I think that they add way too much complexity. It doesn't make sense to try to write technical software tests at such a high level of abstraction.
When you try to test something at a really high level along the lines of "As a user, when I click on X, I want Y to happen" the low level technical ramifications of this can be enormous.
There is a huge amount of underlying complexity which the one-line sentence simply cannot capture and which is prone to change and needs to be tested separately... Probably not in a way that a non-technical manager would understand unfortunately.
Ultimately the tests are not granular enough to be useful as tests for developers and they have too much overhead to be useful as documentation for non managers.
Phase 1
Non-technical and semi-technical business analysts write scenarios as acceptance criteria for the requirements.
They are still so high level that the devs need to re-write or refactor the majority of them to be able to get any sort of re-use out of them.
Phase 2
Writing scenarios is largely seen as a time sink and the test coverage is falling behind. This is then changes so that the BAs write a headline scenario and some high-level bullet points (higher level than they were doing already) which are then fleshed out by devs.
Devs re-write statements in ways that aren't very good english ('When I log in as a "Tim_test5"') and scenarios are no longer good to show to the customer.
Phase 3
Test writing is still slow, so BAs and testers start writing and performing manual tests to close the user stories. Step 'library' and feature files are building up but still only a relative handful of tests are automated.
So yeah, in an ideal world they would be perfect, but I would recommend the following:
* Don't include BDD in your 'definition of done'. You will never close anything.
* Only write BDD for things that are stable - some changes can mean that tests need to be updated, split or take more arguments. Syncing up changes to statements in feature files is extra overhead.
* Deciding on wording to reflect business meaning of a statement and technical meaning of a statement is harder than you think. Factor in time for this.
* If your team haven't used BDD before, create some central resource with statements in that can be used by non-technical users to build up scenarios lego-style. If what they need isn't there, then it can be added. This is especially true when a set of actions is easily abstracted to a single business action ("I send the request" or something which takes many clicks and form filling).
Upon submitting/editing it would create a ticket with the story. We also ended up having it create a branch once it was scheduled by product so engineers could git checkout storyXXX and see where the cucumber steps went yellow.
There is a pretty cool SaaS product that does this now more or less that I’ve only gotten to play with http://www.simian.io/
With a DSL-driven BDD framework (like rspec) you get autocomplete and debugging and every other language feature without really losing any of the descriptiveness.
With cucumber you get lots of pain in both those areas, without anything that feels like an improvement along another metric.
This is the part I could never buy into either. You cannot escape that writing tests is the same thing as programming (you need to break the problem down into steps, formalise what you want to check, use abstractions) so non-programmers aren't good at writing tests without rigorous training. Non-programmers tend to need a lot of help even describing what they want in plain English as well because they just aren't used to thinking like programmers. I've worked with non-technical people for years who still don't understand that "it's not working" isn't enough information to go on.
As for overhead, it might make sense if you're working on one project in a single language for a long time but the setup and configuration overhead has been significant when I've tried BDD tests.
I'd rather just use descriptive function names in tests as then you can take advantage of autocompletion, automated refactoring, jump to definition shortcuts, static checking, language features etc.
That said I haven't made up my mind yet if it is good and I just don't get it yet or if it is a mostly a waste of time.