Cucumber lets you write automated tests in plain language
cucumber.io
cucumber.io
The preconditions are that you have domain experts or business people interested and willing to engage in writing or reviewing these tests. Unless you have this and it's something that those people are going to sustain when the going gets tough you're just making writing tests harder for no real benefit.
- write specs in a cucumber fashion
- write parsers to go from cucumber to whatever objects / data you expect
- finally write your tests
In practice when people try to use it for anything semi-complex they inevitably end up either writing very vague tests or repetitive tests, neither of which are of much interest to stakeholders.
I've used docs generated from hitchstory for having conversations with stakeholders. I've also shared semi-human readable unit and e2e tests - especially when it's, say, JSON snippets in an API with technical stakeholders.
Unfortunately, except for a few domains where specs can be expressed very concisely, cucumber isn't suitable for that.
Going from "Step 1: Do X. Expected result: Y. Step 2: Do Z. Expected result: Q" to a big long "sentence" describing these steps joined with "and" (without the expected results being linked to each step) is strictly worse in just about every way. The actual automated tests are still being written the same way they always were, the descriptions of them are just WAY harder to understand.
It's a two way street. I'd highly recommend never implementing it unless product is the one driving it, and actually makes it a priority.
The other one, and the one that is talked about more, is control of the methodology by the individual dev team, rather than as an external dictate from outside.
But the one relevant to Cucumber is that development isn't an activity that the dev team does alone, but a high-contact relationship between the dev team and the people working in the application domain that the software serves.
Like many Agile tools, it is completely broken when divorced from those organizational principles. Unfortunately, while lots of shops are "Agile", those "Agile" implementations are almost invariably completely divorced from those principles, and are the same kind of consultants-sold-this-canned-buzzword-bingo-approach-to-management-and-now-it-is-imposed-on-the-software-team-we-keep-walled-off-from-business thing that the entire Agile movement was a reaction against.
There is a narrative, pushed by its creators, that cucumber isnt broken it is just misused.
It's a narrarive that needs to die.
It is equally bad as an agile collaboration tool as it is a testing tool.
Just like all the other no-code inventions out there, they fail to reckon with the fact that essential complexity isn’t a problem with programming language syntax.
If readable tests are the goal, I think the time is much better spent cleaning up regular tests and writing well-named helper methods. Especially in Ruby you can get pretty close to tests that can be deciphered on a high level by non-programmers. That said, I've never had a PM or other non-programmer actually want to do that.
Capybara has this problem as well as do other web acceptance tests. Maybe not every time a new matcher but frequent enough. Mostly because of the high variability of HTML paths. You have to design your HTML in a way (use lots of [title] tags and standard HTML) if you want to avoid it.
Then you refactor your common helper functions as natural, but you’re not forced to enclose them in steps.
Cucumber tries to solve the problem of turning customer requirements into 'real code'. In exchange for that worthwhile benefit, it asks you to implement the most terrible, reg-ex based spaghetti code imaginable.
The problem is that it doesn't solve the original problem AT ALL. And then you are left with terrible reg-ex driven spaghetti code. Like the Jamie Zawinski saying, "now you have two problems".
The lesson here is that software development processes have to pass the 'human nature' test.
The software industry has largely abandoned waterfall development because it just doesn't work well in practice. It doesn't work because people don't know perfectly what they want before they build it. Agile processes usually are much more efficient because they are more closely aligned to how humans solve problems in the real world.
Cucumber suffers from the same issue of being disconnected with reality. In theory, you can find a customer who can give you perfectly written use cases and you can cut-and-paste those into your cukes. In practice, that never, ever works. So let's all stop wasting our time pretending it was a good idea now that it has been shown to not work.
It's quite funny that now with AI-based no/low code the same thing is attempted again. Even if the regexes might disappear, it's even more text, with even less structure (assuming anyone checked those prompts into git in the first place)
I'd never ask them to write them, but I will often write a spec/test based upon their two sentence jira and then screenshare and walk them through my interpretation to get feedback early (i.e. before ive wasted time building the wrong thing).
Cucumber/gherkin is awful at this of course, and the regex thing was a terrible idea but it's not the only tool.
The idea that tests should be split into a specification layer and execution layer is a good one that should have taken off by now.
There is a fundamental reason it hasn't:
An actual specification layer isn't any simpler than the execution layer. That's a programmer's fallacy.
What has taken off, and is part of virtually every software project, is a loose, natural language specification, which hints at "more or less" what the stakeholders are imagining. The idea that you can close the gap between this and a complete specification in a way that all stakeholders can still digest is the fantasy of cucumber. Or any other tool that attempts it.
You can't solve the problem in that way. Because, from a high-level stakeholder's perspective, the whole point of the people below them (whether programmers or UX designers or anyone else) is to fill in those details in a way that matches their own hazy expectations, at least well enough.
The point of separation of concerns isnt to keep the simple layer separate from the complex one. It's to simplify the whole thing by only addressing one concern per layer.
Unit tests are often a pain in the ass to read because they are a mess of implementation and specification details. No separation.
>You can't solve the problem in that way. Because, from a high-level stakeholder's perspective, the whole point of the people below them (whether programmers or UX designers or anyone else) is to fill in those details in a way that matches their own hazy expectations
If you crystallize the hazy expectations on their behalf and skip feeding it back to them then you will often find out that "thats not what i meant" after the code is complete.
Those mistakes are expensive and avoidable.
> The point of separation of concerns isnt to keep the simple layer separate from the complex one. It's to simplify the whole thing by only addressing one concern per layer.
With some caveats, I think this is just a fiction. There can be some value in having high-level tests and low-level tests, but not because it removes complexity. It can help with focus and priority. Which is a problem that can be solved in many ways.
> If you crystallize the hazy expectations on their behalf and skip feeding it back to them then you will often find out that "thats not what i meant" after the code is complete.
But this is exactly what they want you to do, and do well enough that "that's not what i meant" is not a big problem. They certainly don't want to read cucumber tests as a way of ensuring you're on the same page. They will tolerate rough, incremental prototypes, per the old agile advice, and this is probably still the best way of solving the communication gap problem.
I think the problem here is that you are substituting standard arguments for my arguments.
>With some caveats, I think this is just a fiction. There can be some value in having high-level tests and low-level tests, but not because it removes complexity.
Separation of concerns isn't about having "high level" and "low level" tests. You don't have "high level" views and "low level" views in MVC. You have views and controllers and you keep them separate because otherwise you have an old-skool PHP-like mix of SQL and HTML all over the place.
That PHP style mess happens to most tests because the code that is intended to describe behavior is mushed up thoroughly with the code that attaches to the app and executes it.
If you separate the concerns, the test becomes readable and even serves as a spec.
>But this is exactly what they want you to do, and do well enough that "that's not what i meant" is not a big problem.
The problem is rarely that they didn't say what they mean. The problem is that once you've built the thing or walked them through a crystallized user story it makes them rethink what was actually needed. In many cases I've walked stakeholders through user stories and they suddenly realized no new feature was necessary.
>They certainly don't want to read cucumber tests
Nobody does, but that's because cucumber tests have abysmal syntax and UX, not because the idea of readable spectests was intrinsically bad. Throwing away spectests as a concept because cucumber is bad is like throwing away programming as a concept because COBOL is bad.
>They will tolerate rough, incremental prototypes
Of course, which is not as expensive as a fully fledged app but way more expensive to build than crystallized user stories. I prefer the cheaper option.
You dont get that for free, but i disagree with it being a 'mess' of regex...Regex for test steps is usually as simple as a basic string or number capture pattern. It's 100% a skill issue if it ends up 'messy'.
If you need something more complicated than regex you use a data source file for parameters with named columns....Which brings me nicely round to how easy it is for qa or management to look at the data being tested and again, reason about it, make suggestions, spot missing cases, etc.
Qa read every test we write. Management reads ones of high importance, sometimes.
> The software industry has largely abandoned waterfall development
Variants of waterfall are essential for large, high-risk safety systems. There is no 1SFA dev process.
Business people don't really care about this stuff. Over the years I realize more and more that we engineers are naive in thinking that business side is concerned with edge cases and race conditions.
Just to give an analogy software people might get better, if you come to a lawyer because, say, you want to buy a house, you are not going to sit down with them and say "given I want to buy a house, when the seller hides water damage costing over $2000, I get to walk away from the deal". You just hope the lawyer is good and will protect you from various edge cases. You have a lot more to deal with than just closing paperwork. You probably are thinking about renovating, moving, getting inspection, etc.
Businesses are just like that with engineering. They don't want to sit down and meticulously analyze every possible edge case. They have other things to do. Especially when stakes are not that high. Most of these errors can probably be resolved with a phone call and a database edit.
I think this is probably for the best. A good engineer will make sure you're standing on a solid ground, and ask the right questions at the right time. They wouldn't need this amount of hand holding. Leave business time to focus on making deals, connections, organizing the whole operation to move forward, etc. Let them give you vague requirements, and crystallize them yourself. It's way better than a micro-managing business that thinks they know exactly how everything should be.
P.S. Also, I'm not sure why everyone is so hung up on regex = bad, it's not like switching to an AST-based language would've made anything better here. Regex is fine imo, just the entire concept isn't.
Specifically I try to make a nice string representation of state, and then tests match off that state.
So for instance I don't want to say expect(x.length).toBe(0) (one syntax, Cucumber goes further in that syntactic direction). Because if the length is 1 then 1!=0 is a very opaque failure. Or if you do expect(x[0].attr).toBe("y") then you've tested one thing but you have to have another test for list length, other attributes, etc. Often you'll leave the others out, expecting them to be ok, but sometimes they aren't...
Investing in a good, stable, matchable string representation will give you a great tool for any number of tests, and it's not just a great tool to verify but also to debug your tests.
Given /^a nice new bike$/ do
expect(bike).to be_shiny
end
have never actually tried it for that reason - it just seems worse in every way
It creates unnecessary work for something that's never to be seen by real stakeholders.
There is a new trend these days using LLM that is similar to Cucumber: Spec Driven Development using AI. You'd be left disappointed again.
Engineers' responsibility begins at the "code behind" the gherkin layer. Someone needs to enforce this.
Sadly, IMO because gherkin looks like structured language (i.e "code"), and because that's what engineers do, they end up doing it. Either they tricked themselves into thinking they should, or the product team tacitly assumed it was a code thing.
If you're an engineer and you're writing gherkin, you should consider this a "code smell" and consider ditching it. Go for a spec tool one layer closer to your main domain. i.e. just write normal unit & integration tests.
I'm not saying "don't do BDD" or "don't do acceptance tests". You should do that! I'm saying "gherkin should allow product owners/managers to take ownership of reporting on how well the app meets their expectations". But most typically, they don't get it, or they don't want to do it.
I recently worked somewhere where we had dedicated devs-in-test. They wrote most of the e2e tests for the application. They wrote it in Gherkin, which meant that they _also_ wrote the JS/Selenium tests behind them. So they were basically writing two sets of scripts for each test. At some point they explored using a 3rd party cross-device testing platform (lambda test, maybe?) who had their own in-house test script that looked like Gherkin, so it needed a JS/TS code-behind to actually run the tests. So they did it all again!! Such a waste.
No, it doesn't.
Or to be more precise: Cucumber lets you write your tests twice - once in something that isn't nearly as close "plain language" as it seems at first glance, and then again in a horrible mismash of regex spaghetti code that will make you wish you'd been born a thousand years before computers were invented.
I was forced to try to use it about 15 years ago and spent literally twice as much time debugging the testing framework as I did actually implementing code - velocity on actual feature development ground to a halt. I think the most hilarious part about this was that it was completely incapable of finding the most common class of problem we tended to run into at the time - cross-browser presentation issues.
One of the big problems you'll find if you try to use this trash is that nobody except a software engineer will be really able to write tests in a consistent enough way that you'll be able to implement them without a truly horrible regex soup: even if you can actually get a non-software-engineer to write cucumber tests for you (dubious), the non-engineer-types will write "given I'm logged in" for one test and then "given I have logged in" for another, and "given I gave gone through the login process" for a third. And it doesn't matter how many regexs you implement to try to cover all their variations in their language, or how clever your regexs are - the nature of natural language means there's near-infinite variations, and they'll come up with new and more annoying variations just when you start feeling confident. And that's before we even start talking about the ambiguity of natural language.
I did manage to find a decent use for cucumber-style language a couple of years ago, though - we started commenting our unit tests using this style:
def test_something():
""" test some thing """
# given some prerequisite
set_up_prerequisite()
# when I perform some action
perform_some_action()
# then some condition should be true, because <reason>
assert some_condition is True
If your job is polishing a turd, make the turd shine like a cucumber. I have seen countless hours wasted on making Cuke/Gherkin "pretty" instead of accomplishing anything for the business.
On a technical level, cucumber is also at odds with the need for a test suite to be easily maintainable. What I mean is that each test (especially e2e tests) will want to do some setup/initialization. This is usually expressed as step definitions.
Over time, an undisciplined team may write several slightly different but effectively identical step definitions. They may also combine multiple steps into bigger steps because usually a spec writer doesn't want to exhaustively define every piece of setup, they just want to write "Given all 200 pieces of input data and mocks are magically in the right place..."
I was able to wrangle the specs into composability using Rule and Background blocks, but at that point we were just programming tests with a shitty layer over the actual code.
Only now instead of this developing brittle generated tests, it will instead be used by the llm as guidance to generate the actual code and tests.
Before people jump down my throat, I know we are nowhere near that today and I promise I'm not pitching this to my leadership because they would gobble it up too fast.
But for us engineers, I think there is an interesting space for thinking of llms as akin to garbage collection, a feature that allows us to abstract to a slightly higher level of thought. Yes we still need to know how to check under the hood, but this is looking like the right level of precision-flexibility ratio that llms thrive in
The idea behind it was that Behavior-Driven Development might be a great idea, but Gherkin was a pain to work with. LLMs bridge that gap now:
BDD in whatever implemenation is a gigantic waste of everyone's time.
Even the US-born engineers cant always agree on how to communicate product ideas!
Once you write it, you can use the sentence for building other tests if you design it for being flexible and variable, i.e. design well enough your code.
Given I'm connected as ___ with password ___.
When I click on the element __.
Then the element ___ is present.
Then the element ___ is present.
[...]
For critical use case, it was enough. For more that's a bad idea: it take way too much maintenance cost.
It's better to (1) write the test, (2) then later once all that's done, extract documentation in a human-friendly format.
Cucumber was all the rage in 2008, IIRC..
I’ve seen (somewhat) experienced engineers get lost in complex pytest environments that used a lot of mock objects. Over time, they lost clarity on what was being tested, how, and why.
The real value is to have an automated set of simple smoke tests.
Frankly, Cucumber serves a nonnegotiable reminder to me that the automated tests should be simple and straightforward.
For anything extremely complex, I’ll write unit tests instead of trying to craft ridiculous gherkin steps only used for a specific scenario. None of this “when it is midnight of a blood moon in a leap year and the second request attempt has timed out…”
To me it’s just another framework to learn - a failed abstraction. It’s always introduced by idealistic devs who regularly jerks off to conference talks.
Buried in the mess there are a couple of good concepts like it moves away for the 'busy-work' of atomic tests.
But shit like "living documentation" makes me want to vomit.
Are people dying quicker than we can teach this is bullshit? Or are people dying stupid?
The idea non-coders might ever be involved with Cucumber is also fucking retarded.
Last I used it, it was a mess, maybe it's better now. Part of that was those arrogant Selenium motherfuckers.
All this shit is why LLMs are talking over. Fucking retards with their stupid as fuck theory's about auto documentation and Selenium refusing to implement X (i forgot) on some BS 'principal'.
LLMs just get the job done, they might fuck shit up, but they are not arrogant autists about it, and LLMs don't care when they are called useless arrogant autists and double down.