The Tragedy of Given-When-Then
theitriskmanager.com
theitriskmanager.com
> The reduction of the tester to an expert translator/typist is a tragedy
> The Given-When-Then detracts from understanding and readability but provides the much prized automation through tools like Cucumber
> Instead of abstract domain models in Rational Rose
Now I see how we got into this situation. All this misery comes from the people who understand the code and the people who understand the business being different, non-intersecting groups of people.
Since the BAs can't read the code, there is a desire to construct a technical representation that they can read, representing a some kind of formal specification, and then derive as much as possible the program from that. This was the promise of Rational Rose and the whole UML project. Had it been successful, it would have reduced programmers to stenographers, "coders" in a very limited sense. As it is, in a "Rational" system of this kind, coders end up producing increasingly elaborate polyfills between the machine-generated parts of the system and the rest.
The target of this is always the "5GL" dream:
- specify program in English
- ???
- automated tools transform to code
GWT does this de-skilling to testers instead; it's effectively TDD with tests in this quasi-human-readable format that gets unimaginatively translated into programs by the testers.
The proposed solution is to do more communication through spreadsheet-prototypes. Since these are effectively programs in Excel, with all the real programming capabilities thereof, you get a real working model of the system that actually behaves like a program.
Many businesses simplify this process by then just deploying the spreadsheet to production.
(I'm not familiar with Cucumber, but it looks like a thing for constructing toy "human readable" DSLs for tests? I wonder if we could get BAs to learn INFORM?)
The old school business guys had a plan (or at least a wish) to get rid of programmers. Ironically, I now see the best programs being written by programmers who have no BA's (essentially becoming BA's themselves). It turns out that it was way easier to just get the BA's to talk the language of the computer, rather than the other way around.
From what I've read, the same is true in testing. The best BA's are people who are actively interested in testing, and willing to learn the language they need to best help out. You cut out the middleman, not by making translators, but by learning to speak the language natively.
As it often is, it's easier to change the people than to change the world.
I know other functional languages like Idris take this approach too where the designers could declare the input/output types and as long as the implementation conforms, you have some higher confidence in program correctness (at least API contractually). Interesting stuff.
[1] https://pragprog.com/book/swdddf/domain-modeling-made-functi...
Businesses solve this problem by hiring and training programmers in their business model and domain.
Coders, are worthless. Coders with business acumen and domain knowledge are invaluable.
This is also why I will never work with anyone with a job title of BA. They are trying to do my job but can only do one half of it. They're a pointless bottlekneck in the system.
And no, there is no place for BAs who can code only those who do. They're called software developers.
At this point I don't know if software engineer, developer, programmer or coder is really the right term any more.
I'm more of a systems analyst. The amount of time I spend coding is miniscule and I don't think I'm alone in this.
I've found BA's are very useful in the following areas specifically
- Communication with the market (either customers or in-market staff such as sales/support) to gather feedback about what they'd want
- Orchestrating decisions (asking all the dozen stakeholders what they want to do and arriving at a sensible conclusing)
- Arbitrating feature/UI decisions (people on the team aren't agreeing on whether we should do something or not)
- Documenting requirements (by this I mean the desired functionality that we believe customers need/want)
- Iterating and getting feedback on requirements
- there's more, but off the top of my head that's all I can think of
Sure, I as a lead developer have business knowledge and can also do the job of business analysis, but a dedicated BA will be able to spend more time doing that - increasing their skills, and freeing up me to do other things. If I take on all of the above tasks in a decent-sized project, it's going to suck up a huge amount of my time - in many cases almost all of it - so then basically I'm just a BA with a different job title. BA's are great if you learn to understand the role and how it can help a team
Similarly, BAs without business domain knowledge are even more worthless because of this.
I.e. consultant BAs are, as most of us know, a waste of money.
BAs are struggling to find a language to describe their business requirements in a way the programmer or computer can understand.
It's code. It's what programmers have been writing all along. That is the god damn language they need.
Given when then? If then else.
Surprise!
If they are cheaper, I guarantee you're going to have problems.
You're literally installing a weak link in the most crucial part of the chain.
And given basic supply and demand they are cheaper.
Anyway read your Ricardo, you're right but wrong about the implications. Your development is an even weaker link in the chain because good developers are even rarer than good BAs.
Plenty of developers are not BAs. BAs are systems analysts with expertise in requirements elicitation (basically, goal-directed interviewing) and technically writing. Plenty of developers are neither systems analysts nor skilled at interviewing or technical writing.
This is a pipe dream in all but a minuscule slice of software projects.
GWT/cucumber is as decent a tool as any for creating automated tests of system actions (and especially interactions) in a mostly-human-readable format that is likely to be understood by BAs, testers, and devs, even if not all of them are expected to be able to write them.
I've used them across many projects with great success, with the understanding that it's not intended to replace unit tests for calculations, nor stories for initial specification.
As endian says downthread[0]: ", G/W/T isn't the solution to domain understanding, conversations are. You should be continually communicating to between all stake holders to maintain a current domain understanding."
The article definitely does not make that assumption. The expectation is that the business analyst (someone with business analysis skills) will specify a small slice of the solution that will deliver value. Based on feedback it is expected that the specification will evolve.
I think you are spot on about readability. A specification in human readable form can easily be verified by a wider range of stakeholders.
Chris
I follow a slightly different technique that I think makes given/when/then also a useful tool for describing data and calculations.
1) Write the test as follows, using deliberately simplified but still realistic data (this is crucial):
>Given <a set of market data and trades in a table> >When <arbitrary event such as calculation is performed> >Then <leave blank>
2) Write the code that outputs the data/calculation.
3) Run the test in a "rewrite" mode that fills in the results of Then based upon actual output (this process is somewhat similar to golden master).
4) You now have a passing test and generated data which you can eyeball to see if it is in line with what you would expect. This test with recorded output can then be shown to the PO (or whomever) and committed to source control and used for regression testing.
This obviously isn't possible with cucumber or other gherkiny tools and it does require your processes to be fully deterministic (a laudable goal anyway), but it works pretty well IMHO.
The benefit of calculating it yourself first is obvious, you get to cross check your result. That tends to highlight minor errors such as -/+ errors.
This is not a problem if stability is more important than correctness of course, but that's not where I am.
Act 2 - is really about process rather than G/W/T (which is really just AAA, arrange, act & assert).
Act 3 - again process, G/W/T isn't the solution to domain understanding, conversations are. You should be continually communicating to between all stake holders to maintain a current domain understanding.
We wrote an article recently on the limits of BDD [1], G/W/T didn't really come up, there are other more glaring issues with BDD when it comes to systems that intersect mismatched understandings of the real world between experts and users. Unrealistic wants and goals are killer. Additionally we started writing a tool to attach metadata to scenarios (G/W/T) so you can capture technical details about things called SpecStack [2]
[1] https://endian.io/articles/limits-of-bdd/ [2] https://github.com/endiangroup/specstack
Interesting article. I would be interested to see if feature injection ("Working Backwards") would help you. I do not think the first of the problems is a BDD problem, its more a "reality" issue. The only way to solve it is to use financial derivatives and I suspect the lack of liquidity in the crypto would make that prohibitively expensive.
Judging from your web-site, you are only 5 mins up the road from me. Would you be interested in a lunchtime session to see if FI would help?
Chris
Having structured format for tests/requirements, in English (et. al.), can be incredibly helpful. I would love to see some innovation around helping programmers and non-programmers reach a shared understanding of what the system should do and the cases we will use to verify it.
I don’t think unstructured conversation, unstructured English, or Excel tables are the solution. This is still an unsolved problem in our industry.
How about structured excel sheets. Excel sheets with quide rails for business people to structure the expression of their thoughts?
That's what I'm getting at.
Chris
Been doing a lot of API testing recently with karate dsl and writing cucumber tests that include json expressions with some syntactical sugar for validation. The tests serve as a specification for the system which is actually quite a bit more precise than even swagger since you can even go back in time and compare the deltas on request/response between test executions to troubleshoot regressions.
Agree that GWT can't help business understand the inherently complexity of a state machine, but individual tests can be used effectively to model state transitions, especially at the api level.
Feature files to me are most effective when they act as a roadmap of the steps one needs to take to effectively test a given set of use cases, NOT a 1-1 carbon copy of the requirements. It should tell a non-programmer what the test is doing without having to dive into the code, while being a scaffold to which a programmer can build their test logic into. If one needs to look at the requirements, one should do just that, read the requirements document, or in the developer's case, read the requirements set forth in the user story. Scenarios are guides to how to navigate what the test is doing, not what the application itself should be doing.
Also, I often see programmers attempt to write a BDD test and run into a case that doesn't quite fit flush into G/W/T, then ask the community of how they might go about writing that test. Instead of understanding flexibility, they are met with an abrupt "that's not BDD, you are doing it wrong, if you did it the BDD way everything would work out". That's discouraging, frustrating, and destructive. G/W/T is not gospel, and it doesn't fit all test cases. I see nothing wrong with fudging some tests to not follow Gherkin to-the-letter if it better facilitates a test while still remaining clear what the test is doing in plain English within the feature file's scenario.
I expect to see is continue to organically grow in popularity as people try it and adopt it. It's both obvious and insightful at the same time.
I'm not sure i like the idea of going through what's essentially a batch job to export my events inside one bounded context. I'm also not sure i like going through an external message bus to operate between aggregates.
I don't have any better ideas, but i wonder if it's necessary to discard my reservations to succeed in the strategic goal.
You can find more info https://gauge.org https://taiko.gauge.org
Could not agree more. That's why Andy Pols and I recast the BA role as a Business Coach back in the day.
Chris
Given all the examples given, it sounds more like Cucumber is the problem. Having specifications/tests written in the form of given-when-then isn't shown to have any issues. Rather, taking those specs/requirements and disconnecting them from the people who need them is the issue.
You are supposed to capture requirements in Given/When/Then form before coding, and they should be written in a way that is independent of the implementation
The complaint seems to be that what happens in practice is "the devs have taken over" the system, so Cucumber just becomes a test automation software
This has led to writing tests in Given/When/Then format which test overly specific elements of the implementation, and are often written after the fact, instead of as a way to capture requirements.
And some other stuff, but that's primarily what I got out of it.
That's what happened in the projects where I used fitnesse, jbehave, cucumber, etc. The users couldn't understand the artifacts and what they're supposed to do, and the tools were highly idiosyncratic. It ended in becoming just yet another hassle for developers.
Whereas using Excel for specification of calculations (by Business Users or Business Analysts) actually brings the business and developers closer together, removing the need for intermediaries. Business Users are very happy using Excel and checking calculations etc. as it tends to be their home environment.
And the reason seems to be that it's requiring us to express ideas in a vastly simplified language. In order to do that, we have to expand a complex idea to many simpler ideas.
That's making us do the job of the compiler! Real high-level languages are rich with many idioms and structures and the compiler then reduces that to simpler ISA.
I agree with this. Writing Cucumber/Gherkin scenarios is extra effort if the Cucumber files themselves aren't used/read elsewhere. -- It'd be simpler to embed "Given/When/Then" statements within test code (like RSpec).
I'd emphasise that Gojko Adzic's "Specification By Example" suggests discussing examples before refining to a specification; that may get around the author's complaint that non-table formats don't allow for important cases.
That said, "Given/When/Then" is hardly magical, so doesn't deserve much praise/criticism itself. Any test involves "do an action, check the result" (with "setup the system" and "cleanup" being implied). Sometimes called "Assemble, Act, Assert". "G/W/T" is just a neat, consistent format for describing behaviour in English. A table of values specifies some computation; the column titles help to describe that behaviour.
- [0] https://github.com/jnicklas/turnip
The reason RSpec didn't include this originally probably stems from the library being designed for testing at the class or module level, rather than an entire application.
[0]: http://spockframework.org/
[1]: http://spockframework.org/spock/docs/1.3/spock_primer.html#_...
[2]: http://spockframework.org/spock/docs/1.3/data_driven_testing...
[3]: https://twitter.com/dminkovsky/status/1116727735399976966
Too bad you can't use whatever specification language feels right. Spock only provides Apache Groovy, with its tacky syntax hacks like long strings for function names, block labels having special meanings based on their name, or the OR and LOR operators being used for drawing tables in the code. In the past when software has provided Groovy for writing specs, they eventually provide an alternative when Groovy's shortfalls become obvious, e.g. Kotlin for Gradle [1], or the Declarative Pipeline Syntax [2] for Jenkins.
[1]: https://docs.gradle.org/5.0/userguide/kotlin_dsl.html
[2]: https://jenkins.io/blog/2016/12/19/declarative-pipeline-beta
Actually, Kotlin does that too. I think it is even encouraged for writing tests.
I'm skeptical of this claim. It would help if there were a year, considering it's not clear if the author means the modern Internet, the ARPANET, or when the modern Internet became widely available to a larger group of people. Based on the mentions of Excel, I'm inclined to believe it's the last option I listed.
The MIT AI lab and other research areas come to mind as places that cared about how the programs were operated and whatnot and these weren't exclusively used by employees. SHRDLU comes to mind. While I'm thinking about it, the Apple Macintosh also does.
I don't believe I was familiar with this Given-When-Then model beforehand, but I also think that's because it's somewhat natural, or at least seems natural. The author has failed to convince me why this is a bad thing. I suppose I can see why these larger business practices are poor, but that has me failing to see why this particular practice is singled out.
Needless to say, it wasted a ton of time and never produced any meaningful value.
* Good luck telling your PM their English has a syntax error
* Once that's out the window and the cucumber files are just maintained by devs, is the regex translation layer worth it?
* A lot of implicit state also gets wrapped up in the test in making these phrases at least semi-readable, which makes for a pain debugging failures.
When you could have just written a simple integration test in 15 minutes, but the Scrum master thought Cucumber would be a good idea.
I like when authors express a strong viewpoint but then also include descriptions of circumstances where their viewpoint may not be applicable. This seems to be alluded in the above quote, but are there specific contexts where Given-When-Then _is_ helpful and the appropriate mechanism to document requirements?
It's an executable specification, but it's also in a format which is readable. If not "documentation", then it's at least "small, verified examples".
I'd find a table of values less readable. I don't mind if a document like this is the output of running the tests themselves, though. (Obviously, the important details are "input program, expected output", which might not lead you to use Cucumber, but I think it's a fine use of Cucumber).
This doesn't feel familiar to me but I've tended to always work in legacy systems where the opposite problem (no BA at all) is a more familiar problem.
I find this such an unlikely sounding claim that I want to reject it without consideration, or at least assume that he's only talking to a very restricted subset of engineering non-programmers.
APL is decades old and very much a business language by origin in IBM System 360, there ought to be decades of people's experience with this on both sides to back it up or refute it - programmer and non-programmer. Is there?