Karate: Test Automation Made Simple
github.com
github.com
I kinda liked the premise, but take “simple” with a truck load of salt.
My first thought when I entered the page ;-)
[1]: https://twitter.com/techgirl1908/status/1136645315090456576
[2]: https://hackernoon.com/the-world-needs-an-alternative-to-sel...
[3]: https://twitter.com/ptrthomas/status/1034794832571510784
[4]: https://twitter.com/ptrthomas/status/1295009968425336832
Pros:
* the resulting code was relatively short
* once we got used to Karate the relatively complex tests were quick to write
* the whole framework is really flexible, you can describe complex json structures using just DSL and you can always call a Java method if you are missing something.
Cons
* some of the commenters here already noticed a lengthy README.md - well, you need to read this whole file before starting any work. The framework is reasonably well documented, but you need to ingest the documentation as a whole before attempting to write a test. You cannot just search for "how can I do ___ in Karate", I mean you can try, but you are unlikely to find an answer this way.
* entry point while short is rather steep. You need about 2-3 days before you begin to understand what's going on in the code
* the code is unreadable for bystanders. Good luck with understanding "#[] #(^SOMEVAR)" or "#[_ > 0]"
* the architecture: the whole thing runs on Maven/Java. Java interprets Karate DSL, which can contain Javascript(!) which can call Java again! Yes, you can make a chain of calls Java->DSL->JS->Java! To be fair, it makes the whole framework really flexible, but still, wtf.
Will I ever use it again? Maybe. I'd say it's specific tool for rather specific tasks. It can serve certain purposes quite well. But if you are wondering what framework you should use in your next project - there are much safer choices out there.
(I've had good experiences with QuickCheck for Haskell, Hypothesis for Python, ScalaCheck for Scala, JSVerify for Javascript)
Their first example for BDD is just painful to look at.
Transforming the "business" language into a "developer" language so that "there is no glue code" is missing the point of BDD.
Imagine putting that example in front of a user and asking: "Is this what you want?". Even if the end user is the developer using your API it makes little sense.
As usual, I mat be missing something here - but it seems it's basically moving the "glue code" to the Gherkin level, making using the Gherkin useless.I feel like this would be the way to go to minimize non-valuable testing while allowing you to discover code thats becoming more leveraged and therefore more critical to test.
2. This feels like the job of a code coverage tool to me (and it's something I would like as well), but most coverage tools I've seen don't quite have the granularity you're looking for. In certain ecosystems you could hack something together pretty quickly (e.g. the `trace` module in python).
Correct me if I'm wrong, but this idea sounds like you'd like to bias your tests away from small units and toward integration tests (and a similar tool could be useful for dead code elimination and a few other tasks). I know they're not a panacea in general, but in your personal experience what problems have you found in fine-grained tests that outweigh their benefits?
I'm a believer in "right-sized" testing, meaning balancing the cost of test creation and updating against the value the tests provide. I believe tests provide the greatest value when they are testing: * public methods * critical business logic * highly reused core logic
in other words, "code for others".
For code that's used for one specific page render or something similarly isolated, I think it's more questionable whether to bother going for 100% code coverage.
The latter category is where I'd like to "trust but verify" with devs permitting them to opt out of unit tests where they believe the code isn't critical, but have some confidence that we would catch code if it became more central and then enforce testing requirements.