Why Recorded/Playback Tests in Selenium Break
blog.automatedbrowsertesting.com
blog.automatedbrowsertesting.com
In my experience, once you stop trying to record clicks, often on elements that will dynamically change outside of your control, and instead use proper coded tests for locating UI elements like Selenium 2+ gives you, it's much easier to have both tests that don't break and tests that are easy to maintain.
But the focus of the academic journal (not the blog) was more of a focus on general record/playback testing tools, not just Selenium IDE.
In my experience, record and playback can work well. Especially visual, screenshot-based solutions like Sikuli or Kantu are actually more stable then most developers think. And the beauty of them is that they can also be used by non-developers, i. e. "manual testers", so the developers can actually work on the main product, not the test suite.
Kinda led me to create https://sourceforge.net/projects/sikuli4net/
Sikuli4net is pretty much a dead project now due to me having absolutely no time to devote to it though. :(
As a counter to your experience I've spent a great amount of time stamping out issues with record-and-playback tests. They're a passable stop-gap if you desperately need testing and, for whatever reason, haven't written any. Sikuli is possibly worse, though it's been a few years since we last evaluated it and quickly decided against using it in any serious capacity.
Using manual testers as discount automation testers is a waste of everyone's time. They're the true experts in actual use of the product. I've seen them find more issues that from a code perspective shouldn't really be possible that it's given me an implicit trust and respect of their abilities. If someone wants automated testing it's another thing they're going to have to spend money on. Doing it half-assed is more than likely going to do more harm than good.
The biggest reason for this is because they don't have the time or knowledge to re-write their scripts to use WebDriver.
So even though Selenium has dropped support for IDE a long time ago, it's still being used by testers.
Skipping anything you feel is obvious/etc, do you think there are any noteworthy gotchas i should know?
Appreciate your input!
Also, I would definitely look up Selenium's implicitWait functionality, as that can fix a lot of "timing" issues within front-end tests. FluentWait is pretty awesome as well (In the past, I have actually implemented my own FluentWait that was independent from the Selenium libraries, because it was so awesome).
If your app supports multiple browsers, make sure you run your tests on all of them when developing -- I can't count how many times I have gotten a test to run reliably with Firefox, only to find out it is broken on Chrome.
Also, for the love of god do NOT use record/playback to generate your tests. The code that it creates, while a time-saver in the short run, is an absolute nightmare for maintenance. (at least it used to be, I haven't used it in the past two years).
Credit: 2+ years writing automated tests, and experience building three separate Selenium test frameworks in two different languages.
1. the biggest bang for buck that we got was the ability to pinpoint which tests to run to exercise which feature. Use as many tools or strategies as you can to make slicing and dicing combinations of tests easy.
2. Think like React and keep everything relevant to a test close to each other with minimal separation into layers. Initially, we split data into a data layer (excel spreadsheets,) UI verification and navigation logic into a package, and a very thin test driver wrapper which pulls all this together. It quickly became very hard to determine what to focus on when tests failed. We have now started migrating to Cucumber where everything is nearby so test failures are easy to debug.
3. Write as little code as possible -- this is generally good advice for all s/w projects
4. Distill UI actions into their lowest level components and expose them as primitives. By this I mean expose things like "open URL," "click X," "Put text there," "Check CSS attr" as primitives. After this, each test is just recipes containing primitives.
* Don't treat your test code as second class citizen. For tests follow the same clean code principles you are following for your app code.
* I saw many dirty workarounds for selenium tests that didn't behave as expected. In major count of it happens due to lack of understanding selenium / test framework API. So read the docs. If something doesn't work - read the docs again ;)
* Don't test everything with selenium E2E tests. Test this way just most crucial scenarios. Leave more detailed stuff to unit tests.
Is making points and adjusting them into counterpoints means a discussion is maintenance of some idea one wants to share, or is it just weird wordplay on your part?
I find this to be the biggest issue. Specifically, the "best" time to write tests is also when the code is least likely to be stable!
It's easy to think "if we had just started writing browser tests when we had fewer features, we could have kept up with a testing suite". But that means your browser tests are constantly breaking and people just give up on them. The app eventually gets big enough, and has enough users, that you have to bolt on the tests later to keep bugs from hurting too many customers.
This seems problem seems to be way less of an issue with regular unit tests. I guess the lack of changing dom selectors would be a big reason why.
So really, the best thing to do would be to make it easier to write and update automated tests.
Recording clicks is of course one way, but the tests are probably more brittle for it.
If you have access to the source templates, you could probably right up some kind of rudimentary compile time type checking and validation of selectors, so you could know when a new feature breaks something.
The best would probably be some seamless mixture of the two - record tests in terms of the templates, with the ability to detect breaking Ng template changes...
We've settled on aiming to build a tool that keeps an index of selectors used <=> page/page structure. Then if something changes, you only have to update it once. So when you create a selector, it'll save it for that page, you can name it and reuse it.
Given that we have the source templates too, I think we'll try to validate that the selector will be valid. We might be able to even add tasks for the front end guys that alerts them if the changes they're checking in will break selectors.
Combine that will an idea for suggesting selectors based off the source, along with e2e script managent, it'd be a pretty slick tool.
It's a pretty ambitious project though. We do hope to make it general enough to open source though. We'll see if we make it (pun intended).
The goal was to make things a bit less fragile by allowing Devs to annotate markup, while avoiding retraining the screen-clicking QA or creating an entirely new locator scheme. This was back in 2010, but I think it generated expressions a bit like:
xpath=(((//.[@data-test-label='foo'])[1]//.[@data-test-label='bar'])[1]//.[@data-test-label='baz'])[1]/a
Messy looking, but it meant only a structural change to your foo/bar/baz structure would break the locator, or a particular change within the baz markup. Switching a div for a span or adding another nested div for some reason? No problem.
Unfortunately we never got to use it in production, the company was bought a couple years later and the new management wanted to skimp/dumb-down QA even further. I guess they wanted to buy results rather than processes.
I might be biased, but QA in Production [1] is my preferred approach now that we have great rollout systems [2][3] and ways to do verification testing in production [4].
[1] https://martinfowler.com/articles/qa-in-production.html
[2] https://split.io
The iron rule of software is that changes in the product have a reason behind them. If your UI tests break because of a structural change, that break should be anticipated while the corresponding story/feature is being developed.
In the past, when we faced test failures which seemed annoying and trivial, we discovered that the real reason why we never anticipated those failures was because we never know which tests to run for the feature or page in question. We are hoping to fix that by using robust, semantic tags for tests. So, where we used to have tags like @TC_123456 we now use tags like @Login_UI etc.
https://github.com/saiscode/8tomation
Just a regular .feature file (with minor restrictions) can seamlessly interact with any webpage without needing brand new glue code.
The framework also supports parallel verifications. E.g., if you want to check multiple items on the target page in parallel, you can do that too.
//I'm using this for my clients right now
[1] https://www.npmjs.com/package/webdriver-manager [2] https://www.npmjs.com/package/selenium-standalone
We use a combination of unit-tests for the things that are straight-forward to unit-test (UIs aren't). But for lots of things that unit-tests can't really cover so well, or for when we have to mock the entire universe anyway, we rely on UI tests.
I'm not affiliated in any way, but I can honestly say that Ghost Inspector[0] made things WAY better for us. Why:
* No more wrestling with capybara. Not having to constantly switch between the browser, test runner, IDE. Most tests don't take longer than 5 minutes to create from scratch.
* We run UI tests on our staging environment, and we deploy all feature branches to staging before they get merged to master. UI tests are automatically run as soon as we deploy staging. And they run in parallel. So they're quite fast.
* We plug the Ghost Inspector results into slack+email, so as soon as something breaks, we know immediately.
Fixing the broken test is usually very straight-forward, and our developers do it (we don't have QA). Even if we need to re-record a test, it's usually much faster than re-writing a capybara/integration test.
This first example is weird. That change has a pretty big impact on the way the form works; browsers serialize forms using the name attribute so that change means the form data that's sent when the form is submitted is going to be different to what it was before. You'd get FullName=<value> in the form encoded data from the first input and Name=<value> in the second. I would expect a change like that to break a test.
If you don't care what the name is then don't use it in your UI tests. For example, use an ID instead. Using <input name=“FullName” id="nameinput"/> you could locate the element using by.id('nameinput'), and then you could change the name <input name=“Name” id="nameinput"/> and your tests would carry on working.
Though personally I wouldn't want to have to start adding ids and names to things in the markup just for Selenium to find them.
The dynamically generated at runtime tests solves the weak locator problem, and it's smart enough to avoid inputing invalid values in input fields. We deal with page loading well, and we're working on being able to automatically deal with alerts (and other modals).
The result is browser tests that are way faster to write (because you don't have to worry about locators), and the tests are way more resilient. We're currently pre-release, but we're looking for companies and devs who'd like to be in our upcoming alpha.
Not to mention dynamically generated tests at runtime. I'm not really sure how that would solve the problem of a weak locator. If you're given explicit inputs and they're weak enough to break then building a reliable test from that seems a tremendous challenge.
I love that you guys are having a go at this problem. I do this kind of regularly. The claims of being faster to write due to not having to figure out locators doesn't seem to hold water though.
We can also handle other common actions like "Login", "Search for ipsum", "Select the first search result" (or "Select the first product/hotel/flight/etc" - that one's really impressive), "Add to cart", "Checkout". When writing the tests, we give you the list of imperative actions than we think your declarative commands map to, and you can correct us if (when) we get it wrong.
We also have some value-adds. Apart from the usual assertions (checking the page loads, looking for certain text or elements, et cetera), you can write assertions checking for the presence and correctness of 200+ advertising and analytics tags, and can included computed assertions (assertions that are computed from the context of the test). We also have natural language date handling, so you can write evergreen tests with commands like "Search for a flight next Monday" and we'll resolve that to the actual date to input into the datepicker. We're also working on malware detection (basically checking all loaded resources and outbounds like on every page we load for suspicious markers).
[1]: https://heliumhq.com
implement a proper pattern, and suddenly im locating commands instead of selecting dom elements which may contain my command.
Just smoke test that all the inputs do something and unit test anything else you care about.
UI testing is an antipattern
https://github.com/saiscode/8tomation
With this, your basic cucumber .feature file (with minor restrictions) can be executed without needing page objects. Take a look if you have a second...