Selenium IDE alternatives for UI regression testing | Hacker News Reader

Selenium IDE alternatives for UI regression testing

screenster.io

105 pointsOst_Goth68 comments

I'd recommend coding the tests instead of using the IDE, much less frustrating.

Protractor (http://www.protractortest.org) is a good wrapper for Selenium: despite the name, you can totally use it with non-Angular pages (I'm a Preact person myself). It's convenient because it handles installing and updating webdrivers (which was especially handy on Windows), and seems well-maintained. And if you're using only Chrome and Firefox, it even has a direct connection mode to skip having to run a Selenium server (example: https://github.com/wildpeaks/threejs-examples-screenshots).

Otherwise if you just want to run tests in a real browser environment but don't want to test in multiple browsers, you could run them in NW.js, Electron, SlimerJS, or PhantomJS.

Do you know of a list of wrappers like Protractor?

If your app is powered by Django, it has LiveServerTestCase utility that runs your site in selenium and lets you click around, and because test is written in backend, you can also test resulting app state or stage test cases for your integrations.

NightwatchJS is a Node wrapper for Selenium and seems to be pretty human friendly in terms of syntax. I found it to be pretty fast in my use cases with it.

My team has been pretty happy with Geb, which has the benefit of running on the JVM.

Nightwatch-Cucumber https://github.com/mucsi96/nightwatch-cucumber is a killer combo for me.

Lately, I was playing with it. Here is a starter project: https://github.com/jumasheff/nightwatch-cucumber-e2e-testing...

The cool bit of an IDE is you can potentially bring non-software-devs into the world of testing. That's not to say it's perfect or simple, but it's an interesting way to bridge a gap.

The same potential holds for cucumber test cases.

Ehhh I don't agree that that's the case. Any time I've dealt with BDD I have to add more / rewrite the backing bits constantly.

For this reason, much prefer test-libraries that read more like BDD cases without actually trying to invent a new DSL for me to maintain on top.

This is hypothetically true but I think if you're expecting your manual testers/business to play mad libs with your defined statements they have to be so resilient that they become fairly inefficient.

If that's an acceptable tradeoff for having anyone be able to create new test cases though, it would probably be okay.

I guess we're talking about two different approaches to solving the same problem (code-based tests can't be used by non-technical people). So we can:

1. Substitute writing code with a visual editor. This is, essentially, what Screenster does.

2. Make scripting abstract enough to a point where it starts to resemble plain English. That's, in a nutshell, the approach of Cucumber.

Each of the two has its pros and cons. In my opinion, the key strength of tools like Screenster in this context is how they address visual/css regression testing (specifically, things like text or image formatting issues, layout issues and other things that typically get broken by CSS updates): http://screenster.io/a-better-way-to-do-visual-regression-te...

> I'd recommend coding the tests instead of using the IDE, much less frustrating.

This is what I do for web, mobile web and mobile app (appium). It takes some time to set up the proper layout[1], but once it's done it's very easy to move the frameworks around for the various platforms. Scripting out the tests also allows me to do more with our API for more than just UI testing, but also functional.

1 - http://www.guru99.com/page-object-model-pom-page-factory-in-...

>I'd recommend coding the tests instead of using the IDE, much less frustrating.

The advantage of visual/screenshot-driven automation tools like Screenster.io or Kantu.io is twofold:

1. It allows non-developers to join the testing team and create automated web tests.

2. Even if you are a developer by training, tools like NW.js or PhantomJS still have a significant learning curve. I argue that developer time is typically better spend on the product than on coding complex test cases.

As the author of that article about Selenium IDE alternatives, exactly!

I mean, if you want to do end-to-end UI testing, you'd probably want manual testers to cover a huge part of the routine. And tools like Screenster help you automate the process and use it for UI regression testing.

Much less frustrating — maybe, but not always economically feasible.

In case with some (actually, a lot of) projects, people who can code will have to spend 100% of their time coding features. Tools Selenium IDE strive to offer a 'good enough' solution for cases when you're trying to automate the work of manual testers or have less technical members of your team involved into UI testing. And tools like Screenster turn 'good enough' into a working solution.

Visual regression testing is, imho, another case for tools like Screenster (http://bit.ly/2kXEEFV), even though I know that, technically, you have some of this functionality with Protractor (http://bit.ly/2kune0t).

On the more technical side: does anyone know of a real alternative to Selenium? ...cause everything seems to be based on it even if you're not using it directly.

I like my web app acceptance tests to be code, readable and maintainable, not UI/OCR thingies... but I've seen so many bugs like "this element is visible and clickable, but Selenium says it's not, because the 3d-css animation + dom-manipulation voodoo of the tested app got selenium completely confused"... and I'm not even mentioning web code with events-on-SVG-nodes or webgl here...

Any non-standard and animation-heavy web-app/page will utterly confuse Selenium to the point that you end up testing by injecting js into the browser (so now you can have bugs in tests only, or tests that pass despite bugs) mingled with test-runner code and heuristic timeouts everywhere...

(I know the "real solution" is "use a frontend framework and write front-end-unit-tests instead", but usually you're writing the acceptance tests for something that already exists).

You may like something like Codeception: you code your tests and it drives Selenium. Code is in php but I'm sure you can find the same kind of things in your prefered code language.

For a more visually oriented IDE Sikuli seems to be a good start but last time I checked there was no easy way to automate its use for CI.

> (I know the "real solution" is "use a frontend framework and write front-end-unit-tests instead", but usually you're writing the acceptance tests for something that already exists).

Eh, GUI tests definitely have their place, although they can be far less specific than your unit tests. They're pretty great for quick and regular confidence checks.

Anyway, TestCafe says it doesn't use Selenium, but I haven't tried that yet: https://testcafe.devexpress.com/

I was about to suggest TestCafe even though I haven't tried either. After reading a bit, it seems to work with a small proxy that injects a script to trigger events on the page via the normal DOM api's. If might be a good solution if you're targeting modern browsers.

How about casperjs?

casperjs is similar to Selenium but it only works with phantomjs. That's useful in the sense that you can do a quick test to see if your code works in a browser, but it's a different prospect to Selenium. Selenium (with WebDriver) works with all the major browsers. If you're automating testing a website/app then you need to be testing it in lots of browsers.

If you prefer coded tests, there is no good alternative to the de-facto standard of using Selenium and/or Webdriver. But as you said, this means that creating a test setup is a development project by itself. So for our team, screenshot-based web automation tools like Kantu (https://kantu.io - or Sikuli https://sikulix.com for the desktop) are the best alternative to Selenium, especially for non-standard and animation-heavy web-app/pages like the one you need to automate.

I find that screenshot-based test cases are very easy to read and maintain - even and especially for the non-coders on your team. New button? Just take a new screenshot. Everyone that can use Snagit understands that ;)

I do wonder about the accessibility of pages like this. If it confuses Selenium, can screenreaders cope?

Ugly and simple truth: no. In real life, in a customer centric approach, you can choose to ignore even 50% of your users anyway, from a marketing pov, if you decide those are not the users you care about (hint: you care about those willing to pay way way more - like when you make an overpriced iOS app but completely ignore Android to give another example). Why care about... less than 1% of users, when you already chose to ignore an even larger percent in your other approaches? Fair? No. Makes sense? Yes.

Expose all data trough a clean and documented API. If screenreader accessibility matters, someone will eventually pay you to develop an alternate interface using same API, or an open-source will be developed at some point...

Solution to accessibility problems is usually: (1) expose your data though an API, (2) wait to be paid to develop a truly accessible alternate "light" UI (if it ever happens), but don't cripple the "main ui" (well, actually more like "have it like the customer wants it" ;) heavy animations and fx are "crippling" in my personal view too...).

I'm just saying that ignoring users and ignoring testers has considerable overlap in this regard.

(And apart from general and professional ethics, local laws might disagree with the assumption that you're free to ignore users for that reason.)

I did an experiment with the Chrome Debugging Protocol the other day. It allowed me to write and run acceptance tests in pure JavaScript with no Selenium, Java or binaries (except for Chrome of course). See working proof-of-concept here: https://www.npmjs.com/package/websteps

>I like my web app acceptance tests to be code, readable and maintainable, not UI/OCR thingies... but I've seen so many bugs like "this element is visible and clickable, but Selenium says it's not, because the 3d-css animation + dom-manipulation voodoo of the tested app got selenium completely confused"... and I'm not even mentioning web code with events-on-SVG-nodes or webgl here...

I just write javascript directly for automating situations like this.

I see this as somewhat analogous to the abstractions provided by an ORM. It won't provide every edge case but that doesn't mean you should toss it out entirely - just bypass the abstraction in the (minority) of cases where you need to.

If you need to bypass the abstraction too much of the time that indicates that something is a bit wrong with your front end code, just as having a database that you could almost never use an ORM would indicate something fundamentally wrong with your data model.

Since this thread hits my expertise... We have developed Usetrace (https://usetrace.com) for a few years targeting teams that want non-techies to automate GUI testing. (Disclaimer, I am a founder). So far the feedback has been positive and we're getting healthy revenue. It seems that teams are desperate for faster solutions than coded UI tests. But they will only use a tool that keeps up with frequent UI changes (automatically or semiautomatically) and is really painless and quick to use.

Some lessons learned that we're solving with the product:

  - creating basic tests must be doable in 1 minute
  - testing is a team effort (testers&devs) - must have collaboration tools built-in
  - must be extendable with code (js)
  - sync/timing issues must be handled automatically (when possible)
  - UIs will change - generate locators and algorithms that adds robustness
  - debugging must be speedy: must be able to debug test steps live with the browser context (html5 remote connection to the browser)
  - randomness is needed - need an easy way to introduce random data (like random emails)
  - testing emails must be as easy as testing a web page - inbox must be an integrated feature
  - must handle frames transparently - user does not care if the element is inside a frame or not
  - must be pluggable to CI
  - must have good, clear reports
  - some tests will be flaky - must be a way to keep builds stable & green
  - no installations / plugins, just a browser
  - fast to update tests when UI changes - the system must do global replacements when it sees duplicate stuff
  - duplicate test step/code will cause a major maintenance burden - must have reusable/parameterizable components
  - people will want to reuse functional tests for stress testing and monitoring - allow that too
  - people need to check for dynamic content - must have a quick and codeless way to use variables
  - people want to have quick builds - tool must offer parallel testing
  - people will want to integrate to their own systems - must offer APIs
  - html pages are responsive - must have support for different screen sizes

While your premise is indeed sound - and let me say I'm not taking away any credit for what you and your team have built - but we did evaluate Usetrace last year and found several basic issues. The system simply wouldn't record several events and user actions and it fell so far behind in a test case that we concluded it was probably not the right tool for us at the time.

But I'll be happy to re-evaluate the tool.

Thanks for the feedback. We have come quite far since last year. For example, now you can record html5 drag and drop events. Give it a try, we'd love to hear if there's a webpage with issues, so we can fix Usetrace.

Thanks - will certainly have another go at it.

We just signed up for Ghost Inspector yesterday. I hope it gives us half of this list... these are a fantastic set of principles.

I'd add that it's nice to have an "eject button" path to run the tests on local infrastructure. Also, it should be possible to run tests against localhost.

Thanks, good points. We have used ngrok for testing localhost. We actually wrote a helpful article on it. This is useful info regardless of the testing tool, so pasting it here: http://docs.usetrace.com/articles/connection/

Wow, the pricing is a little inaccessible for most teams...?

Evan at the "starter - 20 builds / wk" tier, I'm not sure that would cover any team I've ever worked with for an entire day...

The product might be good, blog post might be right. But for a vendor locked alternative to say an opensource alternative as "doing more harm than good" makes me rage.

Yup, this is obviously marketing. They didnt even go as far as posting it somewhere other than their blog. Funny how they promote 2 competitors in the post and make them look more favorable. This company needs marketing help ( if you are reading this email me and Ill help you improve).

You forget to list competitors who will help them improve ;)

Anyway, I appreciate the transparency of "yes, this is our product" as well as inclusion of competitors. Doesn't make it seem as though they're the only other game in town.

Most importantly, their analysis tells me they're watching their competition. Too many companies say "oh they stink" wrt their competitors and take a "we know best" approach.

Being open to watching competitors means knowing when they do something smart and adopting the good parts. I appreciate that in marketing

I'll think about it, thanks

You are welcome. Best of luck with the project.

Sorry for that:)

I didn't really intend to pick on Selenium. Actually, that's the exact wording from the title of the article I used as my reference, so I though it would be fair to leave it the way it is.

But I agree, Selenium is one of the many cases illustrating the huge impact of open-source projects on the technology, and we must give it credit for that.

I think "doing more harm than good" came from a different post... I don't see anything wrong with comparing an open source and commercial product. We'd still be using IRC if Slack didn't come around.

Time is not free so in the end you have to compare the total cost and productivity and choose the best tool for yourself.

Everybody interested in web automation should also check out Helium [1]. It's a wrapper around Selenium, thus inherits its stability, speed and browser support. But instead of requiring you to write driver.find_element_by_id("btn356").click() you can just say click("Download"). This alleviates the need for record and playback tools, because scripts are so easy to write. It also avoids the problem that scripts created via R&P are virtually unmaintainable. Helium is commercial and I'm one of the authors.

[1]: https://heliumhq.com

Thanks for the link. Personally I find Canopy scripts to be readable, but good luck with pushing for more automated web testing.

http://lefthandedgoat.github.io/canopy/

Looks very cool. I'll have to try this out

I went with CodeceptJS [1] instead (Seleniu in nodejs, with most of the async madness abstracted away) for the last project, but, as thangalin says it best "It is a lot of effort to automate user interface testing with Selenium, especially for web applications with complex user interfaces, concealed options, and so forth.".

Advantage is that testrunner code mingles seamlessly with injected-into-browser-js. Problem is... complexity, and twilight-zone-class bugs in the testing tools themselves: there comes a point where the logic of the acceptance tests is so convoluted, and they are so undebuggable and unmaintainable, that fixing and extending them eats the time of the best and most expensive developers, instead of being a great tasks for juniors...

Eager to try helium next, hope it sucks less :)

[1]: http://codecept.io/

Edit+: don't get me wrong, CodeceptJS is amazing compared to the madness of seleniu-webdriver/webdriver.io used "raw", with promises and callbacks, so kudos to its devs. But its far from optimal... though this maybe because selenium itself is far from optimal.

A lot of companies still use Selenium IDE to create and run tests, even though it's basically been deprecated for a couple of years now in favour of WebDriver.

The reason a lot of people still use it, is because it's very easy to create a test; you basically record the things you want to test and Selenium IDE generates the test for you, without requiring any programming knowledge.

( Source: I work for a company where people upload their Selenium IDE tests to run them on a large grid of browsers in the cloud, https://testingbot.com )

A Selenium UI recording tool is a good starting point to help people learn. Quickly though it becomes pretty limiting and specialized testers need to switch over to using code.

The Selenium IDE I like using at this point is Oxygen. It's Atom with a Selenium recorder built in. It records and gives you code with details about the XPath and CSS Selectors provided in comments above each recorded action. It's the best tool I've found for quickly getting off of record-playback dependency and into working with code. The Oxygen IDE provides intellisense and real debugging. It strikes the right balance of allowing you to record when you need to, but then getting out of the way and providing you with a real IDE so you can work with the code.

http://oxygenhq.org

I did a blog post not too long ago about some of it's quirks on Windows... https://scriptmonkey.wordpress.com/2016/06/03/oxygen-for-sel...

> Selenium IDE makes you handle timeouts manually which is a major inconvenience.

you can set the implicit wait setting. this will work in most cases but will make the tests slow when you need to test negative conditions. the behavior around implicit wait varies by driver implementation which can be annoying. in general it's better to use explicit waits.

> Selenium IDE only works with Firefox and may not work for sites or web apps that have slightly different DOM structure in Chrome, Safari, and IE.

selenium ide has webdriver playback[1]. if you don't like that you can generate webdriver scripts in c#, java, python, or ruby and run them against selenium server.

[1] http://blog.reallysimplethoughts.com/2013/02/18/webdriver-pl...

Screenster support for timeouts is much more extensive. It automatically determines the time it takes for the server to process each user action, it uses the visual baseline to wait for the right state instead of timing out or waiting too long.

Obviously you can do everything with WebDriver, Screenster uses it too. The point is that you can get a lot of benefits out of the box when you are using a highler level platform like Screenster vs hand coding to a low level API like WebDriver.

There's some description of Screenster features here http://screenster.io/selenium-alternatives-for-testing-autom...

Where I find such IDEs lacking is their inability to control playback timing and animate mouse movement. It is a lot of effort to automate user interface testing with Selenium, especially for web applications with complex user interfaces, concealed options, and so forth. It should be trivial to generate an instructional video showing how to use the application by re-using the efforts involved with writing the web tests.

A while back, I combined Java's Robot class with Selenium and a simple scripting language to create Scripted Selenium.

https://bitbucket.org/djarvis/scripted-selenium/

https://www.youtube.com/watch?v=tGvcm0Rlm-0&

I really don't like pricing levels that aren't just per user, but per test. That encourages thriftiness at a level where you shouldn't be thrifty.

Never mind that I'd be seriously worried that a lot of these small time players are out of business the next year and I'll end up with useless tests (of course this is better if the software itself is just helping recording tests and it all ends up as Selenium scripts in some human-readable language).

Has anyone used [Nightmare](https://github.com/segmentio/nightmare)? It's built on top of Electron -- the Atom code editor is built on Electron.

It's great, but I can't stand the chaining syntax. I'm a bit biased, but I think that some suite with async/await will be the future. Here's something that I've been working on: https://github.com/KevinGrandon/ghostjs/

If the authors are listening, I think the tool you have in mind is called Sahi (not Sachi). Also, you lump it with proprietary solutions unfairly when you say:

> Finally, Sachi, RFT, and UFT are expensive enterprise solutions that require local installation, certain servers, runtime licenses and lots of config. Being difficult to learn, hard to maintain, and costly, none of these solutions beats the open-source and free Selenium. End of story.

Sahi is open source and is hosted on SourceForge: https://sourceforge.net/projects/sahi/files/

Thanks for pointing this out, I should've done my homework better with Sahi pricing. I've fixed the issue

if we go with simplicity, then I didn't see anything simple than http://sikulix.com/ - you can automate anything you want there.

That's a horrible name for a project

Sikuli is simple but from what I know about it you can't write maintainable tests with it. Relying on the pictures to be always the same and driving your test is guaranteed to cause false positives.

I think the future lies with platforms like Screenster that have recording, collaboration, built-in browser execution, visual verification and lots of other magic, and not with low-level frameworks like Sikuli.

But both have their place :-)

Although a recorder makes it easy to record tests with a growing test suite you'll find it frustrating to re-use or re-factor the tests.

Plus all the data that you record is tangled in the generated code making it hard to locate or change.

We created gauge http://getgauge.io/ (It's open source) to solve this problem by making it easier to author and write tests in your favorite tool, without a recorder.

The tests can be authored in mark down and wired up to re-usable selenium code.

I've been out of the QA role for a few years now but there is a nifty, albeit not maintained, tool that I used to use for generating UI objects for Selenium. It's not really an IDE, but it did allow me to quickly generate files needed for using the PageObjects model.

https://unmesh.me/2013/08/29/pageobject-generator-utility-fo...

Maybe slightly unrelated but what do people use for browser driven stress/load tests? I have been using PhantomJS + phantomherd[1], but boy, PhantomJS sure takes a lot of CPU.

1. https://github.com/nmeans/phantomherd

Definitely not browser based but I've had good luck with Bees with Machine Guns[0] (Python EC2 load test swarm) for load testing. It's not as scriptable as a PhantomJS test, where you can have every actor follow their own script or anything like that, but it can sure generate a lot of requests a lot cheaper than some of the alternatives.

One thing we've done in the past is to generate a large background load with BWMG and then run a few PhantomJS/Selenium tests in a smaller test rig. It's a lot easier to get the load that way vs. trying to swarm Phantom or Selenium.

0: https://github.com/newsapps/beeswithmachineguns

I'd be curious how these test recorders handle mouseover events (eg: you have a multilevel menu which you hover through). The only way I got this to work was record every mouseover event, which quickly made an enormous brittle test, or just don't test stuff that has mouseover.

Having developed a test recorder, I found out that the best solution was to make mousehover an explicit action (the same as in webdriver). Better to have a robust test than try to do something too fancy.

selenium has moveto[1] which will move the mouse to an element. you would use it in an action chain.

[1] https://github.com/SeleniumHQ/selenium/wiki/JsonWireProtocol...

Screenster like a lot of other tools gives the user an ability to send a mouse over explicitly during the recorder.

In the modern web pages that have to work well with touchscreens mouse over in general is de-emphasized anyway.

Hmm .. This is kind of pointed towards the host product. At least include links to the competing products if you want to give an impression of objectivity.

Have you looked at Applitools for visual testing? http://applitools.com/

Reply on news.ycombinator.com