JavaScript Testing Does Not Scale
ejohn.org
ejohn.org
If I might deconstruct his argument:
"As far as I can tell, Selenium Grid requires that you use Selenium to run your tests. Currently no major JavaScript library uses Selenium (and it would be a major shift in order to do so)."
Doh, yes Selenium Grid uses Selenium. And the argument about no major JS using Selenium is not an argument to not use it.
"It isn't able to test against non-desktop machines. Each server must be running a daemon to handle the batches of jobs - this leaves mobile devices out of the picture."
Unless you write the appropriate daemon for those devices and integrate it into the existing Selenium Grid set up. Surely that's easier than starting afresh?
"It can't test against unknown browsers. Each browser needs special code to hook in to triggering the loading of the browser by Selenium, thus an unknown browser (such IE 8, Opera 10, Firefox Nightly, or Chrome) may not be able to run.'
Huh? That's just bizarre. If you've got some new browser then teach Selenium how to start it. It's not that hard.
"And most importantly: Selenium Grid requires that you actually own and control a number of machines on which you can run your tests. It's not always feasible, especially in the world of distributed Open Source JavaScript development, to have the finances to have dedicated machines running non-stop. A more cost effective solution is required."
This is solved by virtualization.
Sure it is. If a developer will never switch to Selenium then Selenium Grid (in its current state) is useless. TestSwarm is completely test suite agnostic - which is huge. I don't know of a single Open Source JavaScript projects which uses Selenium.
> Unless you right the appropriate daemon for those devices and integrate it into the existing Selenium Grid set up. Surely that's easier than starting afresh?
It absolutely is. There are no daemons in TestSwarm. It's a single URL that you can open in any browser. No 3rd party software is required to be installed.
> Huh? That's just bizarre. If you've got some new browser then reach Selenium how to start it. It's not that hard.
So the only solution for an unknown browser is to wait for the Selenium team to add it? That's unacceptable. New browsers must be added instantly and on-demand (which TestSwarm handles).
> This is solved by virtualization.
No it's not, not at all. A virtualized environment requires a few things: A machine on which to run it and enough resources to run as many virtualizations as required. As an Open Source project we don't have that sort of money.
Second, virtualization is insufficient for testing. Within the Mozilla project we've discussed problems with virtualization and how they are not always precise. Nothing is better than running the tests directly on an actual user's machine.
Third, virtualization does not solve the problem of manual testing. We want to get tests in front of users wherever possible - throwing them to some dumb server does nothing to help us.
> for the Selenium team to add it? That's unacceptable. New
> browsers must be added instantly and on-demand
No, the solution, as with any FLOSS project, would be for you (or someone else) to add the browser to Selenium and not wait for the Selenium team to do it.
I don't understand the requirement for adding new browsers 'instantly and on-demand'.
> Second, virtualization is insufficient for testing. Within
> the Mozilla project we've discussed problems with
> virtualization and how they are not always precise. Nothing
> is better than running the tests directly on an actual
> user's machine.
I'd be curious to hear those arguments and to have examples of when virtualization is not sufficient and you need to go to the actual user's machine.
Because the major JavaScript libraries' timetables should not be bound by how willing they are to twiddle config files/write new browser loaders in Java. It shouldn't be anymore complicated than hitting a button that says "Add this browser as a new test base."
> I'd be curious to hear those arguments and to have examples of when virtualization is not sufficient and you need to go to the actual user's machine.
Timing, performance analysis, and CPU/memory availability are huge ones. Being resource constrained in a VM is not an acceptable approximation of what a user's desktop actually is. Nor is running a mobile browser in a simulator. Nothing beats running an actual browser on an actual desktop.
If you were to argue that you could have no resource constraints in your VM, that brings it back to the argument of cost and resources.
Fair enough. Don't you run into the same problem with volunteer machines though? You can't actually control their configuration in TestSwarm so you can't control the resources (e.g. you can't control what else the machine is doing). If you are going to do do any serious analysis of timing or performance then you are going to have to control the machines.
Absolutely - the timing and performance won't be more accurate but it will be more realistic. For accuracy everything has to be done on the same machine in a 'clean' environment (and definitely not in a VM).
Selenium is the flakiest test framework I've ever used.
Though, my solution would use the best of both worlds. jsunit tests fired off by Selenium on a grid.
That's why there are also already solutions, for example buildbot. You can get it to check for new revisions in your VCS and run it on any platform. This is happening all over the world, using different systems (OS X, Windows, Linux, *BSD) with different versions of the OS and different compilers (GCC 4.2 GCC 3.4, Visual C++ etc).
Yes, if you want to be compatible, you'll have to do a little effort, but Javascript suites are hardly the first to figure this out. Everybody is doing it with relatively few issues.
BTW here's my testing :D http://axod.net/browserTesting.png (OSX+Firefox/Safari/Opera, XP+IE6, XP+IE7, XP+IE8, XP+Chrome/Firefox, Ubuntu+Firefox). Thought it might make my macbook sluggish, but it just carried on regardless :)
Pretty easy, just run the tests more than once (on more than one client). If you get caught giving bad data then you get banned - seems simple enough.
There's a big difference between one test randomly failing and someone submitting completely bogus results.