Our application is primarily one page and uses a lot of jQuery to both animate things on screen and to work with the backend via AJAX.
Any hard won tips out there?
disclaimer: I used to work there, and put my blood, sweat, and tears into building it out just for situations like this.
When I click the link that does a 4000ms animation I should see "#mydiv"
Capybara will wait for #mydiv to appear for some timeout period (30 seconds). If it appears before then, it keeps going.
This way you can have complex animation and still assert final state of the page. Capybara allows Selenium, headless WebKit, and other JS drivers. It's one line to swap.
If you aren't in the Ruby world, you might at least steal some ideas from that lib to make your Selenium tests better.
Also, you might drop the folks at Pivotal Tracker an email. They have a fat one-page JS-heavy product, and I bet you anything they automatically test it. They might have cool stuff to share.
The second thing I do is use my tool every day. I use the unstable branch on the unstable server whenever possible. Problems usually become obvious very quickly. I don't push to production until I've used the unstable server for 24 hours without incident. I find this combined with the testing practices described above to be the best compromise between reliability and getting-stuff-done-ability.
Edit: I have a great outsourcing team that builds Selenium for me, and runs in SauceLabs. PM me if you want details.
In comparison with Selenium, I've found unit and integration tests using this setup fast and resilient.
My advice: Do NOT use the Selenium IDE. write your own framework, using Selenium, which isolates changes when they occur. If someone changes the login page, you only need to change a login method in your test suite not all 200 rest cases! Look up the Page Objects pattern as one way of doing this
Updating the tests even when we make significant changes to the website is must easier because of this approach.
One issue we ran into quickly... tests need to verify just one thing as much as possible, not long strings of actions that wander through the whole site. The problem is that there's considerable setup before testing that one thing -- creating users and getting them to the appropriate state, especially (worst of all when test cases require 3+ users to interact).
To address this I've set up some helper requests Selenium can make -- actions that aren't included in our production build -- to create/destroy test users of different types & in different states directly. I'm also thinking of adding a page that'll simply return a full dump of info on a given test user, so we can test that the end result of actions in the UI have resulted in the expected data.
It's all very much work in progress, but I'm convinced that with a little extra elbow grease added as pain points show up, we can keep a full suite of Selenium tests alive and useful.
I found bulky solutions like Selenium to be somewhat hit-or-miss and overkill for most tasks. It's very rare that with proper unit testing I really need a whole additional layer just to click around the site and make sure stuff is working as expected. Maybe some basic sanity checking, but it's really not my preferred way to think about testing--which of course is just a personal preference.
One tip I do have is to disable jQ animations during your test runs so you aren't having to jump through hoops in your test suite just to wait through animations -
jQuery.fx.off = true will do that.
It also has a 'live mode' which is useful for writing up new tests. It basically follows a file and whenever there is a change it begins running the commands from that line onwards again. If it hits an exception it'll pause there and wait for you to fix the broken test/site before resuming.
I wrote a custom test runner as well, though not addressing the same issues; mine is specifically for non automated testing. It's a Java GUI that lets you choose from all our existing tests, tinker with parameters, and run the test live in your own browser against our test server with a few extras (like stepping through the test, go/pause, run all tests in sequence, etc.).
It's partly to let our CEO really interact with our testing -- i.e, see what's being tested, and rapidly explore the site as a whole -- but also is pretty useful as a robot; i.e., when I want to register three new users to try something out, I don't have to do it manually.
Have different environments set up. After every big push, run through a simple user flow to make sure what the user will actually see is what you intended.
One way that I minimize my front-end testing liability is to do as few "big" pushes as I can. The safest change to make is the smallest change possible.
And I agree with simplegeek, selenium should be fine in your case. Although can't imagine how you could test animations... :)
I live in constant worry of changing css and totally screwing up a completely unrelated bit of the web app, and the usual thing I get told is "well, it's very fast to check if it still works", which reminds me of me not understanding why unit testing is useful.
I have often heard two justifications for testing: (a) correctness and (b) developer speed.
Phrasing (a) differently, is X functioning as intended? Phrasing (b) differently, how quickly can I make a change to Y, and if I change Y, how long will it take to manually check Y (with a browser, etc.), how likely am I to introduce a bug (leading to extra development time) and/or will I have to rewrite a test?
If your primary justification for testing is developer speed rather than correctness, does that impact which features (models, controllers, views with div ids, actual view graphics) you should write tests for?
We also have an increasingly broad application; as we add new features, I (as a developer) find that I don't visit older parts of the app, though of course the users still need them to work. Particularly when refactoring basic back end stuff that's called from lots of places, it's too tedious to manually track down every part of the UI that would need to be retested due to a given change (and then test it manually... ugh).
I understand this worry, but it makes me think that it may be a design problem with the CSS. Unrelated bits shouldn't be using the same CSS, related bits should probably change together.
In a more practical note, I have created selenium tests which capture screenshots and then saved those screenshots as artifacts in the CI (TeamCity) system. Having a human look at a bunch of static images makes for a faster and less error-prone sanity test to see if things are totally broken.
But my point is exactly that: I can't just go in and _improve the code_ (extracting classes, rewriting it in less/sass or whatever) because to verify the changes do not mess up the UI requires clicking around many pages in half a dozen browsers in three OSes.
And thank you very much for the practical note, it's an interesting idea!
I hope the automated screenshot idea helps. I personally found it more useful in theory than in practice, but I wasn't doing a lot of UI polish work.
Computers are great at functional testing but you still a Mk1 eyeball to check aesthetics. Luckily, functionality is what your customers will really care about. No one will shout at you if an element is misaligned for a while before you catch it with manual testing
But... it's pretty cool to have a single Selenium test that does nothing but zip through your site and show all of the unique views at rapid fire speed, so as soon as the page is loaded it has already filled in any relevant form and is loading the next page.
Make sure someone with good concentration sits and watches that test run once in a while; anything seriously wrong will pop out, and they can watch it zip along while taking a coffee break.
For user interfaces, it's a lot more effort for a lot less return. YMMV