Hacker News Reader

Comment by jmickey | Hacker News Reader

This has been our experience as well. We invested a lot of time and money in making sure Selenium tests run reliably for our clients. Despite this, the best reliability we managed to achieve was 90% with tests that run for 40 minutes, which is obviously not acceptable.

We have compiled a few tips we learned along the way in our blog post - http://novoit.eu/blog/05-5-tips-when-writing-Selenium-browse...

View on HN

>the best reliability we managed to achieve was 90% with tests that run for 40 minutes, which is obviously not acceptable.

What was actually going wrong during that 10%?

I get something closer to 100% reliability, so I'm feeling a little perplexed by all of this.

Do you make heavy use of sleeps?

Mostly these would be cases where the browser would, seemingly at random, end up in an unpredictable state and all proceeding test scenarios would fail because of this. (Page is white, or a completely unrelated website gets opened. We have seen lots of weird situations so far)

This might be exacerbated by the fact that we use the remote Browserstack Selenium hosting service so that the tests can be executed automatically as a part of our deployment process.

White page and randomly ending up on an unrelated website both sound like bugs.

>tests that run for 40 minutes

This is pretty good actually. It sucks if you're relying on Selenium testing for verifying your code as you're writing it, but before and after deploys to staging and production? This isn't bad at all.

40 minutes from clicking a button to deploy is actually abysmal, especially when you need to worry about things like rollbacks, or deploying at the end of the day, or releasing quick hotfixes to users. In modern build processes, even 10 minutes seems too long.

Reply on news.ycombinator.com