Ask HN: Rate my startup, Go Test It (cross-browser testing) | Hacker News Reader

Ask HN: Rate my startup, Go Test It (cross-browser testing)

50 pointsmartinkl41 comments

Hello HN,

Just recently, at Future of Web Apps in London, we launched our startup, Go Test It. Our goal is to make automated functional testing of web applications incredibly simple and efficient.

We have an infrastructure hosting various different browsers, and we have a fantastic test recorder which can pick up mouse clicks and keyboard events to create test scripts. Test scripts can be converted to Ruby or Python, and run directly on our infrastructure.

It would be awesome to get your feedback on what you think and how you would use Go Test It in your own projects.

Please sign up for free at: http://go-test.it/hackernews (use "hackernews" as invite code)

Thanks :)

View on HN

Nice. And that's coming from someone who has been thinking about an app like this for quite some time.

One minor thing: On the "Welcome to Go Test It!" page (/welcome), you have "I have existing Selenium scripts that I want to upload." and "I do not yet have any test scripts." I bristled at that slightly. I _do_ have a bunch of test scripts (that I worked pretty hard to create), but they are not in a ruby, python or selenese form. Maybe something in the gray text for the second option that acknowledges something like "or I don't have scripts in a Selenium format" would create less of a false dichotomy and help me feel better about myself than selecting "I do not have test scripts" does. Maybe I'm overly sensitive.

One bigger thing: I like the options for the "New Test Script" (projects/*/scripts/new) page. I'll probably go back and install the Firefox plugin when I have more time, but I wanted to see how it works quickly, so I backed up and went to the "Write a new script in a text editor" option. Now I'm presented with a blank text editor, with little or no clue as to what I'm supposed to do there. At the very least you should have links to samples or documentation for the scripting language, but you could do better than that. You know my app's URL at this point. You could pre-populate this script (for new users) with a basic "Is it up?" script, maybe checking the title and some H1 text on the page, just to give me a working example to modify. I think for new users you should minimize the number of steps to a running test. As it stands I either need to install a firefox plugin (a pretty big hurdle, imo, although it is right to offer that) or (I'm guessing) go off and research how to write Selenium scripts and then come back and create one on your site. As a random user looking to solve my functional testing problem, I'm not that motivated to do either. Once I'm off writing Selenium code I'm likely to forget about GoTestIt. I'd suggest that you either auto-populate a basic script for new users, so that I can click "Run Test" right away and see what happens or find a way to record basic scripts directly in the browser without installing a plugin (or both).

(Also, personally I find the "Feedback" tab on the left to be a little distracting, but I'm on a netbook so my monitor is smaller than most. Of course, it looks a little funny when glued to the scroll bar too. I wish UserVoice would let you attach that to the top or bottom of the screen.)

Thanks, those are some very good points. We've tried to make it easy to get started, but clearly we can do better still.

In what language are your test scripts currently?

I have a bit of history with writing automated functional tests and testing frameworks. I've used a large number of tools, most frequently HttpUnit, custom scripts built on top of Apache's HttpClient, a Jakarta Commons subproject called Latka that sort of died on the vine (although I think it may have replaced Apache's watchdog servlet engine compatiblity test for a time), custom Ruby scripts based on Hpricot, and others.

My current project is Drupal based so I'm using a php unit testing framework called SimpleTest. I've found it pleasantly easy to work with for basic HTTP and HTML level testing but it doesn't have any JavaScript or browser-driving capablities, so I'm probably in your target market (i.e. I have unmet needs).

A pay-as-you-go pricing option might be useful. The monthly pricing model works well if you plan to run the tests every day/week, but I'm guessing that a lot of people will want to run the cross-browser tests on every new deployment, but perhaps not between deploys.

That's something we're definitely thinking about. For the time being you can just sign up for a month, then cancel it again -- there's no minimum contract.

I strongly agree with this. Also, the prices seem quite steep to me -- though they may not be. Instead it may be that I'm not seeing how many tests can be completed with the allotted actions at each price level. The "what can I do with 15000 actions" is a step in the right direction, but I think it would be easier to comprehend if you said something like "a typical application of x size can be thoroughly tested y times a month w/ z actions".

This seems really really expensive.

Say it takes me 10 actions to fully test one feature and all its cases. Spread that across FireFox 3 + 3.5, Safari, and IE 6 + 7 + 8, that would take a total of 60 actions for one feature. With a web app that has 20 features testing it once a week I would have to pay $326. What if a test fails, do I have to pay again?

I don't see how anybody can use this with such a hefty price tag.

Just think of the amount of time you'd spend testing all of those manually, and what your time is worth. Go Test It costs hardly anything in comparison.

I have seen a bunch of these and many are priced at this level, which is around US $0.15 per test. It astounds me as it seems ridiculously high.

Most of my automated tests run in about 5 seconds or less. With EC2 prices at about $0.10 per hour, I can't imagine ever using a service with such a high premium. Not only because the price is nominally expensive, but also because it's a commodity service that's being priced at a HUGE premium.

For me, I'd consider the service if I could offload my grid testing to the cloud for around $20 / month.

That way I could run a bunch of smoke tests after every push to make sure nothing broke and a larger set before commits. But that can get to be a lot of tests especially when you re-run for failures.

Cross-browser testing can be one of the biggest time/money sinks during the QA phase of a web-app release. $326 may not make sense for a small company, but medium to large companies will spend much more than that on manual cross browser testing. A 4 person QA team, assuming average costs (salary, benefits, taxes, office space, hardware, etc...) of $100/hour (which I think is low) burns through $3,200/day. $326 is a drop in the bucket.

Looks nice and useful, but scheduling is a bit unclear to me:

I have 1 testscript, 7 browsers, which makes 7 tests. If I schedule to run every "day", will those 7 tests run every day, or 1 a day? I expect the first, but "Our scheduler will try to start one of the tests above with the start times separated by the time you specify." seems to imply the second.

(oh, and check this out: http://vldtr.com/?key=go-test.it )

I love the service. Its a great idea, and the scriptability of it makes integration into an existing pipeline extremely easy.

All praise aside I cannot say I personally would use it much beyond experimenting with it. My project uses google maps too extensively and ends up failing out too much. Thats more my problem than yours as scripting events for GMaps would be overly complicated and useful to only a few.

Thanks for the demo!

Google Maps is actually one of the sites which we use internally for testing Go Test It -- many features of it can be automated quite nicely. Particularly with such complex JavaScript, you don't really have a choice but to test in every browser you want to support!

I like it.

Sign-up nice and concept good.

Couple of comments/bugs: 1) When you first record something and click 'run' it brings up a confusing dialog box with a textbox. I figured out eventually this is a save dialog box, but it isn't obvious

2) Appreciate the need to queue things up, but I didn't find the queue status thing very intuitive.

3) I used news.bbc.co.uk as my test site, and all the IE ones died opening the front page.

Thanks for your comments -- I know we need more work on the user experience. We'll also look into those IE failures.

I did something similar a couple of years ago called Blue Violin. It was a web app that recorded web events, saved them, then ran the test script. The source code is available at http://code.google.com/p/blueviolin/ - you might find some code of use to you. If you have any questions, you can find me at @JoshWatts or josh.watts at gmail.com.

Support of Selenium scripts is great. I was afraid only Ruby/Python were supported from your description.

How can I delete or rename existing scripts?

it would be great to have more browser available: Safari on Windows & Mac, Opera 8 to 10, Chrome, etc., along wit hthe option to disable javascript, Jva, Flash, etc.

I like the concept and the sign-up was easy.

I noted that you can't run a test until you first save it, even though there is a Run button. You may wish to make this piece a little smoother.

I got a little confused by the queueing though. Was I supposed to wait when it said queueing? I did, but nothing happened for a while. Then I selected the browser type again and clicked the button to run the test. This time I had my results.

My problem was that my application requires authentication - in particular Twitter OAuth. Do you have any plans to address authentication in general. I didn't know how to work around it, so I couldn't evaluate the playback.

I tried to leave a comment, but when I submitted it, the message came back "Not Found".

- Scott

Clickable link: http://go-test.it/hackernews

I am in the market for such an application and my budget is up to $150 per month but I find the 'action based' pricing model a huge turn-off. If I am paying $50 upward, I want to be able to automate tests freely and run them nightly without ever thinking how many 'actions' I have left.

The UI is decent, it lacks support for pages that require authentication from what I can see, and I couldn't find any features for running tests regularly (e.g. nightly).

I find it hard to imagine that anyone would sign-up and pay that much money to be limited by an 'actions' meter and for an application that is not mature by any means.

If you adjust the pricing and address some features it could be a good application though.

The signup process is rather effective -- gets you to running your first test nice and quickly.

I agree the sign up process is very slick - I like it a lot.

Agree...I was signed up and writing scripts within 2 minutes. Could be made quicker by using Facebook Connect or Google Auth.

How does this differ from Selenium or what the guys at Sauce Labs are doing?

seems not far off, but a hosted paid for version of the same thing. It's a lot easier to set this up than Selenium from what I've seen. Even if ease of setup is the only difference they'll still be a really popular service.

The Sauce labs guys assume you have already recorded the script. With go-test.it you have a firefox extension that does the recording and uploading of scripts. Jason Huggins doesn't have a good opinion about recorders/IDEs. Selenium is very developer focussed (As JH himself said at one of his talks),but most testers don't code. Go-test.it seems to be based on Selenium and focussed on the average tester. I know for a fact that there are large companies that are willing to pay for this kind of product.

Update #1: Saucelabs records the video of tests which go-test.it currently doesn't seem to do.

Now that is a really useful application! well done guys, if the price is right (bit high at the moment for the individual / small business)

I just ran through bunch of tests for my application. This is very nicely done.

How does this compare to BrowserCam? [ http://browsercam.com ]

(disclosure: I work next to the BrowserCam team)

Well, for one, it sure looks a lot more professional than the BrowserCam website.

Go Test It is much more focussed on the browser automation and functional testing; BrowserCam is more about checking that the design looks right.

BrowserCam does screenshots, selenium based testing, and raw VNC support, but lacks the site design and product focus. They are part of a larger company and don't have the same amount of control over external appearances.

I didn't mean to imply that Go Test It is inferior to BrowserCam, I was actually curious what others throught. There may be a Mac / PC analogy here - both have almost the same features and so the primary differentiators are price, design focus, and product integration. If you aren't a Gomez customer, that last one isn't going to matter.

UPDATE: I'm getting my product names wrong. BrowserCam became a Gomez product called Reality View, and only provides screen shots. Selenium testing is done in a related product called Reality Check.

Very cool. As other people have have said, a pay as you go payment plan would be pretty cool. I'd use it.

Can you make it work with Flex apps? If so, you've got a client. If not, I'll have to pass, unfortunately.

Sadly, we cannot yet easily interact with Flash or Flex apps. It's something we want to do, but not right now unfortunately.

I would use it. I feel that your pricing is too expensive. I would pay $10 maybe $15 as a small company.

what a bittersweet solution ;) "oh cool! look at all these bugs--hey wait..."

really awesome service. very very impressive. price point seems a bit high for another small company to be using, but corporates would jump all over it. hope it takes off, well done.

Does Go TestIt solves any problems which selenium doesnot solve...?

It is actually based on Selenium internally. The main advantages are:

* Go Test It provides a hosted infrastructure, so you don't have to spend lots of time configuring and maintaining various VMs with different browser versions;

* Go Test It has a lovely test recorder which is much nicer to use and works better than Selenium IDE;

* Many little tweaks and improvements, like automatic screenshots etc.

Very good idea, I could see myself using this...

Reply on news.ycombinator.com