Announcing ABTests.com
davidcancel.com
davidcancel.com
WhichTestWon - http://whichtestwon.com/ Testing Thursdays - http://testingthursdays.com/
What is interesting is that it from the guys behind Crazy Egg - Which is a simple but great tool for click tracking
But for business owners here? Who are generally fairly proficient with coding? No, no, just no. Friends don't let friends use Google Website Optimizer. There is almost certainly a better option available for your platform of choice. Use it. If not, write it, it will pay for itself at any sort of scale. (I should know, I had to do it myself.)
GWO is optimized to ease deployment by non-technical marketing folks. It makes significant compromises in user experience to make that possible. In particular, using Javascript for the basic two-alternatives-on-two-urls A/B testing will cost you an extra page load. If its on your personal blog, OK. If not, that just isn't acceptable. Everybody who looks into the effect of page load times on dollars-in-your-bank-account finds the effect to be staggering. Google and Amazon famously have published studies showing measurable sensitivity to differences of 100ms. Causing an extra page load is just about the worst thing you can do for page load speed. It is so bad that Google would swallow broken glass before ever using Website Optimizer, or any technology which required an extra page load, on any of their public sites.
Then there is the 2 URL requirement. This has a lot to anti-recommend it: from a SEO perspective, putting the same content on 2 URLs visible to users encourages them to link to both, splitting your link equity and exponentially decreasing the amount of earned organic traffic. (This is because having 2x the links results in far more than 2x the traffic, because of the steep drop in click-through rates on SERPs.)
Users also might bookmark or link to the alternative you deactivate at the end of the test, which means you get to maintain it for forever. I still have 3 URLs on my site from using GWO years ago which I have to remember to support every time I rewrite my URLs or, whoops, Mrs. Smith's English Resources page breaks. (Good thing I didn't have a web app at the time -- can you imagine having some users bookmarked to a signin page which 404ed? Not fun!)
It also plays very, very poorly with dynamic content. You end up essentially having to rewrite any variant to be displayable in Javascript.
And testing behavioral differences rather than display differences is just downright frightening. Really: you don't want to do it. Flee. FLEE. Save yourself while you still can.
Disclaimer: I wrote Rails plugin which competes with them, well, to the extent that two free things that solve similar problems for different people can be said to compete. Treat everything I say with a grain of salt. You can find these thoughts at more detail and length at my site:
In short, as Patrick pointed, GWO is developed from a perspective of ease-of-use and simplicity. You cannot do advanced stuff with it.
Even though I agree with Patrick with the impact of load time with site performance, the case against using JavaScript must be seen from a broader perspective. Google, Amazon and other companies stress for millisecond-level optimization because their site time is already hyper optimized. But if you see an average e-commerce (or some other) website, a couple of 100 milliseconds due to JavaScript look like a good bargain when it comes to keeping view layer separate from code layer.
But, overall, I think it should be seen from business objectives. If one cares about milliseconds, best to do testing at code level. If you care more about doing tests quickly and not touching your code every now and then, JavaScript based testing is the way to go.