Hacker News Reader

Comment by patio11 | Hacker News Reader

Using variation pages causes SEO and site problems when the test ends unless you manage your 301s properly, and the marginal load for the redirect to B artificially biases test results. Also, I did a bit of an eye popper demo at a company I just consulted with: deploying an AB test live on my site took two minutes twenty six seconds while slowing my typing speed to explain every step, while the company reports their experiments with GWO require 15 minutes at a minimum. That gives the typical engineer 12 reasons not to AB test.

View on HN

Hi Patrick, glad to hear that! Curious what you think about client-side A/B testing products in general (Optimizely being just one of them) vs. server-side products (like A/Bingo). Do you face pain points with A/Bingo that might be solved with a client-side javascript based solution?

In general, I feel server side is far superior for developers and client side with a WYSIWYG UI is a fabulous option for other stakeholders. I could talk on this subject for a while but my plane is about to take off. Suffice it to say you can't match power, flexibility, and speed of server side with client side testing. You can move away from just twiddling text to actually redoing workflows or business logic.

I didn't write ABingo with a pain point included, but for Appointment Reminder it will be helpful to do AB testing in non-Rails marketing pages, too. I wouldn't append too much effort optimizing your solution for the needs of people who have written AB test frameworks though.

I would like to see an A/B test where A is no test and B is a traditional A/B test. ;-)

My guess is, similar tests favor the "non-test" or quicker result/load, though I bet it would take A LOT of views to irk this out. Google always talk about these test on shades of blue, I imagine those tests are similar in the number of runs they require (hint: a lot).

I did this before, I was making the case that JavaScript run tests took extra time and that server side tests were superior for large companies because they didn't need to call out to a resource outside their network. Sure enough conversions were 9% lower (percent of a percent, so from 2% to 1.82%). The funny thing was that when we tested another one of these JavaScript based split testers it was coincidentally in the same server room as the hosting provider at the time, so it didn't negatively effect the conversion rate.

That being said, almost nobody split tests. It's a serious problem because marketing usually needs technical to split test, so even if Optimizely cuts your temporary conversion rate by 9% you will still make that back quickly after a couple of split test iterations.

My number one question for optimizely is when are they going to include some sort of call back api or something to track real conversions. The number one concern SaaS businesses have is visitor -> paid account not visitor -> free account or signup screen.

Reply on news.ycombinator.com