Abba - JavaScript a/b testing | Hacker News Reader

Abba - JavaScript a/b testing

blog.alexmaccaw.com

76 pointsolivercameron16 comments

It's a bit confusing that this has the same name as another (pretty well known) a/b test calculator: http://www.thumbtack.com/labs/abba/

I'm impressed with how nicely this handles the whole A/B testing stack. But the statistics have some potentially important shortcomings -- I posted a github issue about it:

https://github.com/maccman/abba/issues/3

(Full disclosure: I authored the ABBA library mentioned by matsiyatzy, so this is incidentally self-promotional. But I wrote that tool with the hopes of improving the use and interpretation of A/B test statistics in the wider community :)

Cool. But if you already use google analytics and are lazy like me and don't want to set anything up server side, you can also consider ABalytics: https://github.com/danmaz74/ABalytics

The screenshot of the chart shows 217 conversions from 184 visitors, which isn't right... is it?

You're correct - I mocked up that screenshot and looks like I made a mistake, my bad!

Very interesting. I'll have to try it out. The first thing I'll look at is how it deals with the jQuery changes being visible to the visitors (aka flicker/flashing). Do you hide the page and wait for document to be ready?

Do you do continuous checking of the DOM like Optimizely? Diagram: http://support.optimizely.com/customer/portal/articles/45288...

The trick is to place your a/b testing JavaScript right after the relevant DOM elements, and not do a/b testing on DOMReady or window.load. That'll ensure there's no flickering.

I'm just hoping someone comes up with perfect way I can just put a script in the head and not worry about flickering. So I don't have to hide the body with CSS and make it visible when the script finishes. Optimizely seems to be getting there, but it's not perfect. (And kind of awkward, why can't browsers provide a better way to target elements as they come into existence?)

I agree, we use CSP and hence no inline scripts.a method to avoid flicker would be great!

> If the link is navigating to an external page which you don't control, then you have no choice but to cancel the link's default event, wait a few milliseconds, then navigate manually:

Why not have a callback on the `complete` method to trigger the navigate when the request has finished?

Update for clarity: Presumably the call to the server to register a conversion uses JSONP or similar. Why not provide a callback for completion (or timeout) of this call, and use that to trigger navigation away. This seems more robust and it better encapsulated than requiring the application developer to setup a timeout.

You don't want to force your link to depend on receiving a successful response from the server, because that would be a really bad user experience when it does break. You should opt to lose your analytics when things break instead.

To do that, you'd need to set a maximum timeout to wait for the response anyways, so then it's just a (valid) laziness call. If you _absolutely_ need the best performance then go with a success handler _and_ a maximum wait timeout. But realistically you want the maximum timeout to be ~300ms anyways, so why bother with the extra code.

Regardless, it would be nice to be able to pass a callback (whether it's based on a timeout or receiving a response) right to the `complete` method. Not sure why you're getting downvoted.

I wonder if segment.io will be able to interface with this?

Ian from Segment.io here—randomly happened across this and I was thinking the same thing while reading it. I'd love to hear about how you were thinking it'd work, or what the abstraction would look like: ian@segment.io

Thanks for open sourcing. How hard is it to convert server-side from ruby to php ? I was expecting a simple ajax post. However it looks like ruby everywhere.

Thank you for releasing this!

maccman — is not ANOVA the superior comparison?

Reply on news.ycombinator.com