Mixpanel (YC S09) releases macro-level A/B testing tool

The first time I noticed this was wrapping my head around funnels, which is a subject I'm fairly deeply experienced with. The three magic words in that documentation are funnel, step, and goal. In every other analytics tool I've ever encountered, the word "goal" is synonymous with conversion: somebody went through the funnel and reached a result which was positive for the business. In Mixpanel, goal does not mean goal. Goal apparently means "step name": for example, the first goal of the seven step funnel that represents the core interaction for my website is the Dashboard goal. The second goal is the new card goal. The third goal is... you get the general picture. This breaks my brain every time I directly touch the API or docs. I wrapped it in my own API and try not to think about it too much.

That is the prelude to me saying: your A/B tests are something potentially wonderful, but they're not A/B tests. You're going to suffer an impedance mismatch when explaining them to anyone who knows what they're doing in testing because when you say the words "A/B test" we're going to think of something else.

An A/B test involves:

1) A user interacting with an element which is subject to the test. An element can be almost anything: a design feature, a bit of copy, an entire checkout workflow, whatever.

2) The user being randomly assigned into one of two groups and shown the appropriate variation on the element.

3) The user does some stuff.

4) Hopefully, the user converts.

5) You compare the propensity for conversion among the populations of users in each test group to see which performed better.

The solution you have delivered is different than A/B tests in many ways:

1) It does not randomly partition users into groups. That is, presumably, something your customers have to do prior to firing an event with the property (or super-property -- I forget what you call it) testWhatever: "alternativeA".

2) A/B tests are cool not because A and B are cool letters but because they make statistical significance testing really easy, so that if alternative A has 20% more conversions than alternative B you can quickly tell whether that is solid evidence that A is indeed better or merely a possible artifact of random variation. Your testing feature does not implement significance testing.

3) There would normally be some halfhearted bow in the general direction of independence at this point (we mostly gloss over that and hope people forget stats 101 since, in practice, just wishing this requirement away tends to actually work acceptably) but the entire point of your stats feature is, as far as I can tell, taking the independence notion and beating it black or blue or purple and then seeing how that choice of color interacts with six other things.

Again: I like the feature. I think that given an hour or two I will probably even be able to do something with it that will eventually make me money. But if you try to communicate this as A/B testing you're going to cause a lot of unnecessary confusion.