PlanOut: A framework for online field experiments | Hacker News Reader

PlanOut: A framework for online field experiments

facebook.github.io

51 pointsstanfordhci7 comments

As a former user of this system at FB, I'm really excited it's being open sourced. Eytan, Dean et al are some of the sharpest experimental design people going round, and the paper (at https://www.facebook.com/download/255785951270811/planout.pd...) is worth checking out for a more formal introduction.

So do you have to use python to work with this? That's probably a stupid question because clearly facebook's using all kinds of tech on the backend. I'm trying to see how you'd integrate this into a rails or node project...

Found the answer to my question on the readme: "The implementation here mirrors many of the key components of Facebook's PHP implementation of PlanOut which is used to conduct experiments with hundreds of millions of users."

I tried to build the python version of PlanOut with portability in mind. It took only a few minutes to port the base class that does all the hashing from PHP to Python. The the interpreter + a few core operators took about two hours. Developing the Experiment and Namespace classes to work in a way that we were happy with took a lot more time. I suspect that if you knew node / js, and just wanted to do a direct port, it shouldn't be too hard.

We have a little more background on the project at: http://facebook.github.io/planout/docs/about-planout.html and I'd be happy to review any ports to other frameworks.

Does Facebook use this system to turn on new features for a certain subset of the user base to test those features out, or is this mainly for A/B testing existing features?

PlanOut has been used both for evaluating the effects of new product rollouts / encouragements and for more scientific experiments that look at the effects of existing features. You may want to check out Section 4 of the PlanOut paper at: http://facebook.com/publications/563103330455476/

Awesome, thanks! We'd have to port it to rails to use it at work. Right now we're using much more primitive AB testing.

Reply on news.ycombinator.com