The what and why of product experimentation at Twitter

A few brief points:

1. The post talks about most experiments not moving core metrics in a large way. This does not mean the experiments don't ship, it just means core metrics are very hard to move. So "few experiments actually succeed to turn into new features" is not entirely correct -- quite a few of them do ship because they improve a particular feature, or because the team believes they are the right thing to do as long as they don't do any harm, or even an acceptably small amount of harm.

2. If the majority of the experiments worked, we wouldn't need to experiment, would we? :-).

3. Core metrics being hard to move is not a Twitter-specific thing. If you find them easy to (positively) move, you are either so tiny that any change is a large change, or you are measuring something incorrectly. LinkedIn, Bing, and Amazon experimenters contributed to a paper, linked from the post, which states as "rule of thumb #2" that "changes rarely have a big positive result on key metrics." Anecdotally, the same is true for Google, EBay, and Pinterest.

4. Analysis paralysis, according to wikipedia, is "an anti-pattern, the state of over-analyzing (or over-thinking) a situation so that a decision or action is never taken, in effect paralyzing the outcome". Deciding to not ship something is not analysis paralysis. It's a valid outcome. If everything you try works, you are probably not trying hard enough. Now, if you ran an experiment and kept saying there is not enough data to make a decision, and kept analyzing the data and collecting more inputs -- that would be analysis paralysis.