Ideas about A/B Testing Methodology

I'm disappointed that this did not generate a discussion on HN. About two years ago, I was made accurately aware that methodology is one of the most -- if not the most -- important factor in conducting research, and that my methodology was shit. I've spent that time almost obsessed with methodology, hoping to strengthen this weakness.

This is the first post I've seen about the importance of methodology in A/B testing. Given that the underlying distributions can be difficult to estimate, there is a susceptibility to both Type-I and Type-II statistical errors. Consequently, I think this post was much more important then it seemed when looking at the lack of discussion.