I found this fascinating. In the world of "optimize everything" I quickly realized I've been doing A/B testing completely wrong.

tl;dr is to set a sample size before starting an experiment - otherwise risk statically insignificant results.