Statistical Advice for A/B Testing

No condescension perceived :-)

I think that background varies - some people are fluent statisticians and others come from different disciplines / careers. Anecdotally, my experience from browsing A/B testing advice online is similar; some content is very statistically savvy and some is a bit oversimplified.

I can answer from personal experience. I'm a dev at a tiny startup. I took statistics at college 20 years ago but haven't used it since the final exam, and remember little other than sample size is critical. I've figured out through recent experience that you need to let a test run a while - results seem to flip flop for the first few thousand visitors. I use tools like VWO because it makes it simple to run a simple one variable test, and I'm happy with the results (though ignorance is bliss, so maybe others would not be so content).