> Hi, Jimmy from Optimizely here. The practice you describe is actually perfectly fine, so long as ...
Lotsa things are OK so long as you are doing X and Y etc.
Take a look at a portion clinical trial[1] guidance from FDA. Note specifically the basic Stats guidance:
6.9.1 A description of the statistical methods to
be employed, including timing of any planned
interim analysis(ses).
6.9.2 The number of subjects planned to be
enrolled. In multicenter trials, the numbers of
enrolled subjects projected for each trial site
should be specified. Reason for choice of
sample size, including reflections on (or
calculations of) the power of the trial and
clinical justification.
I don't it's recommended practice anywhere to start collecting data, do a simple t-test after each observation, and declare a significant difference after p < 5%.
Of course, if every other patient is suffering serious consequences, or becoming miraculously well on the second day of the trial, you stop. In those cases, you generally don't need a statistical test to tell you that your a priori evaluation of the drug or intervention was wrong.
I fail to see what is so vital about some web site A/B test that one cannot be bothered to think ahead about what defines an observational unit, how many of those one might need to detect an improvement, and wait until after that sample has been attained to test (and, if the web site doesn't get enough visitors to fulfill your sample size requirement for that particular test, that is a different problem entirely).
[1]: https://www.fda.gov/downloads/Drugs/GuidanceComplianceRegula...