The small sample trap in A/B testing

This is definitely an interesting thing to think about when it comes to A/B tests for smaller companies. From my experience working on them it wasn't usually an issue there (since every company involved was a multi-million/billion dollar organisation with the site traffic to match), but it does make me wonder if some agencies would end up misleading potential clients because of it. If you're a marketing agency you want as many new customers as possible, and if you're not mathematically inclined/don't care it could be tempting to just say "everyone should do A/B tests".

More interestingly though, I wonder how this affects most YouTube channels nowadays. YouTube's rolled out A/B testing for both titles and thumbnails, and it's available for just about everyone in the partner program (and maybe even those outside of it).

But the thing is, the average YouTube video isn't popular enough for an A/B test to be effective here. If it gets less than a thousand views on average, there's basically no point testing anything.

So, I wonder how many creators there are falling for the trap mentioned in the article. Making changes to their strategy based on insufficient data due to the platform suggesting a feature they don't have the audience necessary to make good use of.