jlinowski

1) Agreed that the smaller the effect, the more statistical power (usually from a larger sample size) you need to detect them. But to assume that all changes have tiny effects, and therefore not detectable and a waste of time, is a flawed assumption.

Once upon a time we published over 100 a/b tests here: https://www.goodui.org/evidence/ and clearly the relative effects vary (not all single changes have always a small effect).

More so, the effects of a/b tests can be further increased by grouping multiple higher confidence ideas together into a single variation.

2) Short term gains may (or may not) lead to long term disengagement. Measuring micro (shallow) and macro (deeper) metrics would be the right way to answer this.

Facebook is known to run countless online experiments – most of which are kept secret. Luckily for us we have the Internet Archive that allows us to pry into these possible design optimizations.

Does anyone else see a problem with a session based approach to a/b testing (that Google Optimize uses)?

Sure. But the idea is to do a quick ice breaker project that can later turn into a larger one down the road. Hence in that case a $500 - $2000 structured project makes sense, I think.

I don't think customers want to buy YOUR time. They want to buy THEIR results for THEIR business with as less of your time as possible.

The question I recently have asked myself was this: on these structured or productized pages, should we push the customer to a low friction gradual engagement lead generation action, or close the sale with a payment? I'm now trying out the former. Thoughts?

My response to the article: http://wireframes.linowski.ca/2012/12/calling-your-bull-the-... I think it's stretched. Interfaces still have good characteristics.