We didn't use A/B tests to make the iPhone at Apple
twitter.com
twitter.com
A/B testing is best suited for determining user preference between existing designs, not new ones.
If you're at the AB stage you should have a fairly mature prduct where you're optimizing to move forward from the low-hanging fruit customers.
Ken goes on to point out that instead Apple relies on exec demos (true); yet zero mention of consumer focus groups. Unless your idea of a good product focus group is one slightly egocentric billionaire.
And really, the entire argument comes down to: "we don't have enough hours in the day to consider every detail, so we just focus on the stuff execs see." When Ken brags about not consulting data in building his software, I view it as equal parts damnation and praise. If he didn't consult any user metrics in designing the software keyboard autocorrect[1], we now know why its reputation with users sucked for years.
What’s hidden though is that out of the thousands of companies that will try, only a few will succeed. The rest will burn through millions and have nothing to show for it. A/B tests are done because they work in the short term, providing runway for the long term. It’s true that many companies abuse these tests and heavily lean on incremental improvements. Management can become complacent, spineless and rudderless, thereby refusing to make hard choices that go against the data. But, it keeps volatility down and appeases the overlords holding shares and vested interest in the company.
It does make wonder though how, despite being so secretive, Apple got a lot of things right with the iPhone and iPod. So whatever their process was for internal focus groups is definitely worth exploring. We’ve all heard how Jobs was insanely attentive to details, but I still wonder where the overall cohesion of design came from. No one person could have possibly gotten all of that right.
This is genius! It might very well be true.
https://www.faz.net/aktuell/technik-motor/digital/designer-d...