Hacker News Reader

Comment by mehrdada | Hacker News Reader

I doubt there are companies exclusively rely on A/B tests to make decisions, but even if there were, someone has to think about hypothesis and design the right experiment to then confirm with data. You can't just have a computer running around and make experiments and confirm them for youº.

º: you kind of can, if you have enough fresh data, to search for and derive statistically significant results automatically.

View on HN

Maybe it's just me, but I don't think the secret behind this kind of success is just properly designing experiments. Again, if this were true, the best founders would be data scientists. But they're not. They are visionaries. So I reckon it must be the vision.

Vision is obviously necessary - but a big problem folks run into is failing to recognize when something is not working. A/B testing is one way to keep yourself from falling into the trap of thinking 'I know better than my users what they should want'

I agree, but I think this doesn't apply to innovation. When is comes to innovation, people always want faster horses.

that depends on the validity and accuracy of your test metric.

Also a good way to find a local maximum when there are much higher maxima not far removed.

>Again, if this were true, the best founders would be data scientists. But they're not. They are visionaries.

[citation needed]

You'd be paralyzed by indecision if every decision had to be driven by the results of a test.

Reply on news.ycombinator.com