I guess A/B testing will tell you if the horse will become faster if you change from iron shoes to carbon fiber ones.
A/B testing won’t ever get you a Tesla from a horse.
A/B testing won’t ever get you a Tesla from a horse.
Right. But if you were already selling Teslas, and along came a smooth-talking product designer with a dream to "improve" it by building a horse instead, A/B testing would make sure that change never saw the light of day.
A/B testing is not a way to get out of having to come up with good ideas yourself. It's a way to validate that your ideas are any good in the first place before betting the whole company on them.
A/B testing is most critical when evaluating big changes to a product, because those are the changes most likely to completely blow up the business. Otherwise it's left up to the opinion of the highest paid person in the room, and people are notoriously bad at guessing how customers will respond to change.
There’s something very powerful just talking to the user, feeling their pain and working with them to fix it until they say “this is awesome”.
How do you think the eyeball arose from unicellular eukaryotes?
Eyeballs are nowhere near optimal either.
You’re not going to get to a global optimum driven by human choice of what to test (only local optima at best) unless the human setting up the tests is some sort of sage.
For a trivial and well-known example, there is now no developmental or evolutionary pathway that will lead to a vertebrate eye where the nerves run behind the retina. As a result every last vertebrate has a blind spot where all of the nerves dive through the retina. This makes our eyes less efficient because the nerves block out light.
We're therefore stuck with the first design of the vertebrate eye and can't change it. There is no pathway to the more sensible design of the mollusk eye.
This is but one of many examples. For another one, relative developmental timing of growth is fixed across vertebrates. For example the "hand" grows before the "arm". The result is that pterodactyls, whose wings are entirely hands, could fly from birth. But birds (whose wings are arms) and bats (a mix of the two) can't fly from birth. No matter how desirable an evolutionary trait that may be.
No. At least not in the way described. Optimal as considered by whom? Us? We look at something and say this would function better than that, but is something that is optimal perfect? Basic microeconomics shows that perfection is sub-optimal due to the law of diminishing marginal return. Perfection is wasteful and unsustainable. What do we know to be optimal? From what I know, “adequate enough to reproduce” may not be perfect, but it might be brushing up against optimal in its “good enough and no more” / lagom nature. Mollusks have mollusk eyes. Can we know that humans would be better off with mollusk eyes without rewriting our evolutionary lineage for us to be more like mollusks in other imperfect ways?
I'm not necessarily against that as a valid way to measure job performance if it's done intelligently (for one, realizing that there's a lot of blind luck and variance, and it takes time to smooth out) - I mean, if you're in sales and you're not booking any sales, you don't get to hide from that. But it's also really easy to get ranking and evaluation schemes wrong, so I understand why people would be nervous about it and prefer soft-skill-based evaluations instead.