Hidden Secrets of the Amazon Shopping Cart
grokdotcom.com
grokdotcom.com
These meditations on A/B optimization reminded me of a genetic algorithm-based A/B testing framework mentioned not too long ago: http://ejohn.org/blog/genetic-ab-testing-with-javascript/ (I'd link to Genetify itself, but there's no homepage, just the demo)
An interesting thought: Why hasn't this caught on? Machines are good at exactly the things A/B testing aims at: measuring performance, permuting a test-matrix, and reporting on the results. This is not to say Genetify is the answer, but rather that I rarely hear of conversion optimization from developers.
Another reason is that proper optimization requires a good amount of traffic, without which your A/B tests would never achieve statistical significance.
That said, optimization is pretty popular with online retailers and big web companies from whom even a 0.1% change in conversions/signups/etc can result into addition of tens of thousands, or even millions of dollars of revenues. For small-websites, the benefit of 0.1% may not be obvious. Hence, not much motivation for investing in optimization.
Having your users vote is definitely the best way to make decisions. Lately, when my cofounder and I disagree, I say "let's build both versions and ask our users which one they like best." (in most cases, the differences are trivial enough to make this feasible).
Second Lesson (one I feel is more important in general): Don’t copy what other people do if you aren’t fully aware of the business issues involved.
"Back then, ... Amazon, for the most part, sold books."
I just consider them a generic shopping mall now.
All very well, but sometimes I wish they would spend a bit more time on their wishlists - not much has changed since the beginning and some more effort in this area is overdue by now.