Ask HN: Have Amazon's A/B tests gotten more blatant?
I was just on Amazon (US') website to buy some pens. I haven't purchased from them in a while (maybe six months ago or so,) but I was bewildered by their jarring use of A/B testing (or maybe I was jarred by their bewildering use of A/B tests. Let me know which sounds better to you.)
I go about my usual routine of starting two searches ("Lamy Al-Star" and "Lamy Safari") and start opening a new tab for each potential purchase. After narrowing it down to maybe six pens or so, I start adding them all to my cart (where I can compare the prices directly and knock off all but the two I actually want.)
This is all in the span of around twenty minutes from start to purchase, but I'm noticing weird things. Going from one search result page, I enter a new query and all of a sudden the new results page has, after showing me three products (one horizontal line of products,) a pane where some "selected" products are being showcased from someone I've never even heard of before, followed by "More results." Right, more results--that's what I expected by scrolling down.
The products have the usual keyword-stuffing, but that's nothing new. What got me is the difference from tab-to-tab. ~Half of my tabs had a mini-cart-items-horizontal-space-taker-upper on the right and half didn't, in the same browsing session. Some of those tabs were even opened up from the same search results page!
Adding these items to the cart brought more inconsistency. Sometimes it would just bring up a little "added to cart" toast-like notice on the top-middle of my screen. Sometimes it would bring up a horizontal-space-taker-upper where there was none previously and show me my newly-added item. Once it brought me to an interstitial asking me (ironically) if I really wanted to add it to my cart. I couldn't make sense of it all.
Now, what finally drove me off the edge and spurred me to write all this blather is that during the checkout phase (directly after hitting checkout on the cart page, without my further interaction whatsoever) it thanked me for choosing to have my delivery to be on my "Amazon Day" (whatever hellish dystopia that is; maybe it's assigned to you at birth?) I didn't select that. I never would've selected that, and I in fact deselected that.
The (hopefully) constructive bit of all of this is: has anyone else noticed this? Like I said, I haven't been on Amazon's website for some time. Does it seem odd to you? I've never seen it being done to this extent on any other website.
I'm no stranger to A/B tests. I've seen my search results moved around a bit before, or the font color being changed, a button moved to a different spot, et c. I get that they bring a little scientific rigour to the otherwise-difficult-to-articulate design of web pages and when you're at the scale of Amazon, it's practically free data you'd otherwise be leaving on the table. But holy hell did that seem excessive.
Also, for anyone out there on the other side of the glass actually performing A/B tests, I'm curious: is there a "line," in/formal as to how drastic of a change you test? Do you keep moving whatever you're measuring in the direction that shows statistical significance? Do you know when to stop? None of that is criticism or rhetorical questions at all, by the way. I'm legitimately curious.
Anyway, I'm curious to hear everyone else's thoughts on the matter.