Comment by saagarjha | Hacker News Reader

Comment by saagarjha | Hacker News Reader

Parent Full thread

A/B testing is the worst. I'm always on the bleeding edge of products I use and extremely tolerant about changes and regressions–to the point the browser I am using crashes every couple of hours, likely due to a longstanding threading issue–but I just cannot stand constant A/B changes. If you change the font of your website even once, I will know. If you tweak the spacing of some elements, I will know. If you change a color slightly, I will know. Even if you pretend like nothing changed at all, I can tell and it drives me out of my mind because something seems "off" and I must find what it is. I spent half an hour once searching for why Hacker News's text area font seemed to have randomly changed from Courier to Menlo before emailing 'dang in desperation to figure out what was different (turns out it was a new lang="en" attribute). Please, don't change things randomly and silently.

I understand the frustration, but A/B testing is one of the more objectively tools we have at our disposal.

While there are good arguments against A/B testing UI changes and doing p-hacking, much of the modern web's current UX and UI improvements are in part due to this. How else would we truly know what affects user on a broad scale?

In theory, yes, A/B testing sounds great. However, whenever I see it implemented it invariably becomes some sort of annealing process for a metric that doesn't actually make the site more pleasant for end users ("engagement", usually) and you have people shipping apps with seven different UIs in them and users who are accustomed to features moving around and disappearing completely unpredictably. As I was searching for a picture of "Courier" Google gave me a completely different "cards" UI that went back to how it was before when I searched "Menlo". My last couple weeks with Slack on iOS have been a nightmare as it constantly switches between its fairly decent UI and some new abomination depending on which workspace I'm in and the phase of the moon. It seems like nobody really knows how to do A/B testing properly. (Perhaps the companies that do are doing it in such a way that I cannot notice. But there is ample evidence of a lot of products where I can.)

Reply on news.ycombinator.com