Dear Software Makers

Yep, and A/B testing as experienced by uninformed, unaware end-users is a dark pattern.

It undermines the perception of (and trust in) continuity which is necessary to make effective use of a tool. The best way I can describe it to the skeptical is: imagine the dials on your car's dashboard rearrange themselves occasionally overnight, and on some commutes to work you suddenly can't work the radio or the AC while moving at ≥35mph. Of course, since the widespread use of touchscreens, that example became very literal.

So the car manufacturer has figured out the "optimal" arrangement of dials and buttons on their dashboard for their preferred levels of user engagement. Great. How many of those users now associate their car's brand with inconsistency? "I can't trust the damn buttons to be in the same place the next time I drive."

There are small A/B tests which absolutely make sense. You often see marketing sites making small tweaks to banners and copy. It's not that one has worse UX or even that one is objectively worse, just that different users have different preferences and it's often difficult to know exactly what will work best.

Similarly you can be very confident of something, but A/B testing it still reduces risk. Any significant change should probably always be rolled out to a small fraction of the user base first in case you accidentally change something for the worse.

I agree if you're talking about some BS experiment where a company uses A/B testing as an alternative to putting the hours into product design and user research.

Is it unethical to do phased rollouts (where a small percentage get the new version) as a way to do safe deploys? If the issue is that two users making requests at the same time might see different things, then this would also be unethical? Yet, these sorts of phases rollouts is the best way to release something safely. When I worked at a large CDN with 50,000 servers around the world, we ALWAYS did phased releases, to make sure we didn't take down everything all at once, and to make sure we caught any performance regressions right away.

Is your issue that the user might be getting a version that won't stick around? That seems always the case, whether you do A/B or not. You might rollback if there is an issue, and you will certainly roll forward at some point, meaning users will get a new version at some point.

Would it be an issue if the A/B test was temporal? Like all users got one version today, and a different version tomorrow?

I guess I am just confused by this statement:

> If you don't have enough confidence in your changes to make them carte blanche, then don't make them at all.

This seems contrary to so many other complaints we see all the time, that companies push changes out without taking into account what users actually want. So, do we want companies that push out changes with no user feedback because they are confident that they know what users want, or do we want companies that get feedback from users on whether new changes are helping or hurting.

(I have this weird feeling that you're not upset with blue/green deployments...)

> Users don't want their shit changing all the time.

Eliminating A/B testing won't solve this problem. Even without A/B testing, they make updates, etc.

You might as well just say "Updating an online service without asking the user first is unethical."

Oh, and how much are you paying for that service...?

Look I get it feels weird but in practice most A/B tests are stuff like “does this copy change if ppl use this feature”.

The reasons they don’t is the same reason RCTs for new drugs don’t tell patients either. You end up with selection bias.

> If you don't have enough confidence in your changes to make them carte blanche, then don't make them at all

This is a bit hyperbolic, empirics is something that should be used more by decision makers not fit their own sake but for people who don’t understand why they are making them, especially in government (although It’s harder because finding cases where it’s appropriate is hard).

An A/B test isn’t just about what’s better, it’s about understanding all other things being equal how does one change to X affect Y.

A lot of ppl have bad takes on what makes a product better, and they would otherwise have a greater say in the product design.

Some product managers are just really stupid and are there due to nepotism so it’s an external equaliser and allowing the thoughtful ones to have more of a say.

> Users don't want their shit changing all the time.

Yep that’s why you don’t ask them.

When these products operate in competitive markets and not some winner takes all market these are often about improving users experience.

If this was something more high stakes like a medial trial I’d get, but for stuff like filling out a document or watching a piece of media. The stakes for most SASS app are really low.

Just sounds rubbish both for viewers (look how terrible titles and thumbnails are after a/b tests) and uploaders alike

Youtube certainly took a lot of unpopular decisions lately, but being able to stream any hi-res video ever posted in an instant and for "free" is a miracle. I will be happy with youtube for as long as content creators I care about are happy and I can get their fresh content via browser, yt-dlp, or other means.