Hacker News Reader

Comment by smallnamespace | Hacker News Reader

How do you distinguish between unethical science and A/B testing, or even just 'We changed our product, then didn't like the results, and changed it back?'

View on HN

There is at least a dense 50 years of research and writing on the topic of scientific ethics, it's a bit naive to pop into a thread and ask, "so what's the deeeeaaaal with human experimentation?"

To be clear, I think the question is where the line is between 'experimentation' and 'just doing what people do normally'.

Concrete example: I am in the process of trying to hire people right now. I'm not sure what the exact best way to do that is, so I vary how I interview over time and see how well it seems to work. I record what happens in a spreadsheet, and then later on I look at the 'data' to make some judgments.

Am I experimenting on my interviewees? Am I potentially harming them?

Well certainly if I ask question A in one interview, and question A' in another, and question A' is harder and I don't pass that person, then one might be tempted to argue that maybe that person would've done better on question A; therefore, I've harmed someone by 'experimenting' and making it harder for them to get a job. Rejecting someone can certainly make them depressed.

In principle this is no different from A/B testing an interview process, using real live humans no less.

So do I personally have an ethical obligation to disclose to everyone I interview that I am conducting human experimentation on them? Or is the only difference in power and scale—I am just one person, but Facebook can A/B test on millions of people at once?

> Well certainly if I ask question A in one interview, and question A' in another, and question A' is harder and I don't pass that person, then one might be tempted to argue that maybe that person would've done better on question A; therefore, I've harmed someone by 'experimenting' and making it harder for them to get a job. Rejecting someone can certainly make them depressed.

> In principle this is no different from A/B testing an interview process, using real live humans no less.

> So do I personally have an ethical obligation to disclose to everyone I interview that I am conducting human experimentation on them? Or is the only difference in power and scale—I am just one person, but Facebook can A/B test on millions of people at once?

Generally, when interviewing, we ask the same questions of each applicant. We adjust the questions from A to A' as positions are hired, and pools are assembled, but for a specific job, each candidate gets the same questions.

It would have to be different for open recruitment not tied to a specific job, but I haven't had to do that.

There is plenty of gray area in human protections law. Source: I'm a researcher who spent most of last week dealing with lawyers in multiple time zones on exactly how to write a bit of protocol in order to preserve the grey area (so as to avoid setting unnecessary precedent), while honoring the more stringent interpretation (to avoid even the appearance of skating onto thin ice).

Four years ago, having talked to some of the same lawyers about the same topic, none of us had any thought that it was a grey area. An a priori interpretation of the law was pretty clear: a more liberal position was (and is) perfectly legal. But the most anxious minds tend to prevail in these matters.

I enthusiastically support requiring Facebook to have an IRB greenlight any user impacting changes they want to make.

That's a terrific idea.

https://en.wikipedia.org/wiki/Institutional_review_board

There is plenty of gray area in human protections law.

Of course, which is why the conversation continues. If it was settled at the beginning without nuance, it would be defined in black and white.

I'm not sure how your reply relates, though, since I was merely imploring GP to demonstrate any awareness of the state of research rather than jumping in with a lazy question-comment.

Elsewhere this is expressed by the slogan, "I'm 12 years old and what is this?"

Well if your study involves depressed people unknowingly being manipulated with negative newsfeed items that they didn't know they consented to being subjected to in a secret study, then that's unethical. The scientists aren't stupid, they knew beforehand that in a sufficiently sized population, there will be a segment of depressed people that they are influencing to commit suicide by being subjected to increased negative newsfeed material. That is unethical to me any way you cut it.

I would assume a core tenet of ethics and deciding if something is unethical is if it has the potential to harm people. Inducing depression to indiscriminate masses is pretty high on the list. It's not valid to compare the emotional impact of Facebook on millions of its addicted users to something like the color of a box of laundry detergent.

Selling cake has the potential to harm people, a moral framework needs a bit more subtlety than that to be sensible.

do folks "sell cake" to see if it worsens people's depression? this is more like selling cake that secretly has nuts in it which are known to cause allergic reactions without disclosing it.

Well, it's more like if the nuts had an unknown reaction, and the idea was to figure out what the reaction was by slipping it into some cake, and then observing.

which would be massively unethical if not illegal to do as a food manufacturer

Well, including laxative to judge how much people can tolerate before they get the shits isn't far off the mark from inducing depression.

FB - A little negative = meh

FB - More negative = major downer

FB - Totally negative = woah, look at the sorry state they are in now.

Cake - A little laxative = mmm this cake is special somehow

Cake - More laxative = nice cake, special, where is the roll of extra TP?

Cake - Dump it in = people home for days, the shits.

In both cases, not telling them is the ethics problem.

Basically the public trust boils down to people expecting others not to harm them. Depression can become chronic. The shits could damage someone requiring medical help.

Both are clear risks people would very likely avoid, if they knew.

FB - We want to run a depression test. Plz volunteer your feed and see if you get depressed.

Cake - We have a laxative that tastes great, but you might get the shits, plz have some cake.

See the problem?

I think one good discriminant is: Would you want to explain it candidly?

E.g.: "Starting next week, we'll try to affect your mood as part of an experiment". Or "the new version of our product purposefully tries to depress you a bit" followed by "we didn't like the results of it so we've changed it back".

The terms don't apply to the same context. A/B testing is a specific technique (well, not that specific) which can be used to do ethical science or unethical science.

Medical trials are also just A/B testing. waves hands

true, but most of them have opt in. . .. Like people know they are in the test even if they don't know if they are the control or not.

That was my point, apparently that was not obvious at all. That is what I meant by the "hand waving" part. Being apologetic about FB just because what they did was some kind of A/B testing does not redeem them at all.

Reply on news.ycombinator.com