LinkedIn ran social experiments on 20M users over 5 years | Hacker News Reader

LinkedIn ran social experiments on 20M users over 5 years

nytimes.com

32 pointssizzle32 comments

Framing A/B testing as "social experiments" for shock value is disingenuous, but it is interesting to think about the real-world knock-on effects that recommendation engines have.

In this instance, recommendation engines helped people get jobs. If you're a relatively unknown musician on Spotify, being "blessed by the algorithm" and put into algorithmic playlists associated with more popular artist translates to real-life streams and income. If you're an unknown artist pushing work on Twitter or a wannabe "influencer" on Instagram/Tiktok/YT, showing up in the algofeed and getting a bunch of views, likes, and reposts could be what makes you "go viral" and start your career.

There's a cottage industry of people making a career out of posting content on platforms, or trying to, and one of the biggest factors is whether or not these opaque, vaguely understood algorithms happen to select your content to be recommended. Crazy as it may seem, deploying a retrained recommendation ML model could impact whether someone makes rent this month.

>Framing A/B testing as "social experiments" for shock value is disingenuous,

Is it? I've only done minor human experimentation, but had to go through the IRB process. What I did was on par with simple AB testing. The only difference here is that the AB testing is not being done to be published and not under the purview of the IRB. If it were, they would require the same sort of approve for human experimentation. Sure, the paperwork is nothing like a drug test, and I think the IRB is extra heavy weighted on the side of seeking too much approval, but there is a history of human experimentation to explain why that is the case. That new unmonitored experiments are happening under tech employees running loose instead of scientists running loose doesn't really indicate any significant difference and we should be watchful that similar abuses of power as had previously happened don't reoccur. I think we've seen enough with facebook and their influence on children to know it is a real concern and we might already be far enough along the path to be worth raising the alarm.

IRBs are a substantial break on scientific progress today, and need massive reform.

Instituted to prevent repeats of the Tuskegee Syphilis Study, they now claim (slow) pre-approval rights to survey questions.

To me, it shows that gate-keepers are going to gate-keep. Institutions always strive to expand their powers.

Consider a slot machine company simply deploying an A/B test on the literal bells and whistles of their latest one armed bandit. They find that one setup increased any-word-except-addiction to a higher degree than the other, and so went with it. Would you consider this a social experiment?

I ask because I am trying to understand where your objection is coming from. And if you would consider the above a social experiment, might it not simply be desensitization? A/B testing has become a somewhat regular part of 'engagement optimization strategies' at many companies which sounds perfectly vanilla, yet in many cases engagement is just the software word of choice in the above fill-in-the-blank.

NYTimes runs social experiments on users for years:

https://open.nytimes.com/how-we-rearchitected-mobile-a-b-tes...

This is mentioned in the article. But yeah, it is ironic. And given that newspaper headlines are pretty much all some people read, I wouldn't say it's any less harmless than suggesting weak ties for your linkedin network.

Did you read the article? It's right in the article:

> The New York Times uses such tests to assess the wording of headlines and to make decisions about the products and features the company releases.

I read it on Saturday, and it hadn't mentioned it then:

https://web.archive.org/web/20220924090459/https://www.nytim...

Looks like they realized they got caught and updated it. TBH, I could 100% believe that the reporter and editor had no idea the NYT did it themselves, because anyone would be clued in enough to understand that wouldn't have written the article in the first place.

https://news.ycombinator.com/newsguidelines.html

> "Did you even read the article? It mentions that" can be shortened to "The article mentions that."

> The Times also makes a practice of running what are called A/B tests on the digital headlines that appear on its homepage: Half of readers will see one headline, and the other half will see an alternative headline, for about half an hour. At the end of the test, The Times will use the headline that attracted more readers.

https://www.nytimes.com/2017/03/23/insider/headline-trump-ti...

Every company with algorithmic recommendations will run experiments to try to improve it according to various metrics. This is the well known knowledge in the industry, not a conspiracy theory.

Trying to improve peoples chance of getting a job as opposed to something lame like "engagement" is laudable. The key question is how big the difference between the groups. If the difference is marginal, then I think the knowledge justifies it. This will lead to more people getting jobs in the future, so the total utility increases. But if it was night and day, and they really fucked some people over, then it was unethical.

> The key question is how big the difference between the groups.

2x increase in job opportunities.

> They found that relatively weak social ties on LinkedIn proved twice as effective in securing employment as stronger social ties.

> A year after connecting on LinkedIn, people who had received more recommendations for moderately weak-tie contacts were twice as likely to land jobs at the companies where those acquaintances worked compared with other users who had received more recommendations for strong-tie connections.

That's one piece of the puzzle but not the whole thing. It's not the same as saying they got 2x more job opportunities. People can add friends outside of the recommendations for instance. Or they may be added by the other person. I tried to quickly skim the paper just now but I was unable to find the details.

True, but wouldn’t both cohorts be expected to add connections manually in roughly the same amounts?

This sort of knee-jerk reaction to a paper is exactly why companies at large prefer to treat their R&D as 'strictly confidential'.

Can you imagine how much we could advance if the whole of Industry contributed and published their 'non-secret sauce' research in the same way academia currently does.

I think the only real issue is summarized here.

>“The study has an inherent bias,” Dr. Flick said. “It shows that, if you want to get more jobs, you should be on LinkedIn more.”

>Can you imagine how much we could advance if the whole of Industry contributed and published their 'non-secret sauce' research in the same way academia currently does.

How much faster would scientists produce findings if they only needed to follow laws and didn't need the IRB unless they were looking at experiments that might need legal exceptions. There is a reason we slow down scientists, now the question is how do we handle people doing similar but under a different name and funding model.

That's a bit of a non-sequitur. My point is that there is a trove of data and studies that each company's has to keep in a vault.

Not all of it is strategic or secret but companies rarely publish by default.

Note that I'm strongly opposed to a lot of practices by social media networks (I believe LinkedIn falls into the category these days), but I guess any A/B test that they run could be considered a "social experiment on 20M users".

Unsure if the particular tests mentioned are worthy of attention - think there are probably other more relevant controversial practices.

This doesn't really bother me. As noted, the T&Cs specifically say your data may be used in research. I'm also not entirely sure how they're supposed to improve recommendation algorithms — which we're all clamoring for them to do — without running some form of experiments with them. The question to answer is, did they do it as ethically as they could? Just based on this article, I don't see any unpardonable sins.

Linkedin is a private company and they're under no obligation to offer you any quality of service unless you sign an SLA.

Non Paywall Link: https://archive.ph/v2ehO

Thanks!

I think this is extremely common for social networks to experiment with this kind of things. Trying out various blends of user experience, social circles and so on to see what best converts users to premium members.

Not saying it's right, just that it constantly happens.

It should be easier to notify people involved in these studies that they may have be adversely affected in the course of the study, rather than having to go decipher a peer reviewed paper on some specialist academic journal site, for the common user at the very least.

I'd imagine the terms of service we all agree to 'notify' us of such consequences.

I'd imagine the parent poster is asking for something a bit more noticeable than a line or two in the ~6800 word ToS.

Looking at the current state of GDPR compliance I’m not sure this would actually be a good experience for most people.

I ran a ethical experiment and avoided LinkedIn.

I gave up when I couldn’t see how to make my LinkedIn profile reflect the changes and evolution of my career. I’m not a monolith.

And beside that whole job market is a privacy hell when you really search for a job.

dang.

On HN, reposts of an article aren't considered dupes until the article has had significant attention. This is in the FAQ: https://news.ycombinator.com/newsfaq.html.

It's also not the convention to link to previous submissions unless there are actual discussions there. (That's no doubt why users flagged your comment.)

Reply on news.ycombinator.com