Toxicity on Social Media
thenoisyroom.com
thenoisyroom.com
For most of its history, Reddit didn't have an algorithm that promoted stories beyond upvotes and time since posting, that might even still be the case. Despite this, we have still seen a steady trend toward extreme views on the platform. To be fair, it has the reputation (at least in some circles) of being the most redeemable of the major social media platforms, probably thanks to the simplicity of its algorithm. Unfortunately, that's not saying much, it's a low bar to clear. What explains the polarization of Reddit in the absence of a bouncer amplifying extremism?
I think there is a significant percentage of users that do not initiate extreme content but participate in amplifying it. They may even find it problematic, but they really don't like the extreme views they hear on the other side. Or maybe it is the content they came to see out of morbid curiosity, something I am guilty of sometimes. The bar is so crowded because people find it preferable to the empty one down the street that has the expectation that people behave respectfully.
Incredible presentation, but I think the awareness we need to spread is a movement away from social media in general. As a social outlet it is generally incompatible with healthy social functioning and individual wellbeing. Face-to-face interaction has inherent guardrails for avoiding these problems and supporting the kind of social experience that we are really looking for.
id be seperating social media for any related analysis, as two major milestones demarcate distinct usage patterns (first algorithms, then LLMs). imo those factors would influence the discussion about as much as a platforms inherent construction.
toxicity has evolved over time, we have progressed from mere keyboard warriors, to nation states delivering propaganda campaigns with a click, now finally half the internet is bot activity.
Regardless, I think it's fairly clear that twitter is as manufactured as it ever was.
There were 3 conditions that were working and were removed very very quickly
1) it was a web application only. Which enforces an interaction that is more contientous
2) it skewed older. Compared to other sites like IFunny or Instagram the age profile was closer to 30 than 12.
3) the upvote/downvote mechanic was used to upvote relevant content not something you agreed with. And downvote to drown overused jokes, lack of nuance posts etc.
But in 2020 reddit destroyed 3rd party apis and went full head on the app.
Age plummeted, app useage is mroe casual than laptops and length of posts went full brainrot and lastly there was no enforcement to teach people what upvotes meant. So it became thumbs up or down, and the jokes went from heavily downvoted to always the top comment.
150 million users in 6 months is the death of any conversation and reddit did it on purpose to try an IPO.
Honestly, I think that more nuanced setup may help limited toxicity quite a lot. If there's no general upvote/downvote option, people might have to actually think why they like/dislike something rather than treating the system as a "I dislike this because I disagree and everyone should think the same way I do" setup.
It's why I quite like the reaction systems some forum scripts have. Yeah they're not perfect (many still have like/dislike options by default), but giving users reasons for why they upvote/downvote a post makes things a lot more meaningful. I also quite like how for some of them, agree and disagree don't actually change how the post appears or count as a rating. They just exist so people can see how many people agree or disagree with something and that's it.
Length of posts has plummeted, "meme" content and "twitter" like language was repudiated while now its basically the main mode of communication.
There used to be "famous" usernames, not everyone agreed with them but most people considered their input valuable, ending perhaps with the famous Unidan incident.
I would admit that having been in the site for 15 years the degradation has been continuos and small communities were much better than default subs from the get go. But the Eternal September post App release has been irreversible and made the site culture absolute trash
Reddit is heavily botted, including by capital interests, and has been for a long time. This includes basic up/down vote activity.
> I think there is a significant percentage of users that do not initiate extreme content but participate in amplifying it.
Yes, it's probably initiated by bots, and then real users are easily persuaded to follow the manufactured herd.
These issues are not exclusive to reddit, either.
Is that why /r/all is consistently anti-capitalism and anti-business?
See: wsws.org as a longstanding external example
Siloed groups are the easiest to monetize to advertisers, and they're also not a threat to a faux democracy.
rightward_ratchet++;
You can also not read it, you do you, but it's enlightening.
Alternatively, ask yourself, would monied interests be inclined to exploit these platforms? Of course they would. It'd need a solid explanation as to why they wouldn't.
No way, hasn't been true for many years. Try viewing the site from a few different people's systems.
The only point I'd add is that it's not handling time evolution in wicked problems quite right. Agree that the noisy room is distorting the world in exactly the ways described. But what if we've been in there so long, and the world has become so distorted.. that reality itself slides towards the once-extreme positions? Easiest to see this with climate-change controversy since that is the way that sort of thing happens, regardless of whether you think it's happened yet. Cascade, phase change, and collapse don't just call a truce.
So you have to anticipate that, acknowledging the pessimist is actually right, and that systems are a real bitch. Then you point out that if we're already doomed, we have nothing to lose nothing by trying. Systems are complex after all, that's the whole problem.. so if we miscalculated on the doom, then bothering to try actually saves us. Checkmate pessimists.
Also someone once told me - "To get your voice heard in a loud room, you either need to be very tall or extremely loud." Tall in this context are rich and influential people. Basically money and influence buys you height in a loud room
This is a double edged sword. Lots of "thought leaders" on twitter have outed themselves as lacking in both thought, and leadership.
The whole debate could be summarized in a paragraph or two, but the social media environment is unfortunately curated towards diluted opinions (as you said) instead of nuanced ones.
All that to say I'm happy HN is still holding strong in terms of quality as compared to other platforms.
But I also want to argue against the range of understanding argument. Attention has a limit. Anyone who wants develop a deep understanding for any topic would do themselves a disfavor by trying to expand their range aimlessly.
We can't all know everything all at once, so we should just develop some common sense for the most important topics instead. Like "people generally good and against violence". We used to have that once, we can rebuild it now.
1. They're not as visible as the post they're attempting to moderate, so people just won't notice that there's more info.
2. Many people practice https://indieweb.org/POSSE so you now need to duplicate your reply across many separate social networks if you're going to match the reach of the specified content.
The nice thing about a plugin is you can associate the annotation with the underlying content by CTPH hash (i.e. the underlying tech for virus signatures) so it shows up wherever the annotated content shows up, regardless of URL and and with identical visibility since you're going by what appears on the screen not by whatever internal logic the underlying site uses.
Instead of bandaid-hack solutions leading to perpetual cat-and-mouse, why not build a citizen-owned platform from the ground up, as detailed here:
https://www.noemamag.com/the-last-days-of-social-media/
You would barely even need to advertise for it if it was obviously better than any of the existing corporate slop. It would sell itself, and the "profit" would be the end result that everybody can enjoy.
The people who run existing social media didn't start out evil, being in a powerful position made them that way.
I'll be rooting for this user owned thing to stay true to its goals, but if it's shaped like the other ones in all ways but its ownership structure, then I won't be expecting it to do so.
> The people who run existing social media didn't start out evil
Um, not all of them:
> On July 6 instant messages by a 19-year Zuck appeared on Twitter, along with a link to a 2010 Business Insider story about an exchange that took place shortly after the Facebook founder launched the social-media phenomenon in his dorm room.
> “Yeah so if you ever need info about anyone at Harvard just me. I have over 4,000 emails, pictures, addresses, SNS,” Zuckerberg’s message says.
> “What? How’d you manage that one?” a friend asks.
> “People just submitted it,” Zuckerberg responds. “I don’t know why they 'trust me.' Dumb fucks.”
> The Majority Goes Silent - When the majority of people looks at the feed and assumes they're outnumbered, people will often self-censor.
That's not the same thing, is it? Here the majority is, say, anti-, but they are being frightened by a noisy pro- minority. They're moderates in the sense that anti- is the conventional position to take. But they have opinions. (They could also be in the minority, and this fear of speaking up would still be a bad thing.)
Otherwise, if they're truly moderate, but are frightened into silence supposedly, what would they be saying if they dared? "Everybody listen to me, I have no strong opinion on this matter"?
Take immigration or refugees - the obvious thing is that you're either for or against it. But there's so many things in between, so much nuance, etc. And that takes reasonable adults to think and talk about.
> The Majority Goes Silent - When the majority of people looks at the feed and assumes they're outnumbered, people will often self-censor.
> That's not the same thing, is it? Here the majority is, say, anti-, but they are being frightened by a noisy pro- minority. They're moderates in the sense that anti- is the conventional position to take. But they have opinions.
I don't follow your argument (which is different to the one in the article):
There's a small noisy pro-side, a small noisy anti-side and a majority, but not necessarily a moderate majority!
The article doesn't say anything about the majority being moderates, does it?
> Otherwise, if they're truly moderate, but are frightened into silence supposedly, what would they be saying if they dared? "Everybody listen to me, I have no strong opinion on this matter"?
Not necessarily true; there's a noisy pro minority, a noisey anti- minority and a silent majority. Who know if they are pro or anti or equally split?
And even if they were actually moderate, they could see opinions like "everyone should have guns" and "no one should have guns", and keep their majority moderate opinion of "people should be allowed guns depending on whether they cross some objective line into dangerous or neglectful behaviour".
That's both a moderate and a majority position, and yet you won't see it expressed in a forum because all the noise is being made by the two extremes.
The argument you're making is that the silent majority must necessarily be moderates, but that's not a requirement.
Even when people do have strong opinions on a topic (and a moderate opinion can also be strong), most people have better things to do with their lives than to go around blasting their opinions to the world as a hobby. And the few in this camp that do are not very likely to be amplified by the engagement algorithms.
- how you does this handle the fact that a lot of accounts on social media platforms are bots that maybe controlled by a small number of people.
- how do we actually get this implemented?
(then they started having shorts, so I cancelled youtube premium)
In a perfect world it would be great to have a platform that allows open-sourced algorithms for people to choose from, although that's a crazy pipe dream.
I personally find them nigh unusable because of the lack of any kind of filtering. I am on HN precisely because it has a somewhat working post sorting system — a recommendation engine, as an activist who wants to get HN in trouble might say.
Anyway I doubt a regulation as such would fly under the first amendment. Recommendation is expressing and opinion and expressing an opinion is speech. If I think one post is better and deserves to be on the top spot, I believe I should have a right to say it without some guy in DC telling me to shut up.
Hackers might be interested to know that there's an "open questions" section at the end of TFA. Some of it probably wants simulation, some wants theorems.
Camel-ai pubs/frameworks might be related and useful, for example: https://github.com/camel-ai/agent-trust
Several model checkers also have primitives for working with common-knowledge. TFA puts it like this:
> Learning a fact changes what you know. Seeing it displayed publicly — where everyone else can see it too — where you know others can also see it, changes what everyone knows, and subsequently how they act.
An important piece of technical vocabulary, it really seems we need this to talk about a lot of problems lately. Here is Terence Tao talking about some related math for disinformation and politics ( https://mathstodon.xyz/@tao/114866548969775485 ) and summing it up this way:
> we barely even have the vocabulary to discuss, let alone analyze, games in which control of information is a major battleground.
He kinda means in general though I think.. probably we can find heuristics and crunch a case or two
It's the most disturbing thing I have ever worked on, there is much more out there than moste people realize and a lot of it uses deceptive dark patterns.
If somebody is interested in talking more about this or is working on similar things, always welcome!
What we are monitoring are deceptive patterns on a text or transcript level. Deceptive patterns can be things like information inconsistency in one post, context shifts in one post that are used to reframe something, or video patterns like fake statistics or fake headlines that are not consistent with the main content.
All of these patterns are actual science backed psychological manipulation patterns and they are consistently used in the most viral posts we detect. My perspective after one year working on this is that the average media literacy is even lower than we think and that we build an evolutionary system with the social media platforms that is optimized to increase the performance of digital manipulation actors.
I have the following questions, in no particular order - I'm writing this comment off the top of my head in stream of consciousness while procrastinating:
- Why did you decide on a new technical system/platform as the way to go? This might sound like a silly question, but one thing I've noticed when talking to techies who are interested in this problem is that it can veer into 'I have a hammer, look at all these nails!' The reason I ask this is one thing I've noticed in working with the average person (or even very educated non-techy sorts) is that they view themselves as having little to no agency when interacting with technical systems, and I worry that adding another one just further encourages them to outsource that agency (just believing you instead of randos on Instagram/TT). This is versus things like outreach through different channels, formalized educational programs, producing of children's educational material (teach the parents while they read to their kids, for example, which lets them set aside the ego of being lectured to as an adult), traditional/alternative media campaigns, etc. If it's just a case of that's your skill set and resources, fair enough!
- You mentioned in a sister comment the low rates of media literacy. I 100% agree. Do you have/have you found any good ways to handle the combination of high education/socio-economic class and low media literacy? I've noticed very similar patterns across education levels, but my peers with graduate degrees or some manner of social 'success' fully believe themselves to be media literate and in fact some of them could recite most of the deceptive tricks and point them out if asked. They still knee-jerk believe things that confirm their priors.
- Is there any educational focus on heuristics and ways that the average person can satisfice their way to something better than the status quo? A lot of effort in this area seems to assume some platonic ideal of an informed, rational citizen with plenty of time to dedicate to educating themselves/learning better habits. Because of this, they tend to be information dumps. In addition to the low media literacy, there are a lot of people (at least here in America - I can't say for your education system) who lack the requisite knowledge to understand what you're telling them. I know that could sound a little insane, but a lot of people can't manage hypotheticals or understand second-order effects. We've also got the studies about people's attention spans. Going through what amounts to paragraphs of psychological text (or video) presented in a factual way will make people scroll or their eyes glaze over, but actions they can take in their life (e.g. stopping social media use for a month and noticing how their thoughts change, specifically following a small group/topic that you don't belong to/have much interest in to see how conversations change over time without personal investment, etc.) might be more approachable. Right now, the two approaches seem to be 'let experts educate you so you can learn a byzantine system in your 45 minutes a day of free time' and 'just go touch grass/let's go back to 1990'. I don't think either of those are realistic for the average person.
>We Could Do This Now - Platforms already have a lot of these capabilities. They already survey users. They even know how to run sophisticated polls. There are a few technical details to work out (spec here), but this is not a hard problem to solve.
Why do you think something like this is not already implemented? Platforms literally profit from this division, so why would they be incentivised to do anything? What's needed is not a good gesture from the overly powerful platforms, is fast, hard and deep regulation.
And the money decides how to run the circus. Not for the benefit of all.
So it is a really hard problem.
It's obvious from the hyperbole around the discourse alone that this moral panic has reached levels of derangement that far outclass any rational basis for judgement.
Does social media have negative consequences? Sure. Are people assholes on the internet? Always have been. Is social media the greatest and most existentially perilous evil ever conceived by humankind? No.
I think in ten years people will look back at this (on whatever strictly censored and regulated internet replaces this one) with the same bemused confusion as we do the Satanic Panic. And honestly in forty years, if technological civilization still exists, we'll find out how much of that was stoked by the CIA or other interests.
I mean wtf. Is this your parody account?
Every bit of hyperbole I mentioned is practically quoted verbatim from some thread or another here, it is what people believe, and you can't even bring yourself to approach me in good faith because I've committed wrongthink by defending the existence of social media even implicitly.
The CIA and other governments are running influence campaigns across social media. The links between the major social media platforms and intelligence agencies are well known and well documented. And civilization is threatened by numerous factors, such as our over-investment in AI and the mass deskilling and destabilization that will create, creeping fascism and increasing political violence in a multipolar world, climate change leading to mass famine, pandemics in a post-scientific age, etc.
But people want to destroy social media (and by extension, want to destroy the freedom of communication it allows) rather than bother to consider that the real problem is the same problem we've always had - government and corporate interests trying to control our lives and manufacture consent through fear and panic.
They ran the same playbook prior to social media but the process was so normalized because they controlled so much of the media and culture that no one really even noticed it. Now people notice but they can't distinguish between the symptom and the disease.
Congratulations on the endorphin hit. You really zinged me. I need to find where the grownups hang out.
however i agree that the CIA and other governments are running influence campaigns on social media. i think that's been proven actually.
the answer, as always, isn't 'destroy decentralized communication' or public discourse online. it's to have tighter regulations on how algorithms are configured. what's pushed vs. what's suppressed because it's obviously intentionally inflammatory/trolling.
this is an issue requiring extreme nuance. but to say that being worried about how social media today affects society is like 'the satanic panic' is kind of absurd.
1. insanely low-effort to post 2. requires NO discernment, proof, credibility, or peer review to post 3. 'viral' in that opinions circulate because other people have interacted with them, not because they are right or meaningful. so bad news, good news, real news and fake news all travel at the same speed, lowering discernment even further 4. echo chambers are baked into the form. people are more likely to interact with content they agree with vs. content that is true or impactful. this creates circles of people agreeing with each other on increasingly niched-down topics.
it is extremely different from newspapers and television.
You aren't listing problems intrinsic to social media per se, so much as how people choose to use it and how specific platforms choose to operate. The latter of which is a problem when Twitter, Facebook and the like optimize for engagement through controversy, but I think when we focus on social media as a whole we risk throwing the baby out with the bathwater in restricting human rights and the ability of people to network and communicate freely without interference by state interlocutors.
“The medium is the message”.
This stuff’s been around long enough we’ve got a pretty good idea of what its “message” is.
What's this mean?
It's a lot easier to not have to delineate the myriad of effects that each platform has on its users and issue broad-sweeping legislation that will have consequences on how information is distributed and how people can interact with each other according to how information is spread.
We're dealing with technology capable of containing various kinds of media at a scale far unlike what McLuhan observed 60 years ago. That's not so say that he's become obsolete.
Read this and let me know if I'm getting it wrong...
https://web.archive.org/web/20060605204535/http://individual...
Is traveling to Tokyo just to sprint across the Shibuya Scramble for a slightly less-crowded Instagram selfie really a model of the good life? Should someone like Zuckerberg have this level of control over the activities and minds of the human race? Is Mr. Beast a role model for children by industrializing the exploitation of human virtue?
Human social pressure and follower mindsets are part of the human experience but systematically gaming those instincts in real-time so money flows to a social media company at all costs in some strange digital sharecropping scheme is what’s new and the hierarchy of others trying to capture a small piece of that pie creates these distortions.
> Human social pressure and follower mindsets are part of the human experience but systematically gaming those instincts in real-time so money flows to a social media company at all costs in some strange digital sharecropping scheme is what’s new and the hierarchy of others trying to capture a small piece of that pie creates these distortions.
To what I think @krapp's point is: these dynamics are not exclusive to social media. At their core they're led by something far more primal than what social media only exacerbates. Governments are not as naive as the general public. Regulations effected in 2026 to "regulate social media" could have consequences on how information is spread among people in 2040.
Which ones?
1. cheating or being lazy with the sampling
2. Being a weasel with the phrasing to get the desired result
3. Being a push poll.
Still, a "trusted" poll is slightly better than a freeform "community note", especially if it sticks solely to how prevalent an opinion is.
Slashdot used random sampling in moderation 30-ish years ago. It worked OK, except that scores were used for very little (crucially they didn't even sort by them), and they had a more gameable non-randomized system to moderate the random system. And of course it was probably vulnerable to Sybil attacks.
(By the way, I guessed 4% for the number of toxic users)
- A fair number of LGBTQ people don't feel represented by either party and so wouldn't answer yes to aligning with either the Dems or the Republicans. More communists, anarchists, third-party voters, etc.
- Voters tend older than the general population, and a lot of the LGBTQ elders are either dead or not attached to the community. The biggest demographic in the LGBTQ community by numbers are the bisexuals and a lot of 65 year old bisexuals went through their lives acting and living as straight. Likewise, a lot of older people with gender dysphoria never knew being trans was an option, so they may never have identified with it.
I feel like the real problem is the people. Many of us just want to be told what to think to blend in with society, some of us demonstrate Dunning-Kruger publicly and a few of us really want to drive the polarization for clout and attention.
Everyday I see people promote increasingly stupid ideas on both sides, further pushing my believe that the only solution is to severely limit what government can do, therefore making all this discussion pointless.
It's an interesting initiative though. One that I also think could have unintended consequences that would additionally seed greater distrust in the media—which isn't necessarily a bad thing. But I imagine that the people who already sense this distrust and distaste toward the impression of polarization that the media gives are becoming less and less likely to subject themselves to the nude opinions of anonymous strangers online.
There is a fatal flaw with the solution though: inauthentic users.
The solution aims to reduce the distortionary effects of social networks in a market of ideas and conversations.
Inatuthentic users have deceptive participation. There are 2 types: Trolls and organized/motivated inauthentic users.
These users have a strong incentive to protect the distorted perceptions in the exchange, and will adapt to reduce the effectiveness of Community Check.
(I have a theoretical solution for it, however I am not quite sure how to test it out)
That surprised me quite a bit, since the national average is over 9% according to Gallup, and considering demographics (younger people tend to lean Dem and have a higher LGBTQ rate than the average population) that figure is certainly wrong.
This is showing how in the social media system the dynamics play out.
Both Democrats and Republicans estimated 30% but actually.. only 10% of both sides supported political violence
That number is crazy in so many ways and the post is overly nonchalant about it. The "distortion" isn't what's worrying here
I just had an issue with the way that number was completely overlooked
These people are unwittingly working for the platforms to drive engagement, often to the exclusion of any goal they might've had before the addictive aspects of social media kicked in.
I think we get less of this kind of behavior here on HN because each username is not bedazzled with metrics. You can see up vote counts for your own comments, but you can only infer those counts for others. The scoreboard is hidden, so it isn't triggering as much bad behavior from people who can't handle such things.
I think we could get even better behavior out of people if we never showed them raw counts of updoots, but instead only showed them metrics relating to their explicitly stated social graph, plus maybe one hop out:
> Alice and two of her friends like this
> Charlie likes this
It gives a sort of directionality to the feedback. Instead of seeking the high score as granted, likely, by a bot army, you learn something about Alice's corner of the social network. Maybe you should get to know Alice's friends better.
Because that loud 3% that are being harnessed by the platforms to drive engagement via content we all hate... Their primary sin is just that they fell for it. They're like alcoholics, if we want to help them into a mode where they're less problematic, we should hang out someplace besides the bar.
The tiny minority dominates the feeds because that's how the incentives for algorithmic driven social media are structured. Do we really expect Meta, X, TikTok to anything that could reduce engagement?
Good luck having any of the mainstream social media apps add the banner they propose.
Anyway, social media is dead, has been dead for quite a few years now, the majority of us are out back there touching grass, it’s only the fringes (on both the political left and right) who’re still obsessed about it.
The part that annoys me about the toxicity, or repetetive and annoying topics on reddit, HN, etc. is not that I am unaware that the content is produced by a small fraction. (I underestimated the count! I guessed 2%)
It's that people espouse it: They upvote and retweet it.
> Both sides develop wildly inaccurate beliefs about who the other side actually is.
That was a guess I had for a while. People have a strawman version of their out-groups in mind and quickly map people to that if an unknown person says something that indicates they might be part of the out-group.
> What percentage of the other side supports political violence?
It would be interesting to see the in-group statistic as well: "What percentage of your own side supports policical violence?", in my experience people also justify very shitty behavior as long as its from their in-group. (This plays heavily into the first point of espousing all kinds of shit)
---
It would be interesting to see if the community check actually changes anything. But the actual data seems to be only possibly for very generic topics - those we have the data on already. Something that would not be available for daily-fresh topics.
For my personal sanity I simply left reddit and stopped opening comments on certain HN posts - of course that does not help with the societal problems. Unfortunately.
I think something that is not calibrated in the post and also missing in this reply is that believes and actions do not need to be aligned.
Both groups say around 10% of members support political violence, however no democratic president is pardoning wholesale domestic terrorists. And the 90% of republicans who condemn political violence are not repudiating, removing themselves or condeming the fact that far right groups are the most dangerous demo according to the FBI, or that most political violence occurs in rep states, or the direct correlation of the NRA infiltration into rep campaigning and mass shootings...
Like if you say you dislike violence but defend the system that creates the violence and pardon the people who commit the violence and share the table and take the money from the violent people... your "beliefs" are not worth much.
The whole conversation about out-groups is less relevant when discussing left wing policy due to the fact that it is not orchestrated AROUND in and outgroups. Right wing ideology is de-facto a ingroup political theory where some people must be excluded. When you add morality being justified due to being in group you end up with some very concerning politics where actions are judged on beloning to the group and not the morality of the action or the consequences.
See the blue collar protect the children anti abortion crew voting for a new york millionaire owner of a beuty pagent who was best friend with the worlds best known human child trafficker...
The believe system collapses the second you put the right tee shirt on, and that is what makes polling those people irrelevant. They simply will support whatever is in front of them as long as they belong to the in group. War bad in ukraine, war good in Iran. Taxes bad in 2018, tariff taxes good now. Sillicon Valley tech people all leftwing indian soy boys in 2016 now all alpha podcast ai cool guys who fund our president.
nothing matters as long as you wear the tee shirt
Lobste.rs in these regards are better.