What would open sourcing the Twitter algorithm actually look like? | Hacker News Reader

What would open sourcing the Twitter algorithm actually look like?

transitivebullsh.it

142 pointstransitivebs131 comments

"The algorithm" is irredeemable. Its sole purpose is to prolong engagement so they can sneak ads in front of your eyeballs. Open sourcing it would not change its purpose and intended design. The design does not take into account that it can distort users worldview to the point of delusion and hinder or even set back progress of society if enough people are deluded in a group. I left twitter for the sole reason of that I saw a huge group of people feeding their worst tribal tendencies and celebrate the death of a human being. This is the very same mechanisms that has fed into pogroms and other atrocities through out history. "The algorithm" presented me with these celebratory tweets as a recommendation in my feed of only following highly technical people and projects with in software.

But what if "the algorithm" is turns out to be rather basic? Maybe it just gives people more of what they like. Filter bubbles, polarization and herd mentality could be a natural, inevitable consequence of any Twitter-like UI and a central database.

The algo surely is quite basic. I believe JanneVee is saying that a "rather basic" algorithm that tries to maximize engagement drifts towards promoting the more divisive posts that amplify our tribalistic tendencies.

And by giving more visibility to polarizing content it incentivises others to create more polarising content.

The algorithm therefore is a positive feedback loop of polarization. It doesn't just show what we are, but amplifies our worst (or tribalistic) tendencies.

Relevant Howard Stern reference: https://www.youtube.com/watch?v=9G6xu-J_Dmc

The problem is that I tried to put myself into a software development filter bubble but yet the herd mentality and polarization bled through the filter. It is not a single issue of "algorithm = bad". It is multiple issues intersecting with psychology. It is the subversion of willpower by hijacking the dopamine that you feel rewarded to keep scrolling, it is the fracturing the shared reality by presenting different content to different people depending on what they engaged with and a number of other issues... all this makes it a complicated mess where one set of people starve themselves to death because they think they are fat according to their feed and other where people eat themselves to death because fat is beautiful according to their reality. You can fix one issue but then the next issue props up and all of a sudden the complications of human psychology makes the unintended consequences even more dire. All this to make you watch ads and buy things.

Jaron Lanier’s claim is the feedback mechanism (clicks, engagement) isn’t nuanced enough [1].

i.e. If the algorithm had more insight into a given person’s emotional (and intellectual?) response, the outcomes could be better than the status quo.

[1] https://open.spotify.com/episode/3DBNkrAY5sCIX2sqHpZ567?si=M...

The word here is "could"... It can and has been used for outcomes that hasn't favored the users interests. And that is in my view a worse outcome than status quo.

I definitely hedged yeah.

> The problem is that I tried to put myself into a software development filter bubble but yet the herd mentality and polarization bled through the filter.

But you understand that technology have social consequences? Discussing them is vital part of discussing technology. "Moving fast and breaking things" is fun and all, but when you endangering peoples lives and wellbeing, maybe you should think twice about what you are doing.

Technology is a slave to capital.

Talking about technology as though it has any moral imperative other than "make more money" in the West is naive at best. At worst it is not only boring to anyone over 21, but it's counter productive to improving the world. Our newest toys are there to distract from the woman behind the curtain.

In short: let me read what I want and save your moralizing for the people who actually make decisions. None of which are technologists (any more).

>Technology is a slave to capital.

You mean it's slave to people controling capital? Because capital by itself isn't sentient or capable of making decisions.

>Talking about technology as though it has any moral imperative other than "make more money" in the West is naive at best.

Technology have moral consequences, not a "moral imperative". Technology doesn't care if it counts number of Jews, carrots or cans of Zyklon-b. But people should, and normal people do. Tehnology shouldn't be an excuse to do whatever you want, because "computer said so". Someone programed/tauhgt that computer. Someone is responsible for it.

>At worst it is not only boring to anyone over 21, but it's counter productive to improving the world.

Capital, corporations and billionaires are human inventions, so does kings, slavery, white supremacy, misogyny. Kings didn't stopped to be kings just because everyone asked them nicely. It took time but in the end, kings were no more. Because in the end they depend on ordinary people to do their biddings. Jeff Bezos doesn't check if drivers reached their quota of urin filled bottles, people (directly or indirectly) do this.

>In short: let me read what I want and save your moralizing for the people who actually make decisions. None of which are technologists (any more).

Not talking about problematic technologies, decisions made by people in power doesn't make them go away. Pretending bad things didn't happen is delusional, but just talking doesn't help either. Actions are also needed, even if they are inconvenient or dangerous. There are many ways change can be brought, protests, civil disobedience and more. But pretending everything is ok, will make every call to action fail, even before it started.

> But you understand that technology have social consequences?

Yes that is why I started of this whole thread with: > "The algorithm" is irredeemable.

My personal opinion is formed by tweets that where not about the social or societal consequences of software development but totally unrelated events from people I didn't follow.

Bingo. Filter bubbles are nothing new. Newspapers are lo-fi filter bubbles. Each newspaper occupied a different part of the political spectrum with its own biases. The concept of "following" is also a filter bubble. People will naturally follow other Twitter users with similar views, until the point where most of their feed consists of like-minded tweets.

We are independently blaming every social media platform for what is, fundamentally, a human problem.

Well the problem I'm trying to point out is that yes it is a fundamentally human problem. The problem is that a machine is amplifying this. There is a curation of content that is personal that may lock you harder in the filter bubble than a newspaper trying to cater to a wide audience. To cater to wider audience occasionally the newspaper will contain something that outside of your safe little bubble. It is a bigger chance that you expand and/or evolve your views because of this. Of course the option is to be closed minded, but that is a choice by the reader and not the social media platform which do it for you automatically to keep you using the platform.

You re making a good case about why twitter is overrated as a medium and should not be taken seriously. the vast majority are crazies, who don't matter. The rich men are interested in controlling it because there is a small minority in it, the journalists, who are using it as a measuring stick and as a way to coordinate their stories. While that's true, those journalists are not a particularly sticky crowd and will move elsewhere if some or other rich man thinks he can control them. I think substack is their next campsite, and it could easily launch a microblog timeline.

First, everyone matters.

Second, part of the problem is diving into the delusion by picking a side. One such side is men vs women.

I would argue that almost the entire problem with the internet(including Twitter) is advertising. All the problems stem from it.

I ve posted thousands of comments, they never mattered.

I don't even think twitter issue is "the algorithm", but the UI itself. The issues is not even bubbles, there is nothing wrong with bubbles, we even celebrate healthy bubbles thats bring fun, odd topics, etc. The issue is bubble shiftings to extremes with nothing to moderate it. As the bubble become more and more extreme, and showing their behavior in public, the behavior become "normal" and allows it to become even more extreme.

Getting rid of the "trending" block, with something else, like "trending tag within people you follow" would get rid of this normalisation of extreme behavior.

Then you need moderation.

Seeking engagement is not necessarly evil.

GitHub increased engagement, and it lead to more cooperation in the open source software, that's a plus for everyone.

The issue of a lot of social networks, is they want engagement, even if it cause addiction.

I think twitter social network "model" is salvagable in something healty, but it needs a lot of changes.

I use the Minimal Twitter extension to remove everything. I have a chronological twitter feet with no ads and no suggestions. I go in maybe 2-3 times a day for 5-10 minutes, scroll down and like/respond as desired, then leave once I recognize the tweets from last time (or I get bored).

I'd be happy to pay $1 or $2/mo for this directly from Twitter, but until that's an option I don't feel remotely bad removing my sliver of ad revenue from their bottom line.

The UI is like that because of the above commenter's point - the business model itself. It's not that Twitter isn't technically capable of tweaking the UI or giving you more settings to customize your experience, it's that giving you those settings would mean you use them against their own business model.

> Seeking engagement is not necessarly evil. GitHub increased engagement [...]

At this point it really depends on what you mean by "engagement". GitHub's definition of "engagement" is definitely not the same as Twitter's.

Technically, Twitter's goal is not "engagement" but "time spent looking at ads". It just so happens that "engagement" is a very good proxy for that. In Twitter's case, increasing "engagement" is definitely evil.

I also don't believe GitHub's goal is to increase engagement. Their goal is ultimately to deliver features useful to their paying customers - companies that build software, where as Twitter's (and any other ad-supported service's) goal is to serve more ad impressions (or at least make advertisers believe they do).

Twitter, and GitHub goal is not "time spent looking at ads" or "deliver feature" but "make money". I bet/hope that if Musk succeed in taking over twitter, he will implement a patreon style subscription, there is a lot of money there thats twitter leave on the table.

> he will implement a patreon style subscription, there is a lot of money there thats twitter leave on the table

I don't think that's any better than the current scheme; infact, it could be worse becasue the current ad system has centralized control which can check on extremist echo-chambers. I suspect Patreon-style funding will increase polarization[1], with the "content-creators" optimizing for what brings in the most money, not necessarily their thoughtful ideas.

1. With a side of increased brigading, coordinated harassment,doxxing and swatting; but that'll be the price of "free" speech.

Current ad system try to sell weapon related merchs to far right extremist, etc, I don't believe in your "centralized control".

I’ve seen the work that goes into spotting & mitigating TOS-violating ads: you don’t have to believe in it for it to exist.

You can learn a little about how youtube recommendations and facebook's feed work in this interview I did with someone who worked on both of them.

It's nothing magical. It's just a big system trying to find the things it thinks you are most likely to find engaging.

https://corecursive.com/061-reinforcement-learning/

I know that they aren't magical. But the intent is to present engaging content to you so you stick around longer so you can be datamined and targeted with sponsored content. The issue I take with it is that there is a consequence almost of philosophical dimension of free will. There is societal and social cost of that particular business model that relies on driving engagement to present ads because it lacks the dimensions that are harmful.

There are similar algorithms driving advertisement and one of the ads that has been targeted at me lately is a nicotine source two years after I kicked my nicotine habit. https://en.wikipedia.org/wiki/Snus The intention is clear, to make me pick it up again. Given this is simply a personal anecdote but it is eerily accurate sometimes.

I mean in the interview that idea doubted, the maximizing of ad revenue.

At least with facebook, but I assume with others, they are using engagement as a proxy for delivery value to you. That is the goal they are trying to maximize for.

So they want to maximize how much pleasure you get from the service, so they maximize engagement, but they don't necessarily maximize revenue, because that would hurt engagement.

To quote myself "Its sole purpose is to prolong engagement...", so we are on the same page here. But there is downside to this, it often hijacks your dopamine system and can be addicting. It doesn't only do that, occasionally you get content that you disagree with and interact with because of it. That also gives you pleasure to tell someone that they are stupid. This is why there is a such disproportionally high reach of e.g. flat earther content. People engage with the content to tell them that they are stupid all the time, but sometimes it sucks someone in to their rabbit hole. It is a fairly benign example... so what about anti-vaxxers or QAnon? These people are enjoying themselves to the point that they delusions take hold. And I blame the maximization of engagement.

Yeah! Totally agree. I thought you were saying they were trying to max revenue pee user. My mistake.

I always wonder what people expect when they talk about twitter's algorithm as if this some sort of Deus Ex level AI developed in the depths of Area 51. If anyone wants twitter's algorithm I guess all you need is to download some arxiv papers on state of the art recommender systems and it's probably going to look like that.

Yes and no.

Twitter's scale and real-time nature make it a difficult beast. Their network graph contains hundreds of millions of nodes and billions of edges.

And it's constantly updating. So any graph ML algorithms you want to use have to deal an underlying graph that's eventually consistent at best — and oftentimes very sparse in terms of feature availability.

Also yeah, most people talk about "Twitter's algorithm" but have no idea what it is they're talking about — that's exactly why I wanted to write on this topic :)

Yup and temporal graphs are a super hard problem without a lot of great ML solutions currently

I work on FAANG-scale recommender systems. Most of the time, due to the scale of the problem, the algorithms powering these systems are not anywhere near state of the art. On the other hand, they are highly optimised for the situation. You can bet that every aspect of the Twitter algorithmic feed has been thoroughly debated, thought through and most importantly A/B tested.

I worked at one of the FAANGs and the algorithms were definitely state of the art in some ways. Maybe they didn't use the latest models but the sheer size of, for example models, was huge. Easily 500+ features inputted and hundreds of millions of parameters for deep models.

Would you be open to have a discussion about FAANG-scale recommender systems? I have a small website that could heavily profit such a recommender system. With 200k daily visitors there should be enough data to train a model, but our current implementation sucks and barely beats random.

If yes, how can I contact you? Twitter?

Not the person you're asking, but i'm curious what your current approach is. In my first job (ecommerce site) I built a (very slow) recommendation system based on k-means clustering and it did a pretty good job of clustering customers based on interest and suggesting products, definitely better than random.

Building it today would be much faster because there are actually proper libraries/programs for doing this, rather than my inefficient vanilla Python implementation.

For sure - @arshamg_ on Twitter. DMs are open.

People speak of "the algorithm" as if it were an animal spirit or American god to appease and to implore. I suppose this is our zeitgeist. Your explanation helps give me and others a conceptual grasp on the nature of the thing, so thank you.

I wouldn't be suprised if it's just a handcrafted spagetti of condition blocks in PHP .

One thing this blog post doesn’t seem to mention, perhaps because it’s not Twitter-specific, is that all the code that drives “the algorithm” is heavily tied to everything else at Twitter and not easy to extricate and use by yourself. It might be fun to look at, as much as it is fun to look at a bunch of weights, but you probably are not going to be able to run it yourself.

It's likely big neural nets, looking at the weights won't tell you much. What you can use instead to find biases is the training data, especially what sort of data the explicitly exclude/include. It's unlikely Twitter releases that.

Open sourcing the algorithm would be such a huge boon to those wishing to game it. Those with the sufficient compute resources could run billions of variations of Twitter activity to perfect their messaging to get it trending. It'd be an arms race for those with enough money to keep up. Any chance of anything ever making it into trending in an organic way would be lost.

For transparency, it might be nice for Twitter to continually release their algorithm from a certain time period ago, say 6 months

Unfortunately it wouldn't even look anything like that. "The algorithm" is probably a bunch of ML with some shitty product driven hacks on top, and those hacks will be trivial to game. Things like "if follower_count > X: spam_model_weight *= 0.1"

I really like the idea of being able to select my own algorithms

I'm not sure how it works but it works in such a way to frustrate me. I rarely see the tweets of those I follow. Instead I see the ones they liked, I see the tweets of others that follow them, I see tweets of others who are similar to them.

I see the tweets of everyone BUT the ones I follow. Hence I have a chrome plug in that cleans up their junk into a usable state.

You can switch to the old reverse chronological feed (called "Latest Tweets") in twitter itself.

I also use https://github.com/giuseppeg/refined-twitter-lite

But it automatically switches back to algo mode after a while, diametrically against your expressed wishes.

I sincerely hope I never learn the names of the people at Twitter responsible for this user-abusing product decision.

It no longer does this.

That is a very handy and unfortunately hidden feature. I just enabled it, and even knowing the feature exist I had to Google to know I need to tap that funky icon on mobile.

Unfortunately for Twitter I will keep using the browser plug-in which also hides their ads (maybe they get paid for the display, even if hidden)

My way around that nonsense is ;

1. I use Tweetdeck on PC (browser).

2. I also prefer to add people to lists rather than following them.

So on Tweetdeck I have multiple columns open at once, the first column is the people I follow (Tweetdeck presents them in chronological order, no messing about with similar users or liked tweets) and the rest of my columns are my lists. (F1, MotoGP, NBA, Space/Rocketry, Tech).

I mainly use Twitter on PC but whenever I use it on mobile it's to look through my lists. The timeline is pretty much useless on all apps.

I've tinkered with coming up with my own algorithm for YouTube using their API (albeit very basic ones) and I quite liked the results. I _tried_to make a front end and gave up because of the scope creep.

I wonder why platforms wouldn't just offer different algorithms. e.g. historical, maximise-time-on-screen or exploration. Most of the users would be nudged into what the platform wants anyway, and they will be able to claim moral superiority over other platforms that doesn't offer this. Heck, you could even charge for this feature.

https://stratechery.com/2022/back-to-the-future-of-twitter/ has a great breakdown of Twitter's history in this respect.

TL;DR they used to have an open API and various third-party clients, each of which had their own flavor. BUT this made it really hard for twitter to guarantee a good UX and even harder for twitter to implement an ad-driven business model.

I'm not arguing for an open API, I'm arguing for more options in content algorithms. These would be under their sole control.

Aside from that, thanks for the link. That is a great post!

I think at a practical level, opening up their API for non-watered-down third-party clients would be the only true way to add more optionality to twitter's feed UX.

Since I don't see open sourcing the actual feed algorithm to be something that would practically happen any time soon (which makes me a sad panda).

I would imagine Twitter spends a lot of time maintaining the one algorithm. More might involve a lot more work for diminishing upside.

I think the reason they don't offer options is because in their business model, there is only one option, the one that keeps you on the site longest and watching more ads. What you think you want and what actually keeps you there are probably not usually the same thing.

Exactly, any algorithm choice that resulted in lower engagement metrics would necessarily be something Twitter (and any social site really) would try and avoid at all costs.

It's why government regulation is probably the only way to fix the issue, since you need companies to act in a way that's counter to their own interests (but would lead to a better experience for users.)

Stephen Wolfram proposed some similar ideas on this topic. Namely allowing "final ranking providers" and "constraint providers" to modify the final outputs of the algorithms, without requiring full-on transparent explainability

https://writings.stephenwolfram.com/2019/06/testifying-at-th...

Would you care to publish it? I'm interested in the same thing. Perhaps we could even team up

Heya, sorry for a late reply but I probably won't as it's been a while and I'm too busy with other side projects/ life.

I got the inspiration from this article https://towardsdatascience.com/i-created-my-own-youtube-algo...

And was writing FE for it in Electron/React

    SELECT 
       t.* 
    FROM 
       tweets t, followers f 
    WHERE 
        t.author_id = f.author_id AND f.user_id = ?
    ORDER BY rand()

there its open source now

i prefer date DESC

reminds me of the infamous Conway sort (https://transitivebullsh.it/gaming-cs-interviews#083392b655d...)

Sad to see another blog that has no RSS feed :(

People, please, I am begging you - turn those back on. It's the only way we can stay up-to-date with your amazing content.

Anyway, this is what I came to say. A great post, full of insights. We have discovered quite a few of them ourselves, while building https://murmel.social, and I envy the author a little for publishing the knowledge first. Next time ;)

Hey haha I literally added this yesterday :) https://github.com/transitive-bullshit/nextjs-notion-starter...

https://transitivebullsh.it/feed.xml

Great call!

Author here. I needed to split this up into multiple posts because it required a lot deeper dive than I had originally expected.

What I'm really interested in, however, is help with answering the following questions:

What would open sourcing the Twitter algorithm actually look like?

Would it be possible to abstract away all of the engineering complexity that it takes to run a global, production system like Twitter and produce an OSS spec or API that is actually useful?

Would it be possible to produce meaningful results without access to Twitter’s full data set?

What does meaningful even mean here? How would we define success?

What would need to happen in order to make this a reality?

What are some practical proposals that would help improve the status quo?

What do you mean by "open sourcing" the Twitter algorithm?

What are the actual requirements that you're looking to satisfy by doing this?

Great questions.

These are exactly the types of questions I'm trying to get more people to think about — and the reason I wrote the article above.

It's not well-defined, but the goal imho should be to allow more transparency and optionality around Twitter's main feed UX.

Open sourcing twitter's algorithmic feed would be a monumental task that's honestly unlikely to ever come to fruition, Elon or no Elon. BUT twitter could certainly do more to move in that direction.

> What are the actual requirements that you're looking to satisfy by doing this?

If it improves transparency around how twitter's algorithm works.. if it introduces more optionality into twitter's main UX (and isn't just a chrome extension, for instance), if it's coming from twitter itself and not a third-party.. if the effects of different factors on relevancy signals are made more transparent (like being able to play around with a simulated tweet under different conditions and seeing the resulting relevancy scores for a set of example users).. there's just so much that twitter could do here to improve transparency around how their feed works. And I'm generally in favor of anything what will move the status quo.

In Elon's words “Civilizational risk is decreased the more we can increase the trust in Twitter as a public platform.”

Actually the more I trust Twitter the more risk I am exposed to. Avoiding Twitter and discounting anything to do with Twitter is a better risk management policy for me.

Some sort of explanation of why am I seeing this tweet would go a long way.

like tweet was displayed because 50% interest in dogs 20% friends liked it 30% generally popular

Agreed! This is one of the biggest changes that Netflix did to its UX to help users understand why they were seeing certain recommendations. I'd need to look up the source, but iirc it dramatically increased user perception of it's recommendation system.

Other than a lot of people preferring the reverse chronological feed, I think the biggest complaint I see these days is the perception that the algorithm is being manually manipulated to promote/not promote certain topics and/or people.

Given that Twitter/Facebook/Reddit/Tiktok are the behemoths in the room that all have the power to sway public opinion by either promoting or suppressing information, I think they should be regulated so that any topic that was promoted or suppressed by the algorithm or manual human intervention should show itself on the content just like an ad does. "This content has been quarantined by [Twitter Staff / Automated Review] and is only available to those with the direct url." would be a perfect example of such a message.

From there, users should have control over if they want to see such quarantined content or if they want a "safe" experience that Twitter curates for them.

> I think the biggest complaint I see these days is the perception that the algorithm is being manually manipulated to promote/not promote certain topics and/or people.

The scary thing is, this isn't a conspiracy myth. Back in 2014 FB got caught at experimenting at how algorithm changes affected the mood of users [1].

[1] https://www.forbes.com/sites/kashmirhill/2014/06/28/facebook...

This is only possible for content that is flagged in a binary way. Most of these systems will also have a score for content and up/down rank the content based on the score. Using a score is actually a way to reduce the harm done by the algorithm. A low score will reduce the number of people that see content but not remove it completely, preventing false positive censorship.

This is wildly unlikely. the content scope of twitter is huge, the data scale is huge and they have so many engineers that it's difficult to keep "manual human intervention" confidential.

Even something as simple as a keyword search, ie "hunter biden laptop", would require searching tens of millions of strings per hour. Distributed across thousands of servers, integrated with whatever dev ops set up they have. It just seems like a huge pain, for little practical gain.

Their scale is so huge that I doubt twitter has much manual control over what people see, even if they wanted it. It seems much easier to just multiply weights

Why would it be ok to quarantine legal stuff. If i subscribed to it i should be getting it. Otherwise Twitter is falsely advertising its service

I honestly just use Twitter’s historic feed. The day they remove this ability is the day I delete my account.

I use lists which is something they seemed to have forgotten about. If Twitter removes lists I will use something like hootsuite

Lists and list sharing is reminiscent of Google Reader in some way I can't quite remember. The lack of ads is striking whenever I follow a link to a normal timeline.

I use https://tweetdeck.twitter.com/ which has the ordering I expect, allows arbitrary columns and has no ads (though this might be because of ad blocking).

I have five columns: personal account, replies to it, side project, replies to that, messages. Only thing I miss from the old OSX Twitter client is the ability to copy and paste in images.

Yet, any company who open sources an algorithm can simply apply any private alternative algorithms using any alternative/undisclosed data sources to any undisclosed subset of users as they wish.

Hell, whose to say that some people aren't chosen for long term "experiments" with different behavioral/belief modification goals? That's what I would do.

Good point.

Even if there were some sort of open source spec like the W3C specs, for instance, Twitter would still be a black box that could choose to conform to the spec or not choose to.

What we need is something more enforceable. So either you do something completely green field like https://blueskyweb.xyz/, or you spend $43B & roll up your sleeves to try and fix the problems with the current platform.

Even better than open-sourcing the algorithm would be giving people options on which algorithm they get. I bet you could get people to pay for Twitter blue just to get reverse chronological back or whatever it is.

Agreed. The goal should be more transparency and optionality around twitter's algorithmic feed.

Note that everyone has access to the reverse chronological feed called "Latest tweets". See here for how to switch: https://twitter.com/TwitterSupport/status/144797117918529536...?

Also agreed that twitter pro would be a great place for twitter to add more options.

Still, defaults matter and most people will not change their algorithm settings. Ironically, those are exactly the people who are impacted most from an algorithmic timeline that tries to maximize engagement.

"Open sourcing" the algorithm to a public platform will be very interesting... but even getting Twitter out of influence of foreign agents, whether they are bots [1], the 50 cent army [2] or powerful foreign state-level actors [3] will be a good start for Twitter.

[1] https://twitter.com/johnkrausphotos/status/15172153497243525...

[2] https://www.washingtonpost.com/news/monkey-cage/wp/2016/05/1...

[3] https://twitter.com/Alwaleed_Talal/status/151461595698675712...

Imagine if it was just a chronological order of tweets and retweets from your followers. And that’s it. No algorithm.

Twitter would be so much better.

Agree. And imagine if, instead of a single “good for all” black box algorithm, one could simply apply custom filters. “See less of this” or “always show” or “no retweets”.

People are obsessed with personalization algorithms and disappointed when this involves giving people control to personalize. As though personal control isn’t magical enough (in comparison to the magic of the algorithm).

That's how I use twitter. I had to implement this complicated concept myself by scraping tweets from accounts I follow into local database, and then displaying data from that database in chronological order.

That's what the timeline setting does the mobile client at least (not counting the occasional ad)

there's literally a button for that, you click the three stars at the top and select "see latest tweets"

Any idea how to do this on desktop? I only really reply to people on mobile and rarely browse Twitter on there.

On the non-mobile website there's the same icon at the top of your feed. I don't know about any of the desktop clients

still speculation without being an employee (even if it did some research from blogs which seems to be more then most do) twitter should just write their own post and shut everyone up already. idk why the company is so secretive about everything. its clearly not doing them any favors

I'd love for twitter to take a more active role in this, but from what I understand, twitter employees are forced to be pretty tight-lipped about these things.

I tried to reference as many of the official sources as possible (in addition to drawing on my past experienced at Amazon + Facebook).

yeah these companies act like were all protecting the coca cola formula or building nukes or something. like really, its blocks of text being put in order. the hr, pr, comms, leaders or w/e seem so disconnected. im sure there is some bs reason like were protecting the company, i think its misguided being so closed off though and missing the bigger picture of building trust with users.

especially on a platform like twitter :)

its almost like its a place where people can interact directly.

thanks for calling out scale and engineering problems btw! sometimes i think people think we're sitting around all day throwing darts at a wall of pictures of conservatives to pick who to ban next.

> yeah these companies act like were all protecting the coca cola formula or building nukes or something. like really, its blocks of text being put in order

The Coca-Cola formula seems like a pretty good analogy, honestly. Most people can't tell the difference between different cola recipes, and there probably isn't anything in the recipe itself that's responsible for the company's success. Not as much as marketing, and simply being in the right place at the right time, anyway.

Are they trying to prevent people I.e. spammers from gaming the algorithm?

"A former Twitter employee who spoke on the condition of anonymity to discuss private matters said the company has considered an “algorithm marketplace,” in which users can choose different ways to view their feeds. But efforts to offer more transparency have proved challenging, the person said, because of how tied Twitter’s algorithms are to other parts of the product. Opening it up could reveal trade secrets and invite abuse, the person said."

Source: https://www.washingtonpost.com/technology/2022/04/16/elon-mu....

Twitter, in general, is a remarkably open company internally. Because it leaks things like a sieve and has people dig into its products for new stuff, this ends up being the case externally as well. Most major A/B tests are announced via the various official Twitter accounts, for example, and employees regularly tweet about or solicit feedback about new features. Editing tweets, for example, had a concerted effort to avoid it from leaking externally (specifically compiled out of builds rather than being behind a feature flag, codenames, documents mostly put behind “do not distribute internally”) and it was still a somewhat open secret, since you could stumble upon it in the codebase or when searching for bugs. Companies that “care” take far more extreme steps to hide things than Twitter does, or I would argue even knows how to do. It’s not in their DNA.

With that said, the timeline is somewhat secret sauce but mostly it’s just a glop of various parameters that aren’t very interesting to share. Like, how are you going to even describe it to someone? It’s easy to talk about changing a button color, but “we promote retweets 10% more now” is boring and not really something that most people care about.

I have a theory about current state of the are of recommendations, automation or ML in general. I think a lot of the valid criticisms we see with recommendation systems (building of bubbles, extremization etc.) is due to the simplicity of the algorithms.

In reinforcement learning there is a notion of exploration and exploitation. All of supervised ML can in a way be seen as exploitation.

Incorporating ideas of exploration in a clever way into the standard recommendation algos, I think could be the solution to a lot of the mentioned problems.

This notion that open sourcing algo would resolve “speech” issue on social media is laughable. All it will do is reduce the contention down to said algo and how it could be manipulated.

Yes and no.

The goal here is to increase transparency and optionality around twitter's core feed. There will always be difficult, contentious human problems with a platform like twitter — but instead of writing it off as "laughable", there's a lot that can be done to improve the status quo. And imho that's a very impactful goal worth striving for.

How it works is interesting in itself but it is not the reason why these 'algorithmic feeds' have gained the nefarious reputation they have. The real problem with this type of technology is the way it allows the proprietor of the service to present its users a tailored view of whatever content the service offers. The way this view is tailored depends on what the proprietors' aim is:

Twitter uses algorithmic feeds to present its users with an ideologically driven view of whatever is being discussed, this can easily be seen by reading the first comments on contentious issues where they nearly always push some 'progressive' comments to the front, even when those comments are clearly less popular with users than whatever other comments follow. Comments critical of 'progressive' issues often require one or more extra clicks to be made visible. Sometimes this leads to the first page being devoid of comments, requiring a click to access whatever comments are present where those comments are nearly invariably critical of the 'progressive' issue being discussed. This does not happen when there are positive comments on those 'progressive' issues since these are presented directly on the first page.

TikTok and Youtube also use algorithmic feeds to channel users but their algorithms seem to be tailored to keep users on the site/app for as long as possible.

I agree with your sentiments. Just keep in mind that the whole point of this discussion is to move towards more transparency and optionality around how Twitter works.

As you pointed out, not everything is algorithmic. There are definitely human factors at play, but without a more solid understanding of how twitter's algorithmic feed actually works, the discussion will remain hand wavy at best.

Just my 2 cents :)

This was abundantly clear when, e.g., every Trump tweet had the first (always highly negative) replies from a couple of crazy doctor brothers (twins?).

I always interpreted this as someone wrote a script and learned to game the algorithm, you think there’s a line of code in there that says trump_replies.unshift(crazy_doctor_brother_tweets) in there somewhere?

No, I suspect the algorithm takes into account both the sender as well as the content of the message in deciding which comments to push to the front. Reliably 'progressive' senders - especially those which create more 'progressive' engagement - get extra points, this in combination with a 'progressive' message or one critical of any non-'progressive' message seems to be among the deciding factors in what to push to the front. In the absence of 'progressive' senders and/or content the algorithm probably looks for senders who create more engagement over those who don't, adding an extra click barrier between 'progressive' messages and comments which are critical of such.

Popular non-'progressive' senders - Trump and now Musk being the poster child of such - probably get their own comment manager who makes sure there is always some vitriol waiting no matter the message. This does not scale but the Pareto principle makes that it does not have to since there are not that many of such. The same may be true for popular 'progressive' senders where comment managers may make sure there are supportive comments but this seems to be less clear-cut.

Scenes when Twitter algo is literally just what you and your friends like and retweet - not even clicks or anything. Wonder if Jack Dorsey will have been responsible for the end of democracy then.

The easiest way to implement “open source the algorithm” is to delete the AI algorithm, revert the defaults to reverse-chronological feed, then release that.

Sounds great. But it will never happen because of business incentives. I'm focused on trying to move the conversation towards more practical solutions.

If Elon takes Twitter private, I doubt that you can argue that the quickest practical solution is other than as I’ve suggested.

In the secretive context of the current non-profitable public Twitter model, yes, I agree, they cannot do anything useful.

Not sure why you got a downvote though, so have my upvote.

Oh yeah, if Elon takes twitter private, then all bets are off :)

Though I personally think throwing out any form of algorithmic feed wouldn't happen. There are just too many business and UX reasons it makes sense, which is why every major social platform uses them (https://www.socialmediatoday.com/social-networks/truth-about...).

I don't follow many people on any social network and endeavour to read everything those people post. Like you, I want a straight sort of those items. However, I assume the majority of people follow more than they can read, so a huge effort goes in to sifting through the fire hose to provide a filtered taste.

Someone with a short following list can see the drama if they look at the huge list of replies that a popular account might get. So much cruft.

I actually do the opposite. I cap my follows at 100 and allow the algorithm to curate the rest. This way there's no stale/old interests in my recommendations and most recommendations are relevant to something I'm interested in right now or in the past ~2 years or so.

Right, but a lot of people do not wish to use Twitter this way.

It would be pretty easy if they just reverted it to reverse chronological and lose the engagement win they get by having it.

Ok, but would mean people who use “the algorithm” wouldn’t get it anymore.

Correct. Which is what most of the complaints are about, knowingly or not.

Elon Musk is a free speech absolutist and does not understand information in the modern age, these kinds of abstractions seem to elude him. That, or he does, and he's a sinister operator, I believe the former.

'The Algorithm' is not that important to Twitter.

What makes it work, is a semi-coordinated mass of users on a reliable platform.

If that platform could be distributed, and, there was a critical mass, it could work, maybe.

But 'open sourcing' is really not the issue at all. There isn't a line of code at Twitter that's valuable on it's own to anyone.

You could hand over the Source Code to Trump & Co. for their deluded 'Truth Net' or whatever and it wouldn't make a difference.

I don't think there are very many true free speech absolutists. We mostly hold positions as a means to an end, not as an end in themselves. Meaning, people declaring themselves free speech absolutists do so not because for them it's Free Speech, consequences be damned. But because they believe Free Speech is the way to the society they want to see.

A good thought exercise is to try to imagine what would happen if an absolutist of any kind got their way in every way possible, and still hated the results.

Really committing to free speech absolutism requires a very strong stomach very few people truly have. Back when Voat was a thing it was such a cess-pit that when r/TheDonald tried to move there, they couldn't take it. Voat disdained moderation, and as a result filtered for the kind of person who can look at a feed full of conspiracies, the most open and clear hate, fascism, racism and antisemitism, and every kind of porn that's banned on Reddit, and not even flinch at it. The denizens of r/TheDonald were used to setting rules on their turf, and were relentlessly mocked for it until they crawled back to Reddit.

Voat eventually died. I think probably because in the end even the creators grew disillusioned with the community they had created, or because they realized it was only a matter of time before they got into serious legal trouble.

That would imply that twitter becomes a nonprofit

Isn't that Elon's plan? At least he claims it is, so it must be true.

i may have missed that part

I exaggerated, but he said that he doesn't care about Twitter being profitable once he took it private. And he's all about free speech. Not that I believe any of this crap so.

It already IS a nonprofit. /rimshot

Great technical article, but why did you feel the need to add speculative ideological polarization too it?

``` Elon’s motivation is clear and consistent with his modus operandi. It’s the same reason he’s working so hard to build a sustainable colony on Mars, why he’s devoted resources to understanding the potential dangers of AI super-intelligence, and why he’s so insistent on combatting climate change. His guiding motivation is to improve humanity’s chances at a positive future. ```

That's certainly one interpretation.

You could literally get away with murder with that kind of assumption of good-will.

Fair enough.

What I was trying to do was explain my motivation for digging deep into this topic. It has nothing to do with ideological polarization, though it may be a bit speculative.. this is mainly based on Tim Urban's excellent series of articles on why Elon Musk chooses to work on the problems he does. https://waitbutwhy.com/2017/03/elon-musk-post-series.html

Reply on news.ycombinator.com