Moving away from algorithmic curation | Hacker News Reader

Moving away from algorithmic curation

boredzo.org

66 pointsingve41 comments

I think the issue is algorithmic curation uses popularity/traffic/engagement as signals over the actual content. Hence why I think outrage, politics and whatnot gets promoted more often. It leaves content that isn't popular in the dust, and causes people to modify their behaviour and content to optimise for the algorithm unfortunately—I personally think it's a broken system insofar as it's optimising for engagement rather than curation.

Somewhat related, but I've been using Plex's Sonic Analysis[1] and it's been great at recommending tracks in my library (large but not spotify huge), much better than the streaming services, which often would recommend popular non-independent content over gems.

It's "simple" but it works—ideally services ought to let users easily tweak how content is curated for them

>The Musical Universe consists of points in N-dimensional space. But what’s important is that this allows us to see how “close” anything in your library is from anything else, where distance is based on a large number of sonic elements in the audio.

1. https://support.plex.tv/articles/sonic-analysis-music/

>over the actual content

The content are features to the ranking models. Recommendation systems take both what the content is and what people liked the content into account.

This article is missing one important idea - yes, sure avoid algorithmic systems when we don't control the algorithm. But what I really want is an algorithmic feed option (option being important, obviously) on mastodon!

If someone that I have been interacting with a lot had a popular post in the last 12 hours that I have not seen, and I have just refreshed the feed, I would love for that to be near the top.

That is just one simple user story but I can imagine a wide selection of options - controllable percentage of popular posts from the rest of your server or the fediverse. Controllable percentage of posts marked important by my server admin. Controllable percentage of popular posts that people I am following have liked, even if I am not following the person.

People really misdirect their grief when they say the problem is algorithms. I don't think that's really true! We all have trauma and bad memories from corporate-controlled algorithm, because that algorithm serves the needs of the corporation. Users get turned in to products to serve a for-profit system that sees us as tools to be used to extract value. And we all see those algorithms and think: "algorithms suck!" Then people make systems like mastodon with no algorithm at all - no option for one - and people bristle at the idea of an algorithm there even if it is optional. But of course it would be optional, it's mastodon!

Imagine if users had a complex panel (with a few easy pre-sets to try) for various tunings to the algorithm. I genuinely believe it would make mastodon much more useful to a lot of people. I still use both twitter and mastodon and one thing that is clear is that it is very easy to see popular posts from my friends on twitter. Unfortunately there is a lot to dislike about twitter, which is why I want to see people change their thinking - what we need are algorithms we control! Open source algorithms with lots of optional tuning and adjustment and always the ability to disable it completely.

There is some discussion of some of these features in the mastodon issues page. If any of you want to take a look, maybe there is some work you could help with!

https://github.com/mastodon/mastodon/issues?q=is%3Aissue+is%...

I completely agree with you. To handle this for my own situation I wrote https://www.jefftk.com/p/introducing-shrubgrazer but I'd much rather this was a feature of a real client and not my pile of hacks.

I had the impression that cross-server algorithmic discovery is impossible given Mastodon's fundamental architecture. That made me lose interest in the project.

How so? If you optionally allow your server to coordinate with others, I don't see how anything like this could be impossible. Even still, some of the things I want - like giving priority to popular posts that would already be in my timeline but from several hours earlier - would be possible without any server to server coordination beyond standard federation.

I thought no instance has reliable, complete, cross-fediverse search. I thought no instance shows replies from across the fediverse. I've heard the protocol is too inefficient to support receiving everything.

> I thought no instance has reliable, complete, cross-fediverse search.

This is true.

> I thought no instance shows replies from across the fediverse.

I am not quite sure what you mean as I see replies from many different servers in the fediverse. Perhaps there are some replies I don't see, but I suppose then I would not know this.

> I've heard the protocol is too inefficient to support receiving everything.

This may limit some algorithmic features, but not everything. Servers could specifically make accessible their most popular posts of the day from users who consent to this in their settings. I could choose to subscribe to this list to be optionally sprinkled in to my algorithmic feed. I don't need to see every post on the fediverse for this to work. A good heuristic would be to subscribe to popular posts from every server that hosts one person I follow, and their could be some convenience function to select those servers for me. If I find that some servers are producing posts I don't care about, I could unsubscribe from that server in a little option under the algorithmically selected post.

I may only want to see popular posts from something like 20 servers, and I might only want to see the top ten most popular posts from them each day. That seems like a tractable request from my server to the ones I am following, and it does not require full fediverse search.

That makes sense. I guess it goes to show how different what different people want from social media is.

I have no desire to see the most popular posts, I want to see unpopular posts relating to other unpopular posts I'm interested in. Most of the people I follow are people I don't know in real life who have less than a thousand followers but posts things I'm interested in, and I like how Twitter suggests new people to follow to me with similar characteristics.

I would love the ability to tinker with my own recommendation algorithm, but I don't think I could build what I want without an index of the long tail. I also noticed that people I follow on Twitter aren't all on similar mastodon instances.

> I guess it goes to show how different what different people want from social media is.

Well showing popular posts was just one example. I am just thinking of things that can be implemented on mastodon without a full rewrite to create an index.

> Most of the people I follow are people I don't know in real life who have less than a thousand followers but posts things I'm interested in, and I like how Twitter suggests new people to follow to me with similar characteristics.

Sure, that's cool. Thinking about this for a minute, I feel like your server could do some social graph analysis of your history of likes combined with who you follow and who they follow to suggest people to follow who meet the criteria: you do not follow them, but people whose posts you like do follow them.

I guess what I want to say is, I really feel like open social networks are an extremely important concept, and mastodon is the biggest one. Traction and networking effects are very important to make social networks survive. So rather than trying to build a full index on mastodon or build a new protocol that makes this easier, I really want to think about what is possible with mastodon today with simple changes. I believe that is a good path towards success.

Certainly twitter's fully closed system is not what I want to rest on in perpetuity.

That's a good idea. I didn't think of walking out from direct follows. Maybe I'll try and implement that this weekend.

Any pointers for getting started consuming mastodon data?

Oh great I’d love to hear how your progress goes! My email is in my profile if you’d like to follow up there, tho I will also check back here in the future.

No experience myself! I am an open source robotics engineer passionate about open systems, but am sufficiently busy with my robotics work I’ve not yet been able to dip in to mastodon dev. I hear that consuming the data is very easy tho! See the other reply to my top level comment here for someone else’s mastodon project in a similar vein. Good luck and godspeed!

"It increasingly seems to me that the best things you can do with these services—recommendation engines, algorithmic timelines, and such—is (1) don’t use them when you can help it, and (2) lie to them at every opportunity.

Poison the well, and don’t drink from it."

I don't think the problem is algorithmic curation, the problem is _bad_ algorithmic curation.

I personally find quite a lot of interesting content on YouTube's and Spotify's recommendations.

Define “bad”.

YouTube optimizes for engagement and time on site. If that leads you to a bunch of conspiracy videos and radicalizing content then it thinks it did its job well.

Algorithmic feeds goal seek and the problem is that 1) you don’t control the goal (algorithmic cruelty) and 2) the maxima of chasing that goal might have disasterous consequences (paperclip problem).

This is a problem with algorithmic feeds you don't control. But if you could customize your own algorithm, instead of having it be optimized for you by people who have very different interests, you wouldn't have these downsides.

As a simple example, I would like to be able to tell my feed to prioritize posts by my close friends over posts by people I don't know very well.

A controllable algorithmic feed solves problem 1 (kinda) but not problem 2.

The phrase “algorithmic cruelty” is best demonstrated with the most seemingly innocuous efforts. Like reminding you of a special birthday photo with a friend who has died and the computer has no way of knowing that.

The paperclip problem never goes away even if you’re in control. It’s like the butterfly effect. At least if everyone has a slightly different algorithm then the overall system wouldn’t align to produce mobs and harassment quite as easilly. Also without centralized control then mitigating those effects becomes much more complicated.

Even when it works flawlessly with no “side effects” it can have negative consequences. If it’s too good or you have too much fun with it then maybe you spend less time with real friends or ignoring other things. I use a chronological timeline and it gets boring sometimes. That boredom is a useful signal to me to go do something else. ( chronological is still an algorithm we just tend to not think of it as one)

At the end of the day I agree it’s better to have a distributed and Democratic feed then be at the mercy of someone else. That still doesn’t mean ALL of the problems with algorithmic feeds are solved by giving the user more control. It also doesn’t begin to get into the complexity of how a non-programmer would begin to tune such an algorithm without sliders and buttons that go into an algorithm someone else ultimately wrote.

> YouTube optimizes for engagement and time on site. If that leads you to a bunch of conspiracy videos and radicalizing content then it thinks it did its job well.

It doesn't seem to do that anymore though. The amount of complaints about this is significantly down in the last few years and the comment section isn't hell like it used to be.

Anecdotes are meaningful but they’re not everything. Especially when talking about a systemic, emergent problem.

One of the larger problems in this space is: since we cannot view the inputs that created the problem, we cannot verify that the problem is actually gone (rather than, say, shifted somewhere else less visible but just as harmful).

If this effect was achieved by a word denylist for example: then I expect new slang and code words to pop up to combat the filter.

We should ask for independent review and high standards rather than declaring the problem gone when we individually are no longer impacted by it.

Specifically, it's algorithms that by design work against the user's interest in favor of the company's interest (ads, engagement bait, etc).

If the developers of a social network have incentives that are more aligned with their users then it would be possible to create genuinely useful and interesting algorithmic content feeds.

Indeed, I also think we're, as a society, missing a fundamental freedom: algorithmic transparency.

That's the ability, that should be a right, to know in particular for widely available or impactful systems, what is the algorithm behind their feed, from raw data sources (and also the parameters should be public). Choices are made for us that could be very bad or against or interests (for example, to promote engagement or ad consumption), and we have pretty much no way of really knowing that. Let's fight for algorithmic transparency!

YouTube's recommendation system is notorious in several hobby communities for a tendency to point you toward alt-right content with remarkable ease.

A lot of people don't actually know what the word algorithm means or maybe the meaning itself is changing in common parlance.

Seconded; also I don't know exactly what Bandcamp does but I often find good new stuff to listen to through their links too.

> Poison the well, and don’t drink from it.

https://adnauseam.io/ :)

The real problem with all major media platforms is not that they suppress speech through censorship, it's that they suppress it through choosing whose speech to amplify; their amplification of certain people and ideas drown out everything else through saturation.

It's hard for those who control such platforms to resist promoting their friends, acquaintances and (more generally) ideas that serve their personal interests.

I believe that there is a similar dynamic when it comes to startup funding; those who receive all the big rounds of funding drown out those which don't. A lot of our economic problems can be explained by saturation. Too much attention, too much money, too many opportunities and too much power going to a select group of people which deprive everyone else of those things.

I don't think algorithms have to be bad, there's still room for experimentation with algorithms that don't just "maximize for engagement". I'd love a platform where I could customize my own discovery algorithm.

For example, I'd like to try out algorithms that:

+ Prioritize people who post less often

+ Prioritize people with less followers

+ Prioritize posts with less engagement

+ Let me manually choose people to prioritize/deprioritize

I'm not sure any of these algorithms would actually be any good, but the point is I want the freedom to experiment. I suppose this might be theoretically possible with ActivityPub/Mastodon, but I haven't seen any implementations.

The author of this article had put up a whole post on pointers in c which to date is what I use to refresh my memory. https://boredzo.org/pointers/

This along with the pointers by Nick Parlante helped me get over my pointers Heebie-jeebies. http://cslibrary.stanford.edu/102/PointersAndMemory.pdf

The best algorithms are incapable of knowing our thematic tastes or motivations. In fact, the best algorithms do not serve us at all, but the advertisers and the service providers. They follow nothing but perverse incentives, from our POV.

So there's no reason for surprise when a service shovels crap into your face. You're the product.

I don't use many content recommendation systems, so maybe this isn't typical, but youtube's is very good at what it does. It knows I like videos about trains and ships, and suggests virtually nothing else to me. It doesn't need to guess at my motivation for liking trains, all it needs to do is give me train videos.

I think these systems are what you make out of them. If you click trashy content, it'll give you trashy content. If you click politically edgy content, you'll get more of that. If you only click on trains, you'll only get trains. Maybe people who complain about these systems have a totally different experience than me, or maybe they don't like what they see when a content recommendation engine holds up a mirror.

As for their perverse incentives and profit motive... if they're bad at anything, it's getting ad views from me.

I was fascinated by the concept promoted by Pandora: "digital DNA" of musical works. It was an exhaustive list of properties applied to each song they played, and it fed into their recommendation engine so that songs could be typed and matched with more stuff we wanna hear.

It failed me miserably.

Digital DNA, or at least its implementation on Pandora, was woefully inadequate in determining my motivation and reasons for selecting songs in a list. I created a Lenten playlist of penitential hymns, and the cardinal rule was: "NO ALLELUIA" which is not vocalized at all during the Lenten season. Well my recommendations were liberally sprinkled with joyous Easter shouts of the A-word, and my experience was ruined.

I may select songs based on particular lyric themes, seasonal considerations, bands based on their particular location or affiliation; things like that. Recommendation engines just have this kind of sledgehammer that goes "Oh! You like <Heavy Metal>! Here's some more <Heavy Metal> for ya!" when my use case doesn't even care about genre, but I was looking for lyrical themes or topics.

Also unlike Pandora, YouTube has one big firehose of recommendations. It is unable to segregate them within a playlist or a particular session for some purpose. The only way to isolate recommendations is by account or incognito, and that is one reason I have 3 separate personal accounts for different purposes, so that my main account's activity does not pollute the interests and recs of the special-purpose accounts.

Ah right, I see what you mean. Supposing I tried to make youtube only show me videos of red trains, it would probably fail. It has a limited number of content bins and probably doesn't/can't synthesize new bins on the fly for individual users.

I too find YouTube pretty good. But like you say I avoid clickbait and am careful with my likes. If it suggests too much I don't want, I'll tell it I don't want those.

There is a weekly game my friends play wherein one of them crafts a playlist, and then the rest of them attempt to guess the theme that holds it together. It's really fun!

My only contribution was a playlist of inappropriately-named bands, such as the Sisters of Mercy, the Cocteau/Thompson Twins, Concrete Blonde, etc. It stumped them.

I have been loving newsboat; RSS FTW.

I'll recommend Reeder for those in the Apple ecosystem, it syncs via iCloud vs a lot of alternatives using paid services for that

Reply on news.ycombinator.com