Monolith: The Recommendation System Behind TikTok
gantry.io
gantry.io
https://techcrunch.com/2023/03/16/tiktoks-new-feature-lets-y...
like, let's present this user travel or cooking material, that's usually safe
then, let's try things like certain genres of music, we'll see what they like/don't like
what i don't get is... how does that first recommendation on the #foryoupage or discover or whatever it's called, starts recommending you the sex workers who try to post as close to NSFW material as possible, get you to land on their profile, in their bio is a link to their Instagram or Linktree, and then from there it's an OnlyFans link
does the system try to recommend a soft entry into this content and then just pivot away if the user doesn't like it?
Or it tries to match you to an existing profile it has from some ad network data or something.
I get that, but I feel like it starts with "known safe/neutral" material like cooking/traveling/photography/whatever
How can it detect "hey, this person might like if we introduce softcore porn into their timeline"? Like, do they have softcore porn identified on a scale and they introduce the really "safe" stuff and then gradually crank it up? Why are they presenting softcore porn at all? The Apple App Store is cool with that ToC wise?
It's not trying to start with "safe" stuff, it's not trying to "gently introduce" softcore porn.
It's going "This video got a billion views in the last 30 minutes, people must love it, let's keep amplifying it to any account that hasn't explicitly rejected this category of content"
Presumably blank slate accounts are treated as open to anything, until people start curating.
For example, in my mind, not all ASMR content might lead to sexualized recommendations, but a girl in a bikini top with cat ears doing ASMR might generate both recommendations for ASMR and other more cam-girl like content. So I guess my question is, when you're starting off in tiktok seeing cooking videos, do you trend towards ones that feature 'sexier' hosts? They might not be sex workers to you, but they might be making tiktok think you're interested.
Also, what does tiktok know about you to start? What info do you have to give it to start an account?
(all things made up for this example)
cookinglady39 does a beach bbq recipe tiktok, in a bathing suit. You watch it. They give you another cookinglady39 video where she's back in the kitchen, you skip it, they give you a new cooking host also female, also dressed in summer attire cooking outside. You watch til the end. It gives you a man cooking outside, you skip. Nothing you've seen so far has been sexual, but tiktok is probably picking up on some trends that might lead them to give you more and more things done by women, then women in a certain setting, dressed a certain way and so on.
All it’s doing is giving you more of what you show the most interest in.
In the initial set of recommendations based only on overall popularity, there might be a video that's popular that incidentally contains a pretty woman. If the user skips most videos after barely a few seconds, but watches that one fully 3 times through, then the recommendation engine probably looks at users it does know more about that exhibit similar behavior and have higher engagement. It will then recommend videos that those users would probably watch a lot. Now the recommendations are shifted in the direction from "generally popular" to "contains pretty women". You repeat this enough times and the user ends up navigating the space of recommendations until they're maximally engaged (in theory). That means they might end up at softcore porn. Goodness knows that porn is popular if nothing else.
The recommendation engine doesn't even have to know anything about the content of the video. Just know what already high-engagement users that watched that video a lot also watched a lot.
That's at it's most basic really, I'm sure there's additional cleverness on top in practice.
Now, if you fall asleep in front of your phone screen, it takes days or weeks to shake out that mistake?
It just sees that content made by sex workers is popular and puts it in your feed.
Youtube does it too, and my best guess is that it is a form of supervised training and the real question is who's being trained.
Let's use a hypothetical scenario: Someone states that they identify as a man, they're in the 20-25 year old age range, and based on phone location you can gather that they live in Texas. Now you're labeled as a 20-25yo Texas Man. Then you can look at others who fall in the "20-25yo Texas Man" category and show things you'd expect that group to like because chances are, you're more similar to others in the group than being a true outlier. If other people in the "20-25yo Texas Man" group have expressed interest in Apples, NSFW material, and lawn mowing videos, then since you're in that group, it's going to start off with that same material.
disclaimer: i've never signed up for tiktok and have no clue if this is how they do it.
Like binary search, they're really good at finding local optima quickly, and then are rather bad at getting out of them once they get there.
That would definitely do it, basically destroying their most important signal.
TikTok is best in class for recommendeding content and I personally haven't see a dip in quality. Aka I never get trashy videos or anything cringe, just a consistent stream of science/tech, local Toronto restaurant reviews, cat videos, etc
The selling point behind TikTok is that by using the app normally (i.e. watching what you want) TikTok can figure out what stuff you want to watch.
But pretty much every suggestion that comes out on HN for people who complain about the algorithm on sites like this is to do unnatural things to "train" the algorithm: be very careful about how long you watch videos, click on videos that you think are relevant even if you don't want to watch them, watch things you're not interested in to prevent falling into a rut, avoid click on things you're curious about if you don't think you want to watch 100s of them to avoid tainting your recommendations, etc.
If that's "working" I think I'd agree that TikTok was mostly hype.
I spent some time disliking every 5-second video being recommended to me, and the problem went away. It was easy.
I'm another one who's tried TikTok on a number of occasions and it's always failed badly with me, worse than any other site I can think of. It never improved despite people telling me it eventually would.
My impression is that it was making too many assumptions about me based on the wrong signals or something.
But to your point: wouldn't there be a survival bias of sorts with assessing recommendation performance? I didn't like it and left, so presumably the people who remained liked it.
It's obviously popular so there's that, but it seems circular to say it's working for the people who like it and stay. You're losing information about opportunity costs of lost users.
They used to have options for like "don't show me videos with this sound" or "don't show me videos from this user" but now it's just a general purpose button.
They don't need up/downvotes. They have signals such as liking, sharing, watching multiple times, commenting, looking at comments, saving, skipping, looking at creators account and other videos by the creator. If you want to focus on up and downvotes, you end up with reddit.
A bit like how Poker sites give you better cards in the beginning.
As a former online poker pro, this is probably the most idiotic thing I've read on this site.
They say the worst result to get on your first roulette spin is to hit your number.
Obviously, these are just trends among people. Some gambling addicts never won and skilled poker players won't go on tilt after 20 bad hands.
TikTok doesn’t make me feel that way, and it’s that simple. This is the same reason why a ton of people are dialing back their Instagram usage or creating “finsta” accounts - Who wants to feel insecure constantly? Same goes with Facebook.
There’s the argument “Well you’re not using those service correctly. I use FB/IG/Twitter and I’m fine. Follow these steps on how to avoid toxicity on your FB feed…” (articles like this used to be posted to HN constantly).
But the thing is… why should we have to do that? TikTok shows me content that makes me smile BY DEFAULT. I am not being milked for engagement using my fight or flight response like Meta’s lolcow.
If other apps weren’t so toxic (by design), then people wouldn’t be nervous when posting on IG, or avoiding Facebook because of the cancer comment sections and ranking systems/algorithms designed to make you angry.
To me, TikTok just found the proper levers to hit psychologically to keep people’s attention.
I jokingly think, all the creators of each social media app are in a makeshift lab, and one say “wait… if we make them happy, they stay longer!” Then another one goes “Drats! I thought jealously and conflict was the key!”
I predict in the next 5 years we’ll be banning/have a campaign against people under 18 (or maybe 16) from even using these apps, and even go so far as to say anyone spending more than X time on them per day/week, should seek help. (With X being a relatively small number compared to the norm)
Just remember, not too long ago we thought: Cigarettes are a great way to lose weight! Even my doctor recommends them.
https://www.google.com/search?client=safari&rls=en&q=tiktok+...
It's like if Zoom was found to be always recording audio.
The question is, do you trust the PRC with your data?
Personally, I avoid most social media sites since I want to reduce the amount of information about me floating around in databases.
The PRC has the world's most advanced and far-reaching data mining and censorship apparatus to tightly (and quite effectively) control the flow of information to its citizens. Does that sound like the type of organization you would entrust your data to?
Bytedance is beholden to the PRC, and while TikTok data may (or may not) be monitored, the Chinese version of it is subject to the same monitoring and censorship gates as any other social media app in China.
Chained (aka open hashing) hashtables store pointers rather than values in the "main" array, and just put collisions in a linked list on that hash cell. Easy, but indirections have a cost.
Closed hashing (aka open addressing, yes it's confusing) hashtables take hash(input) and just do it again to get a second location, and put it there. Repeat N times for N hash(hash(hash(...))) collisions. Dense, but needs more complex logic to figure out when to stop looking / what to do when deleting because anything could be at location X due to a collision with something else.
Cuckoo hashtables use two (or more) hash algorithms rather than one, and dedicate a portion of the memory to each algorithm. If something's already in the first algorithm's location, put the thing it's colliding with in that thing's second location. On read, check both locations. Dense, relatively simple for both insert and deletion, and tolerant of a few collisions with low cost.
(cuckoo hashtables are a form of closed hashing / open addressing, because they keep all the data within the data-sized arrays, not storing extra info. and all of these are over-generalizing / there are fairly different-looking strategies available, e.g. it's not necessarily pointers or strictly repeated hashing)
And you could just reject the existence of collisions entirely and move to a new, larger array immediately. That tends to perform so poorly in both cpu and memory that nothing really does it in practice, but it is technically an option.
That's not really the common way to do open addressing [1]. Normally you either use (1) linear probing, where you look at `h(input)`, then `h(input)+1`, ... and so on. This is nice because of data locality. Though you need a good hash function for it to work well. Or (2) Double hashing, where you first look at `h_1(input)` then `h_1(input) + h_2(input)` then `h_1(input) + 2*h_2(input)` and so on. I'm not saying repeatedly hashing `hash(hash(hash(...)))` doesn't work, but it's more expensive to call the hash function so many times.
The secret sauce of TikTok’s recommendations - https://news.ycombinator.com/item?id=34836877 - Feb 2023 (138 comments)
Digital nicotine
The ideal social network, to me, is everybody has a fair chance to get enough views for them to seriously produce good content to keep followers retention.
That means, "more money to get more views" technigue should be obsolete in this system.
Tiktok failed at least to me.