Ask HN: With FAANG turnover-why don't we know about actual recommendation algos? | Hacker News Reader

Ask HN: With FAANG turnover-why don't we know about actual recommendation algos?

Recommendation algorithms of Facebook, Google, YouTube are still a black box to us. We don't know what exact features are the input, how much are they weighted and so on.

But the turnover of software engineers is quite substantial, they come and go from these companies. How is it even possible to keep this stuff secret, then?

For most folks outside of the above companies I can understand why know-how would be kept secret after people are gone from the employer: it's quite boring and of little interest to anyone, really. No benefit or thrill if you divulge anything.

View on HN

These aren't "algorithms" in the traditional sense of that word. They are complex, convoluted systems with hundreds of moving parts/services owned by dozens to hundreds of teams. There's no one individual who has a total understanding of the whole thing, and only a few that could speak to it at a high level. Most folks are busy optimizing and improving the microservices their team owns and such. The few folks with the high-level understanding are typically senior enough where divulging this info is neither particularly thrilling nor worth any potential risk.

And yes, recommendation engines are not nearly as exciting as the conspiracy theories around them.

So you’re telling me that behind the curtain it’s probably just Oz deciding that those who like dog videos probably want to see more dog videos?

> For most folks outside of the above companies I can understand why know-how would be kept secret after people are gone from the employer: it's quite boring and of little interest to anyone, really. No benefit or thrill if you divulge anything.

Why would you expect it to be any different here? There's not going to be anything mind-blowing, earth-shattering. They're very big and complex and likely continually tweaked.

There’s not much need for turnover, plenty of companies publish quite a bit on how their systems work. You can look for recsys papers from your favorite company if you want. The Netflix recommendation systems workshop is also good and has many industry talks.

The reality is that there’s a lot of (what is essentially) matrix factorization, and not much of anything nefarious or very interesting to most people who are not recommendation system engineers.

Another thing is that these systems are intentionally hard to game even if you know the weights. The systems optimize for things like P(comment|personalized features), so even if you know there’s a high weight in the scoring function for comments, just commenting a lot on your content isn’t sufficient. You need the system to predict that other people will also comment.

The twitter code (with whatever caveats) is publicly available. What's a concise summary of how their "algorithm" works?

https://tanay.substack.com/p/understanding-twitters-algorith...

https://archive.ph/71TWz#selection-685.0-685.91

While the general workings were well-known beforehand, there were quite a few surprises here. Especially the out-of-network penalty.

Go look in arxiv.org; what I see is a huge amount of research in China and India, you might think there is a “missile gap” in e-commerce. The real action is in things like TikTok and Temu, Amazon and Google are legacy companies like AMC, Ford and GM were in the 1970s.

While the authors may be Indian and Chinese, they are most definitely studying at Western Universities or at least collaborating with western universities in the majority of cases. Not sure what you’re on about.

At this point I expect many of the algorithms have been trained not designed and thus nobody knows for sure what they are doing. And even before that it's a really complex topic that probably can't be easily summarized.

People generally value their financial wellbeing and trustworthiness more than Internet points. It is also likely that no one person knows the algo, and fully describing the algo would likely require theft of IP.

Well Google, YouTube and FB engineers should be embarrassed because their "algorithms" suck.

Nice try, Internal HR Counsel.

NDAs.

Well, duh? And nobody out of the thousands engineers would be tempted to break it, if they could get away with it?

That's my point, really. FAANG employs a ridiculous number of SW folks for outliers to be quite expected. As a reminder, we had Snowden in NSA who at that time was estimated to have a headcount of 20k or so.

Severance packages with claw back clauses that enforce the NDA. Break the NDA and risk paying back the severance plus legal costs. Isn't worth the risk.

Leaves the question of how would they identify a person. I know that information in Apple is quite compartmentalized, but it's an outlier in the FAANG. Surely someone would be a fool to spill the beans just after they've left, but after a sufficiently long period...

> As a reminder, we had Snowden in NSA who at that time was estimated to have a headcount of 20k or so.

Snowden's revelations are infinitely more important to the public interest than some recommendation algorithms, as evidenced by the fact that he had to leave the country and seek permanent political asylum. He didn't do it for fun.

If you're looking for a whistleblower, how about Frances Haugen?

Maybe there’s just nothing interesting or dark about them no matter how badly people want there to be for controversy sake.

That or all the employees who work on the core algorithm are sworn to secrecy in a blood rite cabal with embedded cranial explosive traps if they ever speak or write anything outside of corporate SCIF’s.

Thanks for the reminder, and yeah, duhhhhhh. People actually enjoy keeping their livelihoods. Duh right back atcha.

Reply on news.ycombinator.com