Hacker News Reader

Comment by shrimpx | Hacker News Reader

I don't understand the concept of open-sourcing "the algorithm".

First of all, "the algorithm" is probably hundreds of thousands of lines of code, including all the tedious boilerplate like cache policies and multi-AZ logic.

And second of all, doesn't the algorithm include machine learning components, which are trained on terabytes of data? That data will likely be impossible to open source. And open sourcing the neural nets without the training data is mostly meaningless from a transparency perspective?

View on HN

Open sourcing is in this is not about the implementation or the CS algorithm. It is more about transparency. I think the idea is that the public should know how tweets are ranked, why tweets show up in timelines and which timelines, what makes tweets popular etc. Imagine Google publishing a document detailing how their system ranks pages aka publishes internal SEO rules officially. I don't know if it is a good idea or not. People with enough resources might be able to game the system (if they don't do it already).

The only change is that people with resource would stop guessing how to game the system, and start employing people to ensure they systematically game the system.

> if they don't do it already

That's an interesting point. A practical description of the algorithm from the perspective of someone trying to game it may be more useful than anything Twitter or Google would release.

Probably right but I think trying to be transparent is better than not trying at all.

Gesture carries weight to the users too.

Not sure any big company has tried this before. I could be wrong, but either way looking forward to it / FWIW hope it catches on.

Or is it six cats in a basement with a laser pointer and a mouse?

The point of releasing it is to let people know exactly why they see the tweets they do in the order they do. I hope Elon just goes back to time base ordering of tweets.

I personally know a FAANG employee whose full time job is building tools to try and help understand why the company's recommendation algorithm picks the things it does (and more specifically, predict how changes to the algo will affect that).

Even the people who build these systems barely know what the algorithm is going to do, much less why. It will be a herculean task to try and convey that to an average user.

Right. Also, it takes a new eng hire N years to gain a working understanding of the small subset of code that their team produces.

Disclaimer: I'm pretty new to twitter, so I may be misunderstanding something. On my Twitter Home Screen there are three little stars on the upper right of the central container which allow you to toggle between "top tweets" (ie. 'The Algorithm') and "latest tweets" which is the time based ordering.

What you are misunderstanding is that there is more social clout to be gained by complaining about twitter than exploring the available features. The ability to sort by latest doesn't stop people from feeling self righteous by demanding this feature exist.

You understand it, it's just like instagram, facebook, reddit etc, you let them pick something for you via some algo or you order by date.

Maybe it's a room full of lava lamps.

This is pretty standard in the machine learning world. You'll open-source the code and weights trained on a public data set (these are often licensed specifically for non-commercial use). But in production, you'll be using different weights trained on a proprietary data set.

The whole point of the repo seems to just show that there's nothing called as "the algorithm" It's probably something like 100s of 1000s of algorithms doing different things

You can open source the model code.

And developers will be able to train a model using it on a subset of Twitter data. Just that the quality of the outcome won't be the same as having the full set of Twitter data.

If it's too complicated, there is a good chance Elon will ask to simplify it until it can be open sourced.

Ever heard of Pseudocode which happens to be human-readable? If they are really going do it they will release source code for programmers and in general computer scientists to analyse and on top of that they will release pseudocode which non-techincal people can somewhat understand.

Reply on news.ycombinator.com