It would know what I read and watch. It would find new stuff for me, dedupe it, listen to my feedback,and learn from its mistakes. It would work with all kinds of media, not just youtube or HN.
And it would all be private. I don't mind paying for my content but I do mind being the product.
As a non-program, you can always depend on the means that humans have used for content discovery since time immemorial: word of mouth.
A recommendation engine that works solely based on your own watch history would require a totally different approach using algorithms that are years/decades/forever away from existing.
It does make sense that the job is simpler with everyone's history and inputs, if you can keep PII out of it. Are there any good references on recommendation aggregation algorithms?
Here's a simple proposal that keeps everyone's stuff anonymous and allows a rec service to pay bills.
Distribute the pool of everyone's watch histories/prefs/etc using IPFS, which is a DHT with static names and some storage. User clients put the share data into IPFS and register into a public pool. The client remembers their own hash. Rec services (there can be multiple) pull everyone's sets periodically and aggregate/indexe them. Users can then retrieve a batch of new recs by sending a small fee to the service of their choice along with their hash and the server will place some new recs into their bucket. If the recs are not suitable, they can try another service.
Even without this, I could just check videos you shared on Twitter, follow the same process, and have a decent chance of identifying you.
Anonymization is very very hard. Maybe federated learning is more promising here, keeping all your data privately stored on your own devices.
I think you're the one underestimating modern recommender systems, especially the ones built by advanced teams in big tech companies.
They absolutely use content features. Among other advantages, this counters the cold start problem (how will you learn about a new video if it doesn't get recommended to anyone).
YouTube has hours of video uploaded every minute. A non-centralized, context-less recommendation system would mean that you are offered those crap videos together with ones that are the standard for ‘good’ now. Just get into the long tail on any search results and see if you want those.
A non-monopolized video discovery would be splendid, only who knows how to achieve that. Perhaps it would be more realistic to do the recommendations based on the numbers of ‘retweets’ in federated networks—in a more ‘social’ style instead of views (dunno if PeerTube already offers that). Also, an open system will be incessantly gamed.
Personally, I would gladly donate my (anonymous etc) data for this noble goal.
How do you figure out which data has been faked to drive views to particular content?
Data quality and relevance matter far more.
I use both YouTube and Netflix pretty extensively. These companies are both held up as ML success stories. But, at least in my personal experience, they suck! My front-page feeds consist of (1) content that the platform knows I have already seen because IT IS IN MY WATCH HISTORY, (2) content that is trivially related, like by having the the same channel/director/actors, and (3) random junk I have no interest in watching and always regret clicking on.
It's not that the content isn't out there -- I'm able to find stuff I like through "big spreadsheet of all the Netflix movies" websites, online recommendations from humans, and YouTube channel crossovers/cameos. But I've never had that magical "the algorithm knows you better than you know yourself" experience that every ML engineer I've ever spoken to seems to be convinced their systems achieve.
Are all ML engineers mindless drones with exactly average movie preferences, or am I some kind of incomprehensible weirdo?
if you regret clicking on it, that means you did click on it and the recommendation did it's job; remember that the job of the recommendation engine is not (directly) to suggest content you'll like, it's to suggest content that will get you to click. Ideally those goals would be aligned, but not necessarily. YouTube is simply incentivised (via ad revenue) to get you to click on as many videos as possible.
So while it may not be overly successful on you (i.e. you only click occasionally), if it works on a significant portion of people it could still be a successful strategy overall.
I also still don't understand why yt recommendations are as poor as they are. I've tried taking the time to try removing certain things from the "recommendation history" (or whatever they call it), flag recommendations I'm not interested in etc and it does improve things a bit but still it recommends things I would never watch in a million years (eg right now I have 4 Joe Rogan recommendations on my homepage even though I've literally never watched any of his things and have zero interest in ever doing so, and click on every one and say not interested etc, for a while it was obsessed with trying to get me to watch MMA content even though I can't bear watching other people hurt each other).
And the whole issue of recommending videos where it knows I've watched the whole thing, I just don't even know where to start.
This is a different strategy to Google web search, where the monetisation occurs on the search results page (via ads there) rather than on click-throughs, so there isn't an incentive to try and get you to click several tangentially related links.
>I don't mind paying for my content but I do mind being the product.
This is another thing that bothers me. Being guilt-tripped for not watching an ad, or being a Patreon subscriber, or YouTube Red subscriber. Even if I don't watch ads or pay a fee I am, me, this individual is valuable just for who I am. My data, my habits are valuable to Google just as much as it is for me to pay a fee or to watch ads.
Some portions of a personalized, cross-service recommendation engine are a lot more viable if centralized (e.g. any scraping/indexing, and plausibly the personalization itself). Supposing such a service had no intention of you ever personally being the product, what would they have to do in order for you to still be comfortable handing that data over to them?
No affiliation, but it seems to be trying to do what you want