Implementing your own recommender systems in Python | Hacker News Reader

Implementing your own recommender systems in Python

online.cambridgecoding.com

211 pointsDrLegend24 comments

Google cache link: http://webcache.googleusercontent.com/search?q=cache:1qK6xyT...

Ipython notebook used: https://s3-eu-west-1.amazonaws.com/com.cambridgecoding.stude...

How cool would it be to have a recommendation system for music that follows mood? That could be trained on actual music not by me saying "I listen to songs x,y,z when sad, a1,b1,c2 when it's sunny and i'm feeling good etc". If this will ever exist it would probably increase my hedonic index 10 fold

Songza, acquired by and merged into Google Music, is trying to do that. I've recently converted from Spotify.

I've been working on something similar for a long time. You can get far by using Last.fm to track your own music listening history and analyzing against notes (break-ups, rainy days, etc).

There's certainly a lot of research trying to incorporate things like this into recommendations, but it never really made it to mature products (apart from some experiments on last.fm, 8tracks, etc.)

Spotify tries to accommodate this by offering curated playlists based on moods, but knowing a user's mood to make that type of recommendations is hard. (it's a bit intrusive for a music streaming service to ask you how you feel every time you start it up)

If I could start a service, and trust a skip to indicate "no, you got it wrong, try something closer to the last track I didn't skip", it could often narrow in fairly quickly. Perhaps offer a few more alternatives for skipping that'd indicate reason. E.g. "want something more up-tempo/downtempo/more cheerful/sadder".

Assuming you know what you want of course. But the recommender system could have a mode where it learns which taste you like based on time, number of unread mails in your inbox, skipped songs and moon cycle. :)

But the point is you don't need to know what you want. You only need to think you do. You can let the system adjust what the buttons means too, based on your subsequent behaviour.

Perhaps this could be researched by using a meditation headband (brain sensor), like http://www.choosemuse.com/

I keep telling people that music recommendation algorithms needs a lot more data. Weather very much could play into it, but also time of day, and location. As well as what I let play to completion before and after a track.

I listen to very different music during working hours, during a commute, and near bed time, for example.

I've always thought that music recommendation systems should look at the actual music, and not as much at the surrounding meta-data.

What kind of instruments are used? What key is it in? What structure is the song in? i.e does it have a standard format, is it prog, is it a symphony etc What language is it in? What Rhythms are used?

That information, used properly, should be able to actually recommend music that the listener enjoys, not just guesswork that is usually rubbish. The Spotify algorithm for example...

A very basic test of a recommendation system is the following scenario:

You're a fan of a local band, listen to them a lot. This band is sampled, and actually praised, by a Korean rap artist. Suddenly thousands of Koreans are listening to it. Will the recommender system now recommend Korean rap to you?

Most recommender systems will.

Requesting clarification. What do you mean is the desired outcome here? Should it or should it not recommend Korean rap?

It should not. That people liked this Korean rap artist implied that they might like your local band, but the opposite might not be true (in this example, it probably isn't).

So hypothetically speaking, gangnam style shouln't show up in an English speaking person's list?

Gangnam style became a hit because of the silly dance, not because of the music.

The other direction. Fans of K-pop shouldn't start getting Taylor Swift recommendations when a K-pop single goes viral.

And people that like lots of viral songs should get Gangnam Style recommended.

I used to use rd.io, and had an annoying episode where I liked a Polish electronic musician. Suddenly I was assaulted by track after track of 70's Polish pop-music. It took a lot of effort to get it to stop playing them. The quality of a lot of recommendation systems is still surprisingly bad.

I think it should recommend at least one Korean rap song. And hey maybe you like it and discover a whole knew genre of music. And if not nothing is lost.

I think most recommendation systems can cluster similar users together, and so avoid your problem. But I think very few recommendation systems do "exploration" instead of "exploitation". Ie just recommending whatever you are the most likely to like, and never trying new things.

Yeah, I agree recommender systems need exploration. But user-centered approaches have closely related symmetry problems.

On vector similarity, I'd recommend a custom variant of the uncentered Pearson correlation as it better accounts for a balance between discreet and continuous values with vectors that might vary in length.

If you want to get real fancy you can use variants in combination with the uncentered Pearson correlation including variants of KL distance to RV distance calculations if you can fit your such that those become effective.

It's back for now until the next completely unplanned "hacker news DoS" :-o

good stuff! nice high level overview, would love some more in depth follow ups as well. maybe a discussion about how the different methods scale on huge user-item matrices. keep it up

dang, this looks like it might be good but it seems to be down

Reply on news.ycombinator.com