Using Spotify data to predict what songs will be hits

In the section on dataset features, they include "popularity" (calculated by Spotify) as well as Billboard chart stats like weeks, rank, and a custom-made "score". To me it's not clear whether these features were hidden from the train/test sets or whether the popularity features were only used in their "artist past performance" measures.

If they included these popularity features, it's like asking "can we predict whether a song is a hit just by looking at how popular it is?" If it is the case that they peeked into the future and observed ex-post song popularity, obtaining just 89% accuracy hints at how unpredictable song success truly is. Check out [1] for a famous study of song success which experimentally demonstrates the unpredictability of song success.

[1] Salganik, M. J., Dodds, P. S., & Watts, D. J. (2006). Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market. Science, 311(5762), 854–856. https://doi.org/10.1126/science.1121066

We ended up using the million song dataset, because I'm not sure Spotify gave out this data six years ago, which includes various info about roughly a million songs including artist, length, and supposedly Echonest api results for things like "dancyness". We then merged this with a list of something like 250k results of play counts. We then found out the Echonest data was quite literally all just set to null, so I went out to their api, signed up for a developer key, and spent six days querying to fill out our dataset.

We were massive novices to machine learning, so we basically were just script-kiddying it, and pretty much none of the models we made over a 24ish (because we were dumb college students doing things last minute) period had any significant accuracy. Finally we made a random forest model that was able to, with 80% accuracy, predict the "magnitude" of plays, ie roughly whether a song would get a million plays or a thousand.

When we broke it down (model explainability is an awesome feature) we found that out of everything interesting we had done with feature investigation and data cleaning etc, the model was about 90% based on which artist made the song. In retrospect, that makes sense, in a sort of cynical way; even a great song by an unknown artist rarely makes it big. The moral of the story I guess is that machine learning isn't magic

I still have all the data, and I've been meaning to revisit it now that I actually have a better understanding of the field. It's on my list of things to revisit/do, a very long list

Song popularity in different groups was very weakly correlated. Some of the same songs would sometimes be hits and sometimes busts for no apparent reason.

In other words, even song popularity (in an i.i.d. trial) doesn't predict song popularity.

https://www.nytimes.com/2006/02/14/health/in-music-others-ta...

But now with algorithmic suggestions there is a third trend-motivator, which is music listeners and music producers reflecting on the overall system for statistical trends and correlations. I expect this motivator to push out the other two because it makes labels more (and safer) money and consumers seem to like it. The conditions for success are now highly metricked and quickly A/B testable. It’s like the internet before and after PageRank—and just the same, I would expect a cottage industry of “SEO” to pop up for the music industry.

> In order to balance our data, we randomly sampled 12,000 non-hits from the Spotify data and created a new dataset. This dataset contained approximately 12k non-hits and 12k hits (∼24k tracks total).

Won't a indiscriminate random sampling of the non-hits introduce a temporal sampling bias, as a) music trends change over time and b) music output is not equal across years?