Predicting hit songs with 97% accuracy
frontiersin.org
frontiersin.org
>Small data sets are not appropriate for machine learning as they lead to high bias in their results (Vabalas et al., 2019). To address this, we created a synthetic set data with 10,000 observations using the synthpop package in R (Nowok et al., 2016). This standard automated procedure creates observations by repeatedly randomly sampling the joint distribution of the data. This technique is used when obtaining large datasets is infeasible, including analyses of computer vision (Mayer et al., 2018), sensitive information like hospital records (Tucker et al., 2020), and with unbalanced data (He et al., 2008; Luo et al., 2018).
Isn't this just making up data? How is the analysis of the synthetic 10,000 observations (based on observations from 33 real people) at all meaningful?
In grad school I encountered experiment after experiment, paper after paper where the authors went around emphasizing the parts which were statistically “sexy” while ignoring glaring issues like this
It’s almost like the methodology for creating science is more rigorous than the science itself
Sometimes the sample is too small. Sometimes they assume a correlation that doesn’t exist. It’s troubling and was a large part of why I left academia
In the statistics department, of all places! Where the importance of being forthcoming with your data should be best understood!
Indeed, "predictive accuracy" on simulated data is largely a meaningless metric in any case. The simulation's set up determines the predictive accuracy of any model of it.
The only value of this metric, post-simulation, is to determine whether or not an experiment is worth the ROI -- ie., it's essentially a way of determining whether some set of premises (simulated) are possibly consistent with a given predictive project.
They included a discussion in appendix, but I cannot open that file.
As a statistician it still feels very weird to train on synthetic data, because my evaluation of model uncertainty (parameter uncertainty and predictive uncertainty) is (usually) based on the data the model is trained on. But I understand that ML typically evaluates models based on out-of-sample data, so training on synthetic data makes sense in that case. In this paper it appears they're also evaluating on the synthetic data, which seems problematic:
> One-half of the synthetic data was used to train the bagged ML model and tune the hyperparameters. The other half of the synthetic data was used to test it.
I can't figure out which journal or conference this article was published to, but if I sat on the reviewer board I would have declined their article on this fact alone.
Good find.
Neither of those is what was done here, which was fitting the model once on a greatly inflated synthetic sample.
From the article:
> Nevertheless, less than 4% of new songs will become hits
So 96% of songs are not-hits. If the model predicts that no song will be a hit, then its accuracy is already 96%.
TN = correct non-hits = 96
FP = incorrect hits = 4
FN = incorrect non-hits = 0
Accuracy = (0+96)/(0+96+4+0) = 96/100
This is just a problem with using the metric of accuracy. If your classes are unbalanced, then a few “errors” aren’t going to affect your accuracy.
Imagine a Monty Hall situation with 100 doors, you can predict with 99% accuracy which doors have goats but you only have 1% accuracy on cars.
I'm surprised that one was not prominently displayed in this article. They are standard practice for evaluating binary classifiers.
[1] https://www.dataschool.io/simple-guide-to-confusion-matrix-t...
A machine learning algorithm was trained on 5000 of these observations and tested on the other 5000 observations and the 24 songs.
It got 97% of the synthetic observations correctly labelled and 23 out of the 24 songs.
Unfortunately, it can clearly be seen that the generation of the synthetic data based on all 24 songs means overfitting to the data can easily take place (despite what the authors think their 10-fold cross validation proves)
Without proper separation of training and test data this is a garbage study and tells us virtually nothing.
The albums often contained at least one track that was “catchier” than what was played on the radio. I always wondered why these never made the rounds.
Fast forward 20 years and I’ve seen a handful of those songs come back and make the rounds on pop channels. I’ve always wondered why they didn’t do numbers out the gate. Maybe they were missed at the time or maybe they were ahead of their time and needed tastes to catch up.
I could see taking an AI model that predicts a positive response from “the masses” being unbelievably valuable to anybody sitting on a massive catalogue of historical content. Most of that content is only valuable, at this point, in bundled deals.
But the prospect of there being a handful of diamonds sitting in there that could be panned for with AI? I imagine that’s billions of dollars waiting to be tapped at a near zero cost (plus, possibly, the cost of remastering).
(P.S. for fellow album lovers out there, I built https://audile.blankenship.io for personal use, it’s free have at it)
There's another element at work that gets way too much weight in the charts: Sales figures. As in, "how many people actually bought this single or an album with this single?" This is the main reason why only major labels end up in the charts: They're the only ones allowed to report figures (for several reasons; a few legitimate one's too).
Let's say you make a great song and upload it to Spotify. Within a month it has been streamed 100 million times, making it one of the most streamed songs that month. Your song won't be in the Billboard top 100 because it's not on a major record label and Billboard won't be able to associate the pirate uploads of that song to YouTube (where it also tracks song data) with you, who never registered it anywhere other than Spotify.
There's also entire categories of music that basically have no chance of ever making the Billboard Top 100 because they're "unofficial" (think: Unauthorized but legal remixes).
I am running out of typing time here but just know that the main differentiator is the record label. Unless the song is on a label that actually has the necessary resources, getting into the Billboard Top 100 isn't likely to happen. There will be occasional exceptions of course but the main takeaway is that the Billboard Top 100 is primarily a marketing tool for the big record labels in the music industry. If it ever stops prioritizing their stuff it wouldn't be serving its purpose.
https://www.google.com/search?rls=en&q=how+does+billboard+ca...
The tell that this comment is nonsense is the assertion that only major labels count. That's never been true at all and anyone saying that is telegraphing that they aren't really connected to this issue.
Registering with SoundScan and getting UPC/ISRC is trivial and many services (like HN user founded Distrokid) automate portions of it.
None of this is new. Those of us that were on tour playing music in the 90's got used to filling out Soundscan forms on paper at the merch table and finding a way to fax them in. Indie artists have been reporting sales to the charts for decades.
Also there's no such thing as an "unauthorized but legal remix" that's just someone else's song. The parent comment is "not even wrong" levels of wrong.
I've only heard Rush once - when Neil Peart died.
Similar for other (dead) artists.
Whatever label Rush are on, they ain't doing the payola to get airplay in my locality.
With that said I wonder if they are suffering from a base rate problem.
The problem is that hit songs are VERY rare. If you feed people a ratio of 50% hits and 50% random stuff you might get solid numbers just by asking a guy named Bob what he thinks too. Hits are fairly distinctive when compared to random DIY music. So Bob will just pick the professional sounding stuff and Bob will be mostly right.
To be useful though you’d have to feed the system 20,000 songs, of which 100 are highly professionally produced songs from people who are competent and then identifying which 5 of those 100 are going to be hits, as well as the one fluke hit that’s in the DIY pool.
I bet it can’t do that.
The real task for those who actually do this is being able to sell songs that basically seem like they might be hits from ones that actually are.
The reason is that it costs money and time and effort to try to get attention for the songs to actually make them hit songs.
Most songs are just obviously not going to be hit songs they aren’t even in the same ballpark.
The classic mistake for a budding indie record label owner or producer is to audition 31 people and then say wow this person is better than all of them so that’s the one that will be a success.
Except none of them are ever going anywhere.
Conversely, the problem for a budding major label or high profile executive is to audition 31 people all of whom are actually quite good but only one or two are going to be hits.
They’re all great that’s how they got in the door, but which one is the star?
These are hard problems. Harder than people realize.
So anyways for a machine system have to use songs that obviously have no real strong appeal to anyone at all as the starting point for analysis.
Otherwise it’s just useless.
I would have had more faith had they scrambled the labels and run the experiment of distinguishing one random subset of songs from another random subset. Anything greater than 50% would indicate the model's ability to overfit; I'd like to know how accurately their model would perform.
No offense, but I have seen better ML 101 capstone projects.
You get predictable results like Zeppelin, The Eagles, etc. as the 'most hit-laden' records.
As far as actually measuring the SOUND with machine learning, you could just take a short-cut and study Mutt Lange's mixes, or you could run the computers on it, but this is not actually an interesting question because everyone's been trying to converge on the 'hit song sound' for decades and decades.
You'll get pretty generic and unobjectionable results, and you'll still get blindsided by outliers such as Bohemian Rhapsody (too impossibly long and complicated) and Somebody I Used To Know (very odd arrangement).
Targeting the most mass market possible (which is do-able with AI) is such a shotgun approach that it'll tend to lead to failure because too many talented humans have already beat that approach into the ground.
What you'd want is for your hit-song AI to 'hallucinate' in an interesting way. You want things a little off-model, a 'hook'. In a conceptual sense, not just a 'mathematically optimal catchy phrase' sense.
On further examination it turns out that nearly all the "prediction" was based off a single parameter: The song's artist.
Anyway, it doesn't matter. Science is measured by popularity (citations), and not by truth. With that in mind, it's a nice paper. It has a nice story. It has some complicated graphs. It is difficult for a layman to figure out the problem. So, yes, it's a successful academic paper. It's a great read and very interesting, according to academia.
[1]: https://michaeltauberg.medium.com/power-law-in-popular-media...
ie., whatever the trends are over period X, the subsequent period Y will depart from X (due to, eg., bordem, counter-culture, etc.).
So you cannot predict taste based on an associative statistical model.
There are tons of hit worthy songs that no one ever gets to hear out each week. What makes a tune a big hit now often involves thousands of dollars being dropped on social sites to ensure big industry music trends on front pages of TikTok, IG, and Youtube non-stop... This is what holds other music below it out of view. That is also why hits are often missing authenticity, because big industry music now is written by teams rather than individuals, and marketed with money, not organically or democratically supported by listeners.
You can MAKE a song a hit if it has the basics together and a catchy hook, and then if you drop $20k into google's a platform. Otherwise good luck... Indie music is doomed because they can't compete against big money music...
This is why "Cartel" music (by artists you've never heard of) is usually topping YouTube's charts along side popular (big industry) artists you may know every week. The music biz is not and has not been based on fairness for a long time now.
>Staff from an online streaming service choose 24 songs for this study without input from the researchers. The streaming service also provided the definition of hits or flops. This resulted in a “clean” experiment as song choice could not be cherry-picked for the study and the criterion for a hit was established in advance.
I don’t care much about hits at this moment, but if it can eliminate bad songs even by 50% then it can help build a pipeline to bring humans into evaluating the generated songs
Feels like a way to lock us into a stagnant pool.
I'm sure there won't be any challenging existential side effects of pervasive corporate surveillance, eliminating what remains of the illusion of choice.
I vehemently disagree. This may be true for the general case of music listener but there is a long tail (of both listeners and musical works) that Spotify is very very bad at.
There are also two kinds of people, people who like albums and people who like singles. I'm not an album person because it's often lackluster and repetitive, few bands can really create good albums that surprise me but singles on the other hand can shine and the band knows they can. Just because I like one song doesn't mean I'll like 5 other songs from a band. Most of my favorite music is pretty unknown. I'd imagine hits are just like that hit or miss and good marketing is what really helps a lot of things sell.
To a listener new to the genre, this yields a pretty good hit rate and the perception that Spotify is great at recommendation, but after spending any reasonable amount of time listening to the genre, the same 20-40 recommendations get stale and Spotify is completely unable to surface relevant songs from deeper in the genre catalog.
I will note though that I have been in machine learning since 2011 and generally know how these systems work so I can give them inputs that I know will reinforce their pathways for recommendations in a direction I want it
And yes I agree and expect that non human systems will adjust human preferences over the long run. That’s the point right? A caretaker assistant that is smarter, more capable and can better predict your needs than anything ever