Finding the most depressing Radiohead song with R | Hacker News Reader

Finding the most depressing Radiohead song with R

rcharlie.com

177 pointskevlar181878 comments

So presumably the purpose of this experiment was to quantify just how bad algorithms can be at inferring meaning?

Trying to measure the sadness of lyrics with an algo is tantamount to the early scene in "Dead Poets' Society" where the students are supposed to measure the greatness of poetry by plotting it on a chart.

It's demonstrated in the data: one of the most depressing songs of their entire oeuvre is "Fitter, Happier" but because it's completely metaphorical (or sarcastic), the algo rates it as one of the happiest.

This will always be a big problem in AI. Measuring the literal meaning of words is easy. Measuring the semantics and context of words is a completely different matter.

> This will always be a big problem in AI

I agreed with everything you wrote except for "always".

Considering even actual humans often have disagreements about meanings of songs, I'm not sure how an AI is supposed to deduce the "correct" one.

Neither am I. But 10 years ago I would have said I had no idea how an AI was supposed to recognise objects in a picture, and now they do it better than humans. Tech moves fast.

Recognising objects in pictures is something that, in principle, a machine could do better than humans, because there is ground truth.

I think it's not possible in principle for a machine to do better than humans at recognising human emotions, because a human can't exactly be wrong at this. (Other than edge cases such as not knowing the right name for an emotion, perhaps.)

But suppose a sample of 1000 humans answered a survey on emotional content of a song and a machine was able to predict the distribution within some tolerance. In a sense, this would be superhuman performance, but even if we don't like that interpretation, it would still be very good performance. And it is possible in principle. So the argument that humans sometimes disagree doesn't prevent a machine doing well on the task.

Another possible output for this type of task is an opinion or interpretation and a justification of it. A machine might feasibly be considered "good" if it output an unpopular interpretation with a good justification.

Great point! The same type of argument is going in the AI subfield of computational creativity. A computational system might produce a piece (of art or music, say) which nobody initially likes, but could provide a justification/interpretation which might be convincing. In some ways this would be a big improvement over a pure Turing test-style judgement of the output itself. I think Margaret Boden has written this idea into her definition of computational creativity.

EDIT: "The ultimate vindication of AI-creativity would be a program that generated novel ideas which initially perplexed or even repelled us, but which was able to persuade us that they were indeed valuable.", from Boden, Creativity and Artificial Intelligence, 1998. I believe, based on her other writings, that she thinks of it as a sufficient, but not a necessary condition, for AI creativity.

I don't think that the imprecision of human song interpretation can be used as evidence that AIs are inaccurate.

I thought before writing "always". But I do think it will be difficult for the foreseeable future, because as @Grue3 already pointed out, it's ambiguous even to humans.

The central problem of getting a machine to understand metaphor, irony and the like is going to be Hard by any definition.

Being ambiguous to humans doesn't make it an ill-defined task. A machine which perfectly mimicked the judgements of just one human would be doing pretty well, well enough that we couldn't justify saying "judging emotion is a big problem for AI" unless we said the same thing about humans.

Having said that, I agree that it seems a hard problem for the foreseeable future.

> So presumably the purpose of this experiment was to quantify just how bad algorithms can be at inferring meaning?

That's a bit harsh, but I agree.

Words & music are both key to understanding a song's "mood". Only looking at words can completely strip the irony/sarcasm off a song (as can only paying attention to the arrangements and melody/harmony.)

How would such analysis even work on artists like Dylan?

They use Spotify's valence score as a measure of the mood of the music.

I think the purpose of this experiment was to have some fun.

Indeed :)

I think you will find wildly varying perspectives on whether a given song is depressive or not. To my surprise, when I read the YouTube comments on Radiohead's track Nude, I found many of the commenters found the song depressing, because of the lyrics:

"Don't get any big ideas, they're not gonna happen."

But, as far as I can see, this is only depressing if, first, you expect all your big ideas to happen and, secondly, you associate a big idea happening with happiness. I mean, why would a big idea not happening be depressing, unless you believe this big idea will make you happy?

I quite like the reminder that big ideas usually don't happen, unless you put in an earnest effort. And that the expectation of happiness from a big idea is the only thing that can cause depression if it doesn't work out, which it often doesn't.

In my experience, big ideas come out of happiness, not the other way around.

I think it's the "don't get any big ideas" bit that is depressing.

Well Thom said it's about [believing you're actually wonderful when you know it's not true](https://soundcloud.com/frank-d-3/big-ideas-dont-get-any)

Thom Yorke once said that after touring for OK Computer he was at a local pub and some guy stopped him and said, "No Surprises" is the most miserable song he'd ever heard and why had he written it? At the time Thom said he had to concur with that assessment of the song. I can't remember what interview this was from but I used to have the recording of it where he introduces the song this way.

Well considering it is about committing suicide via exhaust fumes, it must come near the top of the list.

I was hoping to see it on the list. Also morning bell, but I didn't notice either. Was there a complete list anywhere?

I once walked into my baby's daycare to pick him up after work and the babies were listening to a happy glockenspiel version of No Surprises.

It felt very strange.

The makers of those albums are quite prolific: https://en.wikipedia.org/wiki/Rockabye_Baby!

Oh thanks! I'd tried and failed to find it myself.

Cute! But, well, this is obviously bullshit:

> the cheeriest, 15 Step

The title line: "fifteen steps and then a sheer drop". A song which makes explicit references to, well, execution by hanging is...

(And "you used to be alright; what happened?" is a repeated motif.)

> The title line: "fifteen steps and then a sheer drop"

This song is obviously about Thom Yorke's frustration with the Mario Party minigame "Shy Guy Says". Think about it, 'first you reel me out then you cut the string', 'won't take my eyes off the ball again' (ball, in this case, meaning shy guy's flags). It makes perfect sense.

-- lankeyjb202

They're missing In Rainbow's Disk 2, which includes the song "4 minute warning" which I'd consider the "most" depressing Radiohead song: https://www.youtube.com/watch?v=VtotpiSL700

thank you for pointing this out. i only recently discovered IR2, and i found that "4 minute warning" fot my mood very well.

I think this is an example of something machines can't quite do better than humans yet, though it's an interesting problem to solve (trying to analyze how people in general will feel about something).

Personally I'd give it to either How to Disappear Completely, or Exit Music (For a Film).

Exit Music might be their most bitter song, but I'd say Street Spirit or How to Disappear are the saddest. I wonder how many people would have an immediate answer at the ready, and how many would have to ponder and classify ( assuming similar familiarity with the Radiohead catalogue)

So many possibilities, but my mind went straight to Pyramid Song. I think there is no way to objectively answer something like this. Even if you get some kind of majority in favour of one particular answer, that doesn't make it "true".

http://antiquiet.com/music/2012/06/flashback-thom-yorke-expl...

“‘Street Spirit’ is our purest song, but I didn’t write it…. It wrote itself. We were just its messengers… Its biological catylysts. It’s core is a complete mystery to me… and (pause) you know, I wouldn’t ever try to write something that hopeless… All of our saddest songs have somewhere in them at least a glimmer of resolve… ‘Street Spirit’ has no resolve… It is the dark tunnel without the light at the end. It represents all tragic emotion that is so hurtful that the sound of that melody is its only definition. We all have a way of dealing with that song… It’s called detachment… Especially me.. I detach my emotional radar from that song, or I couldn’t play it… I’d crack. I’d break down on stage.. that’s why its lyrics are just a bunch of mini-stories or visual images as opposed to a cohesive explanation of its meaning… I used images set to the music that I thought would convey the emotional entirety of the lyric and music working together… That’s what’s meant by ‘all these things are one to swallow whole’.. I meant the emotional entirety, because I didn’t have it in me to articulate the emotion… (pause) I’d crack…. Our fans are braver than I to let that song penetrate them, or maybe they don’t realize what they’re listening to.. They don’t realize that ‘Street Spirit’ is about staring the fucking devil right in the eyes… and knowing, no matter what the hell you do, he’ll get the last laugh…and it’s real…and true. The devil really will get the last laugh in all cases without exception, and if I let myself think about that to long, I’d crack. I can’t believe we have fans that can deal emotionally with that song… That’s why I’m convinced that they don’t know what it’s about. It’s why we play it towards the end of our sets. It drains me, and it shakes me, and hurts like hell everytime I play it, looking out at thousands of people cheering and smiling, oblivious to the tragedy of it’s meaning, like when you’re going to have your dog put down and it’s wagging it’s tail on the way there. That’s what they all look like, and it breaks my heart.”

For me, Fake Plastic Trees came to mind immediately. Devastatingly sad, but impossible (today) to teach a machine to understand why.

For me, it's Videotape. It captures the mood perfectly (IMO) on many levels, including the excessively long, out of beat automated drum outro.

Motion Picture Soundtrack is also quite glum.

I vote for the happiest song being Anyone Can Play Guitar from Pablo Honey. It is hopeful and encouraging. It speaks of life goals achieved.

This article demonstrates how far AI has to go.

https://m.youtube.com/watch?v=GIWwfWaWuaE

Good choice! I've also always thought 'Lift' felt happy.

If the winning song wasn't written in D-minor, the algorithm will need further tuning.

  > and then used the tidytext package to break the lyrics into words, 
  > eliminate common "stop words" like 'the' and 'a', 
  > and count the number with negative sentiment

Is 'not' a "stop word"? How did it count negations after "stop words" removed, i.e. "I am not forgotten". Positive, negative?

"not" is a stop word. I don't see an attempt to negate the word sentiments.

Would be interesting to see how it handled REM or Dubstar, both of whom have some really upbeat tunes, with really dark lyrics - "The One I Love", for example, with "a simple prop to occupy mind ", or "Not So Manic Now", a happy bouncy pop song about assault...

'Depressing' has a variety of facets made up of sadness, textual content/lyrics, and (at least for me) the placement of the title within its respective album context.

Anecdote: When I previewed Kid A via AudioGalaxy back then (it would not be released for another two weeks in my region) I thought the stylistic choices of 'Everything in its right place' were some kind of encoding error (quite frequent back then) so I trashed the whole album after spending hours on the download.

Imagine my surprise when I listened to the album from the physical album I bought a fortnight later :)

I got my first cassette tape player in 1982 and Prince's "1999" album to go with it. The lead track starts with distorted vocals, and I thought my player was malfunctioning. The intro is only about ten or fifteen seconds, so unlike you, I didn't trash the tape.

Closer to topic, listening to the song "1999" as a kid, I thought it was a cheerful party song. After Prince's death, I listened to the song as an adult and realized it's about nuclear annihilation.

While great to see, this also highlights just how far we are from where data science needs to be. The amount of code necessary to just pull down and reformat data before any real analytic work is done is always staggering; I see it in my work but it's even painful in a fun example like this. I know, 80% of time in prep, 20% in analysis, yadda yadda, but I hope we can turn some of this magical AI into making data transforms easier in the future so we can actually get to the analysis portion faster, along with more accessible "tidy" data to get to the analysis portion faster.

And no, tools like Paxata or Tamr (or http://www.predictiveanalyticstoday.com/data-preparation-too... etc.) are indeed a good start, but are not really the solution yet.

We'll always have to code our way out of some wacko data situations, but I look forward to (and dream of!) better and better libraries and approaches to getting prep prepped faster.

R in general is inefficient with nontabular data manipulation with text data and web scraping, which is why I fall back to Python for those tasks.

But even in the context of the article, the code the author posted is extremely confusing and is an odd mix of base and dplyr code.

"Honey Pablo"?

Definitely Exit Music for a film.

If I can nitpick a bit about your presentation here:

http://rcharlie.com/htmlwidgets/fitterhappier/album_chart.ht...

If you're going put each album on its own point in the X-axis, could you not just label those axis instead of using colour-coding and requiring us to look up the corresponding colour in a legend (that's not ordered in the same way).

They need to do this with morrissey.

As my dad said back in the day(80's), Morrissey is the music you listen to while cleaning your guns.

No, Morrissey has _far_ more irony and humour.

The Spotify Valence score wouldn't be helpful, as the melodies and beats are often upbeat despite the subject matter. And The Smiths? Forget it.

Is there a way to have Spotify users create the mapping, e.g. by using song selection to detect their mood and finding the cluster of Radiohead songs that are only ever listened to in a depressive mood?

Note how listening to all suggestions mentioned in the HN comments but only to few or none of those recommended in the blog post is a good indicator of what you think about the power of ML/AI.

Guessing "How To Disappear Completely" before I read it

and...not on the list. Exit Music and Let Down are pretty glum though

Ingo Mierswa (of RapidMiner) wrote some papers on analyzing audio data, like

http://link.springer.com/article/10.1007%2Fs10994-005-5824-7

http://dl.acm.org/citation.cfm?id=1150523

I'm waiting for someone to find out the most stolen songs from Radiohead's repertoire using R now :)

True Love Waits

Edit: according to the article :)

How on earth is Street Spirit not even on the list?

Yorke himself had called it "our purest, saddest song.", and went on to say:

I can’t believe we have fans that can deal emotionally with that song. That’s why I’m convinced that they don’t know what it’s about. It’s why we play it towards the end of our sets. It drains me, and it shakes me, and hurts like hell every time I play it, looking out at thousands of people cheering and smiling, oblivious to the tragedy of its meaning, like when you’re going to have your dog put down and it’s wagging its tail on the way there. That’s what they all look like, and it breaks my heart. I wish that song hadn’t picked us as its catalysts, and so I don’t claim it. It asks too much. I didn’t write that song.

True Love Waits is a sweet, quiet, sad little song about yearning and the fear of loss.

Street Spirit is pure dark tragedy.

Apparently the second happiest song on OK Computer is "Fitter Happier". (A pig / in a cage / on antibiotics.)

So much of Radiohead's bleakness is in their use of irony.

Q.E.D.

Irony and taunting really confounds sentiment analysis. I mean, street spirit ends in "immerse your soul in love" repeated 2X but is still incredibly bleak.

If, like me, you read that quote and wondered what exactly the song is about, then here is another fragment of the quote that might help:

They don't realize that 'Street Spirit' is about staring the fucking devil right in the eyes... and knowing, no matter what the hell you do, he'll get the last laugh...and it's real...and true. The devil really will get the last laugh in all cases without exception, and if I let myself think about that to long, I'd crack.

(http://songmeanings.com/songs/view/581/)

It is for Thom Yorke, that doesn't mean it should be for everyone else as well.

https://m.youtube.com/watch?v=ZQCYM_cX_pE And remixed into a poppy trance record in 2000

It is also one of their most beautiful.

Creep is the most depressing song for me (as a teen from the 90's).

I can't believe the author got the name Of their first album wrong!

How to Disappear Completely is their most depressing song, ever.

I wonder how Gloomy Sunday would score with that heuristic.

you could have just asked and I would have told you it's "let down"

is the answer ALL OF THEM

(j/k)

This is a repost (syndication?) of a blog post at:

http://blog.revolutionanalytics.com/2017/02/finding-radiohea...

which is a summary of a blog post at:

http://rcharlie.com/2017-02-16-fitteR-happieR/

I didn't know that Spotify has an API. I see mention of a rate limit in the docs, but I can't find the actual limit. If I want to sort the tracks from albums released in a given year (say, ten to fifty out of a thousand-album collection) by Spotify popularity, will the limit get in the way?

Yeah can we please change this to the original blog post at rcharlie.com?

Agreed. I'm not sure why the original link is necessary at all. It doesn't add anything and seems to have been written for no reason other than to write a blog post. The author refers to Radiohead's first album as 'Honey Pablo', so it doesn't look like he actually paid attention to the original article.

Thanks! A moderator updated the link from https://www.r-bloggers.com/finding-radioheads-most-depressin....

TLDR: It's High and Dry.

I could have guessed that. But where is Black Star in the list? That is my go to song if I'm really lamenting the state of the world. ;)

No it's not. It's actually True Love Waits.

High and Dry is 8th.

TLDR; RTFA

You're right. I skimmed it and only saw the first list of Saddest Lyrics.

I'm going to go away now, and listen to Radiohead, whilst I consider my lack of focus, past failures and impending mortality. ;)

Reply on news.ycombinator.com