How to Shuffle Songs? (2014) | Hacker News Reader

How to Shuffle Songs? (2014)

engineering.atspotify.com

78 pointspablode64 comments

Ok, so this article proves that they know how to do it, but they seem to have forgotten over the intervening 9 years - according to some, the shuffle function now prefers similar songs to "keep the vibe", but according to the Spotify support pages, that's a "smart shuffle" feature, and I don't have that because I'm too cheap for the paid plan. Anyway, shuffle should play the songs in a playlist in random order, but make sure to play all the songs (like shuffling a pack of cards and then looking at the cards one by one)! While Spotify's current "shuffle" just seems to pick a random song to play after the current one, with some songs repeating three times while others are never played. Annoying as hell...

if you're not on the paid plan you have ads every now and then, right? how is that not more annoying than the non working shuffle?

Well, if you are using the web version, there are ad blockers (e.g. https://chromewebstore.google.com/detail/spotify-ad-blocker-...)... Actually, if you're familiar with web tech, it's not that hard to write your own ad blocker, then you're not dependent on the extension provider updating it when Spotify changes the names of the HTML elements again. But I'm confident they are already working on a WASM/canvas-based version which would make such attempts much harder.

Winamp had it right.

That's why it whipped the llama's ass.

And they also push songs you recently liked more often.

I'm a paying user so I can't speak about the free Spotify experience, but..

Spotify will only repeat a song during shuffle if you also activate repeat. It will also repeat a song if you actually added the song twice or more to said playlist. You can try it out by shuffling a playlist+deactivate repeat then check which songs have been queued from start to finish.

Smart Shuffle -despite the name- is more a playlist function than a shuffle function. It's like the Radio playlists but with the songs interspersed temporarily in your own playlist. Note that these songs were not already in your playlist - so they do not repeat unless you activate repeat.

Spotify designtes songs that play that way because it's a casino rigged against playing unknown and independent artists. Each time a song is played, Spotify has to pay royalties, by steering money to artists that buy spotify ads, and to major labels, they get major kickbacks.

If they paid indie artists properly, their operational budget would be much lower. Payola is the real reason why music doesn't play randomly almost anywhere anymore. It's even the case with many app music players that you download in app stores, the whole music industry is skewed to favor just a few well known and industry planted artists under the hood. And it's bent on getting worse..

If you hear an already popular TikTok song playing in the supermarket or at a gas station, it's literally driving royalties up for the involved artist. It's also why you rarely hear new music from small artists, and why Taylor Swift dominates news media every time she scratches her nose.

If you are regularly featured on the front page of YouTube, Spotify, and on CNN.Com, you dominate streaming now, and nobody else can get that spot without being backed by major money.

This is why I back up and always carry my own personal music collection. Streaming now is a generically engineered and controlled repetitive hellspot for me without real choice.

This is one of my favourite things about psychology, perception and programming.

When people say they want to "randomly shuffle a playlist", they really don't want that. They want the perception of randomness, but _true_ random doesn't abide by rules.

If you're actually random, there is a probability that it might just play all the songs straight through. Or just pick songs by a single artist back-to-back.

I want all songs truly randomized once, then playing in that order.

Sadly youtube just picks the next one based on the current ones ID + some seed, so it constantly gets stuck in loops

I'm amazed youtube hasn't figured out a way to detect and avoid these loops, it seems like a solvable problem. Even a naive solution like having the client remember the last n played tracks, and not auto-play any of them (picking the next one down) would be a big improvement.

Sony's CD players used to choose the next when the song finished, however it removed the ones it played from the candidate list, so it never looped.

As another neat trick, they have added a small animation while seeking to next track with some lower time bound, so it looked like thinking for a second or so. In reality, the decision was instant, seeking was not.

I have had the (conspiracy) theory for a while that this is by design. Kept happening to me on an account where I've disabled watch-history. Now I'm watching on an account where it's enabled and it hasn't happened even once. And it's been months since I switched.

Actually not sure if it's even that far fetched. Making the logged-out-experience worse on purpose is something all the big internet firms do.

But... You've asked YT not to remember the previous videos, and it doesn't: LGTM.

There could be advanced implementations (like with the remembered passwords, where YT would store a list of seeded hashes of your viewed videos). But they are that: advanced implementations, that impose an extra cost.

Yea I guess you're right that in this case it probably boils down to extra cost.

Though I still think "not remembering watch history" (and adjusting what you are shown based on that) to mean that a functionality that doesn't have anything to do with tracking (because it only requires information from your current session) to be a little lazy.

You wouldn't expect a slide-show of images to be shown in random order, just because you didn't consent to the cookie banner.

But in all fairness - I really like youtube's auto-play and suggestion algorithm when it comes to music. I use it for free and I haven't seen an ad in years so I don't want to complain :)

I have watch history enabled and still get the loops all the time.

I even went so far to seek out a website that does it for me, and which I set up as a bookmark https://youtube-playlist-randomizer.bitbucket.io/

Altough, seems like the site is now broken

That's interesting.

I used to get it all the time and haven't experienced it since switching to an account where it's enabled. I guess it may have just been a coincidence.

> When people say they want to "randomly shuffle a playlist", they really don't want that.

Same in games. "I had an 80% chance to hit and didn't" is a perfect example.

Leading to features like "Karmic Dice" in the recent Baldur's Gate 3

There is a consideration there that isn't necessarily a problem with randomness itself. (Though I agree that the randomness does bring its own problems.)

Say you're playing through someone's electronic D&D module and you try to do something with a 90% chance of success. Then you roll a 2. You didn't make a mistake when deciding what to do, so that feels wrong -- this is a problem with randomness.

What do you do? You reload from right before you tried to do the thing, and you reroll. The game can't actually make you accept randomness if you don't want to. So they have to do something else.

It also depends how well that randomness makes sense in games. In my recent memory nothing was more ludicrous that the modern XCOM games where you were upclose to an enemy and got only 90% hit chance with a shotgun.

To be fair, this is due to game devs giving percentage chances that are straight-up lies. "80% and didn't hit" is actually "92% and didn't hit" in many Fire emblem games, for example (https://fireemblemwiki.org/wiki/True_hit).

Back in the 90s, when portable CD players were still a thing, I had one.

It had a toggle for random play. It indeed apparently played the tracks at random.

By random I mean random, not shuffle. It would play the same track multiple times during a listening session. For example tracks in order like 2, 2, 7, 8, 7, ...

I actually liked that.

If memory serves, it would stop playing after it had played each track at least once. If there were a lot of tracks it would not finish, at least not before the batteries ran out.

Hah, that reminds me of Spotify. If you had shuffle on and it happened to play the last song it would stop playing, because it was the last song in the playlist ...

Rob Pardo (Blizzard, World of Warcraft (2004)) said they decided to model randomness after how players thought it worked.

> ..at least once a week, a developer would come to my office, and say "the random number generator is broken, we need to look at it."--It's because they hit a streak. [1]

[1] https://www.youtube.com/watch?v=FhC0NaB6ock&t=2343s

He shares more details about game design choices in the video.

Video games are rife with randomness manipulation.

- X-Com 2 lies in favor of the playing when displaying a chance of hitting of 85% [ https://www.gamedeveloper.com/design/jake-solomon-explains-t... ]

- Baldur's Gate 3 "Karmic Dice" will prevent streaks of very high or lows rolls

- Tetris games doesn't use Math.random for piece distribution. Most modern Tetris (like Tetris Effect) uses Fischer-Yates with 7 pieces (in the community we call 7-bag) or 14 pieces (it avoids repetition "at the seam" of a bag). The Tetris game I like the most (Tetris the Grand Master) uses a history system (the chance of receiving a recently distributed piece is heavily reduced; it's less predictable than 7-bag). IIRC even classic NES Tetris has a repetition avoidance system.

I don't know if that's still the case (haven't played in more than three years), but Magic the Gathering Arena (digital version of the collectible card game) used to rig the opening hands to make them "more average".

This had consequences on competitive play; they wouldn't even tell you how exactly the algorithm worked, so the only way to build a deck was lots and lots of simulations.

There are also specific physical shuffling techniques used in physical magic the gathering games, in order to distribute mana more "evenly".

Yeah, but those are cheating, and it's not particularly easy to cheat in that specific way. In a high level match, whenever you shuffle your deck, your opponent can (and in fact is required to) also shuffle it, and they're the last to touch it.

Cheaters, like in any game, tend to look for methods that maximize the combination of impact and plausible deniability. Marked sleeves, sleight of hand (pun very much intended) etc.

It can also be used in friendly games, where everyone is aware of it.

there was a whole HN post about the Tetris RNG a few months back. I'm too lazy to search rn but it might be worth a read for others.

previously:

https://news.ycombinator.com/item?id=20872110 - Sep 2019 (41 comments)

https://news.ycombinator.com/item?id=37107346 - Aug 2023 (16 comments)

Jonathan Blow did a good talk on this, but unfortunately the only version of it I know of is concatenated by some youtube uploader with a silly twitch stream (also by Jonathan Blow, but less interesting, and not even sure he is correct about that particular game being rigged?) so just stop watching after the presentation: https://www.youtube.com/watch?v=l0KEDYFWbVc ("Explicit Lying" seems to be the title of the talk?)

A shuffled deck of cards is still random, even if the probabilities of each card draw aren't independent.

An easier solution is to not say e.g. "this attack has a 25% hit probability" but rather say "this attack will hit every 4th time".

Then you are not lying to the users, and your tooltips/docs etc. are correct as well.

The problem isn't that people "don't want random", the problem is that "random" isn't unambiguous. Shuffling a playlist before playing it is just as 'random' as picking any one of the tracks to play next.

> there is a probability

There is a probability that entropy will decrease, it's just small enough to neglect. Playing songs straight through isn't really a concern.

Not so related, but I actually like Spotify’s algorithm for suggesting new music a lot. Currently my daily routine for discovering new music is 1) pick an album that has something to do with what I want, 2) start the last song in the album, 3) skip it. It works pretty well!

I think this gives you similar material to the Song Radio functionality. You could use Album Radio or Artist Radio as a shortcut. I use these and Discover Weekly to discover a lot of new tracks. I know some people don't find discovery via Spotify to be useful at all, but like you, I think it's excellent. I've come across thousands of new tracks, and their strike rate for suggestions is generally very high.

It might help that my bar for favouriting a track is as background noise for the office or around the house rather than purely 10/10 killer desert island tracks.

I once implemented an algorithm that went the entire different direction, because I wanted to shuffle across my entire collection. This is back in the era of the filesystem being the main driver, so I had my files arranged in [Very Broad Genre]/[Artist]/[Album], and what my algorithm did was that when it came time to pick, it picked something out of the current album with ~80% probability, kept the artist but switched album with like ~10%, and switched the genre entirely with ~10%. It also excluded the last N songs played, with that number generally larger than the number of songs in one album, so if it shuffled its way through one album it would be forced to jump up at least one level. The last few albums were also kept in the block list, so if it left one album it wouldn't go back to it for a while. (Genres I didn't worry about so much.)

The idea is that it was random, but that it tended to keep the same mood, and once it did switch, it would tend to stay in that mood for a while. (I suppose I would have also made it sticky once it did switch genres, but I didn't get to use it for long enough to matter.) This also trivially gives you three "No, not this, go to next" buttons, one for the album level, one for the artist level, and one to switch out the genre entirely.

Unfortunately, I have not had time to pour over the mp3 players around to see who has pluggable shuffle algorithms exposed conveniently enough to make this work, and unfortunately it's a lot of work to tag MP3s enough to make this work since those "genre" tags are generally pretty bad, so I haven't had this in a while and I've gotten used to just manually curating my genre mood shifts.

I don't like any of the cloud service shufflers I've used, though. They are concerned about aspects of the song I'm not and oblivious to the ones I do care about and the net effect is about like listening to a radio station, where I end disliking every third song or so no matter how I tune it. Fortunately, I've been building my personal MP3 collection for over 20 years so I'm not even remotely dependent on them.

This is simply is not true. Their "random" algorithm prefers cached songs on the device by quite a margin. I have playlists with more than 500-600 songs in them and can account for it first hand, since I know the songs in the playlist and some seem to never show up.

Do not believe me? Simply clean spotify's cache and look closely at somehow magically new songs start popping up in the shuffle!

The only thing I really want is for albums to never shuffle. The number of times I've started listening to an album in shuffle-mode unintentionally, then only realised by the 3rd track that plays, by which time the playthrough is ruined and I have to just listen to a different album altogether...

More than a few times I’ve wished that starting the first track of an album from that album’s page would automatically turn off shuffle if I had it on. Would be a nice setting to have at least.

> Let’s go back to our users who have also fallen victims to Gambler’s fallacy. If you just heard a song from a particular artist, that doesn’t mean that the next song will be more likely from a different artist in a perfectly random order.

Wait no, it does, right? Since we are not doing random independent events, but taking things out off a finite set. Going back to the colors, if there are 4 green and 4 not-green, there's a 50% chance that the first song is green. But if the first song is green, doesn't that mean that there's now a 3/7, or 43% chance, that the second song is green? And 57% that it's NOT green?

At the extreme, if I have a single song of My Chemical Romance among 10 songs, and it plays first, I expect MCR not to play again (until the list resets, the algorithm describes a finite set, not a rolling one).

When they say "more likely from a different artist" they don't mean "more likely than before your last song" but rather that the probability that it is a different artist is higher than the artist last played (which is what the average person likely expects from their shuffle)

If we're shuffling the set of my liked songs, and 90% of my liked songs are by Taylor Swift, even after hearing 3 of her songs back-to-back on shuffle, chances are I'm hearing her again next

Yeah, you’re right. I think this is a case of simplification in communication over perfect technical accuracy. What he meant was more that song “similarity” (as in artist, album, etc) is not taken into account by the algorithm. But yes you’re absolutely right.

Supposedly, the "pros" use a combination of one-step key shifts, similar tempos, and slowly ascending "energy" (which I've never found a good definition of) to pick the next song. Key-finding algorithms don't seem to be particularly accurate from what I have heard of.

Ideally, you'd use this, plus the year and metadata about whatever microgenre. Use the radio adage of "don't hit me with a brick after you tickle me with a feather."

Occasionally do a "two from Band X" or "three from the year NNNN." I would probably weight the microgenre in the metadata by how rare it is: "rock" isn't particularly descriptive as compared to "doom rock" or "Southern rock."

It's almost impossible to define those microgenres per song because the metadata for music is shit :)

Spotify does try to do it, but it's not an exact science: https://everynoise.com

From https://everynoise.com/EverynoiseIntro.pdf

--- start quote ---

There are other ways to plot music genres, of course, than the one used by the map at any given time. We use 10 dimensions internally, and two completely independent measures of genre similarity. Iʼve flipped the map once already since first publishing it, and I might do so again without warning if I find another configuration that seems more interesting.

--- end quote ---

Oh, yeah, it's messy. Still, I was thinking MusicBrainz for some of it.

No code or pseudocode for the algorithm? Boo.

Something like this would be nice to add to my app JukeLab that turns your tablet into a jukebox, powered by Spotify.

It currently uses Fisher Yates to shuffle for auto play which does feel weird sometime.

Other polish I’m working on in the app…

How to skip skits and interviews during autoplay.

Removing (remastered) etc from album and song names correctly.

Taking total control over playback and the queue through the Spotify SDK which is buggy and hostile wrt auto playing unwanted tracks.

https://jukelab.com

I actually just wish Spotify stop spamming me the very same songs and artists over and over in virtually any "made for airstrike" radio / playlist.

Give me "NOT made for airstrike", please. I often want to start a radio off a song precisely because I'm trying to find new songs that go with that one, not so I can have an excuse to unironically listen to "More Than This" for the 20th time this week...

It seems like the discover weekly playlist does a better job at showing new songs, but you only get that once a week of course, and without any genre control.

There are also genre-based playlists and a release radar.

You can use "song radio" to get a mix of new material and favourited songs that relate to a song, album or artist. It's even been given a prominent icon if you're using Spotify via Car Play or similar.

Yes, Spotify seems to want to force you to listen to the same things over and over, almost as if someone wants to boast about how many millions of seconds of loading time is saved by their clever caching.

It's known that record labels pay Spotify to promote certain artists more (known as industry plants if they are not already established). They found a loophole so that it's not considered payola (which is illegal) https://twitter.com/AbelowRob/status/1641455112345493506

Around a decade ago, the 'radio based on' RATM's Killing In The Name was a good mix of everything, because of the push for that as Christmas no. 1, it suddenly had a lot of people listening to it who regularly had their tastes in all sorts of other genres. It was quite a fun bit of whiplash to go RATM > SOAD > Dolly Parton > Radiohead > The Kinks and so on

Almost felt like a little 'hack' to get a broader selection of suggestions for a few years (though still largely mainstream stuff)

I used to have a Turtle Beach AudioTron [1] (hardware device to store and play audio files to HIFI amp). Device was terrific (for its time) but the shuffle/random play was slightly off - occasionally you'd get deja vu whilst listening to an artist on shuffle - that completely spoilt the device for me, rather than listening to the music I was listening for the next repeat/pattern.

[1] https://en.wikipedia.org/wiki/AudioTron

I took statistics in college and I was a bad at it. So, the coin-lunch reference in the article would have make me raise my hand and ask: Would not be the case that if a coin toss gives me eagle 1k times in a row, there is something wrong with the coin toss? I know the chance if 50/50. The only way to prove it is by testing it. If a casino has now winners for one year and the regulators say that their tables are legit, well, something is rotten in Denmark.

I implemented this a bunch of years ago in a fork of a SubSonic client. Their explanation wasn't sufficient to implement the algorithm but I did figure it out. I did like it. I stopped using SubSonic and now just use the simplistic shuffle in my media player. It's weird that I'm slightly annoyed when two songs from the same artist are played in proximity.

Spotify random isn't well distributed at all, I always listen to my playlist in random mode and I almost never get a good chunks of tracks. Using something like last.fm I can clearly see some tracks get hundred of play, while some never get a single one.

Honestly, I really dislike that I can't get Spotify to actually randomize my playlist. I actually want to rediscover some old song that I liked, but it keeps giving me the most recent songs way too much.

I understand that you need to make something that the average user perceives as random, but please give me some control in the advanced settings, it's literally one of the few things I want to control.

I prefer randomizing lists but not songs. Spotify's user experience could be better in this regard tbh.

Any shuffle algorithm is fine but the problem is Spotify's brainlessness: it routinely plays songs it just played. If you are any long-term user then you have playlists of hundreds of songs and my guess is that you haven't heard some of those songs in ages which means ... the FEATURE IS BROKEN.

I'd prefer a weighted shuffle based on the staleness of a song in a playlist. That way you eventually get through all of the music but in a pseudo-random order.

Reply on news.ycombinator.com