Why isn't iTunes shuffle random? | Hacker News Reader

Why isn't iTunes shuffle random?

apple.stackexchange.com

135 pointsbenihana74 comments

I think this is one of those problems where smart people think too hard. It has nothing to do with non engineers misunderstanding randomness or the human mind intrinsically superimposing structure. All it has to do with is being an algorithmic dj.

Users don't want a random playlist. They want their songs shuffled, and they want them shuffled in a way that keeps songs from the same album distributed far apart. No one presses the shuffle button to get satisfaction out of pure randomness. They press the button because they want to listen to music. If think your users are dumb because they don't understand randomness then you don't understand the job that shuffle is supposed to do.

Exactly. People want their expectations and their pattern matching processes to be confounded, and randomness, counterintuitively, will not do that.

I think that's right, but stated wrong. :) Randomness does confound our expectations. But what it does not do is confound them consistently. It's routine for a random order to play the same song twice in near succession ("I just heard this yesterday, and I have 2000 songs in this thing!"), etc...

But none of that really gets to the linked article, which is that iTunes was using a fixed shuffle order.

You see the same thing when laying tile. If you have a small field tile with fancy glass accents or whatever, the customer really wants random uniformity, not pure random.

This is something any game programmer can tell you. Pure randomness in a game kills the fun. You have to shape the randomness to match human expectation.

That depends on the game and the feature of the game that requires the randomness though. True... pure randomness over spawning points for enemies in an action rpg game, might kill the experience. But if I am playing poker.. I do expect something close to pure randomness when you are dealing my cards.

I worked on a strategy game (think Risk) with an element of randomness (dice rolls) and our hardcore players were very very keen on the true quality of our randomness. I wound up persisting the context and result of every single roll cast so people could see summaries per-game, per-turn, per-user.

I think people are confusing the term "true random". It's still truly random, but it won't have a uniform distribution over all possible values. For example, a common trick in FPS games is to pick a random spawn point, but exclude spawn points that are too close to other players. Still truly random.

Another trick in RPGs is to increase or decrease the odds of a successful hit according to recent hits and misses, bringing the short-term average much closer to the expected value. It's still "true random", but uses correlation between attacks to create an underdispersed distribution overall. (Basically, you want to make the variance lower to make people happy.)

"Truly random" is usually contrasted with "pseudorandom". In that sense almost all games are not "truly random". But you are right in that "truly random" also doesn't mean "uniform".

A lot depends on the game.

Example: My hobby is that I'm a tournament judge for Magic: the Gathering. I also play a bit, because I like the game. And there's an online version... which has a shuffling system that's actually random (i.e., has reportedly been examined by the same folks who certify stuff for casinos, and passed).

But that causes a lot of complaining from players, because real random shuffling is very different from what human beings accomplish when shuffling pieces of cardboard, and is actually a different standard from what's enforced in tournament play (requirement there is that after shuffle, no player is able to know locations, relative orders or patterns within the deck, not that the deck is "random").

At any given time there are thousands of players online, and every deck gets shuffled at least once per game. With those numbers... unlikely results happen. They happen a lot. And they get players really, really angry, to the point that most forums have "complain about the shuffler" permathreads, because those results tend to absolutely screw you over when they happen.

Pah. You clearly don't have much experience with roguelikes. ;)

I don't know. At this point, I kind of expect an orc with a wand of death on level 1.

DYWYPI?

Did you read the first (highest rated) answer? This has nothing to do with iTunes trying to be "an algorithmic DJ" rather than truly random, it has do with the confusing intersection of two features: You can pick the first song in the randomly "shuffled" playlist, and a shuffled playlist holds its order until it is re-shuffled.

The user posting the question assumed that when he double-clicked on the first song in the playlist, iTunes was re-shuffling the playlist. It wasn't. It assumed he wanted to start the playlist again. If he had picked any other song to double click after playback was stopped, it would have re-shuffled the playlist.

The solution was to toggle shuffle off and then on again. Then he'd force re-shuffling.

iTunes does have a separate algorithmic DJ feature, by the way, called iTunes DJ. No need to mess up shuffle. Shuffle is truly random.

> If he had picked any other song to double click after playback was stopped, it would have re-shuffled the playlist.

Pretty sure that's not the case, and the answer indicated otherwise. The order is stable until it's explicitly reshuffled.

I was mis-remembering how iTunes work there. It will auto-re-shuffle in some cases -- you don't always have to do it explicitly -- but it looks like it has more to do with whether you switch away from the playlist after the music stops playing. If I simply "pause" and select another song, without switching away, the order remains stable as you say. If I switch to say the Music library view, and then back, and double click a song (any song), it re-shuffles with that double-clicked song on top, playing.

That was part of the incorrect opinion parent was saying the user had.

That was not his point. He just didn't know shuffling only happens when you activate shuffle mode, so he was getting exactly the same order (not just a "good" distribution of artists and albums).

>Users don't want a random playlist.

Really? Because I couldn't stand it if, after listening to my playlist several times, I could predict the exact order of the songs. That would be extremely boring, personally. In fact, that's the exact complaint of the user who asked the question:

>This frustrates me to no end because I like to listen to my music randomly, but I have a few favourite songs that I always start out with.

Would you ever want the same song to play twice in a row? How about 10 times in a row?

The probability of that happening decreases so quickly that it's not a realistic concern.

Incidentally, today I had some song stuck in my head so I just listened to it on repeat for a while.

Preferably, no. But even if that was a problem, it would only be an occasional one. The predictable playlist would get on my nerves all the time.

I suppose you'll then lecture me on how a random playlist has such undesirable properties.

But I would have to disagree. Writing a psuedo-random shuffle mode that doesn't play songs twice in a row (or even one that doesn't repeat a single song in your playlist) is trivial.

Leonard Mlodinow touches on this issue in The Drunkard's Walk[1]. If I remember correctly, his explanation was something along the lines of this: a truly random playlist is not desirable to most users, which challenges the claim "I do believe the ordering is truly random the first time it's generated" in the top answer. Mlodinow explains that most people really don't understand randomness, and a "truly random" playlist might have the same song twice in a row, play a few songs from the same album in the normal order, and things like that. Some people don't realize that rolling 1,2,3,4,5,6 on a die in order is as likely as 6,4,1,2,5,3 even though the second set of outcomes might seem more likely because it looks more random. To outcount for this, iTunes actually does use an algorithm to make the shuffled playlist "better" than a truly random one.

Of course, by some definitions, even rand() isn't random enough, so I can see why people argue about it. The book doesn't even provide a rigorous definition of the concept, but it was never intended to be a serious treatise on statistics.

1. http://www.nytimes.com/2008/06/08/books/review/Johnson-G-t.h...

This.

I have been dealing with Samsung half-phone half-MP3 players for a while now, and they all choose the next track randomly. I pine for the days when I used to have an iPod that could run Rockbox, so that the "back" and "forward" functions worked in the shuffled playlist. Sometimes I mean "rewind to start" and I accidentally tap the button twice to say "seek to last track." That takes me to a random track and then the original is no longer available as "next". So irritating.

And this randomness-is-too-random crap does happen. I'll put on a list of songs to sing along to, only a few hundred songs, and often after half an hour or an hour I'll be repeating songs which I've already sung. Sometimes they'll be two-in-a-row or separated by only one, two, or three different songs.

(The other great thing about Rockbox was that I had access to folders which automatically acted like playlists. We've had debates about this back and forth around HN, I know, but suffice it to say that it's really nice to have an "Audiobooks" folder where you can download individual book folders. Otherwise you often have to make sure that you don't get bizarre interleaved numberings, chapter 01 from book X followed by chapter 01 from book Y. In Samsung, this is actually governed by the Title of the ID3 tag of the MP3, which can be even worse.

But once those users get through the entire song list they want the order to change. iTunes is just as wrong as picking each individual song at random but in a different way.

I also recall Steve Jobs explaining in a keynote once that they had made the shuffle feature less random in response to user feedback about not wanting to hear the same song twice in a listening period.

Aha... just found what I was thinking of. Quote from article:

As humans, when we come across random clusters we naturally superimpose a pattern. We instinctively project an order on the chaos. It's part of our psychological make-up. For example, when the iPod first came out and people started to use the shuffle feature, which plays songs in a random order, many people complained that it didn't work. They said that too often songs from the same album, or the same artist, came up one after another. Yet that's what randomness does - it creates counter-intuitively dense clusters.

In response to complaints from users, Apple CEO Steve Jobs changed the programming behind the feature: 'We're making it (the shuffle) less random to make it feel more random.' In other words, each new song now has to be significantly different from what came before, so as to conform to our expectation of randomness. Which isn't really random at all.

>In other words, each new song now has to be significantly different from what came before, so as to conform to our expectation of randomness. Which isn't really random at all.

That's the trouble with randomness. http://dilbert.com/strips/comic/2001-10-25/

I like to say "Random is blotchy". If we mean "uniformly randomly chosen with fully independent choices", smoothness is proof that it's not that sort of random.

This comes up surprisingly often. For instance, the mere presence of clusters of some non-communicable disease or something prove diddly. It must be shown that the clusters are unexpectedly large, not merely that they exist. Of course they exist. That's the default presumption.

I remember complaints of the form "My iPod really loves Steely Dan." I can't find the original article, but it's excerpted here:

http://gadgetopia.com/post/5580

In other words, each new song now has to be significantly different from what came before

Mixing opera, country, and black metal in the same playlist would take some getting used to...

I solved this problem by creating a smart playlist that only includes songs I haven't heard in X days. I actually went further and made it include 5-star songs I haven't played in X days, 4-star in X weeks, and 3-star in X months. I hear the songs I like the most, but the constantly updated list prevents the "random" shuffle algorithm from picking the same songs too often.

The top answer is good, but it doesn't explain why iTunes couldn't maintain the seed while skipping forward and back, but reshuffle whenever a new song is selected, or whenever iTunes is restarted, or whenever you switch to a playlist and back, or all the other times you wouldn't expect it to maintain the sort order. So this still seems like something that Apple could (and should) really improve on without breaking important functionality.

In all those "why doesn't it reshuffle" scenarios you mention, I would then have to hear same tracks again. I don't want to. I want to hear everything, some today, some tomorrow, some next week, until I've heard it all.

Unless I do want to hear things again, in which case I toggle the Shuffle button.

By design, I decide. Rebooting, restarting iTunes, picking a particular song then going back to my long playlist, those don't decide for me.

Ideally, what I'd want is a button that generates a new, temporary, randomly-reordered playlist from another playlist, representing it as a virtual child (if I had, say, a "Vacation music" playlist, it would have a child node "Vacation music shuffled with seed 6e43bf".) The virtual playlist would also represent differently the songs that have been listened to since the playlist was created, from the ones that haven't--say, by putting a strike-through through them and greying them out slightly, or something. That way, if I get a whim to listen to one particular song half-way through listening to the shuffle-list, I can go listen to it, then return to the shuffle-list and listen to the rest of it, without having to hear any of the same songs again. This would also allow for the child to track modifications of the parent--if you remove a song from the parent playlist, it would also disappear from the child (unless, perhaps, you had already listened to it; then the child could keep it as a record); and if you add a song to the parent playlist, it would get inserted at a random position within the still-as-yet-unlistened partition of the child.

This is sort of what iTunes DJ will do, if work off a smart playlist based on the playlist you want to listen to, but which defines some criterion by which songs you've listened to won't come up again--for example, if you're listening to songs to give them ratings, you can define the smart playlist to only be the unrated songs in the original playlist; then each song will be removed from shuffle-selection once you've rated it.

Still, the simplest thing to do is to just set a playlist to shuffle, make sure the list view is sorted by playlist-order (the unlabeled leftmost column), and then Ctrl+A, Ctrl+Shift+N to copy the shuffled list into a new playlist, and delete songs from it as you listen to them. It's too bad this is an impossible proposition on a portable device.

iTunes does reshuffle when switching playlists.

> So this still seems like something that Apple could (and should) really improve on without breaking important functionality.

I'm not seeing how any of the suggested changes are an improvement.

Yes, the current scheme isn't mathematically random. That's irrelevant unless mathematically random is a better user experience.

So, let's see some evidence for different schemes that improve the user experience. Note - users don't care about mathematical purity.

I don't know what mathematical purity has to do with anything, I was just thinking Principle of Least Surprise, which is a user experience plus. Of course, I'm just going by what I would find least surprising and most useful, but I've heard lots of people express confusion about the behavior described in the StackOverflow question, so it does seem like there's room for improvement here.

When shuffling I want my player to adhere to the principle of most surprise ;)

That approach could have some random unintended consequences. I shuffle my playlist and listen to a few tracks. One comes on that I don't like or doesn't match my mood, so I peek at the playlist to find on that does match my mood and double click it to start it playing. Unbeknownst to me, iTunes triggers a reshuffle when I do that, and the very next track to play is the one I was trying to get away from.

The better solution (at least from my perspective) is to not pretend that shuffle is a setting, but an action. Simply have a button that says something like "Shuffle this Playlist" or more explicitly "Re-Shuffle this Playlist". Play, stop, forward, back all would work normally within the existing order (straight or shuffled), and if you wanted a new order, you click the "Re-Shuffle" button.

> Unbeknownst to me, iTunes triggers a reshuffle when I do that, and the very next track to play is the one I was trying to get away from.

What are the odds of this happening?

Pretty low if I'm listening to all my music at the same time, but what if I have a playlist of 10 songs?

> Pretty low if I'm listening to all my music at the same time, but what if I have a playlist of 10 songs?

You wouldn't listen to a playlist of 10 songs when there are many of them in it you are not in the mood for?

Well, you only need one for it to be annoying!

Just about every album I have has one song that I will routinely skip over, because I don't like it much.

Yeah, sometimes I like to just toss a album in a playlist and listen to it, and I always seem to end up skipping at least one song

Then un-tick it.

Or rate it poorly and factor ratings into your smart playlist.

And further that could be guarded against fairly easily. Reshuffle would exclude the last ten songs from being the next ten songs, etc.

Pressing Back to get to the song you just played, and double-clicking that exact same song, must absolutely behave identically or it will really confuse people.

It sounds like it's a naming issue. It's not "shuffle mode" - there's a shuffle action.

This is the best, simplest way to describe what's going on. Well said.

Personally, I think this is the ideal behavior. I want a random playlist, but want to play through the whole thing as time goes on. Pure shuffle would give me a new playlist every time I went to a podcast or some other random song, whereas shuffling a playlist means I can go back to where I was in that random list, and not worry about hearing the same stuff again.

If that's the case, then there's also a bit of a UI issue. The current shuffle button is a toggle, not a "click once each time you want to shuffle" button.

Yes, for sure. My speculation (from vague recollections of ancient Slashdot posts I didn't read closely at the time - never having used iAnything for music myself - so take it with a grain of salt) is that shuffle started out randomly picking the next song, but people complained that IT WASN'T RANDOM because they kept hearing the same songs - but of course if it is random, there's no protection against precisely that, and the people for whom it has happened to be well distributed probably arent' going online searching for "itunes shuffle randomness problem" or whatnot. So there may or may not have been a bug in the randomness, but randomness isn't really what people wanted anyway, and upon realizing that they changed it to the way it works now - where you shuffle it like a deck of cards, and then run through that deck repeatedly. Now, shuffle is an action where before it was a mode - and they forgot to change the UI to match.

Yes, but you still need to be able to "unshuffle", so they kept it as a toggle. not the most natural ui, but it works.

"unshuffle" is a sort action

For a playlist, it's "sort by the original arbitrary order that the user put it in" so you can't rely on the usual sort methods - e.g. clicking the column header.

Right, but it's still an action - "put it in some specific order"

I really like the way Winamp handles shuffle. It takes the current playlist (whatever is selected to play), then shuffles it in memory for stable play order. If you select a new song manually, it will play that song, then resume the old play order.

I use ncmpcpp (a client for mpd) and it does the same thing. I've never had any issues with its shuffling; it just works exactly as I would expect it to for everything.

On a sidenote, mpd + ncmpcpp is a killer combination for playing and managing music. I can only recommend that to anybody geeky enough to use a CLI music player :)

This wasn't the question I thought it was. It's actually "why don't things get reshuffled when I expect?"

The answer to the stated question is: because randomness clumps and plays too many songs from the same album/artist/wtfe and bugs people. See also http://news.ycombinator.com/item?id=4301922

I got tired of hearing the same songs over and over during my train rides, so I wrote an app that plays your songs with no repeats ever until you've heard them all.

http://itunes.apple.com/app/true-shuffle-pro/id355913111?mt=... , if you're interested.

I priced it high on the theory that people with lots of music have already put hundreds or thousands of dollars into their music collections and wouldn't mind spending a little to enjoy all their songs in a useful way. I actually tried a free version at one point and it sold the same number of copies as the $4.99 version, so I ditched it and just have the paid version.

EDIT - Don't want to feel spammy here, and wasn't going to include a link to my app, but for those who do want a "listen to all my songs with no repeats" solution, True Shuffle does the job. Hit me up for a promo code if you need one, eh.

iTunes will play all of your songs without repeats until you have heard them all as-is. A user would simply go to their music collection, select the shuffle action, and then begin playing a song. No song will be repeated until all of them have been played, which you should have been able to take away if you read the OP or the comment thread. As for the fact that your app is $5, I will refrain from saying anything besides do not spam the link to it on HN please.

I remember when the iPod first came out and was truly random. People would notice that song X came up twice before having played song Y and conclude that the iPod wasn't truly random. There was a big thing on Slashdot about how the record labels were paying off Apple to weight certain songs more heavily in the shuffle for their own nefarious purposes.

Then again, my memory isn't what it used to be, so maybe the Slashdot crowd was right about that one and I just forgot.

I remember on the Banshee mailing list a while ago (with fear to be corrected here right now) a discussion about how randomness should work. They implemented some awesome random options though: Random by album and random by artist. So you can play different albums or different artists all the time.

But a really random playlist, really sucks. You get incredibly random stuff, e.g.: the same artist 5 times in a row or sometimes, the very same song.

One of the more interesting things I've heard about iTunes/iPod shuffling is the phrase 'it knows me so well', obviously every shuffle isn't a win, but I'm guessing it's not just seeding it pseudo-randomly but also applying some specific constraints on the playlist.

I don't realllly like completely random shuffles as jazz into techno isn't the smoothest transition, but I've never really seen that with iTunes.

I observed this with Google Music too. Nirvana is underrepresented in my music collection (1-2% tops) but I seem to hear multiple Nirvana songs every time I log in and shuffle all (4-6 hour period). I have wondered if Google Music is picking music that is related to previously heard music.

But I have no way to check without looking at the source.

Until Google Music gets smart playlists, I have no recourse I guess.

Not mentioned but I think the randomness is also not equally weighted, 5 star songs will show up way more then lower rated songs. A few years back someone gathered a large dataset of shuffles and plays with songs clipped to a second or less.

I find that hard to believe. The songs are actually "shuffled," in the same sense as a deck of cards is. One a song is "drawn" from the shuffled deck, it is not put back until the deck is exhausted and shuffled again. So in any shuffle, each song must show up exactly once.

There's a setting in iTunes DJ, however, to "play higher rated songs more often."

Yep thanks, I was confusing features there.

This appears to be the article I was remembering (caveat: it's from way back in 2005):

http://www.omninerd.com/articles/How_Much_Does_iTunes_Like_M...

Also, back when iPods had platter harddrives, if it predetermines the whole shuffle order in advance, the iPod can load the few next songs into memory and not have to spin the harddrive up as often. Saved a lot of battery.

Who expects shuffling to be random sampling and not shuffling?

All shuffle modes I've ever encountered have shuffled the playlist or songs on a compact disc and then played them in the once-randomized order.

I had a similar issue with Foobar until I realized that 'random' meant 'won't play song twice in a row' while 'shuffle' was more of a 'won't play song again until everything else is played'.

The bigger problem is that if I select a single album, tell it to shuffle, and click play, it hardly EVER plays the entire album. I've had it play two songs and stop.

If the major reason for this behavior is so that you can press back to play the previous song, it can simply remember a short period of history.

Note that "shuffle" isn't "random". A mathematically "perfect shuffle" of a deck of cards is not random at all, but still provides a pleasing distribution.

Probably the same reason iPhone camera output is filtered (or any other technical term that ends up with photos looking better than they really are).

I had my doubts WinAMPs shuffle was acting on some heuristics, though I never bothered to check.

Reply on news.ycombinator.com