Sad but true: Napster '99 still smokes Spotify 2012
theregister.co.uk
theregister.co.uk
For me, Spotify is a sub par experience because I don't know what I want and I have different music tastes than my friends. The only thing that makes Spotify useful for me is sites like http://sharemyplaylists.com/. I imagine for people who know exactly what they want and just treat it like a giant jukebox, it is a fantastic service. I am not one of those people.
Napster for me, was about finding people as much as music. I was part of a community of people that were super stoked to talk about some new indie band. I would search, notice who had music I was looking for, send them a message and start talking.
I met a lot of people in real life through Napster often at shows. I know others did as well since we received thank you letters and in a couple cases, wedding announcements.
I think the community with its passion for music was what made Napster great, not the massive catalog. All I had to do was enter one of many indie channels and read for half a minute before I had three new things to listen to and a bunch of people to talk to them about.
Not to discount the catalog, I will say that the catalog at Napster will possibly never be duplicated. There were a lot of back catalog works, a lot of pre-release works and a ton of bootlegs. Sure, the quality was sometimes poor, but then I used Napster as a tasting service and then, you know, bought the originals when I could.
My hope is Spotify continues to improve and eventually becomes more community oriented or something else comes along and re-ignites the flame. Music is one of those things that is central to a lot of people's lives and you couldn't ask for a more passionate userbase.
Can you talk about that a bit? What was the underlying algorithm, what was the stack and how much data were you pushing at your max?
The search engine was built on a ternary tree with a custom merging algorithm. I honestly don't know if the merging algorithm has a name as it was something I came up with (literally) while sleeping one night. Because we mostly used ID3 tags and file names, it was completely unnecessary to use a stemming algorithm because if you typed in a misspelled search, given the size of our index, there was probably someone who tagged their file using the same misspelling.
The network biasing code used BGP data combined from a number of looking glass servers to build a map of ip/prefix -> ASN number and ASN->ASN distances. It was then used to reorder search results based on network distance to users so they would bias towards their own networks and save ISPs money and speed transfers on broadband connections.
Servers were linked through a fully meshed network. Each had presence information about every user on the network so that they could route IMs around. The chat system was semi-linked (fully linked on some servers, but we couldn't fully link the whole thing because the client had no administrative functions for chat). If we couldn't send the user back enough results for a search, the query was simply passed around the backend.
The whole thing was written in C++. At its peak, there were about 2.3 million users online at any given time (80 million total users growing by a million every 4 days). The system would be indexing about 17.6 million files per second (and de-indexing about the same amount). The whole system pushed out about 2 Gbps of bandwidth in search results (which were tiny).
Napster was one of the very first services to push past 10K connections on a Linux machine. At peak, I could get over 100K users on a single process (though I'd run out of memory indexing files on the tiny 2 GB machines and blow out the NIC sending search results). During normal operations, each server process had around 40K users on it and between 7-12 million files indexed.
There were a bunch of side infrastructure things no one saw. Court mandated copyright filtering systems, recommender systems (most for play), load balancing servers, bot detection and sequestration systems, analytics reporting jobs, etc.
Nowadays, I could probably fit all of Napster on one big machine. Heh.
I'm particularly pleased by the use of ASN/distancing weights in your results! None of the BT trackers I've hacked around on have had anything like that in them.
Where is the code now? It should be in a museum.
I once hacked up Transmission to re-prioritize based on network distance. It worked rather well when it discovered over the DHT. It is a lot easier when you only have to calculate distance between you and other networks. Storing the graph of distances between two arbitrary users is harder.
The code was part of the assets that were bought out of bankruptcy by Roxio (who renamed themselves Napster). I doubt it is being used for anything.
cheers.
"...it was completely unnecessary to use a stemming algorithm because if you typed in a misspelled search, given the size of our index, there was probably someone who tagged their file using the same misspelling."
Seems to me that the Spotify API opens up all sorts of possibilities for music discovery.
Sharemyplaylist.com is basically what makes Spotify useful for me at all. I still can't talk to anyone, but at least I can have a playlist that sounds roughly like the songs belong together. I'll rarely discover new music on it.
For music discovery, I find http://hypem.com and mp3 blogs to be far more useful. Still no strong community, but at least I actually discover new interesting things.
- The "depth of catalog" is actually incorrect -- Spotify has far far more songs than Napster did (you can see in that screenshot even, most logins/servers hit less than 1m songs totally available, and of course those are just total indexed, in reality almost 50% of downloads would not be available.) Yes, you could find niche stuff on Napster-- as a musician in 1999 I would make sure all my stuff was available, pre-release, demos, etc. But we have Soundcloud for that now, so...
- The discovery on Napster was non-existent. You could browse through a users' collection, the same way you can browse through playlists today. But the only entry into anything was a search box that only looked at ID3v1 data and filenames. There's been amazing leaps and bounds in discovery since then, and it's very clear by the #s that it's what people wanted -- a guided (radio, playlist) experience over a wild west single song retrieval thing.
- Napster was stupid bad at search. There obviously was no catalog resolution and the quality of the results was abysmal. I am pretty sure it was a substring match, for one, and then there was the bad metadata, fake songs, later on a huge spam / "SEO" problem.
- Not going to get into speed, because it wasn't Napster's fault, but even back then it was far easier to get music elsewhere other than Napster if possible. Ratio FTP sites, hotline, and of course bubblecruft startups building customer bases by selling new CDs for $4. It was very clear at the time even that the distributed nature of Napster was a liability, not a promise, as all the single-server solutions were far more convenient and reliable. But this was not their fault and of course their success inspired everything after it.
The backend servers were linked. If your request couldn't be fulfilled on one, your search was forwarded to the next server. The total number of files on Napster at its peek was over half a billion files. Further, that picture must have been from a server that just started because the average server had significantly more users/files on it.
Further, Napster users were ripping everything in sight. There were mp3 encodings of old wax cylinders uploaded for goodness sakes.
- The discovery on Napster was non-existent. You could browse through a users' collection, the same way you can browse through playlists today.
There was an entire curated music website dedicated to music discovery that loaded into the client.
The chat and instant message system allowed people to talk about music which created a massive music-focused community. It was wildly popular.
And don't underestimate browsing. People would search for the one song they were interested in, notice who they were downloading from, browse the other user and then start pulling down their music if they noticed several songs they liked in it. They then could send a message to that user and add them to a friend's list. That was not just music discovery, but friend discovery as well.
- I am pretty sure it was a substring match
The very first versions were substring match when there were maybe 10,000 users. Later version were not and allowed basic boolean queries like term exclusion.
- Not going to get into speed, because it wasn't Napster's fault, but even back then it was far easier to get music elsewhere other than Napster if possible
There was an algorithm on Napster that did network distance biasing. Basically, if you were an AOL user, you'd first get AOL users back when doing a search. If you were an Internet 2 or even @Home user however, your speeds were epic.
* Note: I built and ran the Napster server.
I just want to say: thank you for building one of the most important inventions of our time. (Napster kickstarted the information sharing revolution.)
That said, Napster made it all really easy. They also came along when CD drives were finally commonplace, and Winamp had seeded the ground by having people build up a library of MP3s beforehand.
But its been a while so maybe my rose tinted glasses have munged things a bit! :)
And you're right, BBSes were doing similar things first (although I assume good ones were harder to find than usenet groups or irc chans - don't know because BBSes were before my time!) but of course the mp3 standard didn't exist back when they were at their zenith. MP3s only started appearing en-mass around 1997 I think (??). Before that, I remember every one used to share tiny wav and midi sound file clips on their homepages (my first ever homepage was made to share wav clips of simpsons and monty python dialogue - lol how embarassingly quaint!)
Yeah, in my BBS days I remember downloading MOD files and demo videos. I still remember being blown away by Future Crew.
I miss it.
On a university network Napster was quite fast. I could routinely saturate my (for that time speedy) 10mbit connection, especially to other clients on Internet2.
One other point to your argument is social. The ability to see what friends are listening to and create sharable playlists is obviously something that Napster did not provide.
.edu to .edu transfer was unbelievably fast back in those days.
[1] Although I hate that word.
[2] Daughter - His Young Heart, Japandroids B-sides from the Post-Nothing sessions (Younger Us, Art Czars, and Heavenward Grand Prix), anything from Mirel Wagner or 9mm Parabellum Bullet, anything from The Lonely Forest's first three releases, Gotye - Like Drawing Blood (which is so much better than the album that finally made him famous), etc. Spotify is seriously lacking in terms of selection.
The numbers depend on when one took the screenshot. Here's part of one I took on 02/02/2001 23:32PM CST with more than a million files available:
But I think the essence of what Sean Parker was saying was that there are still many hurdles that a legal system like Spotify will have to overcome. If you really want everything without restrictions, pirating is still a better option.
Spotify and the others are cool because you can stream stuff while on your phone. And getting to the music you want to hear was more seamless and straightfoward.
So in summary, almost, but not quite.
Napster was also one of the first geek-tools to really hit mainstream, and boy was that weird.
Napster was a magical moment where the underground went mainstream for a split second before being snuffed out. Of course these type of (smaller) communities still exist online, and music fans are better off than they ever were, but Napster embodied a sea change that could only happen once.
No, it's because it forfeited the control the labels had over the music. They've showed again and again that they don't mind losing money to maintain that control.
Nowadays I only buy music that I'm reasonably sure it won't feed the RIAA.
All SOPA did was underline yet again how little they understand the actual technology involved.
I'm saving this whole thread re-reading it over and over again, thanks to Aloisius chiming in (giving a valuable and rare peek at the back-end), and the insight into what drives music discovery.
I do credit Napster for awakening a love for music. My father had a deep record collection from the 60's, and he had mentioned some very hard to find items he's wanted for decades that I found on Napster in couple of hours of searching. This was what made it awesome for me. The ability to browse someone else's collection was an incredibly effective music discovery system, and introduced me to music I'd never have discovered on my own.
Granted, this happened despite the music industry's efforts, it took a lot of pressure from Apple to get there. But still, I very much prefer the current situation over when Napster reigned. (Does noone remember the amount of garbage on Napster? The malware, the fake files, the low quality files? Not to speak of the terrible speeds -- many peers were still on dial-up...)
With iTunes, you can find new music easily, download high quality files, and get the album art and liner notes. Downloading and purchasing is fast and safe. You get to pay a fair price and know that the rights holders are being compensated.
After that, it was relatively easy for Apple to go DRM-free (though oddly, they still refused to do it for some time after).
* Note: Shawn Fanning and I founded SNOCAP after Napster blew up.
Napster was one of the first services to give the mainstream world a glimpse at the digital information utopia made possible by the Internet. The paternalistic distribution systems we have now are a mere shadow of what could be without the strangling influence of the media industry.
I wish Pandora could buy one of these services and combine their discovery engine with the catalogues of the subscription services.
Until then, I think I'll just keep buying music.
Here's a Mix I've made, for example: http://www.audiogalaxy.com/mix/87-Guitar%20Gods/?
[1]http://www.geekwire.com/2012/audiogalaxy-music-service-rebor...
It was amazing back in it's (less legitimate) days. It took over from Napster but improved on it in many ways.
I love Pandora for the guidance.
WAH is available on Spotify, which is good (I used them before Spotify), but it's very niche indie discovery.
I can't select a Temptations song and expect to be then offered up a Clarence Carter or Tower of Power hit.
Only Pandora has those "skillz".
Yes that's ignoring the legality of it, but ultimately we repeatedly see that users will ignore legal issues in favour of convenience.
Though, I'm not blaming spotify here, they have little choice but to operate inside a set of imposed constraints. The big labels and the industry are to blame, it's the same old story they manage to derive tons of money from being in control of the distribution, they're not gonna give it up without a fight. Even though it is a fight they cannot win, they will mindlessly fight it till the end, and those who pay for this are the artists and the users.
Ping?
It's not the fault of Spotify that the collection is limited. At least they are trying to move the music industry into the future - whether it will work or not who knows but I'm sure we will all look back in hindsight and see Spotify as the pioneer of whatever comes.
The article glosses over how slow Napster '99 was. I remember waiting a couple hours to download an album. Fast downloads are more important than social features. I do not miss Napster.
Of course, high speed internet is much more prevalent now in 2012 than it was in '99. Downloading an album in '99 sometimes meant you were downloading from PC's with only a 56k modem for connectivity.
So I went to spotify.com, read over their product and information. Checked out their plans. Decided I'd see what they can do with the Unlimited $4.99 plan.
I went to sign up for Spotify. They required I use my Facebook account.
Nope.