Spotify Starts Shutting Down Its Massive P2P Network
torrentfreak.com
torrentfreak.com
I first looked at Red Swoosh and its Firefox extension, FoxTorrent. It would have been ideal, since we using the xulrunner platform for TomTom Home, but Akamai acquired Red Swoosh, and it vanished without a trace. [1] [2] [3] [4]
TomTom's Maps are perfect for BitTorrent distribution, because they're large (1 Gig and growing) and lots of people in the same region need to download the same map at the same time. And it would have been a wonderful legal and practical example of a legitimate use of BitTorrent to point to when companies like Comcast try tricks like blocking BitTorrent traffic.
I visited BitTorrent in San Francisco, and discussed it with Bram Cohen. Their technology seemed ideal for our purposes (it worked a bit like Red Swoosh / FoxTorrent), and they quoted me a price per gigabyte to distribute content over their BitTorrent DNA network, which was a hell of a lot cheaper than we were paying to Akamai (this was before the made BitTorrent DNA distribution free).
A couple days after that meeting, I stopped by the Akamai booth at the Game Developer Conference in San Francisco, and asked them what their prices were for using the Red Swoosh technology they'd recently acquired. Travis Kalanick, the founder of Red Swoosh, gladly told me about his technology and answered my technical questions. (I had previously been looking at Red Swoosh and FoxTorrent, before it was acquired by Akamai, and it would have been easy to integrate with our xulrunner based product, TomTom Home).
Akamai's sales people refused to quote me a price, despite repeated requests, so I told them the price BitTorrent quoted me, and told them we were ready to start testing as soon as possible, and asked again if they could at least give me a ballpark estimate of how much they would charge for distributing content via Red Swoosh, and when we could start testing. They muttered something about not having come up with a pricing model yet, but that I could get into their beta program, whenever that started, which they also would not state.
I was somewhat skeptical about Akamai's support for Red Swoosh, dedication to P2P technologies, and motivations for acquiring Red Swoosh, since P2P technologies are such a huge threat to their business model. It was getting a lot of attention before they acquired it. I was afraid that they might have just acqui-hired Red Swoosh simply to sweep it under the rug so they didn't have to compete with it. And I was afraid that if and when they finally figured out how to charge for it, it might not be such a great deal after all.
So I went back to Amsterdam, integrated BitTorrent DNA into TomTom Home (the desktop app for managing content and downloading maps on your TomTom device, like iTunes for TomToms instead of iPods, which happened to be implemented on xulrunner (the Mozilla "platform" that Firefox and TomTom Home are build on top of -- which Mozilla eventually lost interest in supporting as a third party application development platform).
It was easy to integrate BitTorrent DNA into TomTom Home, it worked well, and I performed a successful beta test with a bunch of users across the world, to measure how well it worked, how long it took, and how much it would save.
Then I did an analysis of the Akamai logs to figure out how much money we were spending on customers downloading maps, how long it took, where they were downloading them from, and how much money BitTorrent DNA would save us.
For example, Australia would have really benefited, because their connection to the outside world and Akamai coverage was terrible, but their internal connection speed and bandwidth was excellent, and everybody in Australia wants the Australia map, of course.
It turned out that BitTorrent would save TomTom about a million euros the first year, and more each year, since TomTom was distributing larger and larger maps, more frequently by subscription, and of course they hoped to get more customers over time. (See: http://www.sadtrombone.com ...)
But then all of a sudden, out of the blue, Akamai unilaterally lowered the prices they were charging TomTom, saving us a lot of money immediately, presumably to prevent us from switching to BitTorrent (after I had made a bit of a scene at Akamai's GDC booth, in front Travis Kalanick and their sales people, about just having talked Bran Cohen and acquired a quote and beta testing agreement from BitTorrent, and insisted that Akamai tell me what their prices were and when we could start testing -- that may have motivated them to unilaterally lower their prices).
So in the end, TomTom middle management decided to can the BitTorrent project, in spite of the fact that it was the TomTom founder Pieter Geelen's idea in the first place, and he'd been micromanaging the entire project and user interface all along, to make sure it would not just save us money, but also not have a negative impact on usability or customer perception.
There are some important non-technical issues to deal with that Pieter wanted to get right: making it easy to use, setting the defaults right, explaining what it is, getting consent from customers to use their upload bandwidth (since they may have to pay for it, have an upload cap, or want to use it for other ahem purposes), and clearly explaining what benefits customers get from it (not leaving the impression that it's just to benefit TomTom financially, at the expense of the customer), and revealing the possible costs and risks.
We put a lot of effort into that part of it, and I don't think it was going to be a problem. It seems to have worked out all right for WoW and Spotify (in retrospect, up to now).
It was disappointing that TomTom didn't use BitTorrent, but I can certainly understand Akamai's mortal fear of it, and I'm still skeptical that Akamai may have bought it just to sweep it under the rug so they didn't have to compete with it, and didn't really mean to roll it out practically and economically, and develop it to its full potential.
Years later, Akamai now has something called "Akamai NetSession", that may or may not be related to Red Swoosh -- I don't know much about it or how much it costs. Can anyone else comment on that, please? [5]
[1] https://en.wikipedia.org/wiki/RedSwoosh
[2] http://www.akamai.com/html/about/press/releases/2007/press_0...
[3] http://gigaom.com/2008/05/03/whatever-happened-to-red-swoosh...
[4] http://techcrunch.com/2007/04/12/payday-for-red-swoosh-15-mi...
edit: The article also says
P2P has been central to Spotify’s success for a
variety of reasons. For one, it allowed the
service to scale up quickly without having to
invest heavily in servers and bandwidth. This
must have saved the company millions of dollars
per year.
edit 2: Thanks to tazjin for the clarification.
Makes sense to me. The bigger you get, the cheaper bandwidth becomes because of bulk deals. At this point maintaining the p2p network and all of the associated code and problems is probably more expensive than just serving the files themselves.
I'd wager that the only reason Spotify was able to offer a service in the early days at all was because it leveraged listener's bandwidth. Even if it could, using P2P will have directly reduced the costs/number of adverts that would otherwise have been required. That's a perfectly reasonable tradeoff, and I guess that the vast majority of users would rather have a monetarily cheaper service.
Hypocrites.
TomTom is highly dependent on the good will and trust of their customers to be willing to share real time traffic condition information with them.
There is an option to anonymize and upload statistics on traffic conditions to TomTom. It anonymizes it by not recording your identity, and not including the beginning and end of your trip, so your source and destination that could identify you are not revealed.
TomTom compiles all the information from different users, and feeds it back into the next map update, so the data you upload about roads you drive frequently eventually benefits you directly, and also benefits other users in your area, and TomTom of course.
(Of course the people who live on obscure roads that turn out to be efficient shortcuts don't think of it that way when convoys of truckers start blasting by their wooden houses all day and all night, vibrating their nails out of their holes and making road pizza out of their pets. But you can't please everyone all the time.)
The benefit to TomTom and its customers is enormous, since it enables their product to predict what the speed of traffic along each road will be at any time, weekday or weekend. But TomTom has to be very clear about how they use the information, and what the benefits and risks are, to maintain the trust of their customers.
A lot of the same issues apply to using customer upload bandwidth to share maps. TomTom would benefit from reduced bandwidth costs of course. And that could reduce the cost of distributing maps, some of which TomTom could pass on to customers, by offering cheaper subscriptions, more frequent updates, richer more detailed maps, etc. Users could also potentially benefit from faster downloads (especially in Australia).
BitTorrent DNA uses the CDN while the P2P network warms up, and if the P2P network isn't performing well, it will fall back to Akamai's servers of course, but there will always be some overhead in the worst cases.
The risks to TomTom in terms of the company's reputation and customer's trust are very important, because if TomTom rolled out a P2P based content distribution network that was perceived as being sneaky and serving its own purposes instead of the customer's, it could make people less willing to trust them enough to upload traffic information, and that's a lot more important to TomTom than saving a million euros a year on bandwidth costs.
There will always going to be people wearing aluminum foil hats with axes to grind, who substitute denominational punctuation characters for letters in your company's name, and who will never trust your company no matter what you do.
Even if they're willing to submit to all kinds of privacy intrusions, administrative fees and long lines, in order to get a license to drive a car, they would rather themselves and other TomTom users spend more time in traffic, more money on gas, and pollute the environment more, rather than upload and share their precious bodily fluids and traffic data.
There will always be people who will get pissed off about TomTom being so greedy to selfishly use their free upload bandwidth to distribute maps, because they're using it for pirating movies and downloading pornography, despite the fact that BitTorrent DNA is very careful not to abuse it (I mean abuse bandwidth, not porn).
There's nothing you can do about those people, but you do have to prevent the situation that a normal reasonable customer gets an unexpected bill from their ISP, and loses trust in your company.
It's really the fault of asshole ISPs like Comcast, who offer unlimited service with limits, throttle traffic, and charge ridiculous fees. Companies who want to use P2P like TomTom can't do anything about that, but understandably get blamed for it anyway. Education and disclosure and consent, and supporting laws preventing ISPs from screwing their customers, are the best ways to solve that problem.
So the TomTom Home user interface for P2P sharing had a page that explained the risks and benefits, and allowed users to opt in to the P2P file sharing feature, at the small risk to TomTom of not having as many people using the P2P network, which was a lot less than the large risk of losing the trust of their customers, and harming their precious traffic feedback collection.
It's ironic (or simply sad) that after Microsoft bought Skype, they switched from using P2P to using centralized servers, so their customers paid dearly in privacy instead of bandwidth.
Privacy isn't such a big issue with Spotify (unless you don't want the NSA to know you're into Nickelback -- hint: they already do, and they probably listen to it themselves). But they're always going to get complaints from non-paying customers who object to ads in their free music, yet hate how the record companies treat artists, and refuse to contribute their upload bandwidth, while at the same time run BitTorrent to pirate movies and porn.
On the other hand, I'll bet there are a lot of World of Warcraft players who would happier watching the real time telemetry of slower P2P downloads connecting to hundreds of other players around the world, than downloading updates quickly and quietly from one CND, just because of the trendy technology with cool blinking lights, and the social cachet of being part of the community sharing a new update through "the cloud". Privacy isn't an issue, and you're going to be using a lot of upload bandwidth when you're playing the game.
So the issues surrounding P2P are different for every company. But good non-technical approach in general is to frame P2P networking as cool and useful, instead of creepy and greedy.
(In Europe/US it's below 30-40 cents per Megabit/s/month now, when buying more than 100 Gbit/s/month. The bandwidth cost of a single average customer for spotify is probably less than a cent per month.)
As their service grows, Spotify should be liking P2P superlinearly more. Since they and other services are apparently not, it something is wrong with my conception of the economics of the internet.
Only if you assume that the price of bandwidth is linear. It is suggested in other comments that when you buy bandwidth in bulk you can get large discounts. If I don't misremember, back in the early days of Youtube, they came to a point where the network operators wanted to get access to Youtube rather than the other way around.
Also, as a company grows and get access to more capital they might be able to invest in other infrastructure that is needed (other than just bandwidth) that combined with the bulk prices, make the opportunity cost of developing the p2p system larger than the opportunity gains.
However, I suspect that continuing to use P2P wouldn't actually result in reduced costs in practice. P2P is complex, and getting a reliable streaming service out of it is obviously a more difficult technical issue that "steaming some bits from a server."
Indeed, I'd expect that with suitable peering and Netflix-style caching servers within ISP networks, it wouldn't even save ISPs money.
I can't remember a case when I've wanted to store a Netflix (or any other) video to reuse.
If Netflix eased in making the ISPS the broken loop to millionaires, politicians, and their families watching movies - more poeple would ditch the major ISPs and have fiber.