Hacker News Reader

Comment by forgotmyhnacc | Hacker News Reader

I'm confused why ISPs would do this for free. Why doesn't Apple just colo like other CDNs do?

It has benefits for the ISP, too. I used to work at a small-medium regional ISP. Something like 50-70% of ALL traffic was streaming video and specifically Netflix. We weren't big enough to get into their version of this program at the time and have boxes dropped at our head ends. But it would have been very helpful. We literally had to upgrade gear and buy bigger pipes from our upstream ISP's to handle the volume of traffic from Netflix. Having the bulk of that content come out of a box already on net would have meant significant savings.

What's to stop them from caching arbitrary content in the same way, for their own benefit, without the involvement of a partner? Is it because of SSL?

ISPs used to do that more commonly before HTTPS killed it but it's an expensive service to operate: very high traffic and if anything goes wrong your customers have a bad experience on the entire internet. The only way to do it is by intercepting TCP connections to port 80 so that system has to be as close to 100% uptime as you can manage.

Site owners generally hated it, too, since tampering proxies were a perennial source of compatibility bugs and protocol violations even before you had things like the ones which tried to “optimize” images by recompressing them, giving everyone on that ISP a bad experience which you don't know about. Stack Exchange has a number of threads where someone was trying to figure out why only some customers had complained months-stale content (Hi, Telemundo!), low-quality images (Hi again, Telemundo!), mismatched languages or truncated/corrupted contents, etc.

Ah, it’s been too long: everywhere it says “Telemundo” above I meant “Telefonica”

Many ISPs do exactly this for high traffic content that is not served via TLS (which prevents it as you suspected)

SSL is exactly the problem. Has pretty much killed caching proxies.

Yes, SSL.

That makes me wonder... I wonder if there is a process by which providers would sign certs to individual ISPs and providers to let them intercept low/medium security content like streaming.

Like, if Netflix is going to serve streams over TLS for philosophical/privacy-from-government/privacy-from-wifi reasons, but wants to lets ISPs cache data, they could create a certificate for each ISP/organization and provide the keys to that org.

Then, if NF can identify you are coming from a particular ISP, they can have your content served from the ISP's netflix subdomain, and the ISP could intercept/cache/re-encrypt the data.

Just a thought.

Could be, but it would be terrible if it went beyond impersonal data like Netflix content. One of the main benefits of SSL is that you don't have to trust your ISP or anyone else in-between with your data. I'm not a crypto expert, but your proposal sounds like a backdoor that could be abused.

There are two kinds of ISPs:

The first kind is the ones that Apple wants - they are the likes of Comcast/Optimum/etc. There are the ISPs that have lots of eyeballs and have captive audience. If Apple has shitty connectivity to them, it is going to be bad for Apple ( because consumer cannot replace such ISP - there's no real competition in those markets ). These are also ISPs that happen to peg their transit PNIs or free peering PNIs as much as possible forcing others to to buy paid peering to have non-shitty connectivity to them. These ISPs are going to charge Apple for colo, power and bandwidth and Apple is going to pay and it is going to pay through the nose ( just like Netflix does and just like Akamai does )

The other set of ISPs are the ones that want Apple a lot more than Apple wants them. There's no way Apple will pay for access to their customers. Those ISPs are going to give Apple space, power and even bandwidth for free. Hell, they may even have to pay Apple.

Source: did that for both ISP side and content provider side (at different times)

Because they will not have to pay for all that repetitive data being peered to Apple's network. It's cheaper to pay the electricity for the free Apple-provided servers than to pay for all the data transit, I guess.

Because it saves them money on transit costs. The Internet Peering Playbook is a good resource if you want to learn about some of the economics behind programs like this one (http://drpeering.net/core/bookOutline.html).

Edit: the flip side of this relationship is also interesting - if Apple or whoever doesn't offload enough of their traffic to the rack, then it isn't cost effective and can really annoy the ISP. I've known some ISPs to boot these caches out of their network when the related company wasn't utilizing it effectively.

The question here is why you would host such a device for free instead of just having SFI peering with apple (which is apparently required in this program). I suspect that the incentive there has mostly nothing to do with transit costs and is mainly about capacity planing inside the ISP's network and thus with fixed hardware and infrastructure costs. (ie. SFI is free, but the 100Gbps port on your router is not)

It may be possible to place caches deeper inside an ISP's network than the peering points. For example, it looks like Apple peers in Dallas but not Austin or Houston so putting a cache in Austin would save bandwidth up to Dallas.

In my experience it will invariably be placed deeper into the network, that is at least into network core of the ISP, which typically isn't anywhere near the edge router placed in some wonderfully expensive colo space associated with some IXP.

Completely right, it would've been more correct to say "saves money" and not specifically call out transit.

Interesting error message on this site. A modal with just "GoogleAnalyticsFailure" and a close button. Site works if you click on close.

Because they can serve 100G iOS updates off their own network as opposed to pushing 100G over the internet.

Because most ISPs have cost associated with getting traffic through their upstream provider(s). So this will save the ISP money.

Just to give perspective, Netflix and Youtube together constitute almost 25% of the total bandwidth traffic. If you could lower your cost of operation by 25% for free, why would you not?

These devices would be in addition to Apple's (likely large) colo footprint. Sending iOS updates to 1 billion+ devices is dissected in a paper here https://arxiv.org/abs/1810.02978

The cost (monetary, and environmental) of electricity associated with transporting data is high (as a recent HN article illustrated). Anything to lower costs is good for everyone. ISPs probably are only in it for the monetary savings, but thats still good.

> I'm confused why ISPs would do this for free.

No peering on immense libraries of HD video content?

ISPs usually don't do it "for free." Placing caches on operator networks is typically part of a larger business agreement that may not involve exchange of cash, but will exchange something of value.

No, it's absolutely done for free. The value ISPs receive is slashing their upstream transit costs.

So... per the upthread comment, "something of value" is indeed being exchanged?

Don't be silly. Obviously this is a business arrangement from which both parties expect to benefit, it's governed by a binding contract, etc... It probably even does have a set of billing rates for stuff at the margin, even. But no, it probably doesn't involve much real money flowing.

None of this is "for free" in any kind of economic sense.

Each party is deriving a benefit from this as in any agreement, but nothing "of value" is being exchanged between the two parties. That's my point.

Can you give defintions for "a benefit" and "value" such that your statement is not self-contradicting? Those look like synonyms to me. And to the law, for that matter: there's nothing that requires a business contract be exchanged with money alone.

Do we really need to be this pedantic? There is no money exchanging hands for this transaction. Apple / YouTube / Netflix puts a rack of stuff in the ISP for no charge. With the right traffic patterns everybody wins:

- End users get a better experience because there are fewer hops between the content and their computer.

- ISP gets to pay less on transit

- The content provider also spends less on transit while providing a better end-user experience

That's the point though. Value isn't "money" -- that's not pedantry, it's a comparatively profound truth. Asserting that this business arrangement constitutes getting something "for free" is missing the point.

This shouldn't be downvoted. The relative benefits to the ISP and Apple are part of a negotiation. Pinning the price to 0 means they're only interested in ISPs who will not be too demanding.

To reduce their transit costs and provide better QoS to their customers.

I appreciate everyone's replies to my question. I don't understand why it deserves downvotes.

Reply on news.ycombinator.com