Dumb question -- how does this work with SSL? Is Apple putting their private keys on these boxes and assuming the ISPs can't/won't extract them?
Exactly how SSL is handled (or not) varies by provider, but one thing I'll mention is that these will typically never have a cert so important that it can't be easily revoked, and the most important data will likely not be flowing across them - exposure is very limited. Using video streaming as an example, one option is to only do TCP termination for example.com (or to not even terminate that domain on the local cache, but back at your main datacenter), then use subdomains with individual certs for the local cache (eg. isp1.cache.example.com). In that case, service calls like login, retrieving the manifest, etc. are secured by the certs you're keeping in your primary dc, then the manifest has a set of https://isp1.cache.example.com URLs pointing to the local cache only for video segments.
Another tricky aspect is making sure that your main network treats them as untrusted so someone with local access can't use it to get a foothold into the rest of your infra.
I think this is specifically addressed with the introduction of TLS Delegated Credentials[1]. This allows the CDN edge to use a very short lived credential in the place of the certificate's private key.
It's already supported in evergreen browsers and in certificate profiles from commercial CAs like Digicert.
The delegated creds draft that regecks mentioned is also relevant. That will make issuing lighter weight, so this sort of 'burn the cert and roll the DNS name' procedure becomes significantly cheaper operationally.
Apple own the intelligent layer, the one that hold the API. Once you query those, it answer with the location and the hash, which allow you to download it from the distributed box and safely verify the content.
- TLS is still important to stop tampering of video content or images, as well as user privacy over what content was specifically viewed.
- Some ISPs have (and still do) intercept plaintext video content -> transcode to a much lower bitrate -> cache that for their users. That hurts the content provider, as they lose visibility (logs/metrics), and the user who may suffer a reduced experience that the content provider can’t easily fix. End-to-end TLS solves that.
Then the client can reject any tampered stuff even without encryption on the transport.
TLS does wonders for preventing transparent proxies, DPI & ISPs from modifying or tracking what you see.
You won't have tampered content if the data is rejected, that's absurd. You can do the same with TLS by the way, that's the whole point of TLS, being able to verify (and thus reject or not) data.
Sure it break the client, but you can do that in any situation, just have to unplug a wire ;).
You bring a good point about user privacy, and for sure a key can help with that, but at the end of the day, there's not much you can do about this once the physical server is somewhere else, TLS or not, you'll need to trust the one that hold physically that server not to void user privacy.
I would still suggest a TLS connection for that server, but it would most probably be self signed with a different root certificate to avoid someone else to be able to make others trust him being Apple over something that wouldn't verify the content with hash coming from platform owned by Apple.
Note that I understand the importance of securing private keys in general. I'm just asking about ISPs because you specifically mentioned them.
Also, there's a surprising number of colo facilities that are pretty lax about who is allowed to come and go.
Being able to sign some things as Apple / MITM some Apple URLs strikes me as interesting enough to get state-security level interest, at which point you fall back to ISPs being made of people, and people being coercible and bribeable.
It's not necessarily that the ISPs will want to break into equipment, just that they are the weakest link bad actors can use to target Apple/Apple Customers.
They announced Keyless SSL way back when: https://blog.cloudflare.com/keyless-ssl-the-nitty-gritty-tec...
I'm a fan of Facebook's take on this, though: https://engineering.fb.com/security/delegated-credentials/
Also see, Google's answer to JWTs: Macaroons-- https://hackingdistributed.com/2014/05/16/macaroons-are-bett...
A single domain name and DNS to route is uncommon because it doesn't give you fine-grained control of load - you need to be mindful of the rack's capacity, and you also need to make sure that most of that ISP's customers go to the rack/people who aren't that ISP's customers don't go to it.
Anycasting isn't going to be great for traffic management or long-lived TCP conns, and if you can avoid the complexity of each rack needing a bgp session into the ISP's network you're going to be much better off.
Typically this is going to be directly routed to the rack via a unique DNS name after some form of service call.
Also they could just be signing content at the application layer like APT does, and the transport is plain HTTP.