Hacker News Reader

Comment by jrochkind1 | Hacker News Reader

Can anyone explain the case from T-Mobile's end?

(Not asking for sarcastic not-in-good-faith explanations of BS reasons that you are imagining.

Asking for anyone who understands more about a cell carrier's needs than I do, to explain what «the feature cuts off networks and servers from accessing vital network data and metadata and could impact “operator’s ability to efficiently manage telecommunication networks.» actually means, to someone who is not a telecom engineer but does understand engineering.

And/or other motives, but based on understanding more of their business than I do, not just wild guesses!)

View on HN

These aren't wild guesses, but I also don't have inside information.

1. Browsing history. We know that Verizon is tracking it for their gain: https://www.wired.com/story/verizon-user-privacy-settings/. It seems reasonable that T-Mobile and others don't want that door to close on them.

2. Video streaming management. Carriers typically restrict video streaming on some/all of their plans to certain resolutions. For example, I think most American carriers limit video streaming to around 480/720p at 1.5Mbps or less unless you have bought a premium plan. VPNs often get around this and I know that my carrier can't detect Netflix access through iCloud Private Relay. Right now, iCloud Private Relay doesn't proxy app traffic, but it could in the future.

3. It looks like mobile carriers are looking to get into "edge cloud" stuff. Verizon has been pushing this and they recently emphasized this in their 5G Ultra presentation. If traffic is going through iCloud Private Relay, buying expensive "edge cloud" services from Verizon is a waste of money since the traffic would be leaving the network to go through Private Relay.

3a. Netflix ships "Open Connect Appliances" that ISPs can hook into their network to serve Netflix content. If your traffic is going through a proxy, you start accessing the content on a server farther away. This mostly doesn't apply given that Private Relay only does Safari traffic, but one could see Private Relay expanding to apps in the future.

4. I think there is a certain knowledge of what is using data that can be helpful to carriers. For example, I worked for a university and they wanted to set different QoS for things like peer-to-peer file sharing vs. web browsing. The university didn't want to punish P2P tech or anything like that. They just wanted to make sure that P2P usage didn't overwhelm other users and uses of the network. Likewise, it could help the university spot patterns like viruses/bots that might be using a lot of network traffic.

4a. I think this can also play into how companies position their offerings. For example, T-Mobile has introduced features like "Music Freedom" and "Binge On" that allowed unlimited audio streaming and video streaming before unlimited plans were a thing. They surely did analysis of network usage of those features before introducing them. You can look at how much video streaming users are doing and then model how much data would be used if you limited it to 480p (including accounting for an uptick in usage due to it being unlimited). However, if you don't know how data is being used, you lose the ability to spot patterns that might be opportunities.

4b. It makes sense to want to offer different QoS for different services. If someone is using FaceTime, you want that to be a good experience. You don't want to prioritize a speed test over someone's FaceTime call. You don't want to prioritize downloading from YouTube over a FaceTime call. That YouTube video can be buffered and if you know that you've transferred 15 megabits worth of 1.5Mbps video, you kinda know that the user doesn't need the next 1.5 megabits of video for 10 seconds.

4c. I know that a lot of people want their connection to be an unbiased dumb-pipe, but I think that people only want that because they tend to see crappy stuff from companies looking for money. Seeing it from a university that only wanted to give people the best possible network experience feels a bit different. QoS can be a positive thing and a dumb-pipe isn't always great.

I'm a bit surprised that T-Mobile would go this route at this time. iCloud Private Relay doesn't proxy app traffic at this time and I haven't seen that they have a similar browsing-history program like Verizon's. Still, there are reasons to want to be able to understand your traffic both for business reasons and for a better customer experience. Again, I'm surprised because it seems like the reasons today are slimmer. I think the Netflix OCA use case is a good one since it reduces network usage in a way that simply helps the parties involved, but wouldn't really be possible if the traffic first went via another external server.

I'd emphasize that nothing here is to say that T-Mobile is doing the right thing. It's just to bring up areas where a company might want to know more about its network access patterns. Some of that can be used for good like the Netflix OCA system or giving higher QoS guarantees to FaceTime. Some of it can be used for bad like knowing using browsing history for advertising.

Very good point on cache servers for Netflix and YouTube. Telcos save a lot of money when they terminate the traffic in their network. Private relay may affect that. Although apple’s solution could be clever about it.

I think this is honestly the biggest legitimate concern for the carriers: where the bits go.

Right now a hypothetical metro area might have a 20G circuit to Carrier A, a 20G circuit to Carrier B, and 20G of private peering to {Netflix, Google, etc}. With private relay they need to rip all that out and replace it with a 60G pipe to Apple.

Apple's solution could be clever about it, and it currently doesn't even block edge routing on mobile apps. It currently only supports Safari traffic, and most streaming will be via an app with things like Netflix. I suppose if YouTube routes to an edge that may be affected if viewing via Safari, but I'd be surprised if peering agreements aren't made when Private Relay goes out of beta.

The reason is right in the "what's new" section of the T-Mobile privacy policy: https://www.t-mobile.com/privacy-center/our-practices/privac...

> "However, starting April 26, 2021, T‑Mobile will begin using some data we have about you, including information we learn from your web and device usage data (like the apps installed on your device) and interactions with our products and services, for our own and 3rd party advertising, unless you tell us not to."

T-Mobile sells browser history data to advertisers, and Private Relay blocks that revenue stream. They are on the offensive to protect their new-found profit center, and most likely are doing this now to show Apple that this is not a feature that they want to see be turned on by default.

It's the beginning of the same saber rattling that Facebook did when Apple announced it would simply ask customers if they wanted to allow apps to track them

T-Mobile, Verizon, AT&T, and other ISPs joined together and successfully lobbied Republicans in the US government for permission to record what their customers do online and sell that information [0, 1]. Apple's proxy service takes away that revenue source.

[0] https://www.techrepublic.com/article/the-real-reason-behind-...

[1] https://mashable.com/article/how-to-stop-tmobile-att-verizon...

On cell networks, video content is by far the largest consumer of bandwidth. And the default for video generally is to auto-adjust the resolution to the highest quality that the network supports. This kind of sucks, since bandwidth is a shared resource for all users of a given antenna on a cell tower.

Though Speedtest on your cell might show your connection speed as 100 megabits/sec down, cell networks special-case video by identifying it as video and rate-limiting it to something like 1 megabit/sec. This is considered "efficient network management". For T-Mobile, this based on the plan (https://www.t-mobile.com/cell-phone-plans), they sell either "SD streaming" or "4k UHD streaming". "SD streaming" is a fancy way to express that they rate-limit identified video streams to 1 megabit/sec.

They identify video streams by watching the IP your phone is connecting to and/or the hostname mentioned in the TLS SNI header and checking if it is Youtube, Netflix, etc. Sending video content over a VPN removes their ability to understand what the content is.

Then couldn’t apple add some metadata on the user’s behalf that this traffic is a video stream? Of course it can be spoofed by the user, but if the hard-to-change default is well-defined by apple, then networks can depend and use that info.

There is a solid, technical problem with VPN usage on such a massive scale. Carriers, like T-Mobile, can arrange traffic exchange with big content providers. Majority of traffic generated goes to a handful of providers, like YouTube, Netflix, Facebook. It's not even about direct, financial incentives. It's a win-win for both ISP and content providers to peer directly and limit the amount of traffic routed through paid uplinks. It's a win for users too, since they can get their content with less hops, through bigger pipes. Even Tier-1 network operators (https://en.wikipedia.org/wiki/Tier_1_network) can optimize traffic by making the direct inter-connections for traffic-heavy content.

When everything is encrypted and goes over the ISP just to the VPN endpoints, they can't do anything. In the end, they will have to arrange peering not with content providers but with VPN providers, who works for Apple.

PS. There is a lot of tension in current setup, even without Apple stepping up. In the old fashion market, the last mile is the king. Big grocery chains have direct access to users, so they are the strong side in the relation with producers. They can position brand X over Y, if they have better margin. They also create their own brand Z rip-off and sell that directly. Just look what Amazon does in that space. When it comes to ISP, they have direct users and have very little to say. They are basically dump pipes, just like the power line.

T-Mobile was very vocal in the past in that space. They often wanted the MANGAs (heh) of the world to pay them a share from their ads. I remember T-Mobile threatening, that they might replace some ads with their own ads. Since they provide the users with phones, they can install their own certs on devices. Chrome has SSL pinning not only, to save users from hackers, but to save their own business model being attacked by ISPs.

Wouldn't most carriers have peering relationships with Apple for Apple TV content? Presumably iCloud Private Relay traffic could be sent over that connection as well, meaning that of all the VPN-esque solutions, Apple's is the most efficient from a traffic exchange perspective.

(Of course, it was likely never about the traffic exchange itself, but rather the ability to route, shape, and track traffic dynamically on a host-by-host basis, as others have speculated.)

I am guessing that AppleTV traffic is very low compared to Netflix or YouTube. Anyway I guess this setup must be super complicated. Apple is probably partnering with tons of entities to provide this all over the world. It must be a huge challenge at Apple scale. I wonder if they have any long term goals, which are not obvious.

the apple vpn provider is cloudflare, which just do free peering

Well the ISPs and content providers can cry me a river about their network optimization. They brought this all upon themselves. Remember when nobody used to use VPNs?

I belive this functions like a VPN in some ways and blocks video throttling. They use traffic inspection to throttle video streams down to 480p unless you have the most premium of plans.

I've never heard that IPS (not the content provider) is throttling down Video Quality by altering the traffic. Do you have some links to back up that claim? This doesn't make much sense, as they would have to download the high quality video anyway, then invest massive CPU power to downscale this. Most content providers will scale down the quality if they detect bad network conditions. If ISP would want lower quality, they could just artifficaly slow the connection.

This has been happening for years. They just limit bandwidth to video sites, so your player selects the corresponding stream quality.

With Apple Relay this might get way harder, if not impossible.

Cell carriers can detect video streams by bandwidth and traffic patterns even with encryption. https://security.stackexchange.com/questions/172212/how-do-m...

Revenue from tracking customers for advertisements.

Non-cynically, it probably does introduce some issues in these legacy telecom systems.

For example, if you run out of data for a month, many carriers will continue giving you access to the internet APN, but then block access to "external" websites. This is so you can easily open your browser and "top up" on data to continue using your device.

Or the usage of HTTP (not HTTPS) was relatively common back when I was in the space (7-10 years ago). There wasn't a need to use HTTP because the carrier was in full control of the pipe between the device and the server. Adding in a VPN that somehow tries to intercept that traffic (that was supposed to exist entirely within the telecom) is not going to work.

But if that were the only reason why, then couldn't they just turn off Private Relay in that specific case, instead of all the time?

https://www.t-mobile.com/marketing-solutions/marketing-platf...

Reply on news.ycombinator.com