AT&T Wireless traffic shaping apparently making some websites unusable
adriano.fyi
adriano.fyi
This I think is because they throttle ip addresses for known video streaming sites. That is one of (the only reliable) ways an ISP can get the streaming provider to drop the stream to a lower quality one by default. Since fast.com is a Netflix ip, and the isp can’t distinguish whether it’s video that is being transferred or a file to measure throughout, the speed test gets caught up in it. Said the other way around: fast.com is great to see actual throughput from Netflix as opposed to some fake throughout from a dedicated speed test site for the exact same reason.
Yup. This is pretty much the reason Netflix created fast.com. They wanted a speed test service that couldn’t be gamed by ISPs. Many ISPs will prioritise traffic to know speed test services (like Ookla’s speedtest.net), making their services appear faster than they’re under more normal usage.
By placing fast.com on Netflix IPs, ISPs either have to prioritise all Netflix traffic (which they’re very unlikely to do), or accept that fast.com is going to provide a more realistic measurement of their performance.
Another fun part: when netflix IPs was blocked by this ISP, it's pretty much impossible to use netflix because the only way to get around the block was to use VPN, but netflix itself blocks VPN access.
Surely it would essentially saturate your fibre if there was net neutrality, unless you don't pay for full fibre speeds?
Internet2 is used by hospitals, among other things, and as such has higher robustness requirements than Internet1. The pandemic created much higher demand for bandwidth from hospitals, forcing Internet2 providers to scramble to keep up in areas. As such, blocking high-bandwidth sites which are clearly lower priority than medical traffic might have actually been a reasonable move.
It might have been difficult relative to the infrastructure at some hospitals - but that is to my point that hospitals in general are not very demanding bandwidth wise and I would be a bit surprised they’re more than a rounding error on the Internet2 in general - especially compared to all the academic research involving large models and just video streaming in general that blew up during the pandemic.
I’m curious if the GP was speaking from some firsthand experience or is just making some conjecture.
Remote Desktop and even PACS are not huge bandwidth consumers in this day and age of 4K streaming. At my large academic institution usually only the radiologists use the PACS client directly on workstations - all the other clinicians are using a “zero footprint” viewer over Citrix like everything else.
Neither of them actually ran any traffic over it.
Thinking about it, it seems like watching habits could make streaming be more or less "efficient" compared to downloading once and storing it locally (depends on how one chooses to calculate efficiency tbh).
People that have things they regularly re-watch are obviously gonna benefit from having a local copy they can access entirely on their own terms.
It hadn't occurred to me until recently, but the way I watch stuff has me going through a pretty large amount of data yet I'm also excessively unlikely I re-watch nearly anything that I've seen in the previous 2 years. Whether something is streamed or stored local, it has to be downloaded the first time it's watched. If it'll take 2 years before I re-watch something, it's a waste of time and money to keep a backlog of multiple disks worth of video
Very wasteful if I did much streaming.
They may have an ISP-local netflix cache, from Netflix themselves not some home-grown hack, so they can achieve that with some reliability without it costing as much as it otherwise would for bandwidth peering.
(PDF): https://openconnect.netflix.com/Open-Connect-Briefing-Paper....
But they're intentionally otherwise indistinguishable from any other real movie on Netflix.
Mb = megabits
1 byte = 8-bit
Perhaps HNers know better, but among lay people, pretty much no one talks about megabits. For example, people know that an MP3 is about 5MB. They know that they can email attachments up to 20MB (or sometimes 10). But I've literally never heard a lay person talk about bits, or bits per second.
And the fact that ISP's transfer caps are denominated in bytes makes things all the more confusing.
(As a historical side note, there was even a weird transitional period where a “1.44 MB 3.5 floppy” held 1.44 * 1000 * 1024 bytes, or what everyone at the time would have called 1440 kilobytes.)
Personally I think it’s unfortunate that standards bodies and even courts have sided with the disk manufacturers. I also find the SI binary prefixes utterly revolting on an aesthetic level, so I’d much rather join whatever parts of the tech industry are still holding out.
It is really trivial to do basic traffic snooping and see what people are looking at. I'm surprised it isn't more common.
I figured it would be harder, or perform worse, but I easily wrote a little piece of software that filters the TLS ClientHello for arbitrary domains. Maybe 10 years ago hardware wouldn't have been able to do this, but I bet it's no big deal now. So your filter chain just looks like <Netflix IP range> -> <has fast.com ClientHello> -> unthrottle. You don't need to do packet inspection on every packet, just ones that you might be interested in (e.g. Netflix IPs).
It's crazy to me that the many people who care about privacy and censorship in tech haven't pushed ECH (encrypted ClientHello) harder. It's such a gaping hole in web privacy that you can still passively snoop domain names sent in cleartext. It makes DoH/DoT almost pointless.
Incidentally, my speeds on fast.com are always terrible (about 1/8 of what I get elsewhere), despite the fact that I'm fairly confident it is not being throttled. That's because the speed I see is >100 Mbps, which is like 4 Netflix UHD streams. Wouldn't be much point in throttling to that speed, you'd want 10 Mbps max, and less on wireless.
The correct mental model is that it’s good enough to convince 1990’s US internet users to type their credit card into a web page. (Where the downside of a breach is that you have to dispute some charges and change your CC#.)
If you need stronger security than that, then many, many caveats start to apply.
For instance, by default, anyone that can reliably man-in-the-middle port 80 on your website can get an acme certificate for your domain from a reputable certificate authority.
Also LE actually knows this attack is possible and mitigates it by validating the challenge from multiple sources so the attacker would need to be in the middle of all the LE validators and your servers.
https://portswigger.net/daily-swig/lets-encrypt-deploys-new-...
You might have access through editing a proxy rewrite rule, for example.
In the attack above you use your own SSL provider for a cert (LE not involved) and you overwrite the cert in a sense that existed before. You choose a provider that just validates with a file location. It's an attack which already requires compromise.
It’s not hard to monitor cert transparency logs, which will show this new cert.
I think it’s more likely you are not considering the full picture of the TLS ecosystem, or are making arbitrary distinctions like “cert transparency logging isn’t actually part of HTTPS” or something.
Consider that Symantec basically did what you suggest (mis-issue some certs) and not only was it detected and mitigated, they lost their CA business entirely over it.
You are leaving out a huge caveat here - exactly where the MITM is taking place matters a lot. In 99% cases this isn't possible unless the victim server network is effectively compromised.
Sure, they could apply some kind of advanced DPI assessment based on packets sizes over time and other crap, but if anyone complains it's much easier to just say "well, does speedtest.net work? What about Google's speedtest? Hmm, strange, must be fast.com being slow then!". If they just check for fast.com in the SNI header, I know I'd be loading the speedtest every time I want to watch Netflix.
ECH is slowly coming along, but it's still perfectly possible to detect these speed tests. It just takes much more time and effort to set up right.
I'm honestly surprised the article shows enabling a VPN actually working to fix the messed up network here. That would be a perfectly valid solution in my opinion; if I notice my ISP is messing with my network like this, there's no way I trust them enough not to use a VPN.
They don't count the "shaped/throttled" sites against your data plan limits, so I can see some people liking it.
Some throttling by default for video actually is sort of for your benefit, at least indirectly, although I don't know if carriers are throttling more than that amount.
A surprisingly large number of people nowadays watch movies on their phones or small tablets. On screens that size at the distances they typically hold them from their eyes 4K and usually 1080p won't offer any visible improvement over 720p. On some devices even 720p probably won't offer an improvement over 480p.
It is to everyone's benefit if other people on the network don't stream at a higher resolution than is necessary to reach the limit of what is visible. Since most people aren't going to change any settings from the default, defaulting to low speed streams maximizes benefits.
Therefore, I have a background bash loop just looking up fast.com every 30 seconds all day long.
I’m working on using a Smarty (Three UK-owned budget label MVNO, for those unaware) SIM card in a 5G modem as a backup line, and haven’t been super impressed with speeds — so curious about trying your tip.
Technically the website fast.com resolves to an Akamai IP, but the speed test it loads uses a Netflix owned IP.
Company A was suing another big cable company (Company X) b/c A claimed that X was throttling customers without telling them and advertising what were in effect unthrottled bandwidth numbers.
In order to prove this, the company I worked for went out and hired people to get cable modem subscriptions for Company X, install desktop that were locked down so the people couldn't use the desktops themselves and then collected ping and HTTP request data for months.
My job was to take that data and analyze it for throttling patterns.
This was my first job out of grad school and involved lots of:
- data cleaning and organizing using Perl
- analyzing statistics using Excel
- creating charts that could then be shown to the lawyers and in turn could be used in court
I didn't know much about databases or stats packages in Perl which, in hindsight, would have made my life easier.
I still remember being on the call with the lawyers and walking them through the data that proved that Company X was indeed throttling bandwidth at peak usage times. The one lawyer on the conference call said "Wow! This data is amazing. You guys are now basically national experts in this."
I was both:
- flattered that someone would call me a national expert in something
- scared shitless that I would end up being cross examined about how we processed this data
Turns out that the court case never happened and they settled out of court.
The reason I mention this story is that it's FASCINATING to me that this is still occurring today. I guess some things never do change.
As long as it's an out-of-court settlement for a moderate sum, it's just a cost of doing business, a cost of R&D of sorts, only in the legal area. (Similar with taxes.)
Your company sounds like they knew the worth of most high-end developers than most big tech-companies do (or, in light of all the mass layoffs, did).
What changed before the last couple of days? My laptop was frequently dropping WiFi connection- a lot of times in the middle of zoom meetings. We use a router provided by AT&T. My wife complained of the same thing, so I thought I'll ping their customer support.
The customer support person proceeded to tell us they wanted to reconfigure our router to enable "5G". After some bewilderment we realized they were talking about 2.4G vs. 5G WiFi. And they will do it remotely. Ok fine go ahead, we said, and forgot about it.. until I got the call from my office IT security.
Apparently the AT&T support person left our router in passthrough mode. According to their K/B, "Placing a device in passthrough mode will remove firewall protection provided by the AT&T gateway."
I reset my router to default settings, and got myself a 100ft Ethernet cable to fix the issues.
Did... did they seriously turn the router into a DMZ*, without your consent? Where every port of your computer is just open to the internet?
That's scary.
*that's what it was called on my router, the "forward every single port to one device" mode. Not sure if that is the correct term for it.
If only we would finish the switch to IPv6 we could pull the rug out from under those bots.
Oh absolutely, I used that trick once in order to use a router that I could configure over web interface instead of Over The Cloud With A Spyware Mobile App (aka Internet Of Shit).
Unfortunately my ISP smartened up to this and started cutting the line whenever I tried to use my router. Like, it would work for a while, then they'd shut off service, and it wouldn't work anymore (even if I removed my router) until I called them up and complained.
They would act oblivious like the problem was with my equipment. Every time I asked them why my internet stopped working they would say there was some problem with my hardware. The hardware was brand new and could not have been more fine; they were mistaken. It would also work perfectly when I actually had service; the issue was the ISP kept cutting it because they're petty bastards.
Eventually I gave up, which is probably what they wanted, but it's not like I'm suffering too badly with their equipment, even if I have to use a rooted phone in order to pry away all the spyware permissions from their stupid app.
They are a monopoly so I have no choice.
It was a really shitty situation that should never have happened, but the executive team was what all customer service should strive to be. A single person, with ample time in their day to fully understand the problem and blow past all the usual roadblocks on the way to a solution. Was still a few hours on the phone, but at least I wasn't treated as though it was my fault.
About Xfinity in particular though, assuming someone else uses them, I did learn just recently that at least Xfinity allegedly offers a special decoder box (or something) that is, I believe, free of charge, and lets you hook up your router directly rather than putting it through theirs as a DMZ, and it's supposed to get them to not cut the line. Some self-install kit or something. You can only get it by asking over phone call.
1. You had a modem that wasn't on their approved device list. It needs to be on this list to receive firmware customized to their network parameters. https://www.xfinity.com/support/devices/
2. (the most likely scenario) You were switching the macaddr of your router's WAN port every time you connected. For IPV4 Comcast was always set up to only allow one macaddr to bind to a public address at a time, you needed to trigger a dhcpcd release on the old macaddr before reconnecting or clone the mac of the old device if switching.
Business class had launched new speed tiers and it broke upstream bonding on my modem, of course they blamed me. So I bought 2 more modems (same model and a different one) and the same things happened as soon as they got provisioned on the account (prior to this on Comcast's provisioning walled garden they'd show the upstream as bonded, but once activated on the account upstream bonding would turn off and speeds went to shit). By this point I had records and emails with like 4 or 5 different people, a useful thread on dslreports, and a very worthless thread on reddit.
I took all this info and CC'd everyone I spoke with in an email to the CEO of Comcast. Executive response had someone up in Massachusetts call me the next day confirming the issue was with their provisioning boot files, and they credited me for like 3 or 4 months of service. Overall I was pissed it was this difficult to escalate but at least once it was in the right office they handled it. It probably helped that this was a business class (which overpays for better support vs residential script readers).
Same applies for my other two computers (my MacBook runs Little Snitch on 10.14, so it's a kernel extension that Apple can't bypass). Not the rest of my devices though. Though it's not like anyone could get through fail2ban on my server, or compromise my Android device, or hack any IoT devices since I own none, etc.
[0]: https://safing.io
asking because: Your comment reminded me of one article / discussion about Apple Devices causing huge network load , and hence delays, due to some map ping thingy.
[Apple Maps location scan spikes WiFi latency every 60 seconds. 677 points, 172 comments]
https://news.ycombinator.com/item?id=31356730
(I hate the '5G WiFI' that most ISPs seem to be calling it everywhere. )
"yeah... we're gonna have to go ahead and reconfigure your home to operate in passthrough mode. This is going to require techs onsite so we went ahead and deployed some generic subcontractors. What they have to do there is, well kinda counterintuitive. They will be using brute force techniques to uninstall and didpose of ALL of your doors. So.... there's a bunch of little wireless packet things, like the kind use for 5G, or the other 5G, flying around everywhere. They always bounce off things and they get lost a bunch. Now that home passthrough mode is enabled, the packets can just 'pass through' and get where they wanna be. Sometimes other things, like humans, or ocelots, 'pass through' too. How exciting! After all, at ATT no one gives a ** if all kinds of critters go through your crap whenever we 'fix' something. "
When your ISP screws up your router settings: 1) It's highly likely you won't be aware there's a major security problem with your network. 2) potential attackers include, but are not limited to, massive numbers of ill-intentioned, yet impossible to identify, individuals quckly become aware there's a major security problem with your network.
When your ISP performs a literal brute force attack on your domicile, leaving you completely doorless: 1) You KNOW, 100%, that there's a major security problem. Your doors are gone. No doors. The physical equivalent of a router in passthrough mode. 2) Potential attackers are within a relatively limited geographic area, with numerous ways to learn more. They've likely noticed that things in your home life have been a bit weird.
As long as I'm not messing with the network, it's all great, and I know where to start looking if there is odd traffic going on.
I've been working on a high performance metaverse client. All Rust, all multi-threaded. Designed to max out a gamer PC. Supports Second Life and Open Simulator. Second Life had a reputation for being sluggish. Fixing that.
I'm pulling content from the servers at 200Mb/s, sustained. 400Mb/s in tests, but don't need to go that fast. The servers (AWS front-ended by Akamai caches) can handle that just fine. Gigabit fiber can handle that just fine. The 3D world appears in high detail in seconds. Looks like an AAA game title. Not like low-rez Meta Horizon or Decentraland.
5G can handle that, right? Says so right here in the promotional materials.Verizon: [2] AT&T: [3]
The carriers said "unlimited", right? So they can't complain if you're downloading 100GB per hour.
[1] https://www.forbes.com/sites/michaelgale/2022/05/24/how-5-an...
[2] https://www.ericsson.com/en/blog/2022/4/why-metaverse-needs-...
[3] https://www.verizon.com/about/news/5g-makes-metaverse-real
[4] https://www.xrtoday.com/event-news/5g-networks-crucial-to-me...
Just charge it to my offshore. Cheers
Limited buffet hours. Most Indian buffets are lunchtime only, and the restaurant is open, say, 11am-1pm, and then closes until dinner. Indian buffets are often leftovers from last night's dinner, too, so there's a limit to the amount of food that they won't cook more.
Size of plates, bowls, glasses. This is an easy one, and you can use it to go on a diet. If you have large plates or bowls, throw them out and get small-capacity dishes. Buffets will make you use a certain size plate, and so each trip you can only pick up so much food, and this helps you notice that you feel full before taking more.
You may also be required to use a clean plate each time you dip into the buffet. This might have various effects on how you regard the food you took originally.
* Truly Limited for heavy users.
[1] https://www.csis.org/analysis/accelerating-5g-united-states
[2] https://thehill.com/business-a-lobbying/business-a-lobbying/...
So there's a good chance there's a shaper letting through about 320kbit/s as it's relatively even throughout the capture.
There's one other person from HN doing some analysis with me via email. Maybe something will come of it. You both came to the same conclusion based on the pcap data. They're also analyzing the iphone tether pcap data I just provided in an update.
I took a very quick look at the iPhone capture. I don't see any notable issue (which is expected since this is "working" behavour), and the main noticeable item in the comparison is an apparent lower latency to the server. This appears to match the speculation I saw in the updates as well.
One thing you might be able to try, if you're able to change the APN on the hotspot, is to see if you're able to connect with a different APN from the consumer side. This is pure speculation, but what some carriers do is use a different APN for their business services, that will route to enterprise only packet gateways. Some carriers may not necessarily disable access to the consumer APNs though that might go through a different P-GW that's closer to you. The iPhone looks like it's using a path about 120ms shorter than the hotspot. And just by chance, going through a different P-GW you might hit different traffic shaping equipment or configuration.
Edit: although be aware that the carrier could also charge different APNs separately. I have no idea what AT&T does.
I went ahead and gave it a try anyway, switching from `broadband` to `NXTGENPHONE`. I wasn't able to authenticate with `NXTGENPHONE`, so I switched it back to `broadband`.
After re-authenticating, I'm no longer being throttled. I'm both happy and deeply dissatisfied with this outcome because I think now I'm less likely to get to the bottom of it.
I'll add this to my updates and keep monitoring the situation.
Thanks, I think!
By disconnecting / reconnecting by trying the APN switch, it might've just put you on a different P-GW that has a different traffic shaping configuration or current state.
I think I read somewhere about someone who did exactly like this in a software company. A new employee showed up, immediately fixed a long-standing bug, then submit their resignation.
Agreed that segment offload is probably kernel-side coalescing.
From shunting this into Wireshark, you can see that the sequence number grows fairly linearly (often a good indicator of a rate limiter) starting from 1.508s up until the 1.95s mark, but at a rate closer to 25-30 Mbps (1.5MB in 0.45s).
https://i.imgur.com/s2JBrCf.png
edit: this was for iphone-capture.pcap. You see a similarly strong picture of shaping for capture.pcap [0], and indeed that corresponds to the ~320kbps range you've noted.
[0] https://i.imgur.com/g9cN95d.png
edit edit: of course, under normal situations, your bandwidth will get saturated, and steady-state will look fairly linear. If you're using "classic" congestion control like CUBIC or New Reno, you'll see this interspersed with drops (search for "tcp sawtooth"). Under BBR you'd expect to see drops in bandwidth due to latency spikes (per bufferbloat). Neither of those seem to be in play here.
Years ago, I had to troubleshoot a very mysterious issue where sometimes a huge percentage of the PCs in the company I worked for would have trouble loading the company's e-commerce site.
Everyone was going through the same array of 2-3 proxy servers, and the e-commerce site had Akamai caching in front of it. The proxy servers would hold onto a DNS lookup result or a TCP connection for longer than they were supposed to, and so they'd continue routing traffic to the old IP address even after Akamai had rotated a node out of service.
This is plausible. Nothing I saw in the packet capture specifically pointed in this direction, but I don't really have the information I'd want to rule this out totally.
I've seen issues like this when the TCP stack on the server effectively runs out of allocated memory. This can cause the server not to use all of the available tcp window.
The only reason I think this is less likely, is I'd expect a company like AWS to monitor and tune the memory on their CDN, and notice something like a kernel bug here or block an attack exploiting the kernel memory.
Some network providers, especially wireless providers also deploy TCP acceleration equipment, so that network speeds are faster. On a wireless carrier if you've got temporary bad signal and drop some packets, if the carrier can retransmit them instead of going to the internet, this makes for a faster connection. Crappier implementations of these TCP accelerators, do a sort of transparent redirect to the kernel TCP stack. So if the ATT proxy has run out of kernel memory, it could be providing a slow connection. It wouldn't surprise me for a telecom carrier to miss and not tune or monitor kernel memory.
If the issue was caused by a proxy, I'd expect more to be broken then a single server, and other customers to also be complaining.
If ATT is using these accelerators, it also means the proxy could be hiding packet loss and delays between the proxy and upstream server or network. Because the client TCP connection is only to the proxy, we only get to see what happened between the client and proxy. So this precluded drawing strong conclusions from the client side capture only.
There is one thing about the packet capture that suggests a shaper to me. And that's how even the traffic is from server to client. While I didn't spend a bunch of time timing out all the packets, it looked a bit like I'd expect to see for packets getting released by a token bucket algorithm. This isn't necessarily safe to conclude, because the client has an offload that is buffering and merging packets using a receive segment offload, which causes us to miss some timing information in the capture.
So based on the above, shaping looks a little bit more likely to me, but I don't have the right information to rule out a broken or miss-configured server. An overloaded or otherwise broken network link like errors on a link doesn't appear likely to me at all.
But apart from all the data breaches, I was also able to verify T-Mobile doing this (arbitrary blocking of texts containing innocuous URLs) on my plan. Although they seem to have fixed it now. https://news.ycombinator.com/item?id=29744347
Even if this were technically feasible, it sounds like a massive infrastructure investment with little to no value. T-Mobile would need to have enough compute and network horsepower to DPI all outbound traffic, intercept every video stream, detect if it's over some resolution threshold, and re-encode it at a lower resolution or bit rate, all in real time.
YouTube over mobile for me is downscaled to 720p. In exchange, that doesn’t get counted in my high speed allotment of data.
I can opt out, but then my data can get deprioritized over a certain threshold.
I can opt out, but then any streaming goes against my high speed data limit.
I think it’s actually a pretty fair trade off.
T-Mobile is also completely hit or miss with service. If you are in a location where it works, fine. Good luck if you travel.
If you can tolerate T-Mobile, Google Fi is better in almost every way security wise -- and I am very anti-google services these days. It uses the T-Mobile network.
As for me? ATT has provided the best service of any carrier while traveling, so I will use them. When I need security, I flip on my VPN.
Be careful there, because T-Mobile rot spreads to any downstream MNVO. Sim-jacking is still possible and data breach is happening above Google Fi level. There were articles about that IIRC.
ATT has provided the best service of any carrier while traveling, so I will use them.
You literally land in a new country, turn your phone back on and you get a "Welcome to [country] - your data rate is the same" message almost anywhere.
Personally - I've flown from Taiwan to Brazil to Amsterdam and then back to the US and I don't have to think about my phone. It just works.
---
Outside of the travel use-case, I would also probably pick something else, but if I know I'm going to be travelling, I'll switch back to Fi.
And the key thing is I just don't have to think about it. I can't forget to register a new account, I don't have to worry about esoteric sign up requirements for certain countries (ex: Brazil wants a CPF for fucking everything), and I can't get stuck without a connection and then not be able to setup the next step.
Since before the merger, it used Sprint and T-Mobile, in addition to US Cellular https://techcrunch.com/2018/01/17/googles-project-fi-now-cap... (ctrl-f sprint)
Discount MVNOs increase their margins by buying wholesale deprioritized data while Google Fi has negotiated the no deprioritization.
0: https://old.reddit.com/r/GoogleFi/comments/ulc1t5/perks_of_f...
>QCI 6 is applied to all of T-Mobile's postpaid and prepaid plans (except for Essentials) and Google Fi which also has QCI 6 as well. This means if you want the absolute best from T-Mobile, you want to get a plan directly from them. Even their cheap $10 prepaid 1GB Connect plan has priority data.
You can apparently confirm this with a rooted phone: https://coveragecritic.com/2019/09/17/how-to-find-qci-values...
>My Google Fi service had a QCI of 6 during regular data use.
As a tmo customer I know I’m essentially off-grid in much of the country when I’m outside many metropolitan areas.
But tmo is great for traveling internationally.
ATT has a great plan for the Americas, south America is all covered, Canada is covered, and many other places. It worked really well for me in Brazil.
That said, when I start traveling to Europe and Asia more, I may switch back to Google Fi.
For domestic use, while traveling / on the road a lot? I would rate as follows:
ATT > Google Fi > T-Mobile > Verizon
Keep in mind, if you are mostly stationary it is better to use the carrier known for good service in your fixed location.
(T-Mobile customer for 2+ decades, least terrible option imho)
EDIT: > Google Voice app may be slow but it does work every time
Until it doesn't! Fair critique though. I can also recommend the "Unlisted" iOS app for this purpose.
You had my interest...till i read the reviews of the DIGITS app on the app store. 1.9 average, most complains about it not working at all. Google Voice app may be slow but it does work every time
I'm curious about your reasons to say that AT&T provides the best service "while traveling". I'm guessing it would be domestic travel, because AT&T roaming charges are the second highest among the big three (I believe Verizon is actually even more expensive).
I was a customer and an employee of AT&T for a while, and I find T-Mobile to be better in almost every aspect, except perhaps for coverage in very rural areas, which I don't mind. T-Mobile 5G coverage and speed is also significantly better.
Many ATT plans include usage in all of the western hemisphere, excluding the Caribbean islands. And for countries not included, it is $10 per day ($5 for other lines on your plan) up to 10 days in a billing cycle, and after that it is free until the next billing cycle.
Not the cheapest (I think T-Mobile has international at $5 per day), but not terrible either for a quick jaunt somewhere and not having to worry about SIMs or changing phone number or whatever.
I’m on T-Mobile right now, and the plan equivalent to the one I had with AT&T (all inclusive and unlimited data), does include 5GB of international data per month. AT&T was $10, even for employees, so not a great deal for frequent travelers.
AT&T by far has the worst network, even if you have great service. They NAT millions of devices behind a single /64 network, you're usually routed to IXs very far away from you, and they do the most packet manipulation out of all the carriers. Verizon is better, but they can do some wacky stuff if you aren't in New England or California.
Their focus is speed and density. The other two have much more advanced network management, which as a customer is both good and bad.
When I switched to an ISP with local peering, ping times between my home and my server downtown dropped to a few milliseconds. It's like being on my LAN.
If this is still an issue for AT&T, traceroute+reverse DNS of each router may indicate it.
This poor peering isn't directly a throughput problem, but really increases the chance of being routed through a congested link, which you could sometimes guestimate from ping times and peeringdb.
Some national players do have ports at domestic IXs, but only as backup links or for negotiated (paid?) access.
The hazards of ISPs also being backbone providers.
(It looks like Rogers has gotten better with this behaviour)
Too bad the indie ISP market has been decimated. Ebox & Distributel --> Bell. Start --> Telus. Oxio --> Cogeco. VMedia --> Quebecor (Videotron). Mostly only Teksavvy left standing
Indeed with Bell & Telus, you could only peer freely on US soil:
https://www.peeringdb.com/net/1550 https://www.peeringdb.com/net/76
This is no different than me approaching Verizon or at&t and demanding they peer with me.
For several years the TELUS and Bell peering PNIs (OC12s back then) would run hot regularly in domestic Canada - for years.
You'll never get them to peer openly, but at best you can get them to offer market rates for paid peer (which is what folks have tried to do in the US).
Maybe I’ve got it wrong, but don’t my peeringdb links suggest Bell and Telus do peer openly, but only on US soil?
And yet ATT will get away with this fraud or theft of selling something it is not delivering because no one in government wants to hold them accountable.
Now I just need to figure out what to tell our users that are having these issues...
Update: For what it's worth, the "above the fold" portion of the content loads down to "All Debuts". After that, there's a long blank space with a Loading indicator near the bottom. Eventually it loaded. I didn't keep track of how long, but it was almost certainly longer than 1 minute.
Support has been awful, best of luck if you go that route. They took us off their unlimited plan without consent, took 6 months to get it back. Also, throttling doesn't actually seem to happen once over their limit on the advertised unlimited* data plan, it's just most of the Internet is awfully slow most of the time going directly through this, ah, ISP, which they claim to be.
* Not actually an unlimited plan
But yes, AT&T throttles a wide diversity of content, even on their very highest tier plan (Unlimited Elite, now named Unlimited Premium). This is advertised as never throttling, no matter how much data you use (all other customers are supposed to get throttled first).
Also the throttling happens with mobile hotspot, I have to use VPN on my laptop with many many sites as well, even when I'm inside the "40GB/mo of unthrottled hotspot data".
The speeds instantly go way, way up once I hop on VPN.
That's not traffic shaping, it's just dumb. Traffic shaping is slowing down types of traffic like torrents or video in order to prioritize regular sites. But this is degrading regular sites that are usually what benefit from traffic shaping if there's overall network congestion.
I'm not questioning anything the author reports, it all seems extremely well documented.
I am questioning what AT&T is trying to accomplish here though. It makes no sense, unless they're in some kind of negotiation with Cloudfront right now, punishing them until they pay up or something? But that doesn't explain why it would only happen on a data-only plan. I'm mystified.
So I'm left with doing (dumb) traffic shaping by destination and target.
If I had to guess, I'd say that they incorrectly thought that some specific IP address (range) serves predominantly one type of data. So they throttle by the only data point they have, destination ip, and the collateral damage is everything else hosted on that ip address.
So to detect bittorrent, they'd build a profile about how many bit torrent clients operate, the packet and connection creation patterns used, and then slap a throttle on. Looking at some independent analysis, these products might only detect 50% of the bittorent traffic, and have a false positive rate, especially for bittorent users also doing something else. And the ISPs don't care, they get what they need if they clamp 50% of the traffic.
So I'm not disputing that everything encrypted is a good thing, just pointing out that because it's encrypted doesn't necessarily mean the shaping equipment can't figure out enough to throttle bit torrent.
> If I had to guess, I'd say that they incorrectly thought that some specific IP address (range) serves predominantly one type of data. So they throttle by the only data point they have, destination ip, and the collateral damage is everything else hosted on that ip address.
This is plausible. As I recall, the way some of the equipment worked was it would sniff out DNS requests, and then mark the IP address as this destination. So if someone set's a rule for example.com, it might accidentally apply to alice.com using the same IP address.
My knowledge on the industry is out of date though.
I imagine lots of people are or have spent lots of money and time trying to figure out the type of data or connection from patterns as you say.
A more nuanced and correct statement would have been to say that it's much harder to do than it used to be, when you could just look at the mime-type or similar to figure out what to throttle.
Most providers that use traffic shaping don't care about content, and the encrypted traffic classification is enough to make traffic policy decisions.
I experienced the same thing when building a large scale video service, but with Comcast. I did extensive testing to isolate it to any direct connection to aws (CloudFront or other aws services).
I'd probably double check assumptions on the cloudfront issues. Switching carriers or adding VPN might connect you to a different edge node.
Some web assets being throttled for specific AT&T accounts seems a little too targeted to just be traffic shaping. I'd expect them to throttle traffic for all users, like they do with the speed tests.
First time seeing it being invisible, mostly they just disable it. Terribly annoying practice. I often select text while reading articles.
I think it helps me read faster online, but it's probably just a fidget-y habit.
Even if you have any amount of 'fast data' left you're throttled to SD speeds for streaming video unless you pay for the higher tier of their plans. Data isn't equal any more. Their higher plans say "HD video is up to 1080p" so I suppose no 1440p/4K/60fps/etc either.
It's honestly fucking infuriating but there's nothing that can be done except use a VPN I guess. I'm on the previous generation so "data is data" and they don't throttle me yet but if I ever change plans it's there.
I'm not totally sure if the other carriers do (or their 'competition' in forms of MVNOs that they themselves own) but they tend to have a habit of copying each other, at least after a little delay.
https://www.bell.ca/Mobility/Cell_phone_plans/Unlimited-plan... (select your local area or just choose Ontario)
bonus fun bs:
every so often when you try to log into the web portal they try to trick you into letting them build a profile off your browsing/usage data so they can make even more money off of you. a popup with 'advertising is a reality in today's world' comes up with the nice attractive 'get this out of my face so I can do things' blue button being 'yes please opt me in'. shady dark pattern!
back when, they used to opt people in by default and make you explicitly opt out... but then the regulators said hey that's illegal. so now they resort to stuff like that to get the numbers back up.
here's some of the blurb if anyone is interested: "Advertising is a reality in today’s world, and people find that they receive ads that are irrelevant to them. With our tailored marketing program, Bell will work to ensure that the offers participants receive when using our services may be more relevant, rather than random marketing ads. In other words, participants won’t see more ads, just more relevant ads."
full text: https://pastebin.com/ESskYEUy it's honestly super gross what they collect and how they try to trick people into agreeing.
My guess is that Netflix is throttled, and given that fast.com is a Netflix site, fast.com is throttled too.
- maximum transmission unit (MTU)
- TCP maximum segment size (MSS)
- different DNS responses leading to different edge servers
- TCP reordering, which may now occur on the tunnel layer
- lost packet retransmission, which may now occur on the tunnel layer
- time to live (TTL) hop count on the packet
- IPv4 may be used for some connections that were previously using IPv6, as some VPN services are IPv4-only
- different peering between the VPN server and the edge server
ISP traffic categorization is only one variable.
Oh come on. There's plenty of ICs out there who'll happily do slapdash work because nobody cares if you close tickets well as long as you close them good enough. So what if the fix only lasts 2yr before needing to be refactored for performance. Lord knows if the code will even be in use then.
You could try changing the DNS server on your hotspot to a different public resolver like 1.1.1.1 or 8.8.8.8. If CloudFront is using DNS based geolocation to route you to the nearest data center, a different DNS server may get you routed around the issue to a different data center.
I'll add a traceroute from the phone and one from a device connected to the LTE router to the updates section next.
[update] Traceroutes added
I suppose it should be noted. The iphone antenna and router antennas are no more than a meter from each other. They're unlikely to be hitting different towers.
You can use Quad9's EDNS [0] so your client is properly routed, any privacy concerns aside.
If the author is reading this, I recommend putting an update at the head of your article with this. It will be kinder to your readers.
The Strava javascript file used for speed tests is 1.68MB uncompressed. But in a browser and most all other situations it should be requested with `Accept-Encoding: ...`. In Chrome, Strava responds with a gzipped response that's 463kB in size.
This doesn't really matter for the CLI speed tests, when its requested without compression, but it does mean that the speed test performance won't correspond with the actual in-browser performance and the traffic shaping may not be comparable when the request is made without compression. Adding `--compression=gzip` to the wget command will fix this.
To quickly show the difference:
$ curl -s 'https://web-assets.strava.com/assets/federated/find-and-invite-friends/827.js' --compressed -w '%{size_download}\n' -o /dev/null
462525
$ curl -s 'https://web-assets.strava.com/assets/federated/find-and-invite-friends/827.js' -w '%{size_download}\n' -o /dev/null
1759662But consider this. When I load Strava's dashboard, open the Network tab and search for "cloudfront", these are the metrics:
> 374 requests 15.80 MB / 7.07 MB transferred Finish: 4.89 min
This is not a good time, and no amount of compression is going to help the situation.
For you a speed test is just once in a life time event.
For a wireless provider this is a thing what clogs the shared media of a radio channel for quite a lot (modern speed tests try to push 100 or even more MBytes?) and cause a disruption for everyone else on the same channel.
I don't know why ATT throttles (if at all, not a customer) that site, but wire and wireless providers are different.
A lot of mobile services (at least in the US) offer "unlimited streaming" which is really just bandwidth-limited to offer SD-quality streams and offer "HD" (read: less-limited) streaming as a paid add-on.
This coverage from when Netflix launched Fast.com in 2016 alludes to the reason why the results on Fast.com may be slower than on something like Speedtest.net. ISPs like AT&T throttle connections to streaming video providers like Netflix, in order to force a lower quality stream. By offering a speed test from the same infrastructure that delivers your TV show, they are revealing the 'traffic shaping' that is degrading your connection. Sites like Speedtest.net are all excluded from this throttling, so it will always appear that you are getting 'full speed' when testing your connection.
[0] https://variety.com/2016/digital/news/netflix-fast-internet-...
I think most ISPs throttle known streaming sites (netflix, youtube, etc.) by default for mobile plans. Some even advertise it [1]. And really what the ISPs have noticed is that if you throttle the throughput, the streaming services will switch to a lower quality stream (e.g. 720 or even 480 instead of 1080p) to allow for non-lagging streams.
So in some weird way the streaming sites are the ones that provided the tool and the ISPs figured out how to exploit it.
The two could have just worked together on a solution. Here they are pointing the finger at each other. ISPs don't want to have default 1080p streams for 4" device screens (which makes sense,) and streaming services don't want their network traffic throttled (which also makes sense.)
[1] https://www.t-mobile.com/offers/binge-on-streaming-video
We do...? Google Global Cache [0] and Netflix Open Connect [1] are both the result of ISP partnerships wherein we deploy cache nodes in the ISP's datacenter to reduce network load and improve our users' viewing experience.
Whether a specific ISP chooses to partner with us or not (and the breadth of their deployment) depends on their willingness to sign the relevant contracts.
[0] https://support.google.com/interconnect/answer/9058809?hl=en
No doubt it can be modernized, but just making Google search as good as it used to be would be helpful. And why not better than it used to be with the traditional interface? That would still be worth doing even if Google adds on a superb AI.
Search is the proverbial goose that laid the golden egg and Google is killing it, and I don't mean "killing" in the contemporary ironic usage.
(My corporate network used to have similarly exaggerated latency on a bad old wire.)
In the article you mention "Cloudflare" a few times, but I think you mean "Cloudfront". For example: "web-assets.strava.com resolves to Cloudflare, see the writeup on my site for details"
I'm assuming this was a typo based on the names sounding so similar and both being CDN's. Is that correct?
https://members.calyxinstitute.org/enroll/membership
I've done nearly gigabit symmetrical over 5G with it before and several Terabytes in a month. No issues.
I've added an "updates" section at the bottom of the post.
That looks like the problem to me. Throttle the fuck out of that shit.
Eventually switched to Mint and problem went away. Maybe a fluke, but I wouldn't be surprised if it was some shaping BS
Deactivate priority for Zoom, and now I have full speed.
What are you using for your DNS resolver?
I've used `1.1.1.1`, `8.8.8.8`, and others during this debacle, which I can't recall.
Here are some query responses that I received during the incident, but I can't tell you which resolvers were in use during the queries.
``` host web-assets.strava.com web-assets.strava.com is an alias for dgpcy4fyk1eox.cloudfront.net. dgpcy4fyk1eox.cloudfront.net has address 99.84.208.46 dgpcy4fyk1eox.cloudfront.net has address 99.84.208.56 dgpcy4fyk1eox.cloudfront.net has address 99.84.208.10 dgpcy4fyk1eox.cloudfront.net has address 99.84.208.115 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:8000:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:e600:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:c600:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:b600:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:d000:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:6400:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:f000:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:400:17:4613:2840:93a1 ```
``` host web-assets.strava.com web-assets.strava.com is an alias for dgpcy4fyk1eox.cloudfront.net. dgpcy4fyk1eox.cloudfront.net has address 99.84.208.56 dgpcy4fyk1eox.cloudfront.net has address 99.84.208.10 dgpcy4fyk1eox.cloudfront.net has address 99.84.208.115 dgpcy4fyk1eox.cloudfront.net has address 99.84.208.46 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:4c00:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:ae00:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:c000:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:7200:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:1000:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:b800:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:9400:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:9e00:17:4613:2840:93a1 ```
The only difference I've seen is order, which is expected.
[edit]
I actually spot some different IPv6 hosts above, after having taken a closer look.
It is for me. Is there something better I'm not aware of?
Yet another Trump triumph that I voted for Biden to fix, that he has shown absolutely no interest in fixing.
goto here: https://www.att.com/support
log in.
cancel.
tell them why.
port your number and sign up.
and you're done!
You can't legislate good behavior. You can only hit them in the checkbook. Wireless fortunately doesn't have the cable providers do, where local governments have entrenched players into local monopolies. Switching away is the most powerful thing you can do.
Well, you absolutely can. It's just that the US voter regularly votes in politicians that do absolutely nothing to mandate at least fair behaviour.