Hacker News Reader

Comment by elliekelly | Hacker News Reader

I only know enough about networking to be dangerous but I am convinced Comcast is doing shady shit with my modem when I change the DNS settings to use non-Comcast servers. Every once in a while I’ll attempt to use Wireshark to try to make sense of what’s happening but I’m pretty clueless and don’t really know what I’m looking at/for.

If anyone knows any good resources to learn about the ISP nuts and bolts that make internet magic happen between my modem and everyone else’s servers I would be most appreciative.

View on HN

The recommendation of _The UNIX and Linux System Administration Handbook_ is a good one.

As far as Comcast, I'm stuck with them, too. At least in my experience, they don't monkey with DNS - I run and use my own DNS servers, and have never seen interference.

They do run deep packet inspection, and if they detect you, for instance, torrenting commercial media, they'll inject scary messages in port 80 traffic. Given that nearly all web traffic is encrypted now, the main effect of this is to break things like automated `apt-get update`s.

One thing you can do to detect transparent DNS hijacking is to ask a nonexistent server a question. Something like `dig @13.14.15.16 news.ycombinator.com` should not give you an answer. If it does, someone's spying on and/or gaslighting you.

Curious: how can they detect whether you're torrenting commercial media if you've enabled Bittorrent protocol encryption? Surely all they can see then is the outer (envelope) of the packets...?

This is a bit of a misconception. Copyright holders have always gone after seeders based on people connecting to swarms, tracker info, and crawling DHT. There’s no reason to use DPI when the list of uploaders is just given out by trackers and DHT for free. See: https://www.usenix.org/legacy/event/woot10/tech/full_papers/...

Exactly

Bittorrent protocol encryption is only useful to protest against the use of DPI for bandwidth shaping, it has no influence on privacy.

Even with (the weak) encryption, connections to trackers and DHT nodes are easily identified

Thank you for clarifying this!

You're right that is how it generally operates, but in the case of Comcast I think this meme doesn't want to die because in the late 00s Comcast really did do DPI to interfere with torrents: https://www.techdirt.com/articles/20071029/020756.shtml

Fairly googleable with "Comcast sandvine". Afaik they haven't done anything like that for years, though.

Bittorrent trackers by design have a list of all IPs in the swarm and give to anyone who asks (that's how peers coordinate).

>I am convinced Comcast is doing shady shit with my modem when I change the DNS settings to use non-Comcast servers

Well that's vague. What are the symptoms? How would comcast even know that you changed DNS settings? It's possible to infer that from DNS queries to their servers dropping off and traffic to 1.1.1.1 or 8.8.4.4 increasing, but I doubt comcast is competent enough to build that sort of detection system.

On my home network I just run a transparent proxy and direct all outbound traffic bound to port 53 to my local dns server, it’s not hard.

Interestingly enough, this is almost exactly how ISPs do it when they really want to get your attention. A couple years ago I forgot to update an expired credit card that I used to pay my spectrum cable bill. One morning every DNS request resolved to their "your account is about to be closed due to nonpayment" page. As I also use my own DNS sever I was surprised by this, and sure enough everything going out of my network on 53 was being grabbed up by their CGNAT and sent to their DNS server.

I just block all outbound port 53 traffic, any device or app that doesn't honor my DHCP-provided DNS resolver can suck it.

Looking at you, Chromecast that tries 8.8.8.8 40 times an hour even though you know perfectly damn well that 10.10.10.1 is working

The DNS provided by many ISPs is not to be trusted, as per this thread, so how else can your Chromecast act to find a trustworthy DNS?

And with newer decides that use DoH, you can no longer prevent devices from contacting their own DNS provider without totally firewalling them (or perhaps using some IP blacklist or whitelist, if available?)

https://en.wikipedia.org/wiki/DNS_over_HTTPS

When I said blocking all outbound 53 I meant no exceptions, my local forwarder already uses DoH to an outside resolver.

Everything that I don't have complete visibility into the network stack of goes on a VLAN that does not forward traffic to the internet, it advertises a proxy via WPAD and DHCP option 252. I have a whitelist of hostnames that each device is allowed to make CONNECT requests to, so far there is only one.

If it's not a plain unencrypted HTTP request to my proxy, or a CONNECT request involving a server/device pair I've decided to trust, it's not going anywhere.

This breaks a lot of things that I would just as soon rather do without. I can't change my universal remote hub settings from the vendor portal, boo-hoo. I can't view my cameras from the hardened VLAN or from the internet (unless I VPN in first since the only copy of the recordings is on my local NAS)... good.

I should think they just detect traffic from your IP address on port 53 to any IP address that's not one of their nameservers.

I agree with you that comcast is incompetent, but everything becomes cheaper and easier over time and network hardware/software products that perform "deep" packet inspection at line rate as well as provide analytics on that returned data are now trivial and pretty much table stakes for Cisco, Juniper, Palo Alto et al.

Specifically for detecting if a user is not using their DNS, yes you could correlate a user's http requests (unless you are using ESNI the requested domain is in plaintext by design) with traffic logs on their DNS server and observe that there was no DNS request to the ISP DNS server before a request was made, I don't think that would be necessary. Most users use the ISP default DNS - that's your baseline. If most customers hit your DNS X times per Mb of web traffic, then someone using a custom DNS is going to stand out like a sore thumb.

Again, 100% agree that ISPs are not very technically competent (to put it mildly), but as time marches on the ability to both capture and more importantly analyze and report on that data is becoming cheaper and easier. ISPs want to get value from (sell) your data and vendors want to sell ISPs subscriptions to analytics and other platforms that bring them reoccurring revenue. Data from customer DNS is one of the most valuable sources of information an ISP has and I would be surprised if there was not at least an attempt to know how many customers did not use it.

Sorry, I didn't get into details because I wasn't intending to ask HN to troubleshoot for me. But, since you asked...

I have an "XFi Gateway" combination modem/router provided by Comcast (perhaps my first mistake) so the DNS settings are restricted and cannot be changed. I have the Comcast modem/router set to bridge mode and connected my own router where I can control the DNS settings.

My understanding is the DNS settings closer to the client control. So in addition to having set my router to Cloudflare's DNS I also set my devices as well. One day, maybe a year ago or so, I'm on HN and I click an archive.is link, read the article, and go to the discussion thread only to see several comments about how archive.is is blocked by Cloudflare DNS. I checked the DNS settings on my MacBook and router and I was indeed using Cloudflare DNS but for some reason I was able to access the "blocked" address.

So I went to the terminal, cleared the cache, and checked nslookup archive.is and it responded correctly. Then I checked a nonsense DNS server: nslookup archive.is 5.9.3.7 or something and it still responded correctly. I tried the same with different websites and got the same result. So I searched "see my DNS server" or something and found a few websites but they all showed Cloudflare. Very odd.

When I logged in with my VPN, Mullvad, and changed the DNS settings on the router and my laptop to Mullvad's and repeated the experiment it finally returned NXDOMAIN. Then I disconnected from the VPN but left Mullvad's DNS settings, repeated the experiment again with the same results - even when I was using a totally bogus DNS server it was returning the correct IP address.

That's when I installed Wireshark and, lo and behold, I could see the requests that should have been going to 1.1.1.1 or 5.9.3.7 going to 75.75.75.75. Comcast.

A call to Comcast was, as expected, a complete waste of time. First they told me it was using their DNS settings because of "their firewall" and then they told me that if I used their built-in router rather than mine + bridge mode I wouldn't have the issue at all.

Messing around in Wireshark I eventually determined the issue had something to do with one specific port that was making the requests (I can't recall how but I think because I could see Mullvad VPN was using a different port for DNS?) so I fiddled around and forced (or maybe redirected?) my router to use that port too and that finally worked in avoiding the Comcast servers. But, knowing just enough to be dangerous and not entirely sure what I was doing, I didn't keep the forced port and decided I'd have to get my own modem and use my VPN in the meantime.

Before I had gotten around to buying a new modem (this was somewhat early in the pandemic) I saw a post on HN about NextDNS and decided I'd see if I ran into the same issue. I didn't, as far as I could tell at least. When I run Wireshark now (I still use NextDNS) I don't see any contact with 75.75.75.75 or 75.75.76.76. I think this is because NextDNS uses DoH? But who knows.

Like I said, I only know enough to be dangerous so perhaps I just had something configured in an odd way that made the Comcast servers step in as a fail-safe and there's a totally innocent explanation. But based on my experience as a Comcast customer I don't really think they're deserving of the benefit of the doubt so I've definitely got a bit of a tin foil hat when it comes to them secretly messing around with my traffic through the leased modem.

I found The UNIX and Linux System Administration Handbook (5th Edition), chapters about networking and DNS very instructive, and they list a ton of additional references if you want to dig deeper.

Comcast isn't doing anything to your DNS. They're the largest ISP in the country, there'd be a huge uproar if they were doing something like that. There are plenty of experts who are subscribers who'd be able to figure out exactly what's going on.

There's already a huge uproar around Comcast. But Comcast isn't losing any customers, because they have monopolies.

Yeah, uproar around prices and speeds and stuff like that. Nothing like screwing with third-party DNS requests.

Actually, I have heard that claim more than once, about various providers (up to, and including, “when I changed my DNS settings and the traffic slowed down, the tech got me to change them back, and the traffic sped up”).

It's less common, I think, because more people know how to check their speed than change their DNS.

The only way I can think of that working is if the provider is intercepting DNS requests for popular speed tests to redirect to an internally-hosted version that would be faster. Otherwise, I can't think of any realistic way DNS settings can affect actual throughput.

If the ISP is checking for DNS lookup of speed test websites, then allocating higher bandwidth to the connection for a brief period of time?

Or, somehow more cynically, the ISP makes money from selling the data collected from DNS, so punishes people who use a different DNS provider. (DNS is plaintext-by-default, so I don't quite see how this would work, but it's possible.)

Or perhaps the system uses DNS lookups as a proxy for “is a human browsing the web”; if there aren't enough, it's clearly some kind of automated computer program that doesn't deserve internet access.

Your first example would be dead simple to detect and take advantage of to get those boosted speeds all the time. Your other two examples are a bit wild.

The first example is a real-life example. The other two are speculative, because I've heard a case where it wasn't the first example.

So, suppose I'm Huge Video Streaming Corp X, and I get a DNS request asking me for the address of my servers. Well I have over a thousand servers around the globe, which one do you need? Any of them would work, but you likely want the fast nearby one, right? So I can try to guess based on the IP address the query came from...

I know the best answer for a Comcast DNS server in New York is the server I physically installed in a New York Comcast rack, but when a public DNS server asks me from Paris, maybe I suggest a London server, 'cos that's pretty close to Paris, shame that New York isn't.

EDNS Client Subnet is a feature that lets a DNS server say OK, I'm asking on behalf of somebody from 10.20.30/24 and so my system can do the same trick with ECS. But doing this unwinds most of the privacy benefit of using a public service, so several famous public DNS servers explicitly do not use ECS.

Obviously the cheap bulk host used for some Single Serving site like "Is pizza rat mayor of New York yet?" isn't affected, that is only one server and it is wherever it is, but somebody like Netflix absolutely is affected by this because they have their machines close to the customers to deliver better performance and if they don't know where the customer is that inteferes.

QUIC has an optional feature called Connection Migration to help improve this, the remote server is like "Um, now that you're connected to www.example.com here in Glasgow, Scotland, I notice your IP address is from Tokyo, Japan, and this is just a suggestion, but maybe talk to my identical twin also named www.example.com in Tokyo, Japan for better performance? Here is the IP address to try"

That's not what I meant by "actual throughput." The fact that a download is slower from a server halfway around the world versus one in the same datacenter where my ISP has a peering agreement near me isn't because my connection slows down when I'm hitting the far away server.

This explanation makes a lot of sense. It also has a slight feeling of Hanlon's Razor, although there isn't necessarily incompetence involved (unless you count the technology's inability to find the absolute fastest/closest server [for whatever reason] as incompetence).

"Come on, what are you worried about? I'm sure it's fine, somebody must have inspected it."

Well, I can tell you that I'm a Comcast customer who doesn't use their DNS, and I have no issues. If I did have issues, I also have the expertise to figure out what's going on.

While absence of evidence isn't evidence of absence, some guy's anecdote isn't really evidence of existence.

I find Comcast’s fuckery to be limited to their business practices. Their actual IP network seems to be very solid.

100% agreed. And I've been a Comcast customer long enough to have seen the days when that certainly wasn't the case. They've made some pretty big mistakes in the past, but they seem to have learned their lesson.

Weren't they the ones who pioneered DNS hijacking of unknown domains to serve their own recommendations and ads?

No, that was VeriSign back in 2003: https://www.icann.org/en/announcements/details/advisory-conc...

Ah, OK. I didn't realize verisign did that too. Comcast followed not long after...

https://arstechnica.com/tech-policy/2009/08/comcasts-dns-red...

Comcast used to do a lot of messed up stuff. As I mentioned in a comment somewhere close to here, I've been a customer long enough to have seen those bad days and how they've managed to change since those days.

Have they really though? They still do DNS hijacking and bandwidth caps and speed boost for a few seconds only and automatic price increases and worthless bundles and such, don't they? They were so bad they had to rename to Xfilthy or something but their practices didn't really change as far as I can tell. What are some examples of things they've improved?

That's aside from all the political shenanigans they pull, trying to kill municipal broadband and net neutrality. There's scarcely a more evil ISP.

I switched away from Comcast as quickly as I could, and never had trouble with either smaller cable companies or fiber providers.

That couldn't be more wrong. They literally published an IETF draft standard on how they do it.

https://datatracker.ietf.org/doc/html/draft-livingood-dns-re...

That draft is referring to the operation of their own DNS servers, not messing with third-party DNS.

"... except in reasonable and justifiable cases where a user has been placed into a so-called "walled garden" for reasons of abuse, security compromise, account non-payment, new service activation, etc."

Their own words

What's your issue with that? In that scenario, the user doesn't even have Internet access. If they didn't force the DNS to specific servers, the user would only see that their service isn't working with no indication as to what's going on. It's clearly not something they do with normal, functional users and I never said that they didn't have the capability to do it.

That was a pretty rapid shift from "Comcast isn't doing anything to your DNS" to "So what if they are? There are times when they should!"

Yeah, wow I guess I should have included the caveat "Comcast isn't doing anything to your DNS... except when you literally don't have Internet access and couldn't reach a third-party DNS server anyway"

If you wanted to be honest you could have said "I have literally no idea what Comcast is doing with DNS, but I will attempt rationalizing everything they do as I am gradually informed of it"

I'm getting this second hand, but I've heard that some isps will redirect dns traffic to their servers based on it going to port 53.

Reply on news.ycombinator.com