Hacker News Reader

Comment by lucb1e | Hacker News Reader

Which ISPs are so bad that you want to use external services, which are further in distance than your ISP, for speed? When I test with my ISP, they beat all of these services (both IPv4 and IPv6). They're simply closer to me in terms of hops.

My router is another story though. The Fritzbox (>200eur router) adds 6ms of latency, and that's what is advertised over DHCP. (Might still be fine, since cached queries are faster than the ping time to the ISP.) Note that my tests were all with uncached queries (random subdomains of a domain), so it always had to go out and ask an external server (though it could cache the NS record for the domain).

View on HN

My isp got the brilliant idea of rolling their own YouTube cache servers. It's great in theory but in or active they're under powered and so at peak hours I can't even stream 240p on my 500mbits connection. I've had to block their cache servers in my firewall for YouTube to be butter smooth at 1080p consistently.

Another example is bell Canada who used to mine your DNS queries to profile you for ads, or ISPs that high jack the nxdomain result to send you sponsored results of vaguely similar sounding websites.

It is common for ISP to host instances of the Google Global Cache (GGC, see https://peering.google.com/) which are used for many Google services, most importantly YouTube.

In fact, in many cases Google itself "suggests" to ISP that they host a few GGC servers.

They are directly monitored by Google, and the ISP has basically no say in how they are run. Capacity is managed by Google directly.

Yep, Virgin Media did this in the UK and messed it up badly. Every day at 6pm YouTube would stop working until the following morning.

It appeared to work by inspecting DNS packets and replying with overrides if necessary. I didn’t like it but I could understand that.

What I did not agree with was the fact that this also happened for other DNS services. Google DNS and OpenDNS both experienced the same issue, as did a few other “famous” DNS providers. Random little ones wouldn’t return the caching servers, and also enabling encrypted DNS for Google/OpenDNS would stop it happening too. I’m fairly sure it was some badly thought out deep packet inspection.

Again, it's probably _not_ Virgin's fault or responsibility. Capacity planning is handled by Google directly.

Also, I think that virtually every ISP with more than a few tens of thousands users is hosting a GGC instance nowadays (and a Netflix OpenCache, etc. etc.).

Nowadays, the vast majority of the transit&peering of ISPs is not going to the Internet, but to a few racks of local caching servers managed by OTT operators.

Bandwidth-wise, at least in prime time, the Internet is much less connected/realtime than people think :)

I would suggest that the deep packet inspection is Virgin's fault. Perhaps it was Google under provisioning the CDN inside VM's network, but I would suggest that's also probably Virgin's fault in part, as they will likely have more network information available to them that may have suggested network congestion, but that was not acted upon.

As someone else suggested, this is likely part of our GGC program. If you can give me info on which ISP + Geographic region you're in, I can take a look and see if there's anything in our logs to indicate a problems; you can email me details at myusernamehere at google dot com with details, since I don't routinely check Hacker News.

If you can reproduce a bad experience, and right click on the player, click "Get Debug Info", and share that result, it's the most helpful thing for us to dig into problems.

FYI, something that makes my Hacker News experience much nicer is email notifications when my comments are replied to.

Get your own at http://hnreplies.com/ - built by Dan Grossman

>My isp got the brilliant idea of rolling their own YouTube cache servers

Are YouTube videos not served using https? Sounds like it's google/YouTube servers but they're underprovsioned

The YouTube cache is probably there not to increase performance for you, but to reduce ISP cost (ISP pays for all data received from outside of their network)

Even for data not delivered via their cache they'll get free peering with Google. But having cache servers at several locations reduces the capacity they need to peering points. E.g. in the UK, most traffic gets peered in London so having cache servers in Northern England reduces the bandwidth they need for their internal network within the country.

I don't think ISPs pay much for peering in Europe.

That's a strange way of looking at it. ISPs take all sorts of actions to both increase performance to customers, and to reduce their costs.

A competent ISP operating in good faith will execute and expand peering agreements when it makes economic sense to do so.

An ISPs #1 goal is to generate money.

When there's a competition this often equates with making customers happy so they will stick with the ISP and perhaps more will switch from competitors.

When there's no competition and customers have no choice who to use it's purely all about squeezing as much money as they can. This means raising prices for the service and reducing cost by paying less to others services even if it would reduce quality of service. The goal is to have service that's good enough to keep people still use them (I mean there is price point and/or quality that people would just not to have service at all).

If a specific action that ISP with monopoly does and it improves quality for customer that's just a coincidence.

> A competent ISP operating in good faith

That's probably the funniest thing I've read all year

My own tiny neighborhood ISP thinks they are smart and have their own CDN for caching everything under the sun. Of course it doesn't work well, not only it caches DNS and HTTP requests like kids collecting candy in Halloween but it also breaks HTTPS sometimes.

As an anecdote of how bad ISP caching can hurt you, I am a web developer and I was debugging an issue in a legacy system. I had fixed the stuff and deployed a new version. Erased all the caches in my machine, refreshed the page, broken. I thought that maybe I fixed the wrong thing and started fixing more... now rinse and repeat over five hours before you realize that the cache is happening at ISP level and that things are fine in the live version on the real internet. This ISP is called PredialNet, web developers from the city call it PredialCache...

I know this is about DNS and not HTTP but their cache servers will also cache DNS requests, so moving some domain to a new machine and IP is also a quest under their network. For a while I paid PIA so that I could side-step their cache servers.

Sometimes it's not even at the ISP level. I experienced a similar adventure a few months ago. I turns out that my cellular hotspot caches PDFs. It's not even at the ISP level, it's in the little box in my pocket.

Naturally, it can't be disabled.

Holy shit!!!!!!! That is evil! I just imagine the tortuous debugging process that led you to discover this. Wow.

Which hotspot is that? It'd be fun to poke at.

All ISPs in Denmark are required to implement a filtering list of somewhat arbitrarily chosen websites, including a number of torrent sites, illegal pornography and probably others. There is a very reasonably fear that this could be used for political purposes.

This filter list is implemented through DNS, making third-party DNS services the most practical workaround.

The same thing happens here in Portugal since 2015. The fact that the block orders don't even go through a court first makes it even worst.

One of the issues is that many ISPs don't actually run their own recursive DNS servers... They outsource to companies which provide a "monetized solution," operating the recursive resolvers and paying the ISP in exchange for the customers' data, which they sell. This will become an illicit activity shortly in Europe, as the GDPR comes into effect, but there are many ISPs that have become very used to receiving those payments. And most of them already don't admit to selling their customers' privacy.

So privacy is why many of the people who use a different recursive nameserver are doing so... In order for that to work, you need to use one of the ones which support link-level encryption, like Quad9 (DNS-over-TLS) or OpenDNS (DNScrypt). Other reasons are performance (which at least in Quad9's case should be no worse than your ISP for real-world queries, since it's back-to-back with many of the authoritative servers already, and all of them will give you a large pool of other users sharing the same cache, which helps performance immensely), and security... Going back-to-back with the authoritative servers also collapses the attack surface between them, minimizing the possibility for MITM attacks between the recursive and authoritative servers.

The ones that cooperate with state-sanctioned censorship? Malaysia blocks anti-government news sites via ISP DNS entries; using a 3rd party DNS bypasses the block easily.

Accusing government officials of mismanagement or outright theft of public funds is an easy way to get your domain black holed over here. That, piracy, and porn. Though I wouldn't be surprised if the latter two were thrown in solely to make the program look more legit.

https://cilisos.my/8-sites-the-malaysian-gomen-blocked-in-20...

Perhaps most ridiculously, Medium.com is blocked for simply hosting an article posted by a censored website. Yes, you read that right; they banned an entire domain, the vast majority of which covers nothing even remotely related to Malaysia, over a single article.

I try to be grateful they are as inept at censorship as they are oversensitive to criticism.

It’s best to just pretend those DNS things are a good way for bureaucrats to achieve whatever it is because they’re so easy to circumvent and the alternative is far worse.

We have only ourselves to blame for allowing the socialists and fundamentalists a foot in.

NXDOMAIN hijacking is enough for me to switch, and most ISPs are doing it nowadays.

Brighthouse did it before they became Spectrum. They did allow me to turn it off though. I've heard that this is still the case with Spectrum, but I haven't tested.

Spectrum is still doing it (N Pinellas Co FL). Not sure if it can be turned off.

That’s quite a bold claim. Got any data to back it up?

Source: I've yet to see this on any ISP I've used anywhere, sans free airport wifis. Travelled pretty much every continent on earth.

Cox does. And I think Kabletown does, too.

> I've yet to see this on any ISP I've used anywhere, sans free airport wifis. Travelled pretty much every continent on earth.

I guess you've never been to Turkey.

CenturyLink, major telecom in 37 US states. DNS requests for all nonexistent domains go to a server that delivers a dumb "search" page to web requests. They're currently the sole fiber-to-the-home provider in my neighborhood, and Comcast is the only broadband alternative.

So one or two ISPs in one country out of 300+ worldwide.

Hardly “most” ISPs as claimed in the post I replied to. Not even a fraction.

Here's an incomplete list with dozens.

https://en.m.wikipedia.org/wiki/DNS_hijacking#Manipulation_b...

T-Mobile does it and that is an enormous ISP.

AT&T does it on their home Internet service. Annoys me to no end.

EDIT: lots of ISPs do it, in fact: https://en.wikipedia.org/wiki/DNS_hijacking#Manipulation_by_...

Time Warner (Now Spectrum) in Texas does this on residential as well as "business class"

So this is a US problem and therefore “most” ISPs worldwide does this?

That’s hardly evidence if any.

> numerous examples of huge ISPs with more users than some European countries have residents

> "That's hardly evidence if any"

I'm sorry what?

Comcast has been on-and-off about doing it. They've switched between proper NXDOMAIN and hijacking it a few times.

I don't have Comcast anymore (not even available in my area), I can't tell you if it's current practice.

Time Warner/Spectrum, the largest provider in NYC (and the only one that services my apt building) does this. They let you turn it off in your account settings, but it doesn't actually turn it off.

I live near Toronto Canada and Rogers my ISP hijack's DNS requests for non existent domains. The data to back up my bold statement is a simple Google search which reveals that they have been doing this at the very least from my Google search was from 2008.

It is possible to turn off this feature, But it is set by a cookie and once that cookie is deleted you have to do it all over again.

Verizon Fios

That's quite the aggressive reaction. ;)

Windstream in Ohio does it.

I'm pretty sure I had a previous ISP that did it too, but I can't remember which one now.

TalkTalk in the UK. last chance I got to test it was about a year ago.

I know you're asking in jest, but here's an honest answer: Google Fiber. Google's DNS servers are ~15ms further away than a local ISP who runs public DNS servers as well (xmission). Part of this is because google fiber has a peering connection at SLIX (slix.net), and so does xmission, whereas Google's DNS servers appear to be in the bay area.

Whether or not it's noticeably faster, I'm not sure, but I've always used xmission's dns servers without fault for the past 10 years.

--- 8.8.8.8 ping statistics ---

4 packets transmitted, 4 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 17.763/18.264/18.803/0.425 ms

--- 198.60.22.2 ping statistics ---

4 packets transmitted, 4 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 2.906/3.457/3.871/0.408 ms

I've had several ISPs in different countries that do horrible things. Not using NXDomain is one, but I've had some that return NXdomain when they shouldn't, then cache that result!

8.8.8.8 is consistent and easy to remember, and now so is 1.1.1.1

Queries to my local Spectrum DNS service are 4 times slower than 1.1.1.1, and they redirect you to stupid search pages instead of `NXDOMAIN`. I switched the whole network over this morning.

Assuming Spectrum is your ISP, wow. Just wow, that's impressive. On Spectrum's part I mean.

In the U.S., many of them are mining and selling our data. Getting off of their DNS service is one step in mitigating this.

Also, as others mention, they can and do monkey with the results.

In other words, here, your ISP is in part a hostile entity. At least, in my perception -- and I'm not alone.

P.S. Of course, there's the argument against giving Google all your DNS usage, as well... I use a different DNS service from a company with a good reputation that says it's not collecting usage data. Even then, and with the state of our State, who knows...

They can (do) mine it even if you use Google or Cloudflare or whatever since DNS is trivial to intercept.

Mine's over a VPN. And if you use a VPN, make sure your DNS traffic is using it (along with some other types of traffic that can bypass it, if you're not careful).

> Which ISPs are so bad that you want to use external services, which are further in distance than your ISP, for speed?

When I upgraded my WiFi equipment at home a month ago I thought it was defective. Multiple times every day I'd seem to lose connectivity to the internet. After two weeks it occurred to me that my previous router was configured to do a couple things for me, one of which was overriding what DNS servers were assigned with DHCP. I reconfigured my new equipment to use those DNS servers and suddenly everything works normally.

To answer the question asked: Spectrum.

To answer the question asked: Spectrum.

I have Spectrum and have been fighting this exact problem for months. Trying out your steps now and hoping for success.

So did it work?

Not really :( I'm looking to replace my router entirely soon anyhow, and hoping this does the trick. My hardwired devices don't seem to have this problem.

Does no one bother to run their own DNS server? It's not hard---in fact, it's easier than running an authoritative DNS server, since you are only using it for local resolving.

I ran the provided script, added my own local resolver, and after the first run, got incredibly fast results.

                   test1   test2   test3   test4   test5   test6   test7   test8   test9   test10  Average 
    cloudflare     35 ms   38 ms   39 ms   33 ms   34 ms   39 ms   28 ms   33 ms   28 ms   39 ms     34.60
    cloudflare2nd  34 ms   27 ms   34 ms   34 ms   34 ms   27 ms   34 ms   28 ms   28 ms   28 ms     30.80
    google         42 ms   27 ms   27 ms   40 ms   28 ms   40 ms   28 ms   40 ms   40 ms   28 ms     34.00
    google2nd      42 ms   27 ms   28 ms   43 ms   41 ms   39 ms   28 ms   41 ms   41 ms   28 ms     35.80
    quad9          65 ms   64 ms   54 ms   65 ms   63 ms   53 ms   59 ms   52 ms   60 ms   53 ms     58.80
    opendns        45 ms   32 ms   56 ms   51 ms   29 ms   74 ms   32 ms   48 ms   45 ms   32 ms     44.40
    norton         92 ms   73 ms   56 ms   152 ms  48 ms   144 ms  68 ms   40 ms   77 ms   83 ms     83.30
    cleanbrowsing  33 ms   28 ms   33 ms   28 ms   33 ms   35 ms   28 ms   32 ms   33 ms   29 ms     31.20
    yandex         174 ms  174 ms  185 ms  178 ms  184 ms  179 ms  179 ms  199 ms  176 ms  186 ms    181.40
    adguard        143 ms  147 ms  146 ms  142 ms  148 ms  143 ms  130 ms  130 ms  138 ms  137 ms    140.40
    neustar        62 ms   63 ms   70 ms   80 ms   40 ms   74 ms   73 ms   151 ms  58 ms   65 ms     73.60
    comodo         60 ms   137 ms  59 ms   60 ms   59 ms   59 ms   64 ms   62 ms   60 ms   59 ms     67.90
    localhost      0 ms    0 ms    0 ms    0 ms    0 ms    0 ms    0 ms    0 ms    0 ms    0 ms      0

Most people are not very interested in testing how fast their local machine can talk to itself; regardless of how fast your local cache is, it will have to talk outside pretty often (for expiring TTL if nothing else).

People are praising CloudFlare for not bothering to log requests. Yes, well ... you get the same results by using your own DNS resolver, and you don't have to be shocked when five years later a different governing team running CloudFlare decide to reverse their stance and log everything ...

But what do I know? Apparently I'm in the minority for running my own infrastructure (DNS, web, mail).

Where do you think your DNS resolver gets the answers it gives you? You're just introducing a caching layer between you and whatever external resolver you're using. The first time a machine requests a domain (and every time a TTL expires; look at CDN TTLs sometime) you are effectively talking to the internet just like everyone else.

Do you ever wonder if such activity (basically trying to be invisible) will put you under far more scrutiny than if you were to just 'be normal'? Thinking from the other side of it you would be a high value target to monitor as closely as possible if I were law enforcement. I know it sounds ridiculous but you may actually have more privacy (unless you're actually breaking the law) by hiding in plain sight with the rest of us.

If you talk to an upstream resolver through an encrypted link (both CloudFlare and Google support DNSoverTLS and DNSoverHTTPS, at least) then you at least limit the potential privacy threats.

If you live close to a peering point, both CF and Google will have mean ping times <10ms, often even <5ms. I doubt there's any real world difference between using localhost and either of them. As they cache results, using them could actually be faster than your own server.

Comcast... before changing DNS on my network, I tested a variety of DNS providers including Comcast. While Comcast had the best individual test minimum and average times, Comcast repeated testing showed lost packets and the occasional worst max times.

I don't know why they faired as poorly as they did, but once I realized some network issues could be pinned against DNS packet loss / terrible performance spikes, I changed over to 9.9.9.9 and 8.8.8.8 as the fall back. Home network has been a bit more stable since the change over.

Indonesian ISPs are required to filter certain domains, but go further than the Denmark case - they also block access to third-party DNS servers. Using DNScrypt or a VPN (or hosting your own DNS on a port they're not blocking like 80) is the only workaround.

Prior to third-party DNS servers being blocked a lot of tech-savvy people were using either Google's DNS or OpenDNS anyway since they tend to be more reliable than the local ISP's (even before they started filtering).

I'm no longer in Indonesia, but I could imagine the results people are posting here would be useful to folks back home so they can adjust their DNScrypt config.

My ISP is Comcast, a company I can trust about as much as I can throw it, so I use DNSCrypt when I'm not otherwise on a VPN, and have a number of fallback DNS providers for when DNSCrypt doesn't work.

Reasonably speaking I don't need very much from a DNS provider: <15ms latency, basic DNS security features and the ability to display an error rather than redirect me to a sponsored results page. I can do without the extra filters, but as long as I can get those three things, then I would rather use a different service than my ISP.

Do you use Netflix? If so, how do you get access if you use a third party DNS service?

Netflix doesn't care who your DNS resolver is. They might care if you access it through a VPN, which I don't.

It does. It’s how most Aussies were bypassing their geoblock about a year ago.

VirginMedia UK nameservers are notoriously poor, they just stop responding or disappear for hours. At least with Google I’m more or less guaranteed a response, as slow as it might be. I’m in an area where VM is the only cable operator, so I couldn’t switch without either losing on speed or spending a ton of money for someone else to cable me up.

I do try to use VM dns, but I periodically have to reverse to Google and often forget to switch back post-outage.

I run a local caching server on my network 192.168.1.22, from there I now forward to cloudflare, then to opendns on misses. This makes DNS resolution blazing fast for things we frequent.

I use my ASUS router's DHCP to teach all my dynamically configured network devices how to find the bind9 DNS caching server.

All the statically configured ones just have 192.168.1.22 hard coded with 1.1.1.1 as backup.

"Which ISPs are so bad that you want to use external services"

Any ISP that tries to redirect me to an ad-filled half-baked web search whenever I run into an inactive domain would definitely fall into my "so bad that I want to use external services" bucket.

Verizon's DNS seems to have improved, but a few years ago I ran into consistent problems resolving even the biggest / most common domains. My brief never-to-be-repeated dalliance with Comcast in 2016 was even worse.

BT in the UK. Their DNS is so unreliable that using it isn't really an option, so it's not even in the running.

Reply on news.ycombinator.com