Level 3 are now hijacking failed DNS requests for ad revenue on 4.2.2.x | Hacker News Reader

Level 3 are now hijacking failed DNS requests for ad revenue on 4.2.2.x

james.bertelson.me

150 pointsdoctorshady74 comments

Seems reasonable. Those servers were never intended for widespread public use, so they may as well make back some funds for upkeep, and maybe encourage some more-technical users to switch away.

Here's a blog post with some background on these servers: http://www.tummy.com/articles/famous-dns-server/

I'm fairly sure that the "4.2.2.x was never meant to be public" line is a myth. Though the NANOG thread cited in that post is good historical background, it is contradicted by more modern sources:

"[...] DNS infrastructure is largely split into two types; open (public) and closed (private). Open DNS is provided by companies like OpenDNS, Google and Level 3. You can use it wherever you are on the Internet with no restrictions or authentication required."

- Mark Taylor, VP at Level3

http://blog.level3.com/level-3-network/a-flawed-study-of-cdn...

I can't find any cite where anyone else who I would consider a reliable source in the DNS world (Vixie, &c) repeat this claim. To the contrary, Level 3 is often grouped with Google, OpenDNS and others in discussions of open public resolvers [1][2][3], and those in the know never seem to speak up and say otherwise in these discussions.

That being said, I have absolutely no personal knowledge on any of this.

[1] http://www.maawg.org/system/files/Fergie_DNS_Open_Resolver_M...

[2] http://markmail.org/message/gh7f2wvfbn5mpvuq

[3] http://www.circleid.com/posts/87143_dns_not_a_guessing_game/...

Presumably it takes non-0 costs to maintain 4.2.2.x DNS servers. While I'd want to believe Google and L3 just try to help out the public at large with free DNS services, I suspect they are not doing that just purely out of altruism.

Google's motivations are long-range, but simple - more ad dollars. Faster DNS means more people using the web (rather than give up in disgust - and if don't believe that happens, let me introduce you to comcasts's DNS servers...); more people using the web means more page view which means more ad revenue.

So no, definitely not altruistic.

I just thought it was to track user actions via DNS - they can see which sites you visit without needing tracking bugs on those sites. Better profiling means better ad serving for Google ... profit.

They specifically promise not to do anything like that.

https://developers.google.com/speed/public-dns/privacy?hl=en

"We built Google Public DNS to make the web faster and to retain as little information about usage as we could, while still being able to detect and fix problems. Google Public DNS does not permanently store personally identifiable information."

I'm surprised Google hasn't done it yet

"Those servers were never intended for widespread public use, ..."

What is the test for "intended for widespread public use"?

If a server is configured to accept queries from any IP address, can we conclude anything about "intent"?

If the server admin does nothing to stop widespread public use (which is trivial to do of course), can we conclude anything about intent?

Is there some other clue we need to look for?

Under your reasoning, it's "reasonable" to provide open resolvers and hijack NXDOMAIN responses, so long as there is no "intent" for widespread public use.

Interesting.

I heard years ago that Level 3 were trying to encourage people (non-customers?) not to use these DNS servers. I guess this is one way to ask people not to use them.

Having said that, 8.8.8.8, Google DNS, has been planted firmly in my memory as my go to "is this machine up?" IP.

My issue is that level3's dns has always been very fast, and more importantly, up... When google's dns is slow, level3 is fast... when my isp's dns goes down or wonky, level3's is up... I'd pay them $10/year to use them without the ads.

Why not use OpenDNS then? If you have an account you can configure them to behave however you'd like.

And then they have your entire history of internet activity, matched to you name, address and CC number. Fine if all you want is reliablility. For those who want to imped surveillance it is as bad as the ISP.

So does Google, for what it's worth.

No, they very clearly and explicitly promise otherwise https://developers.google.com/speed/public-dns/faq#privacy

Their DNS-only service is free.

@karlshea was responding to @tracker1 who said that he/she would pay for the service. If someone is already willing to pay for the service, then I doubt this their major concern. You responded as if @karlshea was ignoring some requirement of the parent post...

Heh, ironically enough, OpenDNS hijacks NXDOMAINs.

http://www.opendns.com/home-internet-security/premium-dns/

To be perfectly clear: It is not hijacking when you are sending them queries for which you should have no reasonable expectation that they service. If you are actually a Level 3 customer, call your sales rep, but I believe this is only for non-customers.

EDIT: By the way, this is the actual company operating the "service" behind the scenes for Level 3 http://www.xerocole.com/searchguide/

If you're not going to service a query, there are perfectly good ways to do that. Since they didn't do one of those, I'm happy to call this hijacking.

and one of them is returning bogus junk. why should they (the NSP) care? especially if you aren't paying them for a network service? it's not hijacking -- 4.2.2.2 isn't a public resolver and never was. it just happened to become one.

It's still hijacking, but they're under no obligation to provide service.

I'd say they should care because the Internet only survives through cooperative effort. Breaking something so they can pocket money is greedy and dickish.

But if they are being jerks rather than just being thoughtless, then maybe that isn't enough. In which case, my fallback answer is "bad PR". It would have been easier for them just to deny service to anybody they didn't want to serve. They went to a lot of trouble break something in a profitable way. To me, that says they might not be a trustworthy vendor, and thousands of nerds are now aware of that.

Yes, it can screw up caches and return unexpected content.

Inserting an ad where an error should be is detrimental because the implementation often break standards endpoint and intermediate apps depend on (http code, DNS query answer/s). An app that was expecting just JSON now dies in some horrible (maybe silent or end-user confusing stacktrace) way.

> Yes, it can screw up caches and return unexpected content.

Then the cache is broken. Obviously not getting a response is going to confuse things, but there is a way to send back REFUSED without giving a positive or negative answer.

Take HN's authoritative DNS server and ask it for information on yahoo.com.

    $ dig @sue.ns.cloudflare.com. yahoo.com.
    ...
    ;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 28595
    ...

They are perfectly right to refuse service to non-customers, but I think it's bad network practice (legally right or not) to return incorrect information.

It's hijacking. Try and talk your way around it all you like, but call it what it is.

I'm going to go sit in your car and wait for you to start driving to the store, then start screaming bloody murder that you are hijacking the car and not taking me to work.

This like you ask me for the nearest mac donalds, I ask to my buddy and he tells me in this place there are no mac donalds. So I refer you to my own restaurant telling you it is a mac donalds. My restaurant, of course, does not serve burger and all the meat is crap.

I have used the DNS servers of the Swiss Privacy Foundation for some time. The IP addresses are not easy to remember but it is great to have uncensored DNS from a Swiss non-profit organization:

77.109.138.45 (Ports: 53, 110; DNSSEC), 77.109.139.29 (Ports: 53, 110; DNSSEC) and 87.118.85.241 (Ports: 53, 110; DNSSEC).

https://www.privacyfoundation.ch/de/service/server.html

(The Swiss Privacy Foundation operates Tor exit nodes too.)

I have a newbie question, what would an end user do with the (Ports: 53, 110; DNSSEC) information?

I've set my machine to use those three IP addresses as the DNS servers, is there something else I'm missing? Thanks!

Normally DNS is on port 53, but if your ISP is preventing you from DNS requests to servers other than theirs on port 53 you can use the other one.

'DNSSEC' means DNSSEC is supported by the server if your resolver can use it - it's a digital signature regime to prevent DNS forgery (disclaimer: look up criticisms of it as well as selling points).

Yep, I'm seeing what should be NXDOMAIN results returning the IP 198.105.254.11 which brings me to a page like http://searchguide.level3.com/search/?q=http://198.105.254.1...

Does anyone know if actual Level3 customers see this page, or is it only for off-network requests? Up until the end of last year, my employer had a Level3 internet connection and we legitimately used 4.2.2.1 as our DNS recursive resolver. I'd be pretty pissed if they returned spammy results to their customers, but to non-customers, well, I don't care: That's what you get. Use a DNS server that somebody says you're allowed to (8.8.8.8, maybe)

Querying from one of my personal servers on a Level3 DIA circuit, I am getting NXDOMAINs for non-existent hostnames.

Suggestion: if your network provider's recursive DNS service sucks so much that you cannot bear to use it (and even if it doesn't, quite frankly) your next best bet is probably to install unbound (https://unbound.net/) listening on localhost on your workstation.

Not only does this give you known-good DNS resolution, but you can also enable DNSSEC validation and be fairly confident that it'll actually do its job in preventing your local machine from resolving poisoned zones.

I set up a caching recursive DNS server on my lan, but had to add a second entry for a caching public DNS server since:

1) Most DNS entries these days seem to have very short TTLs

2) Occasionally the recursive queries would fail

Would you mind clarifying what you mean by "second entry"?

When their DNS seemed to go down a few days ago I also noticed this behavior and immediately switched to a local independent ISP's DNS that namebench spit out.

Doesn't seem too out of character, if they're returning this for actual customers, from the company that most likely allowed the US government to tap into Google and Yahoo's fiber lines.

http://www.nytimes.com/2013/11/26/technology/a-peephole-for-...

That is a pretty big, and very uneducated, accusation.

Since the 1970s the CIA has been installing taps on undersea cables using specialized submarines. This is well documented, and it is well known that the NSA prefers to use methods that involve the least amount of interaction from uncleared individuals even if it is at a much greater expense.

the NSA is the one installing the taps. CIA tried to do a radio tower once -- or maybe that was the FBI. anyway, when it comes to sigint, it's NSA all day every day.

Nope. The Special Collection Service is responsible for deployments. It is a joint program with the CIA providing the field resources and management and the NSA providing the toys.

The article sources "three people with knowledge of Google’s and Yahoo’s systems who spoke on the condition of anonymity," so I wouldn't really call it uneducated.

Is the accusation that Level 3 has a 'black room' agreement with the NSA totally out of the realm of possibility? We know AT&T has such an agreement after Mark Klein exposed Room 641A years ago.

I have to plug OpenNIC[1] anytime I hear of a DNS hijacking story. OpenNIC is a peer run network of DNS servers that are open for public use.

[1]http://www.opennicproject.org/

This looks cool, but why should I use and trust this over something like Google's Public DNS?

Honestly, I have no argument for OpenNIC over Google's DNS that will convert you right now. It really depends on who you trust more.. strangers over the internet or a large corp who was the target of espionage. That's really up to you to decide the lesser of two evils.

A cool thing about OpenNIC is that they offer alternative TLDs that aren't part of ICANN's gTLDs. Also, the owners of the public servers strive to be as open as possible with their policies and features, such as no logging or using DNSCrypt. One of them even offers DNS level ad blocking, though I don't like it because I prefer the internet at its purest form and that policy doesn't seem to flow well with their anti-censorship mantra.

Can everyone else reproduce this problem? People from different locations and ISPs should try it.

I'm not a Level 3 customer in a any way and I'm on a German VDSL connection provided by Deutsche Telekom. And here the Level 3 resolvers still return normal NXDOMAIN answers:

  ; <<>> DiG 9.8.3-P1 <<>> thisprobablydoesntexist.com @4.2.2.2
  ;; global options: +cmd
  ;; Got answer:
  ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 44948
  ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

I can not replicate from Australia.

    ; <<>> DiG 9.8.3-P1 <<>> nxdomain.horse @4.2.2.
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 58067
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

Not sure if I'm misunderstanding this but my ISP (Virgin Media - not recommended, by the way) has always done this.. I assumed all ISPs did that?

If you're in the UK and using Virgin you can opt out. Link on the error page, ISTR

Lots of well meaning kids go and setup Level 3 resolvers on grandma's home network, then the ISP has to deal with support calls when someone elses DNS servers go down.

As a result quite a number of networks "hijack" 4.2.2.0/24 and route it locally to their own resolvers.

How can one detect whether this is happening?

I'm not sure if it's a certain way to tell, but try running a traceroute. If your traffic seems to go into Level3's network, that's a good sign that it's not getting rerouted.

Here's what I see from my DigitalOcean droplet.

root@derpy:~# traceroute -I 4.2.2.1 traceroute to 4.2.2.1 (4.2.2.1), 30 hops max, 60 byte packets 1 198.199.122.1 (198.199.122.1) 12.055 ms 12.123 ms 12.314 ms 2 xe-10-3-3-100.edge3.Newark1.Level3.net (4.28.6.69) 0.948 ms 0.959 ms 0.959 ms 3 ae-31-51.ebr1.Newark1.Level3.net (4.69.156.30) 1.396 ms 1.477 ms 1.478 ms 4 ae-10-10.ebr2.NewYork1.Level3.net (4.69.132.97) 1.530 ms 1.630 ms 1.659 ms 5 ae-62-62.csw1.NewYork1.Level3.net (4.69.148.34) 1.465 ms ae-82-82.csw3.NewYork1.Level3.net (4.69.148.42) 1.464 ms ae-62-62.csw1.NewYork1.Level3.net (4.69.148.34) 1.390 ms 6 ae-1-60.edge2.NewYork1.Level3.net (4.69.155.16) 1.363 ms 1.389 ms 1.395 ms 7 a.resolvers.level3.net (4.2.2.1) 1.456 ms 1.466 ms 1.421 ms

On the flipside, when the local ISP's DNS goes down, many a time will the support person have no knowledge of it, and run you thought the "do you own a computer? did you turn it off and on again?" spiel.

You act as if most ISP support actually admits to problems on their own network and/or that troubleshooters have access to this information.

> At the least it’s leaking, in clear text on the wire, things that I expected to be sent to an encypted DDG search. If there was sensitive search terms or information in that query, it just dropped into Level3′s logfiles.

He must not realize that even if the DNS server was working correctly, the original request that should result in NXDOMAIN is also passed in clear text over the wire and naturally potentially logged by the DNS server. The lesson is not to rely on DNS security. Your ISP can see what servers (IPs) you communicate with anyway.

I'm rather curious. I thought this kind of predatory network shenanigans was par for the course in the US?

Its common in the UK as well. Virgin Media (pretty much the only significant cable ISP in the UK) hijack NXDOMAIN, but let you turn it off via a link from the pages it serves.

By residential ISPs, sure. Level 3 is not a residential last-mile provider. I'd be surprised if you could get any sort of service out of them for less than $1k/month.

Why Am I Here?

The Example Net Web Helper has been enabled to provide helpful searches from web address errors. You entered an unknown name that the Example Net service used to present site suggestions which you may find useful. Clicking any of these suggestions provides you with Yahoo! search results, which may include relevant sponsored links. Why should I use this?

The Example Net Web Helper makes finding what you are looking for easier and more convenient. The service uses the entered non-existing website name to determine useful search results. Often, you will see a desired website or page that meets your needs. Do you track my Internet usage?

No. The Example Net Web Helper simply redirects queries to non-existing domain names to a useful search results page instead of a cryptic error message page or browser-defined page.

The "Example Net" huh?

don't use L3's 4.2.2.x resolvers, as they aren't meant for public use, unlike google's public dns

You say that, but if they weren't meant for public use, they wouldn't be accessible to the public. Time Warner's DNS, for example, are not available from outside their network.

And if your car is unlocked it's meant for joyriding?

If my car became very well known for being unlocked and immensely popular with people taking joyrides, then yes, my continued unlocking of the door would constitute tacit permission.

Some of us pay Level3 four or five figures a month. I checked and receive NXDOMAINs (like I should) but I would immediately open a ticket if they were fucking with my DNS.

That said, we run our own recursive servers and don't rely on Level3's.

Comcast also did this to me. Not one of the several tech support people I talked with seemed to be aware of Comcast's non-hijacking DNS servers at 75.75.75.75 and 75.75.76.76.

Fortunately they haven't done this for 2 years. They killed it when they flipped on DNSSEC because the practice of NXDOMAIN hijacking is incompatible:

http://dns.comcast.net/index.php/help#faq2

Now no customers from Comcast suffer this.

... JavaScript and HTML injection when you reach a cap limit in a throttled market or when you get a cease and desist for pirating, however, is another matter.

I use mdnsresponder nomulitcastannounce -> dnsmasq 127.0.0.1#53 -> dnscrypt-proxy 127.0.0.1#54 -> an encrypted DNS proxy that does dnssec. All of which is locked down by a minimal whitelist leak-preventing fw ruleset like little snitch. I have a script which checks for authentic internet access to allow captive portals to work which leaks temporarily (which I prob need to toggle between rulesets to only allow the captive portal agent to work and deny everything else).

Can't really blame them since they have been telling the public not to use their service forever.

Still, I am guilty of using them too.

Not really thrilled with the idea of using Google DNS.

> Not really thrilled with the idea of using Google DNS.

You can use your own dns server. Just install a recursive dns server on your own network like https://www.powerdns.com/recursor.html or https://unbound.net/

unbounds https cert is broken. The identity of this website has not been verified. • Server's certificate is not trusted. • Server's certificate cannot be checked.

Not a good first sign.

I noticed this a few weeks ago (not sure exactly, but I don't really mistype domain names all that often either) and I can remember the first time that 198.x IP showed up I was rather shocked since the domain I mistyped was my own site's (!), but they seem to have stopped doing it now.

Perhaps L3 themselves have a lot of stuff both within and outside their subnet that depends on these servers behaving correctly.

Is this only affecting Level3's 4.2.2.x nameservers (when it happens), or are their 209.244.0.x ones doing this as well?

I can't reproduce this behavior from the UK. In any case, DNSmasq (the DNS cache daemon that is part of OpenWRT) has an option to filter bogus NXDOMAIN responses if you can get a list of the IPs.

Alternatively, run your own recursive resolver and cache, it's worth it.

While in search of a new DNS server because of this and for lower latency, namebench is your friend: https://code.google.com/p/namebench/

The tracker should now be blocked also in Easyprivacy.

https://hg.adblockplus.org/easylist/rev/303e65d3a2bd

Reply on news.ycombinator.com