Hacker News Reader

Comment by jaas | Hacker News Reader

I'd say there is a 98% chance this is a bug in some firmware and a 2% chance AT&T is intentionally trying to block Cloudflare DNS.

I get why people are paranoid about ISPs blocking content and net neutrality, but let's not cry wolf prematurely. The technical details here strongly suggest a bug rather than intentional blocking of 1.1.1.1 DNS traffic.

View on HN

Then the odds appear to not be in our favor.

CF CEO tweets that 1.0.0.1 is also blocked.

https://twitter.com/eastdakota/status/991718955021623296

Others have confirmed that the ipv6 address belonging to CF appears to be blocked.

Do you have a reference for the ipv6 address being blocked? That would be a much bigger smoking gun

https://blog.cloudflare.com/dns-resolver-1-1-1-1/

> For IPv6, we have chosen 2606:4700:4700::1111 and 2606:4700:4700::1001 for our service. It’s not as easy to get cool IPv6 addresses; however, we’ve picked an address that only uses digits.

For me up in Canada, ping 1.1.1.1 works. But

    ping6 2606:4700:4700::1111
    ping6 2606:4700:4700::1001

shows "connect: Network is unreachable". Am I using ping6 wrong?

We also need to confirm IPV6 works outside AT&T's network.

Edit: Just tried Google's DNS. 8.8.8.8 works, but their IPv6 doesn't, so I guess this was a bad test.

Edit2: Learned about nslookup, but it does not seem to work with either Google or CloudFlare's DNS.

    nslookup reddit.com                       # Works
    nslookup reddit.com 1.1.1.1               # Works
    nslookup reddit.com 1.0.0.1               # Works
    nslookup reddit.com 2606:4700:4700::1111  # Does not
    nslookup reddit.com 8.8.8.8               # Works
    nslookup reddit.com 2001:4860:4860::8888         # Does not
    nslookup reddit.com 2001:4860:4860:0:0:0:0:8888  # Does not

Edit3: Apparently my ISP doesn't support IPv6 yet.

You're using the IPV6 address correctly, does https://test-ipv6.com report everything's dandy for you? If it does maybe they're blocking traffic or there's something else going on.

> No IPv6 address detected. Connections to IPv6-only sites are timing out. Any web site that is IPv6 only, will appear to be down to you.

Okay, guess my PC/LAN/ISP doesn't support IPv6 yet.

If you're in Ontario, Rogers doesn't support IPv6 yet. If you want IPv6, then your only option is Bell (or a reseller, like Teksavvy).

I'm using Bell in Ontario. It could be either my Router doesn't support it, the Apartment isn't wired up to support it (if that's required?), my ISP doesn't support it in my area, or my Bell internet plan doesn't cover IPv6...

I'll ask them about it when they ring me up next time asking for more money.

Hmm... looked at this again and it looks like Rogers may have rolled out IPv6 last year.

I recall on Teksavvy I had to pay extra for a "static IP" to get IPv6. Not sure if you're with Bell directly, though.

Everyone in Ontario on TekSavvy should have IPv6 now without having to pay for a static IP address but at least for me it's still wonky at best.

most modern linux distros regular `ping` will work for ipv6.

(US based) frontier, vz, and spectrum all can ping that ipv6 address (though all have way over 10ms latency)

I tested out both addresses via my phone's web browser just now.

Connecting to WiFi (Time Warner), I got a 403 from cloudflare (presumably there just isn't a web server set up on that address).

Using mobile data (AT&T), I got ERR_ADDRESS_UNREACHABLE. However, 1.1.1.1 actually works on AT&T cellular, so I'm not sure what to think.

ipv6 supports traceroute too

> Am I using ping6 wrong?

I'm pretty sure the other end has to be running `pingd` to get a response from ping. Some do, some don't.

I might be wrong but that's always been my understanding.

You are definitely wrong. No daemons have to be running, ping operates using standard ICMP echo messages that are a part of any complete IP stack. Any meaningful OS will respond to pings unless prevented from receiving them by a firewall. It wouldn't surprise me to find that some embedded implementations skip that part for size reasons, but even in that category most devices I have available to me still respond. It's a basic network connectivity diagnostic tool.

What is unfortunately common though is people blocking ICMP at their firewall, either at the host level itself or further upstream. Sometimes they just block echo requests, but often they block ICMP entirely which breaks things in very weird ways from time to time.

Blocking ICMP in any way is generally to be considered harmful. It's not 1997 anymore, the "ping of death" is not a thing on any OS you should actually be connecting to the internet.

Missing brackets probably:

  ping6 '[2606:4700:4700::1111]'

fwiw, I am an AT&T customer in Atlanta on their fiber service.

the nslookup reddit.com 1.1.1.1 does not return for me, if I connect to work via VPN it does. 1.0.0.1 and 8.8.8.8 do work without VPN. while the AT&T modem shows IPV6 I did not test.

System Information Type Value Manufacturer Pace Plc Model 5268AC

Just curious - can cloudflare blackhole all of Att traffic?

Could they physically? Yes. But they'd be screwing over their own customers who rely on that traffic.

Isn’t AT&T screwing their own customers by blocking 1.1.1.1 as well ?

AT&T isn’t blocking 1.1.1.1, just tested it on my uverse connection. As much as I hate AT&T their internet is pretty solid with the exception of datacaps

A more interesting use case though it would have its dangers is them showing a message to AT&T users that their ISP is doing things to damage the internet and that they should call and complain. People got mad at the idea of CloudFlare slowing down network requests by FCC members in protest of their shenanigans.

I have AT&T internet, and the BGW-210 gateway with the latest firmware. And my area was upgraded to native dual stack ipv6 about a year ago. So I tested it out and the ipv6 CloudFlare DNS (2606:4700:4700::1111 , 2606:4700:4700::1001) works perfectly fine. https://imgur.com/a/grUzeDD Its only the ipv4 1.1.1.1 that dose not. And AT&T made a statement why that is.

""With the recent launch of Cloudflare's 1.1.1.1 DNS service, we have discovered an unintentional gateway IP address conflict with 1 of their 4 usable IPs and are working to resolve the issue,"

https://arstechnica.com/information-technology/2018/05/att-i...

A few of you will be disappointed to know its not a evil attempt to block you from using it. Same way they have literally never blocked the ability to use any other DNS service before.It's simply a bug caused by the way the BGW-210, and Pace 5268AC operate and make use of 1.1.1.1 internally in some way and it will be fixed with a firmware update.

1.1.1.1 was working for me on AT&T after Cloudflare released 1.1.1.1, then shortly after that it ceased working.

Maybe the firmware update has a bug, but it's very suspiciously timed. Notice that the OP is dated April 2, while 1.1.1.1 was released April 1.

This is what happened to me as well. It worked for a day or so and then stopped.

I have ATT U-verse internet service and use their Arris BGW210-700 gateway

One interesting thing is that if I go to the gateway management page, and use their diagnostic tools, I'm able to ping / traceroute the address - but I can't from any devices connected to the gateway

From gateway diag page:

PING 1.1.1.1 (1.1.1.1): 56 data bytes 64 bytes from 1.1.1.1: seq=0 ttl=64 time=0.568 ms 64 bytes from 1.1.1.1: seq=1 ttl=64 time=0.156 ms 64 bytes from 1.1.1.1: seq=2 ttl=64 time=0.164 ms 64 bytes from 1.1.1.1: seq=3 ttl=64 time=0.144 ms

--- 1.1.1.1 ping statistics --- 4 packets transmitted, 4 packets received, 0% packet loss round-trip min/avg/max = 0.144/0.258/0.568 ms

traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 38 byte packets 1 1dot1dot1dot1.cloudflare-dns.com (1.1.1.1) 0.285 ms 0.177 ms 0.090 ms

The times on the pings make it look like its hitting a loopback address instead. Pings to 8.8.8.8 from the diagnostics page take about 23 ms. No way 1.1.1.1 is completing in under 1ms haha

Yes, 1.1.1.1 is in use on your Arris device, the same issue with the 5268AC since day one.

I was using 1.1.1.1 with AT&T Fiber and it stopped working. I didn't really question it, I figured maybe something went down at Cloudflare so I just switched my Mac back to using the defaults again. It never even occurred to me that AT&T might be blocking it.

Maybe stupid question, but why would AT&T block it?

They want you using their DNS for traffic snooping?

Pretty sure they don't block 8.8.8.8 though.

They can snoop on your DNS anyways.

Not with DNS over TLS. EDIT: Which CF supports.

So does Google DNS (using DNS-over-HTTPS), yet they haven't been blocked.

Same thing happened to me using at&t fiber.

A few others have mentioned this already, but 1.1.1.1 has become a colloquial private address, used either as a blackhole or as a destination for internal traffic. Sort of like how 555-5555 technically isn't reserved (only 555-01xx is, according to Wikipedia), but practically, it's not really a workable number and phone companies don't hand it out.

According to the announcement post, part of the reason that Cloudflare was allocated the 1.1.1.1 address is that they were ready and willing to handle the expected inundation of all kinds of bizarre traffic.

It seems that one of those "off-label" uses of 1.1.1.1 is an internal / network control interface on [some?] AT&T networks. I'm just speculating, but it's definitely possible that 1.1.1.1 suddenly becoming publicly routable and pointed to a real thing caused some problems. "Patch it out" may be an acceptable emergency response depending on the breakages, but not really acceptable long-term.

Not an acceptable thing to do silently though, in any term.

and the reports of 1.0.0.1?

If they were so determined to block it, why would they do it in firmware and not upstream? I think people are reading too much into this.

It's cheaper to do it on the mobile?

A possible explanation is that the traffic from active use of 1.1.1.1 caused some backend service to get overloaded with traffic due to a faulty assumption that the address would never be used by customers. Anyone keep traceroutes while before the patch to see if there were errant stops or delays?

They had the choice of "fix the whole backend" or "block 1.x on the user end".

Guess we know which one was easier. If all this wild speculation is true, maybe they're working on a fix to the root cause and will roll back the patch when complete.

This would make the situation both due to incompetence and intentional.

1.1.1.1 is well known (based on the announcement from cloudflare anyway) to have tons of random traffic. That's part of the reason it wasn't implemented by others as a valid address for anything. Could the fact that they're simply allowing traffic at that address cause additional stress on AT&T's network?

I ask because I don't know. I figure any traffic headed that direction would go anyway it just wouldn't get routed very far with no valid destination.

Yeah. And there's also a lot of traffic going in Facebook's direction, for example. Hey, let's blackhole that too - and alleviate the stress on our network that comes from people using it. (In non-sarcastic tone: that doesn't make any sense.)

Based on what I understand, the amount of traffic headed to 1.1.1.1 is much more significant. I agree with you though, that wouldn’t be justification to block it. It looks like they’re also blocking 1.0.0.1 and the relevant ipv6 addresses which shouldn’t have the same traffic issue.

I doubt it's all that significant, it's a really small portion of traffic compared to a web page, javascript, css or images... and with caching even less of an impact.

The problem isn’t DNS traffic. The problem is that for years people have been using 1.1.1.1 in the configuration of software and devices when they didn’t have an up address to configure. The result is that when 1.1.1.1 becomes routable all that additional traffic flows there and AT&T along with other provides carries that traffic. I was wrong that AT&T was blocking it for honorable reasons but this is a still a significant amount of traffic.

You're absolutely right about this. This is almost certainly just there to block people who mistakenly paste in an example configuration somewhere.

Back in 2010 there were problems that came up when IANA started allocating out of 1.0.0.0/8 (e.g. [1]). Things that were once assumed to be unused started being used, leading to strange issues.

Also, why on earth would AT&T block 1.1.1.1 and not Google DNS and OpenDNS?

[1] https://bgpmon.net/issues-with-allocating-from-1-0-0-08/

According to the thread the timing on this looks pretty bad since those DNS IPs were previously working on the earlier firmware.

How would it make sense to block it only on a small fraction of their entire network? It wouldn't accomplish anything.

when 1.1.1.1 was first announced a few weeks ago, many people pointed out at the time that it was already blocked because so many people had effectively polluted it by over-using it for demo examples and testing traffic. CF announced they knew this and intended to do a project analyzing the data. Perhaps this done, whether conveniently or not, with the same intention. We'll see if they reverse it.

Can someone link to the firmware? It shouldn't be hard to binwalk this and figure out wtf is happening.

Also- If this was intentional- I'm betting they'd filter it for the mobile network as well. This has got to be a fuck-up.

Having it seem like a bug would be an effective way to block it intentionally. The timing of such an unusual regression is suspicious. The fact that 1.0.0.1 is also blocked is also suspicious.

A conspiratorial Hanlon's corollary: The most effective malice is that which can be ascribed to incompetence.

Have you seen the Underhanded C Contest? http://www.underhanded-c.org/_page_id_5.html

"Bugs are worth more points if, once discovered, they are plausibly deniable as an innocent programming error."

Genuinely brilliant.

> Having it seem like a bug would be an effective way to block it intentionally.

Just like how only the true messiah denies his divinity, it doesn't give innocent bugs much of a chance.

In fact, now we can show that all bugs are suspicious, with apologies to the interesting number paradox:

The least intentional looking bug is the most effectively hidden, and therefore should probably be suspected of being intentional. Since it's now suspect, it's longer the least intentional looking bug, so the next least suspicious bug suddenly deserves a bit more scrutiny, and so on.

That's not my point. It's suspicious because of its specificity and timing; its bug-like presentation isn't evidence either way.

This is an unrelated yet related question. I am trying to access apple support, I use at&t. When I go to support.apple.com I get an error message stating : Access Denied. You do not have permission to access "http://support.apple.com" on this server. And gives me a long reference hash. This is at&t denying me access?

A university I went to used 1.1.1.1 for its WiFi loginpage

I thiught thats what early cisco (after they bought air-something) used c. 2005 or so

It's either malicious or a major fuck up. Either way it's worth shouting about.

It's the latter. Here is the CEO of Cloudflare tweeting about it: https://twitter.com/eastdakota/status/991718955021623296

Where do you see that this was a mistake? That tweet thread doesn't have anything definitive on it.

Blocking 1.1.1.1 -> 98% chance it is a bug

Blocking 1.1.1.1 and 1.0.0.1 -> what are the odds here?

Only one may be coincidental. Two is enemy action.

No. Block 1.0.0.0/8 due to internal use. That block includes both of those addresses.

the problem with this sentiment is what "one" and "two" imply. it's possible to refer/block/whatever multiple addresses with "one" action.

everything from 1.0.0.0/8 to 1.0.0.0/15 would encompass those IPs so who knows what but my guess would be some routing or other strange internal usage of some of those subnets

Anyone work at AT&T who could give us the inside scoop on these firmware changes? Snapping a photo of the blocking code would be a valuable public service.

Remember to scrub EXIF data!

More than likely:

- If the action was malicious, the people involved in writing this code are likely okay with it and not likely to leak details of it.

- If the issue is a bug, the people involved in writing this code are probably working to fix it, and not likely to leak details of it.

- People not involved with making it would likely leave an internal access trail (independent of EXIF data) when they access that code.

Which is to say, expecting an Ed Snowden every time a company does something unethical is kinda silly, otherwise we'd have Google's search algorithm by now.

1. Probably.

2. True.

3. Unlikely; it's likely in a big repo that's synched all at once.

Alternatively, we can just obtain the firmware from a device and diff it against the last-known-working version, to see how the routing is failing.

My router stopped working a few weeks after using 1.1.1.1. Weird things are happening with it.

I really hope that's the case, was 1.1.1.1 allocated before CF acquired it? Was 1.0.0.1 also blocked?

1.1.1.0/24 was reserved by APNIC for research use.

Lots of cisco example config use 1.1.1.1 for router internal identifier / DHCP server / OSPF dummy network .

Not suprised if it break anything.

Makes sense. Thanks!

I didn’t know this and it sure seems to corroborate what others suggested about it being intentional, but for completely different reasons.

What's that saying about not attributing to malice, what is more easily explained as stupidity or incompetence or whatever? (Occom's Razon and all that).

AT&T routers also don't let you use a 10.x address at home (possibly to prepare for carrier grade NAT, although there is an official 100.x address reserved for that; so fuck you ATT).

I'm so sick of my AT&T router/modem for various other reasons. I hate how you are required to use it for many of their offerings (including Fiber to the home).

There are a number of tools out there for putting their router behind your Linux box. Most of them configure ebtables or use scripts to forward the 802.1q authentication packets to/from the router.

> What's that saying about not attributing to malice, what is more easily explained as stupidity or incompetence or whatever?

Hanlon's razor: https://en.wikipedia.org/wiki/Hanlon%27s_razor

Wouldn't it be possible to use your own router and treat the AT&T router essentially like a modem? I ask because I'm about to move to an address that can get AT&T fiber.

Yes: https://www.att.com/Common/storefront/resources/pdf/att_brid...

However, I still can't ping 1.1.1.1.

Sort-of. It has a DMZPlus mode, but all it does is assign the public IP to an specific internal device and uses NAT, as well as forwarding all ports, to make it look like that device is onthe public Internet (even though the modem has the same public IP). You can still plug in other devices and they get private IPv4s or parts of your IPv6 prefix and it NATs (the IPv4) those as well (it's to support their VoIP phones and TV service).

It's a shitty hack and it adds a weird layer of indirection that's kinda buggy and doesn't always flow traffic the way you think it's being flowed. The IPv6 stuff gets confusing as well because the modem is still dishing out public IPv6 address, so if you want to advertise them as well, you've got to start slicing up your prefix.

You could swap out those two odds and you would be just as right...

They could swap those odds with anything but 100% to 0% and they would still be just as "right" once the answer comes out.

exactly... why pull out random numbers out of your ass?

Reply on news.ycombinator.com