AT&T Fiber Internet Privacy controls are horribly broken | Hacker News Reader

AT&T Fiber Internet Privacy controls are horribly broken

github.com

72 pointsnewlinks75 comments

Don't use your ISP's default DNS because they will mess with stuff. You should probably just set your DNS servers on your device because you probably don't want to use whatever random DNS you get from public WiFis either. I like to use Quad9 for DNS because it's independent and has some privacy/security features.

Also lobotomize their equipment into passthrough and use real equipment. I have that same router on AT&T fiber and just pass-through to Ubiquiti gear. For DNS it's all NextDNS. Or for another step, DNSFilter.

This is a great idea. Apparently some people have reverse-engineered the network authentication to allow you to bypass the AT&T modem, but it looks pretty hairy.

You can't configure DNS at the modem for AT&T fiber, which is the annoying thing. The setting is there, you can read it - but you can't modify it. I have no idea who is responsible for configuring it, either during install or at the factory. So if you don't want AT&T's garbage, you need to configure it on every device and/or at the router (and don't use their built in router).

Thing is, with DNS, the ISP can just intercept your DNS requests anyway. Turn on DoH imo

If you really care, run your own DNS locally. Not all devices/applications support DoH.

Running your own doesn't prevent interception for most domains.

You can run a DoH-to-regular-DNS bridging resolver locally.

My router supports that out of the box, but unfortunately it's somewhat unreliable compared to regular UDP resolution and I had to turn it back off.

It more or less does if your local DNS is just presenting DoH as normal DNS to every device on your LAN, since most devices let you configure DNS per network (even smart TVs, which is nice) but may not have any option for DoH. But at some point you have to trust someone.

> It more or less does if your local DNS is just presenting DoH as normal DNS to every device on your LAN

That's more of a proxy than running my own.

> But at some point you have to trust someone.

If I do my own recursive queries from multiple networks, I don't really have to trust anyone. (I mean, that's still trusting authoritative servers, but arguably they're correct by definition.)

Though I could also ask multiple diverse DoH servers to get a similar effect.

For any domains really.

Your best bet is something like Dnscrypt or DoH that exposes a resolver locally on your full network.

using DoH or DoT just shifts who you have to trust, from your ISP to another company, quite possibly one with a greater interest in selling your data or being mass-surveilled.

To use DNS whatsoever you have to at least trust the DNS host, though. There is no such thing as perfect trust on the internet.

It's quite insane that this tampering is allowed in the first place. We'd still be moving towards a more encrypted-by-default future, but there would be fewer meddling middleboxes that threaten less knowledgeable users.

For anyone that needs this (copy pasted from a comment 2 weeks ago)

Your DNS can also do things like block malware, adult content, trackers, social media, or other things. Here's info about cloudflare and mullvad.

  # Cloudflare
  ## Standard
  1.1.1.1
  1.0.0.1
  2606:4700:4700::1111
  2606:4700:4700::1001

  ## Block Malware
  1.1.1.2
  1.0.0.2
  2606:4700:4700::1112
  2606:4700:4700::1002

  ## Block Malware & Adult Content (not useful for this case)
  1.1.1.3
  1.0.0.3
  2606:4700:4700::1113
  2606:4700:4700::1003

You can find cloudflare information here[0], and remember to make sure you setup DNS over DoT (TLS) or DoH (HTTPS). Especially for them they will want to have encrypted DNS.

Mullvad also offers free DNS[1], which also supports encrypted DNS

  # Mullvad
  ## DoH is port 443 and DoT is port 853
  ## Standard
  dns.mullvad.net
  194.242.2.2
  2a07:e340::2

  ## Block Trackers
  adblock.dns.mullvad.net
  194.242.2.3
  2a07:e340::3

  ## Block Trackers + Malware
  base.dns.mullvad.net
  194.242.2.4
  2a07:e340::4

Mullvad also has block for Adult + gambling and social media (so there are 6 total configurations). You don't need the Mullvad VPN to use these.

I should also mention, as this frustrated me a bit, that your browser may implement its own DNS and so just setting these in your router (or pihole) may not completely resolve the issue. In Firefox, go to Settings > Privacy & Security > (scroll all the way down) Enable DNS over HTTPS using > then under either "Increased Protection" or "Max Protection" you can set a DNS resolver (or turn it off). They have defaults for Cloudflare (default!) and NextDNS. While you're there, also check your settings at the top of that page about "Enhanced Tracking Protection"

I am NOT a network/security person and would greatly appreciate replies to this comment with additional information. Especially about setting up things like piholes, TVs, browsers, encrypted DNS (especially this!), host files, and so on.

[0] - https://developers.cloudflare.com/1.1.1.1/ip-addresses/ - https://developers.cloudflare.com/1.1.1.1/setup/#dns-over-ht...

[1] https://mullvad.net/en/help/dns-over-https-and-dns-over-tls

It's probably also worth checking your ping to these servers. For me, Cloudflare and Quad9 are around 10ms, while Mullvad is around 40ms, so I take that into consideration when choosing a DNS provider.

Great point.

FWIW, when I pinged, quad9 was around ~10ms, cloud ~30ms, and mullvad ~150ms. I'm not sure your 40ms would be too meaningful of a difference, but >100ms definitely will.

I wonder if anyone knows a way that I could script up a means for doing this dynamically? Like I can have a preferred order (let's say mullvad, cloud, quad for example since this is reverse my timing) and then ping occasionally and reorder based on that or a threshold (like <50ms)? Could be useful for like a pihole?

Unless you watch movies online. I remember switching away from my Comcast/Xfinity DNS and immediately experience significant degradation of Netflix/Amazon video quality. Apparently they use DNS for traffic optimization when video streaming from popular video sites.

Netflix has hardware on site at ISPs for this to accelerate your streams.

https://openconnect.netflix.com/en/

I really hope that we'll move past DNS load balancing for large media files at some point.

It's definitely better than nothing for small/one-off fetches, but VOD streaming sites usually have to fetch a license file from some API server anyway – so why not direct the viewer/client to the best CDN host based on the client IP address instead?

This used to be a problem but I haven't experienced it in a few years. I think the streaming providers must be using anycast instead of DNS techniques now?

Perhaps using something like a split-horizon DNS setup would alleviate that

Depends on your ISP. I talk to mine on IRC, they are xkcd806 compatible, they don't do anything to my traffic. They do provide DNS, DOT and DOH servers, or I can use whatever server I want. They're happy to have my host my own DNS server too, as long as it's not configured to allow amplification attacks from random IPs.

They actually had a mea-culpa with them a week or so back

> Our DoH/DoT resolvers were intermittently failing DNS lookups. It seemed to start over the Easter weekend. Our DoT/DoH front ends are DNS aware proxies (dnsdist) to back ends running unbound. dnsdist uses TLS to speak DNS to the back ends. Some of the back ends had failed to reload their TLS certificates after renewal, so although the certificates were valid unbound was still serving old certs and they eventually expired. This resulted in broken back ends in the pool, which dnsdist kept trying to bring back into service. The intermittent nature of the failures meant that it wasn't obvious to users, as clients generally retry silently in the background. Of course our monitoring should have caught this! We've fixed the underlying problem which caused unbound not to pick up the renewed certificates, and we've improved monitoring to catch similar problems should they occur in future.

I assume that's https://en.m.wikipedia.org/wiki/Andrews_%26_Arnold

The minute they come to America I'm switching.

I'd be curious about other DNS services which compete with Quad9. I seem to have an inkling of a memory that there used to be some kind of truly "open" DNS (not OpenDNS I don't think), but I can't recall what their name was or what their paradigm was, precisely.

Mullvad has a public DNS service that I'd trust.

You might be thinking of NextDNS?

That's not universally true. The ISP I run doesn't futz with DNS, and using our DNS servers is beneficial as it shaves about 10-20ms RTT off many DNS requests since our DNS servers are local with forwarders in the city that most of our transit providers are in.

Getting downvoted for an on-topic relevant comment is pretty disappointing. The fact of the matter is that there are still small internet providers around that care about the privacy of their customers. Furthermore, anyone who thinks they are enhancing their privacy by using one of the large public DNS servers needs to do a reality check: you pay for your use of "free" public DNS servers by sharing the information about every single website that you visit. All an advertiser has to do to de-anonymize your browsing history in that case is to put a single unique domain name in a web page and trigger the browser to do a lookup. Do folks think that AT&T is the only large company that has a really shitty privacy track record when it comes to DNS?

I also stand by the claim that local DNS servers are beneficial. Not all of us are 1ms away from the major DNS providers. Transport networks are complicated, and those of us that do not have millions of dollars to build our own fibre directly to a data center where peering is available have to pay a latency price for sub-optimal transport routes. It is a fact that having a local DNS server will save you multiple round trips in such cases when the deployment is architect to do so. I did that because it makes browsing a little bit snappier. Saving a few 20ms RTTs might mean nothing to people living in a large city with a local internet exchange, but it does help those of us in rural remote communities that are distant from that infrastructure.

I had this problem and complained to AT&T support. I know it doesn't make sense to replace the modem because this is a software, not hardware, issue but they very quickly offered to replace the modem and now my DNS isn't getting hijacked. Would recommend trying it!

Wonder if there's been a quite hardware-rev and they can't/don't want to update the firmware on the old units. I ran into that once on Spectrum - I bought the "same" modem to replace one, and suddenly my IPv6 config was borked.

Turns out Spectrum (in my region) actually pushes in their config file to disable IPv6, even though their dual-stack network works great, and has been working for at least 8-years now. Some modems apparently "override" that directive (e.g. ignore it and try to configure the IPv6 stack anyways) and you get fully functional IPv6 service. Other modems play goody-two-shoes and you're stuck with only IPv4. The new modem I bought was sold/marketed as the same model but was internally a totally different radio chipset. It was pulling a different firmware rev which had evidently been patched to actually obey the IP provisioning mode.

Spectrum support told me, basically, that if I have working IPv4 connectivity then my service is considered functional and there is nothing they can do. I gave up playing the support game and ended up exchanging until I landed on a Motorola modem that gleefully ignores that config parameter.

I wish I knew how to actually state my case to someone at Spectrum with the authority to actually fix their busted provisioning profiles, because it's kind of crazy to me that they've basically bifurcated IPv6 in this market based on whether or not your modem feels like reading the whole config file ;-P. (What's even funnier is the modems they install must be spec non-compliant, because IPv6 at the office works fine and that's their leased equipment.)

What modem did they replace it with?

My initial takeaway from this rant is this person doesn't understand how DNS works.

If your ISP making a change is breaking resolution on your local LAN then you are doing it wrong, the problem is 100% on your end.

If you are running properly functioning internal DNS, your ISP cannot make a change that will impact your LAN.

Any OpenWRT device can do this out of the box with zero additional configuration.

I think a surprising number of people probably just use an ISP-provided modem / router combo, allowing the ISP to set the LAN's DNS, decide how to hand out IPs through DHCP, and other configuration options that give them way too much control. A shrinking number of geeks use their own router, and a tiny fraction of that shrinking number even uses their own WAN interface.

Many of the posters here seem to be missing the point. 99.9% of customers using ISP supplied equipment don't care about internal DNS resolution, grandma just wants the internet to work and most consumer grade peripherals like printers just use zeroconf anyway, bypassing DNS completely.

Those 99.9% customers would still benefit from not getting "helpful" ads shoved into their face when they make a typo in a domain name.

Sorry, but this really comes off as victim blaming and "domain knowledge"-eliteness.

Victim blaming because AT&T shouldn't be doing these things in the first place. Yes, you're right, you should set things up in a better way to prevent this from happening, but you're excusing the bad behavior by placing the blame on the user.

The elitism comes from the fact that we live in a specialized world. Not everyone is an expert in everything. If you think you are, I'm here to tell you you're just lying to yourself and thus undermining yourself. There's even plenty of good programmers and people who work on other technical domains that don't have all the knowledge around DNS. This is true for any technology or any specific task. If you're an expert is any domain you're probably quite aware that getting things right ends up being very nuanced. So if you recognize that, then you should recognize that's true outside your main domain.

Now if your intention is to help people and educate them then may I offer another way to formulate the comment?

  If your ISP is making changes that break resolution of your local LAN you can do some configurations to prevent this. You should probably do this anyways. OpenWRT will perform this for you out of the box, but if you can't or don't want to flash your router you can <insert explanation here> or I found this guide [<insert link number>]

  [<insert link number>]

Even if you think people are dumb (__especially__ if you think people are dumb) this message will get more people to fix their problems. You may want to get in the habit of writing like this because we all know that there are things that other people do that affect us. Or chose to do/use things that move the markets and politics and whatever that we then become subjected to. Attacking or blaming them doesn't help us get what we want. Being nice does. At least being nice is the easier way (and you'll probably be less stressed too).

Spare me the kindness lecture.

The entire rant, most of which is lacking in technical depth and understanding of name resolution despite the overconfidence, could be completely avoided with an entry in /etc/hosts (or its Windows equivalent).

Which for a small home network and for someone who doesn't understand or want to fiddle with DNS - is perfectly adequate.

I remember a story in HN folklore from someone who worked at a large enterprise many years ago that used an insane massive deployment of hosts files in lieu of DNS.

Give me the lecture, what do I put in /etc/hosts? How does this prevent AT&T fucking with me? How do I make this work on all my machines to prevent this issue? Like my smart TV?

IPAddress <tab> HostName1 <space> HostName2 etc....

Just copy the same hosts file to all your hosts.

Why does your smart TV need to talk to internal hosts? If so, is it terrible to just use the IP? Or use zeroconf/bonjour which was developed specifically so normies would not have to deal with internal DNS.

Your ISP router may implement DNS proxy or more likely doesn't do any name resolution at all; or their embedded DHCP server just passes the ISP DNS IPs to the devices and the one you get is dependent on if you turned on their privacy thing. In which case you don't have internal DNS and the fact that it worked at some point in the past was coincidental and due to some fallback mechanism.

The shitty-equipment issue is not unique to ATT.

The best solution is to do what others suggest; place the ATT gateway in bridge mode and get your own router that runs OpenWRT; you can program your own local hostnames using the web GUI without touching the hosts file or knowing how DNS works, and everything should just work.

TV should talk to internal hosts so it can play content that is hosted by internal hosts.

Sure, shitty-equipment is prolific but how do we fix it and how do we pressure them to stop making shitty equipment.

> The best solution is to do what others suggest; place the ATT gateway in bridge mode and get your own router that runs OpenWRT;

How? Do I just link them with an ethernet cable?

> Any OpenWRT device can do this out of the box with zero additional configuration.

Except you must use AT&T's modem when you buy their fiber, and you can't install OpenWRT on it. The rant is lamenting that the all in one modem/router isn't configurable at all, and you need to be using an aftermarket router for your LAN to have any kind of internal DNS.

AT&T allows you to configure their Gateway in a passthrough mode that will allow you to use your own router without double NATing yourself.

Sure, but the point was that you have to buy another gadget. And there's still a box you don't control that has to live in your basement (or wherever) doing some not insignificant lifting.

Wouldn't it be nice if you could just configure the box you're paying for already to do the right thing?

Do you have to use their modem or their router?

I've been using my own router across many ISPs over time, and almost all of them offered a "bridge" mode that would just hand a public IP to my own router over DHCP.

You have to use their modem*. You can bring your own router which is what every nerd should probably be doing anyway, but there's no option for an off the shelf modem.

* for fiber

> The only reliable solution appears to be running your own router behind AT&T's BWG router-modem-all-in-one.

This. I wish they had an ONT setup that allowed me to fully bypass everything of theirs.

There are people that have successfully added their own ONT setup to by-pass AT&T awful BGW-320 500/505 gateways. But it seems that its an uphill battle. dslreports.com has an entire thread dedicated to this.

I myself looked into it and realized as much as I'd like to have true bridge mode, it just wasn't worth the headache if AT&T made a change on their end. So I have BGW-320 configured for passthrough mode and an OPNSense box behind it. With Unbound recursive DNS resolver and Pihole - I fortunately don't have the problems as described in the github writeup

In the past, I got AT&T FTTN, and their router somehow managed to break my NAT firewall. I've never had this problem before or since, and spent over a day debugging it. If I had to use their stuff at this point, I'd probably set up an SSH tunnel from my local router to some machine elsewhere, and then run ppp or similar over it. I'd also stick their wifi access point garbage inside a faraday cage.

Annoyingly, I had these problems in California when network neutrality was in full force. One of the rules said that ISPs were not allowed to discriminate against customer-owned network devices. Clearly, it didn't apply to AT&T.

If you have had ATT fiber for years the the ONT is separate from the router/gateway. They use 802.1x port security but the key and cert are easily extracted from the gateway due to firmware bugs and they are probably available online too (but doing the former is easy enough). No need to use any AT&T provided equipment other than the ONT.

If you have the newer combined device you can put it in bridge mode which is probably sufficient for most needs though some have procured and installed their own ONT only setup (SFP modules are like $50).

Even if you don't have a separate ONT you can bypass the BGW320 completely with an SFP GPON-ONT-on-a-stick into SFP+ interface or a 2.5g media converter if you are in a GPON area. If you are XGSPON the common option is to buy a WAG-D20. Either way you still need certs for 802.1x.

>WAG-D20

The WAG-D20 is no longer supported/works if you're on AT&T. At least according to the discussion on dslreports and also 8311 discord server.

https://discord.com/invite/8311-886329492438671420

Also, even Baltic networks has a warning about using it with AT&T.

https://www.balticnetworks.com/products/azores-1x-10gbe-1x-2...

The new working ONT is the WAS-110.

> The WAG-D20 is no longer supported/works if you're on AT&T. At least according to the discussion on dslreports and also 8311 discord server.

The WAG-D20 is basically not recommended generally due to chipset bugs on newer revs and reduced speeds, nothing specifically with AT&T, though the VEIP issue is a specific sticking point.

https://docs.google.com/document/d/13gucfDOf8X9ptkj5BOg12V0x...

The Baltic link you provided also is not limited to AT&T, and that is more a disclaimer because they are probably sick of return attempts by people with just enough to knowledge to find a cite for WAG-D20 but otherwise clueless. I doubt they want to encourage any residential customers for any model regardless if it works.

The WAG-D20 still works ok for existing setups.

thanks for the clarification. Do you know if WAG-D20 chip issues have been resolved? Back when i had first looked into replacing the BGW-320 505 with my own ONT setup, the WAG-D20 instructions I found were straightforward. The newer setup process with ONT like the WAS-110 seem to be a bit more complicated to me.

Yes, I did mention that.

> Either way you still need certs for 802.1x.

Actually for GPON the 802.1x is enforced on the ONT, so if you use an ONT SFP stick you do not need 802.1x. I already had pulled mine years ago and they’re good until 2038 or something so I haven’t bothered bypassing the ATT ONT.

Yeah, I meant for newer customers. The ONT is built into the BGW320 with no separate ONT box. You still need the certs from a BGW-210 etc. in that scenario.

Does having control of the ONTs allow them to throttle traffic/abuse in a GPON/passive fiber system?

I downshifted my ATT Fiber recently, I was getting a full gigabit when I was the first customer in the neighborhood but lately I couldn't do better than 700. So why pay for the full boat?

I was under the impression that you could clone the ONT... SFP ONTs are only like $50. Their router thingamajig was always hot garbage.

> The only reliable solution appears to be running your own router behind AT&T's BWG router-modem-all-in-one.

I'm running Eeros (mesh) with the first one connected by ethernet, and the AT&T router's wifi is disabled, so I don't think I've seen these issues.

Came here to say this exact thing. I have the exact modem/router mentioned in the article - my reliability, speed, and coverage all improved when I switched to my own $100 router.

I point dnsmasq on my home router at Cloudflare DNS and then add two exceptions:

  server=/att.com/68.94.156.11
  server=/att.net/68.94.156.11

(These used to be needed for their account portal to work properly but I don't know if that's still true.)

I also used to point my streaming devices at AT&T's DNS servers in order for CDNs to properly serve from the closest location, but that no longer seems necessary, so I guess the CDNs have gotten smarter (anycast maybe?) since Cloudflare doesn't send EDNS Client Subnet information in its queries.

> These used to be needed for their account portal to work properly but I don't know if that's still true.

Do not believe it is, as I have not needed these exceptions and can do bill pay, usage data, etc no problem.

Does this work for any device that is using the router?

I assume I have to have OpenWrt installed?

OT: > A few useful reference links:
> https://gist.github.com/CollinChaffin/24f6c9652efb3d6d5ef2f5...
> https://kagi.com/search?q=bwg-500+local+hostname+not+resolvi...
> https://kagi.com/search?q=att+fiber+local+hostname+not+resol...

> Google is apparently now useless for finding info on the Internet beyond very basic queries for stackoverflow or reddit, I had to use Kagi to discover anything useful on this topic.

I hadn't thought about this, but is surprisingly true for me. Google search quality has gotten so bad I could either do without and search SO, reddit, HN, Wikipedia directly or use any other search, and if that doesn't work get rich higher quality responses from an LLM. Entering search terms on Google is merely a convenience not necessity/quality.

Does anyone have a comprehensive-ish tutorial on how to circumvent this/just have your own router without having to deal with AT&T. My dad has AT&T fiber and I've been wanting to fix this for him but haven't gotten around to doing the research. IIRC there was some reason why it was a good idea not to use the built in bridge mode + second router and the "better" (ugh) solution was to allow the AT&T thing to do the security handshake and then reroute all the traffic by placing an OpenWRT box between the AT&T router and the fiber box. Would love it if someone could point me in the right direction so I don't have to sift through google.

When AT&T was using an ONT + Gateway there were ways to circumvent traffic going through their Gateway while still allowing the Gateway to talk to the ONT and unlock it. But they've transitioned to deploying a single box (BGW320s) that are a single unit.

You can still put the Gateway into passthrough mode and disable it's features.

He's got the two box solution, so I guess my memory served me correctly about the workaround and I'll try to check the discord in the sibling comment for more help, thanks!

Look up the 8311 discord.

I wish their ONT was a separate device. I haven't had any issues using their passthrough thing but it irks me to have it at all. Plus it sometimes randomly reboots in the wee hours of the morning.

I tried to order a static IP address from AT&T and most of their knowledge base article was pointing to a button / link in customer portal that doesn't exist anymore.

Haha, just tried it and it really does work like that. Amazing.

I don't think that running your own router fully solves this issue. I've had this disabled from day 1 of having fiber and running pfSense behind the BWG320. I also have a backup TMobile connection. When the the ATT line goes down (like when the fiber is disconnected...or cut) my DNS won't resolve and just goes to the ATT DNS hijack. For failover to actually work I have to unplug the BGW320. I have posted on ATT forums and pfSense and nobody knows how to bypass this.

How is that possible, if you run your own router you control DNS. Make your pfsense not use AT&T DNS?? I don’t know what to tell you but it’s your LAN, you can simply not use AT&T DNS at all.

I don’t use pfsense specifically but I can tell you that you have it misconfigured.

That sounds like a failover problem.

My router has dual WAN and will fail over if the WAN port fails to renew its DHCP lease. Ethernet link is still live, but no worky, so fails over.

You probably need to examine your Pfsense settings carefully. I'm running OPNsense with the BGW320 , with Unbound setup as the resolver. No issues.

Reply on news.ycombinator.com