Hacker News Reader

Comment by jon-wood | Hacker News Reader

I’m responsible for a bunch of IoT hardware, and every firmware spec I write includes a note on not using the DNS servers provided via DHCP. While sure there are companies explicitly doing this to avoid filtering, at least in my case it’s because a significant proportion of DHCP servers are configured to send DNS to your ISP, and ISP provided DNS is almost universally terrible. They’ll ignore TTLs, rewrite NXDOMAIN responses into the IP address of their ad-laced web search, and occasionally just highjack every single query to send you to a page saying you’re approaching your bandwidth quota. In the face of that sort of behaviour you inevitably end up with technical support having to field angry customers who just don’t get that it’s not our fault the hardware isn’t working, and the response to that is hardcoding a set of known good DNS servers which we can rely on.

Please, if you have a Pihole, redirect all DNS through it as described in the article. Just be aware DNS over HTTPS is a thing now, and while the devices I’m responsible for aren’t going to try and evade your redirects, the companies that are trying to make sure ads get delivered will absolutely switch to DoH which will be much more difficult to work around.

View on HN

> and ISP provided DNS is almost universally terrible. They’ll ignore TTLs, rewrite NXDOMAIN responses into the IP address of their ad-laced web search, and occasionally just highjack every single query to send you to a page saying you’re approaching your bandwidth quota

Universally? Perhaps in the US? But IoT devices are sold worldwide.

While I do run my own recursive resolver I checked my ISP's and they're behaving fairly reasonable and do none of the above and I have a direct (contractual) relationship with them and we reside in the same jurisdiction so at least in principle I could apply pressure to them if they do something shady. The same can't be said about google or cloudflare.

> Just be aware DNS over HTTPS is a thing now, and while the devices I’m responsible for aren’t going to try and evade your redirects, the companies that are trying to make sure ads get delivered will absolutely switch to DoH which will be much more difficult to work around.

I'm sure google had only our best interests in mind when unleashing that on us.

It’s not just the U.S. A problem report once lead me to discover that the Spanish company Telefónica cached DNS results for years longer than the TTL. My error monitoring also picked up content injection (prior to HTTPS) and DNS hijacking around the world.

Before going with the conspiratorial take note that Mozilla shipped DoH early and both Microsoft and Apple implemented it. Untrustworthy ISPs are a real problem even if Google deservedly gets suspicion about their motives. This isn’t another AMP.

At this point Mozilla is just the Non-profit Arm of Google, They do what google tells them....

and DoH is absolutely designed to get around network based security and filtering, both for Ad's and other reasons.

> At this point Mozilla is just the Non-profit Arm of Google, They do what google tells them

You can just say you don’t follow this closely. Mozilla is not perfect but they do push for privacy, with an increasingly limited amount of negotiating power.

> DoH is absolutely designed to get around network based security and filtering, both for Ad's and other reasons.

This is similarly reflecting a poor understanding of the situation. DoH can’t get around network filtering - if you block packets, there’s no magic trick to bypass it. It’s great for preventing ISPs from tampering with traffic or monitoring activity (this will also require eSNI to complete) but it’s not giving an attacker any capability they didn’t already have. If you’re concerned about security you’re fooling yourself if you don’t have endpoint management and some level of network segmentation and egress control. Attackers have hard-coded DNS servers, C&C endpoints, etc. for decades.

DoH absolutely can get network filtering: if I block port 53 outbound to control DNS queries within my network, DoH (and other tunneling technologies) is bypassing my network filtering of DNS.

By the same logic HTTPS is bypassing your network filtering of every service which happens to have an alternative available over HTTPS. So do you block HTTPS?

I’ve been thinking about installing a proxy with my own root for this reason.

I would not want to use your network if that was the case, and I wonder if you similarly would use someone else's network if it was configured like that. I don't think network operators who provide access to the Internet can realistically expect to control what their users do on the Internet, unless the network operator is also the administrator of those workstations.

Maybe, but my rule is: my home network, my rules. I can’t really do this invisibly anyways: any device I don’t control will get certificate validation errors.

The problem here is even the device owner is not the "administrator" of the device as we have lost ownership rights, locked behind EULA's, Patents and copyright

The point of the original story is "your" meaning a device you own, is ignoring your network controls

it is highly unlikely that the TV manufacturer is going to allow me to install my own custom root certs to inspect their traffic to HTTPS, so yes DoH and other things are a threat to network security, because if the TV become compromised I have limited administrative controls to prevent it other the blocking it completely which is a poor response to the problem

DoH is a solution in search of a problem that can be solves in better more user friendly ways

They could do that any time without DoH, as has been done for years. Your true choice is allowing network access or not.

Yes, from your IoT devices maybe you do.

The usual approach to setting up a firewall is a default of "block everything" and then selectively allow only what is needed.

Most people cheat and only do this on inbound connections, allowing everything on the egress side, because it's easier. But if you want to block your IoT devices from making outbound https connections, you easily can.

There's nothing really new going on here. It's always been possible to tunnel one protocol over another, or use nonstandard ports, and use encryption on the traffic to hide what you're doing.

I would absolutely love to have control over HTTPS traffic on my network, specifically to enable my Squid proxy to cache HTTPS pages, but unfortunately not every device or even program supports custom CA's. I'd be the man-in-the-middle between the internet and every device I own.

I think that is a pretty user-hostile attitude and I suspect you probably wouldn't really love it if every network operator was doing that kind of thing.

What’s user-hostile is every consumer device insisting on pushing its defaults and ignoring any network policies I’ve configured.

Yes, I agree that is also user-hostile, since it should be configurable. The problem is not about network policies though, since DHCP is explicitly not designed to be a policy and is purposely meant to be optional for the client.

For this reason I wouldn't recommend buying a device like the Chromecast, in which the user can't configure the network settings. Instead maybe consider something like the Amazon Fire Stick which is not as user-hostile.

I read that as LanternLight83 is both the network operator and the user in this case, and if so it’s not really a problem.

I would also love to MitM myself in some cases, mostly because it’d be interesting to see what’s going on.

Yes, but my point is that if you feel it's justified for your own network then you ought to expect every other network operator will feel that way about their network too.

So before applying that mentality, it would be wise to consider what your experience would be like if all your neighbours, friends, colleagues etc also did that on their networks.

I have no problem with that: if my workplace MITMs traffic, I’ll use my cellphone connection and a personal laptop for sensitive data. If a friend’s house mitms traffic, same deal.

Is the point that I sometimes use these networks? Then I somewhat agree - I would set up a separate guest network without shenanigans for guests to use. This avoids both the ethical sketchiness and having to explain why their web browser is shouting at them

True, that could be a good compromise. Although there are still some disadvantages like creating an SPoF for yourself and increasing your attack surface (e.g. anyone who compromises your internal CA has access to all your encrypted connections)

You don’t need to MitM. Squid can use SNI to allow whitelisted hosts, for now at least.

Hopefully that will be fixed with widespread usage of TLS ESNI/ECH

Network filtering means blocking traffic at the network level. Trying to use DNS for this leaves you trusting the client - and there are decades of precedent for clients bypassing that for various reasons, such as this post shows.

The solution is to start doing network filtering: if you block packets to unapproved servers, you can actually stop this. You’ll need to run your own proxy, of course, but that’s always been the only way to actually accomplish that goal.

That isn’t really possible, though: CDNs mean that blocking by IP just doesn’t work. The most effective method I’ve found is transparently redirecting all traffic on port 53 to a DNS server I control. DoH means that I might as well setup a transparent HTTPS proxy.

You’ll note that I mentioned a proxy. That’s why: you need to force all traffic through a proxy you control or you’re just hoping that a client doesn’t use hard-coded IPs or an outside API of some source. If your network allows the client to send traffic to port 443 anywhere, any blocking is on the honor system.

My question here is how are you going to install your root certificate on a $300 smart TV? Or, if that is not required because the TV does not verify DoH certificates, how bad is that for security (which we already know is awful on these devices)?

Well, it’d just block internet access there. I wonder if it’s possible to push a root to an Apple TV

The Checkrain jailbreak [0] supports several Apple TVs, so I’d bet you could use use certbot to obtain a LetEncrypt [1] cert for it.

Of course, jailbreaking opens up other security issues, so it goes back to what you can tolerate.

[0] https://checkrain.org/ [1] https://letsencrypt.org/getting-started/

I know that you can install iOS profiles onto the Apple TV, I have done this for 802.1x support, which included a CA root for the 802.1x controller.

I would imagine you can use this to push any certificate that you can also push to an iOS/iPadOS/macOS device.

This is what I was thinking about.

This comes back to the core decision: do you care about controlling the network enough to block access? If the device can't be managed / is no longer supported the safest choice is not to allow it online at all. Different people will have different risk tolerances – it might make sense to put, say, a remote-control power switch on an IoT no-man's land network but if it has access to personal information or cameras/microphones it's not unreasonable to say it should just be blocked unless you're actively using those features.

"That isn’t really possible, though: CDNs mean that blocking by IP just doesn’t work. The most effective method I’ve found is transparently redirecting all traffic on port 53 to a DNS server I control. DoH means that I might as well setup a transparent HTTPS proxy."

This is a very good point and I am dealing with this myself on my home networks.

Like any household/family we have some number of dubious/untrusted devices that still need Internet access.

By establishing my own recursive resolver I can act as a chokepoint (and monitoring point) for their behavior online. It's a very elegant solution, actually, and I have created a nice integration between my datacenter-hosted resolver and nextdns.io as the adblocking upstream DNS.

DoH breaks all of this.

I have no interest in diving down the "MITM my own network by inserting custom certs into embedded devices that may or may not use them".

Since we're talking about it, though, it occurs to me that you could quickly do a DoH lookup to every single new IP connected to, outbound, from your network - and then block all IPs that answer your DoH query. You're basically pre-testing all new SSL connections to see if they are to a DoH resolver that you (presumably) don't want to talk to ...

This solves the CDN problem ... does it solve the problem entirely ? I have only just thought of this moments ago ...

> By establishing my own recursive resolver I can act as a chokepoint (and monitoring point) for their behavior online. It's a very elegant solution, actually, and I have created a nice integration between my datacenter-hosted resolver and nextdns.io as the adblocking upstream DNS.

This only works for the subset of devices which use the local DNS. If they use any of the well-known techniques to avoid that filtering it's completely ineffective.

> Since we're talking about it, though, it occurs to me that you could quickly do a DoH lookup to every single new IP that initiates a new connection, outbound, from your network - and then block all IPs that answer your DoH query.

It doesn't solve the CDN problem: CDNs will route traffic based on the hostname and blocking them will have a degree of collateral damage which most people can't work with. Setting up your own HTTPS proxy avoids this.

"This only works for the subset of devices which use the local DNS. If they use any of the well-known techniques to avoid that filtering it's completely ineffective."

If you also block all port 53 after allowing your own resolver ... you may have some headaches with devices that refuse to use the DHCP provided resolvers but you know they aren't going to other resolvers.

That kind of control is what DoH breaks and I'd love to find an elegant (non-MITM proxy) solution for it ...

There isn’t an effective solution for a device which ignores local network policy other than returning it so the manufacturer pays the cost of designing a bad system.

With the side effect of your local vendor refusing to do further business with you, "the problem customer", with your "unreasonable demands" and technobable.

I use iptables on an openwrt router to redirect all traffic on port 53 to my own DNS

DNS over HTTPS has been a thing since before IETF standardized it, the technique was just in the form of a non-standardized API running on some benign domain.

Thinking about it, if you’re willing to give up TLS 1.3, you could probably just break all https connections with encrypted SNI and then filter based on the SNI information.

Just to nitpick: TLS 1.3 still uses plaintext SNI by default. You need to explicitly put public keys in DNS to enable the encrypted SNI extension.

And in the context of pihole and such, avoiding that means editing the DNS response to remove those public keys. Which takes us full circle back to "do I control DNS for this gadget, or not".

Well, it’s easy to control the DNS if you can block DoH and VPNs: just have your router forward all traffic on port 53 to a resolver you control.

Right now you can block the common IPs for the DoH servers and be fine.

So I need to mitm https traffic for devices I can't change the ssl trust store on?

You have two choices: trust the device or don’t let it on the network. Voluntary measures like local DNS only work to the extent that the device maker wants them to.

The issue isn't network filtering, it's encryption.

Standard DNS is unencrypted. DNS-over-HTTP is encrypted. Or DNSSEC or any number of newer standards that secure the DNS lookup. At that point, filtering will require MITM proxies, whether it's for DNS or HTTP or any other protocol.

It's a trade-off with security on the open network meaning harder penetration and control in your internal network. There's no easy answer.

DNSSEC is not encrypted. Moreover, between end systems and DNS servers --- the scenario we're discussing on this thread --- it isn't even authenticated.

Sure, DNSSEC provides authentication and integrity rather than encrypted traffic, which makes spoofing or rewriting the responses hard.

Why do you say it's not authenticated? If they're using the newer standards then that's what it provides. If they're not then there's no issue with network filtering as usual.

Again: the article discusses an environment where machines on a home network are refusing to use the DNS servers the network is configured to use. DNSSEC authenticates requests between servers. But between DNS clients ("stub resolvers") and servers ("full recursers"), there is no authentication, just a single bit in the header that says "trust me, I authenticated this data".

It doesn't matter if you're using your ISP's servers, 8.8.8.8, 1.1.1.1, or a custom server you set up on Digital Ocean somewhere: an on-path attacker can forge DNSSEC responses to you. It's a ridiculous situation.

Your technical description doesn't really respond to the meat of the critique of DoH, which is that it prevents the technique described in this article - transparent proxying for all traffic of a certain service, in this case DNS.

The problem is that your relationship with your network devices is logically the same as a totalitarian country's/company's relationship with their citizens/users. So any protocol that prevents censorship/surveillance by ISPs also impinges upon bona fide network administrators. Corporate networks have the same dynamic, although there are few tears shed when it becomes harder for them to tamper with users' traffic.

The right answer is to make sure devices that have any Internet access run code you control. The root of the problem here is buying a "smart" TV, hooking it up to the network, and then expecting to tame all of its user-hostile anti-features by policing its communication. The only way to use black box IoT devices is to remove all general Internet access from them, allowing communication only with hosts you do control. For instance I've got a network of tp-link bulbs that are all controlled from a Home Assistant instance, and they never have and never will get a packet out to the larger Internet.

My point was simply that DNS was never an effective measure. Corporate operators don’t need it since they can configure proxies and install endpoint monitoring, but the people building untrustworthy IoT devices or malware can’t be relied upon to cooperate.

I agree that the core problem here is blocking egress entirely – and that’d be a good area for home routers to add UI polish so you could easily allow your TV to hit Samsung.com if there’s an update you need before turning it back off. Unfortunately that’s going to be a losing game for many devices and that really hits at the root cause: we need strong regulation controlling privacy because trying to stop a well-funded company with purely technical measures is almost always a losing game.

Well DNS has been a pragmatic measure for quite some time. I've definitely seen jailbreaks of embedded devices that start off by MITMing DNS to proxy all traffic. DoH is changing that, and I can see how that's annoying.

> hit Samsung.com if there’s an update you need

Why would you need an update? Updates are mainly necessary for security, which you don't need if the device isn't on the Internet. If the device doesn't have all the features you expect out of the box, return it within the return period. There's a small corner case where an update could carry significantly increased functionality, but it seems easier to ad-hoc address that down the line rather than plan for it. Carelessly doing updates is a good way to break your device.

> Unfortunately that’s going to be a losing game for many devices

I don't see how it's a losing game if you play it correctly. Fine grained policing of types of traffic is a losing game, but wholesale denying transit isn't. There is little difference between my network of tp-link bulbs and a local modbus network.

> Why would you need an update? Updates are mainly necessary for security, which you don't need if the device isn't on the Internet.

You've never had a problem which was fixed by an update or something which added support for, say, a new model peripheral? I have, which is why I allowed for the possibility of wanting to do this on the schedule of your choosing but not the default case.

> I don't see how it's a losing game if you play it correctly. Fine grained policing of types of traffic is a losing game, but wholesale denying transit isn't. There is little difference between my network of tp-link bulbs and say a local modbus network.

I was thinking less narrowly than devices which never need to be online. A TV connected to other players can run entirely offline but there are many other things which legitimately need connectivity and there's no good way to prevent that. For example, think about a device like a Chromecast or Fire TV, or those Facebook video chat appliances — people buy those to stream content so the most you can do is force the vendor to send marketing stuff through the same endpoint they use for your content, and that's increasingly hard to filter (think how useful a “it goes to an IP in AWS. Block y/n?” prompt is). That's why I said it'll require a legal fix since a large fraction of the most invasive devices either already do or could trivially be modified to mix other data in with the traffic needed to function.

> You've never had a problem which was fixed by an update or something which added support for, say, a new model peripheral

For embedded devices? No. I can imagine it happening in general, but I don't think I would ever buy into a proprietary ecosystem so hard that there would be peripherals, and newly released ones at that. Still I would be cautious about doing said updates, lest they ruin the device I already have. Like I've got a newer Marantz receiver that works great and hasn't seen the Internet in several years. Even if they developed some new desirable feature, why would I want to let it reflash itself and possibly break, or even just get slower (software bloat)? I'd rather just continue using it as I bought it.

> there are many other things which legitimately need connectivity and there's no good way to prevent that

I sort things into categories. A TV would be in the category of "wtf would you ever hook that up online" - Internet access can only enable anti-features. A Chromecast is a different category - single purpose disposable device that if it turns into shit you just throw it out. Ads and surveillance are part of its price, and if your goal is to avoid them, you should just setup a Kodi box and call it a day.

Legally I don't really see what you're getting at here. I can see a law for my TV category, but leaving it disconnected or pulling the 5G modem will also solve that. How would you even begin to solve the Chromecast problem with a law? Maybe in the EU you could convince them to mandate unbundling ads from a service, but in the US exploiting consumers by shoving ads at them is one of the most popular business models. I don't see that ever changing via the legal system.

Denial is not just a river in Egypt

Follow the money .. watch the actions not the words

You mean actions like the Mozilla engineers advocating for users in standards groups or industry coordinating teams? Or the ones building privacy-oriented features?

You’re going to have to make a more substantial argument to get anywhere with this.

> At this point Mozilla is just the Non-profit Arm of Google, They do what google tells them....

As someone who has used (and still use Firefox) continously since around 2005 it surely feels that way sometimes and some of the decisions I see would make much more sense to me if I knew top management was somehow in Googles pocket.

(That said

1. for my workflows I still consider Firefox the best browser.

2. switching would only make it even easier for Google.

3. I always hope something will change and Firefox will become really really great again or someone will fork it.

> some of the decisions I see would make much more sense to me if I knew top management was somehow in Googles pocket.

Consider.

[0] https://en.wikipedia.org/wiki/Mozilla_Corporation#Google

[1] https://www.theverge.com/2020/8/15/21370020/mozilla-google-f...

They are in google pockets for 95% of revenue. But I think as a browser with a smaller market they join things and change things to be like everyone else.

There was a recent change to facebook picture albums that lazily loads pictures as you scroll. In large albums this grinds to a halt after a few pages and each scroll takes 30 seconds to load the next set of images. Chrome handles this smoothly. For social media I had to switch to chrome.

Firefox is still better for privacy and I use it when I can even though it feels like it's slowing down after each upgrade.

> 3. I always hope something will change and Firefox will become really really great again or someone will fork it.

Abrowser (from trisquel) is what firefox should have been. Best fork out there imo.

Extraordinary claims require extraordinary proof. Of which I presume you have lots to share with the HN community?

For anyone else not in the know, "DoH" means "DNS-over-HTTPS"

Here's an article from November 19, 2020 about the rollout and some of the criticisms/backlash:

https://www.zdnet.com/article/fearing-drama-mozilla-opens-pu...

Can you link to the results of your study?

> Universally? Perhaps in the US? But IoT devices are sold worldwide.

Ugh, the situation is even worst in countries with censorship laws. For example, in my country, all ISP are required to intercept all DNS requests and filter all requests to any blocked domains found in the government block list. At least they're transparent about which sites are blocked though (the list is publicly available to query or download), but the fact that all DNS requests are intercepted causes various technical issues and some ISP are trying to profit from it by redirecting the blocked query to their own ads-laden landing pages. They even went as far as inspecting http host header as well as randomly injecting scripts on unencrypted http requests.

Curious which country is this? A public list of censored sites seems an odd thing to exist

OP talks about somewhere else, but Poland have public registry of domain names that every ISP must block/redirect to Ministry of Finance website in their DNS resolver: https://hazard.mf.gov.pl/ (gambling sites running without proper license) (this is only about ISP resolver, no intercepting going on)

India definitely does this. It is usually porn and piracy etc related

Indonesia. The official block list is available here: https://trustpositif.kominfo.go.id/

ISPs will also enforce other things using DNS. For example when Spectrum was fighting with Netflix, they sent traffic to some overloaded peering connection.

DNS over HTTPS is an abomination sold as snake oil security.

> I'm sure google had only our best interests in mind when unleashing that on us.

Mozilla did more to unleash it than Google. Someone from Mozilla coauthored the RFC for it, and Mozilla had browser support for it first.

The absolute worst version I had of this was in China (and I think Korea as well).

I was troubleshooting an issue where a company's client computers which connected to the company VPN wouldn't have internet access post-connection.

The problem turned out to be that instead of sending a NXDOMAIN they'd return their ad server and then _always_ send back some JS to show ads.

The company network used PAC files (these are a piece of JS with a single FindProxyForURL() function) via an internal-only URI to steer most requests to our proxies, while keeping some internal.

The problem came about when clients would first start up the OS would attempt to access the internal-only URI before the VPN client finished connecting. In a normal network it'd get nothing and life would carry on. With this problematic ISP they'd get something that /should/ have been a PAC file, but because it was some other piece of JS without FindProxyForURL() it wouldn't work as a PAC file and thus the client wouldn't go to the proxy.

The expiration on this piece of JS was set to some absurd amount of time, so when the client would eventually try hitting our PAC file server again (happens every 20 minutes on Windows) it wouldn't get our file because it thought the garbage one from the ISP was newer. And the ISP updated their hijacking JS more frequently than we updated our PAC file.

There were two possible solutions to this. One, routinely touch the PAC file so it's date was newer than whatever the ISP put out. Or two, set up an external A record for the internal PAC server name to keep the hijacking from working. We went with the second.

My ISP in the UK returns NXDOMAIN for records pointing to 10. ips. I had to go through a lengthy process to have that issue fixed with them. Their support had no idea, and was trying to tell me it's nothing to do with them.

In Germany, Telekom used to hijack nxdomain. A court actually found them guilty and since a half year or so they are no longer doing that.

I think that actually supports my point. A customer has a relationship with the ISP and can actually sue them. If people start putting google or cloudflare DNS into lots of devices and apps and those start doing something questionable then you have no recourse and even if they did they'd have to go after dozens of different vendors to change their behavior instead of the single ISP.

It doesn’t matter to the device OEM when the customer demands a refund because the device stopped working, even though the root cause is the ISP hijacking the DNS request and the device OEM has nothing to do with it. Easier to work around this apparently common ISP practice.

Deutsche Telekom hijacked NXDOMAINs for over 10 years, though (maybe even longer). It's not like they did it for a few months and then got sued.

> I'm sure google had only our best interests in mind when unleashing that on us.

Laughably, DoH is from the same gang of A-record squatters that refused to incorporate SRV into HTTP on the (now very evidently spurious) grounds that it could, in some scenarios, require an extra packet, and they couldn't work out how to make it backwards compatible.

Having been comprehensively hijacked by the interests of advertising companies, my view of the HTTP WG has never been lower. And that's a shame because there are some smart people there, tasked with slowly eroding away the last semblance of end-to-end transparency.

> Universally? Perhaps in the US? But IoT devices are sold worldwide.

The ISP provided DNS in South Korea is absolutely terrible too. Setting 1.1.1.1 or 8.8.8.8 always serves far better experience than the ISP's.

As a counter-point, my UK ISP uses Cloudflare 1.1.1.1 last I checked.

What DNS do you hardcode? Google's? Or do you advice the use to set it up himself?

I am very suspicious of the push for https and the like. I feel it is mainly about hiding the payload from me not any third party.

You're right to be suspicious. The DNS-over-HTTPS model favors those who run the servers (because they get exclusive access to monetizable end user name resolution data) and those who control the resolvers.

You might control the resolver on your personal computer (for now). You probably don't control it on your phone. You most likely won't control it on your embedded devices.

The root evil here is that you can't change the root certificates in such devices. Even if you controlled its DNS, the device could still just be programmed to fail if it doesn't reach its analytics/ad/whatever server.

The Amazon Fire TV does this already :(

If it doesn't see internet it just blocks itself and goes to a screen "Oops I have no internet".

So you can forget about watching movies from your local server using the VLC app as well. Ridiculous.

makes it utterly useless in an IPv6+NAT64 environment, incidentally.

You can go to settings > installed application and lunch VLC or anything else directly from there

I'm pretty sure the last time my internet was down I couldn't get into anything but the wifi settings. But I'll try it next time, thanks for the tip!

You can go back from network settings back to settings or something like this. just poke around. Discovered it when comcast went down for 4 days and wanted to run kodi on firetv

Amazon also tends to hide options until you "try" connecting to your network. My device refused to work without internet until I "tried" connecting to my network using an incorrect password. When I did that and the device failed, an option to skip network setup appeared. In small font at the bottom of the screen, of course.

The IKEA Tradfri "smart" lighting gateway will stop responding to commands if it can't phone home to some IKEA server. I noticed this when I changed my router to use NextDNS, which blocked the IKEA lookups. I was ready to return the device as broken until I realized this. I've also had issues with Bang & Olufsen speakers in the past, and inclined to believe it's for the same reasons.

I think it's insane that devices can effectively be bricked if they can't phone home. It's nothing short of waste, and I think environmental legislation should require device manufacturers to supply ways of disabling or overriding these mechanisms such that devices can continue to operate regardless of whether home servers are blocked or otherwise out of reach, e.g. company goes belly up, censorship etc.

> I was ready to return the device as broken until I realized this.

Actually you probably should return such devices as broken.

I’m inclined to agree, frankly.

I did just that with a DJI Mavic.

It was kind of hard to send back a really nice device that I had just opened up and was ready to fly.

Thing is, some companies just use it as a way to fire their customers.

I tend to cut DJI a break, because customer (non-)compliance with no-fly zones is a class-1 existential threat to their business selling consumer drones. Pinging DJI servers to check for altitude restrictions at every power-up cycle is intrusive, but I honestly don't see that they have much choice.

However, they are also playing these sorts of games with other types of devices, where no such justifications exist ( https://www.eevblog.com/forum/eevblab/eevblab-83-dji-pocket-... ) That needs to be answered by returning the product as defective.

When I installed PiHole a few years back I blocked my tradfri gateway from connecting to Ikea's servers and everything kept working! I wonder if something has changed since then? Ikea devices are kind of nice because they don't actually rely on the internet at all and work completely locally (at least, they did a few years back).

All I can say is when I had NextDNS configured on my router it blocked requests to some IKEA domain, possibly smetrics.ikea.com from a cursory search through he logs, and my Tradfri gateway would just straight stop responding to anything at that point. I googled around for a while and found other people having issues with DHCP and QoS with Tradfri gateways, so I made sure it had a static IP set as well as all QoS “features” being disabled, but this didn’t help. It would work at first, for some period of time (30 min maybe?) and then stop responding. Once I saw the blocked DNS lookups I disabled NextDNS on the router and flushed any caches on the router, rebooted everything and it’s worked fine now for a good month or so.

I will admit I haven’t done any further investigation, but simply concluded that the gateway at some point started phoning home and if it didn’t receive a response went into some catatonic state. Maybe I’ll dig deeper at some point, time permitting.

Then you return the device as defective and demand a refund. What you chose to pay for is up to you.

I believe that Google pushed DoH to track your cross-site browsing. TLS hides your URL, and blockers can break adsense tracking and/or any other call-home backlinks.

Using DoH, especially one served by an advert company is just signing up to be their open book.

Chrome didn't change the resolver, though. It just enables DoH if it's on a whitelist of known DoH-capable resolvers. It doesn't send your data to Google unless you already used Google's DNS.

I still prefer DoH giving "exclusive" access to resolvers, because the alternative is sending that data in plaintext for everyone along the path to read?

If your ISP is large enough it is only sent to ISP's name server which probably has everything you need cached, and if it isn't it might blend in with other queries. And your ISP can sniff SNI or guess target domains from target IPs already.

Which is also being worked on via esni (now ECH for encrypted client hello) - https://crbug.com/boringssl/275

> The DNS-over-HTTPS model favors those who run the servers (because they get exclusive access to monetizable end user name resolution data)

Hold up. You are claiming that the fact that DoH prevents DNS requests from being visible in cleartext network traffic is a bad thing?

...what? In a world where the choice is between one party (the DNS provider) having access to my DNS requests and everyone on the network including my DNS provider having access to my DNS requests, I'll choose "DNS provider having exclusive access" every single time.

Hold up. You are claiming that the fact that DoH prevents DNS requests from being visible in cleartext network traffic is a bad thing?

It is when its my network. If they cared about people sniffing they would use DNSSEC, but still use the network DNS server. DNS over HTTPS is just a way for shady companies to hide what they're doing.

DNSSEC provides authenticity, not confidentiality.

Uh, DNSSEC is only signatures, it has no encryption.

DNS over secure channels is actually good - it limits bad behavior by ISPs.

The problem of shitty devices on your private network is a different one.

At this point it is really tough to find devices that don't leak data like a sieve.

I recently bought a car, and could not find one without a cellular modem and microphones. Removing the modem voids the warranty. The period where you can opt out is mostly over.

The Magnuson–Moss Warranty Act of 1975 ensures that’s not the case in the US. I suspect something similar is true in EU.

For the manufacturer to avoid a warranty claim due to a modification or aftermarket part, they must show that the defect was linked to the part or modification.

They might have to show that it was caused by, but in any case, if your paint fails prematurely, they can’t say your warranty is void because you disabled the cell connection.

To get 5 stars by EuroNCAP (and AFAIK EU law demands it too) the car have to call 911/112 automatically if it detect a crash. I doubt you can remove telemetry without also disabling this. If you do the car is illegal.

Illegal to sell by the OEM or illegal to drive? Two very different things.

Because it is probably part of the vehicle type approval, it would be a modification that needs to be authorized by whatever institution does that in your country.

Otherwise it is probably mostly equal to driving without registration. It may not be a felony, but it will be fined.

Given what I see driving around on US roads, there is epsilon-squared chance this would be found let alone fined here. (I’m also not nearly convinced that it’s even illegal in my state and likely not so in any of them, especially given that my car had a dealer service to remove the SIM card as a recall item.)

It definitely wouldn't be illegal to remove a vehicle's cell modem from your own car in the US. The US is very friendly to vehicle modifications compared to much of the world. At a federal level, pretty much the only thing you cannot do to your own vehicle is remove emission equipment. The US does not even have universal requirements for insurance or safety inspections. And even in the states that do have safety inspections, there are typically very few pieces of safety equipment that are required.

Yep. My state has no inspections of any kind. No safety, no emissions, nothing. Insurance is mandatory, but not really verified in any reliable way.

Theoretically illegal perhaps but I'd be very very surprised if this ever actually gets picked up by anyone even on an annual roadworthiness inspection. Maybe during some OEM servicing but then again maybe not.

Interesting point. I wonder if an insurance company can argue the modification made the car lose the safety certification, and in effect the user made the car more dangerous by removing the SIM card, and that it would not pay out for injuries sustained in a crash.

You don't need to remove the modem, just find the SIM card and pull it. It's probably in your armrest console compartment or glovebox.

It's not just shitty devices. By putting it on the HTTP layer which usually is even available to sandboxed applications they made it possible for every single application to bypass your own resolver. Pretty much a trojan horse.

I've hated DoH from the beginning, and not just because shoehorning everything into http is a silly idea, but as I suspected, we now live in a reality where you have to keep an adblock-esque list of DoH servers. Now you have yet another internet arms race.

DNS based blocking was already trivially bypassable before DoH was ever conceived so I don't see how DoH or its use by browsers is at fault for that problem.

As mentioned in TFA, configuring a firewall to redirect DNS traffic from broken or malicious software is also trivial. Or are you talking about hardcoding IP addresses?

There are plenty of other alternatives. The simplest (but least flexible) would be hardcoding an IP of the final server.

Somewhat more sophisticated would be hardcoding an IP to a server with a REST endpoint that returns the real final IP. (Basically just like what DoH does, but without calling it DoH).

Even more sophisticated would be hiding the final IP on some kind of public web service like Twitter or Github.

Note that malware of various sorts has done all of this for years. It used to be reasonably common to get command/control server info via IRC, basically DNS over IRC. DNS blocking has never been an effective network management strategy, and it never will be.

You can/should block connections to IPs which were not returned by DNS recently.

Should be done by your host/gateway/router, not each client app.. Same goes for SSL SNI filtering if that ever gets accepted... router should know where requests are going...

Currently we’re using both Google and Cloudflare’s DNS, providing resiliency against one or the other being unavailable, but I’m looking at potentially using our own resolvers just to reduce the amount of data being exposed to third parties.

So the company you work for will be fighting for my privacy? I'd rather have a PiHole doing that.

These devices do usually have an UI. Why not provide some options to the user? Let him choose among different types and providers. I'd set mine to use the one provided by DHCP or enter the address of my resolver manually.

Have you ever had to do technical support for someone who changed settings they didn't understand? Or maybe someone who decided to "clean up" a bunch of system files that they thought were wasting space?

In my experience, product design is generally done with a well-meaning attempt to protect the average user from themselves. People who can and do manage their own networks in sophisticated ways are, unfortunately, far less common than people who have no idea that their ISP fscks with DNS lookups.

Personally, I'd bury this setting pretty deep in an advanced-usage-only tab and behind a notice SCREAMING about how using these settings is unsupported. And then tell support staff that they are not obligated to support whatever crazy configurations people cook up for their home networks.

I understand not doing it at all. A few people will complain, but the number of people who will refuse to buy a TV because it doesn't play nice with their pihole is almost certainly too small to register on any material financial statement, and attempting to please them will generally run into some other point they are unwilling to budge on. The number of people who screw up an advanced setting they don't understand will show up in support costs.

So if I have a firewall level block to both 8.8.8.8 and 1.1.1.1 your devices will not work for me at all?

If you can block those ip addresses, you can also redirect those ip addresses to other ip addresses.

Closest 2 servers in OpenNIC.

My company makes IoT systems to support low power devices that use mobile (NB-IoT and LTE-M) connectivity and we have some similar problems having to do with mobile networks and APNs.

The way we solve this is that we assume any knowledge the device has about the outside world can become obsolete, so we do we have a two layer approach.

The bottom layer is that we have a set of semi hard-coded fallback values that are likely to work in the forseeable future. Updating these fallbacks requires an over the air firmware upgrade which isn't a terribly big deal since we regularly upgrade firmware over the air. The goal of these values is to make sure we can get the device online and direct it to somewhere where we can trigger firmware updates.

The second layer is that one or more times per day we ping a config server that sends a packet with configuration data to the unit. This is typically API endpoints etc. The configuration data is essentially a prioritized list of resources, so if one won't respond it will go to the next on the list (while still trying to determine if a higher priority resource becomes available).

Last week we got a chance to see how this failed over beautifully as multiple resources were removed and a fleet of devices just adapted as they should. (The shutdown of these resources were planned, but presented a good opportunity to do a fire drill).

DoH will just be the start of another arms race. If you would like to control DoH also user LinuxBender responded to a question I asked about DoH blocklists with these repos in another post. I haven't got around to trying it out yet though.

[1]: https://github.com/bambenek/block-doh

[2]: https://github.com/Sekhan/TheGreatWall

Edit: I am mostly curious to see what devices/applications it will break.

I can attest to this. Story:

Many years ago, a old friend of mine purhcased a new Panasonic Smart TV. It was when "Smart TVs" were just becoming a thing.

I hooked her TV up for her; wiring it into the ATT uVerse modem directly. Other devices worked, but this one did not.

After resetting the modem, factory resetting TV, and making sure the ip address on the TV's menu were displaying properly and matched the router's config (they were), as a young naive tech nerd, I just said:

"Looks like they sold you a dud. Thankfully you kept the receipt!. Either way, you still have warranty to get it replaced."

My friend replied: "Shouldn't I call them first before taking it back?"

I said I didn't think it would help, but go ahead.

About 30 mins laters, she was talking to Panasonic tech support and they asked her to manually enter the DNS entries [I believe it was 75.75.. so Comcrap's], and voila, the TV was online again.

We were very happy it was an easy fix; but that day I deftinely did a LOT of reading on DNS servers.

Till this day, DNS entries are something I always check over when troubleshooting (as well as setting my router to Cloudfare's).

I've been running PiHole-like software on my network for a few years now. A couple of years ago, it would block over 40% of traffic consistently. I never saw ads, and it was nice.

In the last year, blocked traffic has dropped to about 15%, and I'm increasingly getting ads on my phone and Chromecast despite tunneling my traffic through my ad-blocked network and blocking Google's DNS at the network level.

Search up some more blocklists online which are still being updated. It can use the same block lists browsers ad blocks use.

Run 'update your block lists' on your Pihole to make sure the lists are being updated correctly.

I'd think most of IoT interfaces have some form of user-definable configuration. So most of the time it should be possible to make DNS user-configurable (hidden in some "advanced settings" area). Of course, it might lead to some more support calls eventually and requires a little more money (as in paid work) to implement, but you'd be playing nice and have a balanced compromise of default-hardcoded-DNS with an option to appease power users and/or orgs+corps, and you'd also avoid angry-me calling your support hotline and ranting at your people :P

> ... and the response to that is hardcoding a set of known good DNS servers which we can rely on.

Netgear, at one point, also decided that hard-coding an IP address (for NTP, not DNS) was the best solution [0].

[0]: https://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse#Ne...

As opposed to? I thought you had to specify your DNS server as an IP address. How would it resolve itself?

> As opposed to?

As opposed to the IP addresses that your DHCP server told you about.

Sony did the same thing with a Blu-Ray player for NTP, effectively DDoSed that particular box because they also didn’t think to stagger the polling so like clockwork millions of consumer devices would check in at the same time nightly. From what I gathered from the electronics guys a few floors down they eventually resolved that in later revisions, or so I was told.

I worked on an IoT platform for a while that struggled with this. Our workaround was to tell support to tell people to not set automations to run on the quarter, half, 3 quarters or hour mark. Something set to run at 7:14 or 7:16 PM would run every night, something set to run at 7:15 was a crapshoot.

I agree, if it is an embedded device specifically designed to connect to a limited set of services, there is no reason to expect it to follow DNS provided by DHCP.

Conceivably, there's no need to it to even use "real" DNS at all, you could just run a server that responds to queries like "updateserver.ecorp" and save the hassle of even announcing these servers to the public DNS at all.

Also, DoH or not, there are plenty of other ways to ensure that ads get through a DNS filter. For example, a local hosts file could be included in firmware updates and they would just need to make extra effort to ensure that the server IPs didn't change (an elastic IP or load balancer in AWS would be all you need, then it can persist even if the VM has to be deleted).

It's not perfect, but I find blocking all traffic in/out to global dns servers with pfBlockerNG to be an okay way to limit the DoH bypass of simple port 53 blocking. There are still ways around it, but it has caught a lot of other interesting traffic (snmp, ntp, etc.) leaving my network and that a simple port 53 block misses.

This is the list I use: https://public-dns.info/nameservers-all.txt

Out of curiosity, how do you handle situation where outbound tcp/udp port 53 is blocked? Do you fall back to dhcp supplied dns server?

> and occasionally just highjack every single query to send you to a page saying you’re approaching your bandwidth quota

It boggles my mind it's still a problem in some countries. Last time I saw this kind of notice it was in the nineties, and on a web server, definitely not on a client endpoint. People have had unlimited bandwidth for at least a decade now.

This has regressed. In the NetZero days you would be limited to some hours per month depending on your plan. Then came unlimited cable. And now we are back: my residential AT&T DSL connection is limited to 1TB per month in the Bay Area

Whoa. That would give you one day of laundry washing per month in the IoT age.

https://old.reddit.com/r/homeautomation/comments/k72lzq/sowh...

Comcast in my area has a 1TB/month limit. When I used their service I hit that limit nearly every month.

How do you deal with NAT64? It's a real thing with some consumer ISPs.

I can see myself using it as well, maybe even just to see which devices don't use DNS.

Here's another idea: generate a unique IPv6 address per DNS request, route them to the correct destination, filter other IPs. Not really scalable, but usable on small networks/VLANs.

Currently, we don’t. The devices don’t support IPv6, so it’s not a problem we’ve had to deal with so far.

So you'll only start working on the problem when IPv6-aware devices start failing?

Why wait?

Not the OP, but you should also be reminded that many of these limitations are brought out by cost-cutting measures that we didn't have control of and could be used to fire us immediately, and even when IPv6 is implemented most IoT backend servers only operate over IPv4 anyway. Another one that you might be angry about is only including 2.4GHz-band WiFi even though 5GHz WiFi is already ubiquitous (although only including 5GHz WiFi is much more questionable ;).

Exactly this, we’re a small team and have to pick our battles. In practice very few people have IPv6 only connections, and even fewer of the people who buy our products.

and then there are providers/networks who straight up drop outgoing DNS queries, like the university I went to (including dormitory network).

What a terrible state of affairs we're in now that you can't trust the network's own DNS. Please consider making it an advanced option to switch back on the DHCP provided address.

> Please consider

When you are contracted to build something to spec and you don’t build to spec, you don’t make any money. If you push back on the requested spec, the client will leave and go to another company thus leaving you without any money.

What I’m trying to say unless you are writing the spec, you usually have no chance to change anything

I think that’s a gross overstatement about ISP DNS.

Certainly in Europe I’ve not seen anything even close to that.

I’d love to see a good country-by-country survey on ISP DNS to see just how common this manipulation is.

That's not really a consumer-friendly solution. It's also borderline illegal in the EU IMHO (if you hardcode DNS servers in the USA, for example).

The toxic ISP issue isn't so dramatic where I live, it used to be worse 15-20 years ago. But the solution has always been: if the ISP is messing with you, you just buy your own router and configure it for your network, with a VPN tunnel if necessary. Ignoring DHCP makes this unnecessarily harder.

How come you don't specify that it be configurable?

Because no one has asked for it. The people who care are redirecting port 53, which the devices cope fine with, because if you’re doing that you probably have same resolvers upstream. The people who don’t care are never going to decide they’d like to change the DNS resolvers.

I've recently setup AdGuard Home on my Raspberry Pi, and noted that requests made from an Android phone weren't going through the DNS server at all. Turns out it uses a default IPv6 DNS first (which I found no method to modify) and then falls back to the manually entered IPv4 DNS server.

There is no justification for bypassing users network settings, and especially doing it for your own convenience. At very least it should be opt in. If you want to violate customer privacy to save money on support, then I don't know why this is worth even discussing.

This sounds like a game of cat-and-mouse. Lists of 'known' DoH DNS servers will be collected, and PiHole-esque blocking will take shape via more advanced filtering.

I’m responsible for a bunch of IoT hardware, and every firmware spec I write includes a note on not using the DNS servers provided via DHCP.

And that along with DoH is contributing to making my life a pain in the butt. How exactly do you folks who avoid our DHCP's DNS expect us to comply with legal filtering requirements? Also, what happens when your hard coded DNS servers are shutdown?

> How exactly do you folks who avoid our DHCP's DNS expect us to comply with legal filtering requirements?

I would argue that this is more a problem with the legal requirements than with the equipment - the law(maker) has expectations you can’t reasonably fulfill.

Doesn’t make your situation any better of course, the law is the law even when it’s impossible

Oh, we can fulfill them. We'll resort to TLS inspection and force you to trust our CA on your device if you want to continue accessing our corporate network. And now we get to see (almost) everything again, like in the "good old days," not just your DNS queries.

Clear text DNS is the ultimate compromise, a gentleman's agreement if you want, that benefits everyone. We can see just enough to filter what we are required to by law on a best-effort basis, but we never see what you are actually doing thanks to the prevalence of TLS. DoH just broke that agreement.

It's a sad example of how a privacy solution like DoH will eventually result in less privacy, at least in some environments. And I'm not even considering how DoH will be the excuse for totalitarian regimes to up their surveillance antics.

Yeah. The pre-DoH world was good for both. I could say its all filtered for the kids on the locked down machines and the adults who knew something about technology could get on with their lives. Now, we are entering a world where we are going to end up locking down everyone. Good job.

I'm damn sure once I have to do the trusted CA path that someone is going to sell a deep packet inspection solution and present it at some conference where someone in charge will hear about it and then it will be off to the races.

Its not a law problem. Its an expectation that technology isn't as random or stupid that it cannot keep a headstart kid from going to PornHub because some tech folks don't trust their ISP.

I think the classic "the network treats censorship as damage and routes around it" applies here.

Designing a device to connect to something over the internet even if the network it's connected to behaves strangely isn't random or stupid; it's just in conflict with your goals. Incidentally, last time I ran into a network with legally mandated filtering, I checked whether a google image search for "tits" worked. It did.

"behaves strangely isn't random or stupid"

Nope, it behaves fine. The owner of the network is serving under age kids. Push too far and its white lists only and block all other IP and I'm sure we'll get deep packet inspection forced on us. Some folks have serious problems with Google Images Search, but you can actually deal with that.

I would also say anyone hard coding DNS into a device is just absolutely unprofessional. Its basically a red flag that any filtering the owner of the network doesn't matter to them.

From the perspective of the device maker, a network causing a DNS lookup to return something other than an accurate result is behaving strangely. That may keep a device from working, so the device maker guards against it. A quick scroll through this thread reveals good reasons for device makers to do this, mostly ISPs behaving badly.

I'm generally inclined to think an "always use this manually-configured DNS" option is desirable in that situation. Of course, many devices may have a financial incentive (ads) to actively resist the network owner's attempts at filtering.

Filtering is inherently adversarial, and I expect a reasonably sophisticated user on your network could find a way to access some proscribed content. I also expect the users of concern on your network are under five years old and that most of them lack advanced knowledge of networking. Is there an established standard for what qualifies as a reliable-enough filter?

What kind of legal requirements are these, which country?

Well then just use a hardcoded one by default but allow the user to set one manually and then use that.

> ISP provided DNS is almost universally terrible

Yet that is what internet actually is for a lot of people...

> Just be aware DNS over HTTPS is a thing now

As is DoT :)

Where do you want them to send it to?

Wow, thanks for breaking established standards then...

Go beat your ISP first.

> DNS over HTTPS is a thing now

I'm one of those "idiots" whose been whistling in the wind against encryption of everything and all kinds of security lockdowns and it's because of this sort of thing. The theoretical threat of someone sniffing my traffic is just not a concern to me compared to the very real and increasing threat of handing all control of my computing to centralized user-hostile powers.

> In the face of that sort of behaviour you inevitably end up with technical support having to field angry customers who just don’t get that it’s not our fault the hardware isn’t working

If your hardware isn't resilient to network failures then it absolutely is your fault.

I wonder who's the first sponsor for that thing, dns-over-https...

Reply on news.ycombinator.com