DoH Privacy Enhancement: Do Not Set the User-Agent Header for DoH Requests
bugzilla.mozilla.org
bugzilla.mozilla.org
Today I discovered that I couldn't just kill the disqus.com requests with my usual method by sink-holing them in /etc/hosts because for some reason firefox ignored the /etc/hosts file. While Chrome seemed to work fine and local dig requests also returned
disqus.com. 0 IN A 0.0.0.0
I was able to override this in the about:config as described in [1].
I really don't like how mozilla ignors what is defined in /etc/nsswitsch.conf because this breaks all the blacklisting rules a user had set up themselves. At least they should have produced a huge warning telling users that this ignores any local blacklists that are in place.
[1] https://ungleich.ch/en-us/cms/blog/2019/09/11/turn-off-doh-f...
Its not just here, its unfortunately common with firefox. I still cant believe that firefox just silently disables addons if they are no longer supported instead of giving a giant colorful warning.
I’m sure there is some smarter way to do this, but when you’re handling network configuration at the application level, you’re going to miss local changes (including blacklists).
Maybe he best way to handle this would be to change the system level resolver to use DoH?
assuming you means the user-space applications, I agree.
I am not against DoH itself. But the assumption being made here by mozilla is that all users trust the US or US based corporations by default. This might be the case for some people (including myself: in some cases, but not as the default case).
This could have (imo) solved with a couple UI tweaks that educate the user about the implications. Having been protected by my browser extension ad-blockers from most sites it still rendered my second layer of defense (/etc/hosts[1]) useless with this new default. The worst thing is I was unaware of /etc/hosts being useless for several months.
I wish mozilla would make it a core policy that anytime a change makes a trust assumption on behalf of the user in all countries[1] that this is communicated not just in the changelog or blogs but as part of a guided tour, or other visual aids which really spell that trust assumption out.
> Maybe he best way to handle this would be to change the system level resolver to use DoH?
agree, but I also see the difficulty on making the FOSS world agree very quickly on how this is supposed to be implemented as some kind of default (could take a decade or longer, while just pushing it under the radar gives immediate results - that they need to form agreements with various providers, e.g. cloudflare)
[1] they seem to be rolling out different features anyway depending on which jurisdiction you are, as shown in the case of "Mozilla: No plans to enable DNS-over-HTTPS by default in the UK" (which purpose isn't to give freedom[TM] but to comply with the UK nanny-law to ban free speech (and porn, but mostly free-speech)). https://www.zdnet.com/article/mozilla-no-plans-to-enable-dns...
The files module on your system could read /etc/dns or the blarg module could read /etc/hosts but in JSON format.
The thing you're annoyed about is that Firefox with DoH enabled doesn't use NSS at all when performing requests but there is absolutely no way that Firefox can do both -- it either can send requests through the NSS gauntlet and use whatever it returns or not use it at all -- "respecting /etc/nsswitch.conf" is impossible to do correctly.
Linux systems don't provide the right facilities to supplement application level DNS in the way you want. If you want to do DNS in your app you must ignore libc DNS.
https://support.mozilla.org/en-US/kb/canary-domain-use-appli...
It's good to know that there is a method but it looks a bit strange imo (see this article[2]):
>> Now this seems a bit weird: DoH to a public resolver seemed intended to defeat an untrusted local resolver, but an untrusted local resolver can disable the use of DoH to the public resolver in this fashion. In fact, if you have content filtering in place on your DNS resolver, Firefox will also use that as a metric to decide whether or not to enable DoH to the default public resolver.
>> In other words: if the browser detects that one of the things that DoH would help protect against (censorship / content filtering) is in place, then it will not enable DoH. ?? Well, for you as the network provider, that's good news -- you remain in charge, as so often, it boils down to having to trust the things you have to trust -- if you (the end user) don't trust your network provider, you can't use their services to bootstrap a trusted environment.
[1] just add the line for the canary to /etc/dnsmasq.conf:
address=/use-application-dns.net/
Plus it is useless, may allow hacky discrimination, and wastes bytes.
'Thou shalt provide exactly this, no more and no less'
Depends on whether we want to consider later 'improvements' to be silent extensions or an entirely new protocol version. Given how easily abused HTTP headers have been in the past, might be worth the inertia to limit extensibility by design in order to limit MITM tracking.
But MITM middleboxes will of course start to rely on that...
I suppose that if TLS is broken for this part of the system you're rather screwed anyway, but defence in depth is always worth considering.
Meaning if you connect to a public WiFi network, many will only allow captive DNS and block other protocols including DoT. DoH is difficult to distinguish from other HTTPS traffic and is more likely to successfully resolve (since they cannot block HTTPS and still be useful).
Therefore you've created a secure, flexible, and reliable DNS replacement with only mild technical downsides on modern hardware (a lot of the complaints are ideological or relate to specific incompatibilities with existing solutions like the lack of HOSTS file support in the initial offerings).
So I guess, everything I just said is "the point." Rather than trying to use every single possible combination of HTTP headers.
Microsoft has already committed to implement DoH in Windows itself. Browsers just need to stop trying to be their own DNS clients.
That's an anti-feature, something one shouldn't even call internet access. Engineering entire protocol stacks around it sounds like enshrining it as if it were good practice.
It's reinforcing an inadequate equilibrium.
Once absolutely everything is running through opaque https requests we might see middleboxes go away since they aren't able to do anything anymore.
Actually solving the problem is always better than enshrining it by providing an ugly workaround. Even if it takes more time.
Imagine how quickly IPv6 would have been adopted if NAT didn't exist. DoH is kinda like that, a cludge that ultimately hinders progress.
Are you referring to HTTPS-intercepting proxies? (i.e., if you want to use that network, you'd also have to install a custom root CA cert)
On a typical "coffee shop" wifi (where users are not expected to install custom certs), shouldn't DoT and DoH be virtually indistinguishable to the network as they are both shielded by TLS?
No, I am referring to what I said. Public WiFi networks only allow their own DNS resolver and block all others and most other non-HTTP/HTTPS network traffic. This breaks DoT and even unencrypted UDP DNS to your choice of resolver (outside of the WiFi's captured DNS/Middlebox DNS).
A lot of free public WiFi today break DoT and work with DoH. DoH is indistinguishable from other HTTPS traffic, DoT and DoH are highly distinguishable from one another. DoT is trivial to block (often without even targeting it specifically).
One (semi-ideological) concern I believe is valid is that this makes it increasingly complex to understand which HTTP features can or cannot be used somewhere - and when writing a naive implementation, there is the risk of accidentally enabling features that are supposed to be disabled.
E.g., when WebSocket was developed, there was some discussion how much HTTP should be allowed in the initial handshake - specifically whether or not redirects should be followed: Formally, following redirects would seem reasonable as, until the handshake is completed, you are speaking "ordinary" HTTP, which redirects are a part of. Practically, though, it didn't make a lot of sense as the requirement would have added a lot of complexity to clients - even though no there was not even demand for the feature in the first place. In the end, I believe it was decided that the handshake is actually a tightly limited subset of HTTP and any server response except 101 is invalid.
However, last time I checked, browsers in the wild do send cookies and user-agent strings in the handshake, even though you could argue those have nothing to do with WebSocket either.
So with DoH, we now have another "dialect" of HTTP with its own subset of features: On DoH requests, the User-Agent header must be omitted and cookies must never be stored. I can already imagine future privacy exploits if some DoH implementations just used a standard HTTP library and e.g. forgot to disable cookies.
I believe using an actual custom protocol would make it a lot easier to understand which features are and are not available.
Considering that most users would keep default setting and there are two browsers with two default DoH servers, you could just block few IP addresses and effectively block DoH for >95% users.
without the user-agent header, browsers would have been forced to standardize earlier.
Maybe runtime browser-side feature detection can be fast enough. I haven't benchmarked.
The other tricky thing that User-Agent strings are good for is working around browser bugs. Every once in a while a browser will ship with a totally broken language feature that's essentially impossible to detect at runtime.
A particularly bad example was from an old version of Edge, which had a JIT bug where class constructors could – in very particular circumstances – return the class object itself rather than an instance of the class. In that situation, if you want your site to work you have to detect that version of Edge and treat it like IE: send it the bigger, more compatible blob rather than code for modern browsers.
Agent-Capabilities: 1; 97464bde5d94a54f6e199309489a4b60
You might say that it is better to do run-time checks on the client for features you want, and have the client then load workarounds if necessary for missing features, and that indeed is a good way to go in many cases (maybe even most cases).
That method, though, only works going forward. Sometimes I want to know what capabilities past visitors to my site have had. For example, I'm going to be redoing some pages at work and want to know if I can require certain CSS and JavaScript and HTML features.
I could make a list of the features I want to use, add some JavaScript to those pages to test for those, and send the results back to the server...and then I'd have to wait weeks or months or longer to get a good idea of what the consequences would be of requiring those features.
With User-Agent strings, all I had to do was pull up the site logs. Our shopping cart logs the User-Agent string, and page flow through the cart. It was an easy matter to pull up the last 18 months of those logs, and write a script to find all the successful orders and make a table showing how many there were for each browser. (I limited it to successful orders, because those would almost certainly all be real people in regular browsers that are reasonably honest about the User-Agents). I could then look up the features I was interested in on https://caniuse.com/ and see which of the features were not on browsers that we still get significant traffic through.
Dropping User-Agent without adding something to allow that kind of analysis, such as Agent-Capabilities, would be very annoying.
Let's throw down this challenge to the browser companies: MAKE YOUR STUFF SOLID ENOUGH AND STANDARD ENOUGH THAT WE DON'T NEED User-Agent.
Will the endpoints for other TRRs have special subdomains.
Perhaps another "privacy enhancement" would be to avoid using a browser-specific subdomain.
1. ESNI is still experimental and if I am not mistaken it is disabled by default in Firefox.
Didn't someone post his experience getting more expensive flights because he used a Mac?
Yes! Please, take UA out.
The user agent isn't very important. What we should start being worried about is that ISPs are now offering API access that maps your IP address to your phone number and demographic information. (Verizon does this right now, but you can opt out in your account settings.)
Maybe all those VPN providers have a point. Delete the user agent, share an IP with thousands of people, and maybe you can browse in peace.
detection based on payload size alone.
Any implementations with a random padding ballast header?
It is not so straightforward as always being an improvement. For example, a user can be uniquely identified through session resumption session ID whereas with plaintext UDP behind a firewall with many others doing lookups unique identification of a user can be close to impossible.