Hacker News Reader

Comment by colmmacc | Hacker News Reader

Disclaimer: I work at AWS, on Amazon Linux and our VPN products; those aren't impacted by this issue.

The attack that the researchers describe is very impressive, and using traffic analysis and error messages to find the details of an open TCP connection is extremely clever.

Unfortunately a similar approach can be used even more practically to target DNS on the VPN: https://www.openwall.com/lists/oss-security/2019/12/05/3

Encrypted DNS queries and replies can be profiled by traffic analysis, and the reply "paused", making it easier to ensure that a DNS spoofing attempt will succeed. This is a good reminder that cryptographic protections are best done end to end; DNSSEC does not help with this attack, because it does not protect traffic between the stub resolver and the resolver. It's also a good reminder that traffic analysis is still the most effective threat against network encryption.

View on HN

Hi Colm, were still working on a response to your email, but we appreciate the insight you provided and look forward to our conversation.

This disclosure only deals with the specific threat against active TCP connections, but there are more coming.

An actual high-quality DNSSEC implementation would protect against this. Using an untrusted stub resolver is a mistake; end-user OSes should verify DNSSEC responses directly.

Applications are the real "end" in end to end, and TLS is already e2e. Good TLS does thwart this attack; an attacker still can't generate a valid certificate.

It's taken a lot of focus and attention to make TLS reliable enough to make it a default in browsers, and DNSSEC is not particularly close. DNSSEC supports out-of-date cryptography and has no negotiation mechanisms to avoid them, which makes it very hard to use as a default e2e security protocol. It also doesn't encrypt anything.

Yes, it is correct that you cannot easily hijack connections that use TLS, but you can make inferences about active connections that use TLS. This is still fairly devastating for a large number of vulnerable users that rely on VPNs in nations with authoritarian information controls.

e2e TLS guarantees that you received a DNS answer from the DNS server you requested it from, no one was able to see the answer in transport and no one was able to change the answer in transport.

It doesn't validate that the answer provided was correct according to the owner of the domain your query was regarding. DNSSEC tries to provide this validation. Roughly it is similar to signing a message with PGP.

You can argue that DNSSEC is not up to today's crypto standards and no longer trustworthy for its intended purpose, but without some protocol which provides the guarantees DNSSEC tries too e2e encryption is a 1/2 solution to DNS security.

The parent commenter definitely understands what DNSSEC is.

His point is that DNSSEC doesn't work the way you appear to think it does. Conceptually, it's meant to prove that a DNS record in a response is actually a record created by the owner of the zone. But in practice, the cryptographic signature isn't valuable to stub resolvers on the end system; instead, end systems trust their "DNS servers" (their resolving cache server) to perform DNSSEC validation for them. The success of that validation is conveyed in a single "AD" bit in the header of the DNS response.

Since this attack happens from a vantage point in between the end system and the resolving cache, DNSSEC isn't in play; the attacker will simply set the AD bit in their responses.

What end-user OS does this?

None.

Fedora has a prototype:

https://fedoraproject.org/wiki/Changes/Default_Local_DNS_Res...

I ran it for a while. It was problematic from a UI perspective (nothing to do with DNSSEC per se — it was just immature), and it wasn’t fantastic with captive portals.

Also, it noticed that I visited an internet connection where the ISP was hijacking google.com. So it worked!

So by "actual high quality DNSSEC implementation", you mean one that exists only in prototype form for desktop Linux users.

I’m saying this should be done, not that it is done. The AD bit is garbage.

DoH actually addresses this attack, directly, regardless of whether the affected name is in a signed zone (most queried names aren't!), and it actually works on mainstream operating systems. Why would anyone waste even a moment with DNSSEC?

DoH moves the attack point to a different place - either between the recursive resolver and the authoritative servers, or (with DNSSEC) to the recursive resolver host itself. This is certainly a step up for security against malicious neighbors, but doesn't address malicious infrastructure.

DoH with client-side DNSSEC validation would be nice. I don’t know whether the protocol supports this cleanly.

DoH moves the attack we're talking about to a place where the attack can't be carried out.

Modifying DNS results can certainly be carried out ahead of or at the recursive resolver.

DoH without DNSSEC means the results are modifiable by authoritative servers, recursive resolver, and any of the links between them. DNSSEC with a local recursive resolver prevents this. (Of course if you want privacy from your ISP, you then have to tunnel the recursive lookups through VPN(s)).

If DoH is the best one can do, then do DoH. But it's wrong to shout down DNSSEC until an alternative method for authentication comes along.

The same argument applies to DNSSEC: if the DNSSEC authority server is compromised, it can fake records, the same way your DNS recursor can fake records if it's compromised. If you do DoH across the path, at each step, DNSSEC does literally nothing, rather than practically nothing.

Regardless: for the attack that we're discussing on this thread, DNSSEC is not relevant; or rather, this particular attack is a good illustration of how irrelevant DNSSEC is to realistic attack scenarios.

The DoH protocol is not designed to run “at each stage of the path.”

Same things like DNScurve do that but somehow nobody got excited about that.

DoT is already being used [https://engineering.fb.com/security/dns-over-tls/]. The value proposition of DoH is to "blend" in the HTTPS traffic.

Not that DNSSEC is useless, but we should worry about tree validation AFTER having encoded every stage of the path.

I wasn't aware that DoH was currently being touted for authoritative servers as well. So, comparing the best possible adoptions of each protocol to each other on their unique strengths:

1. DNSSEC (with clients doing full recursive lookups, ofc) is able to have zones signed off of the authoritative server, meaning a compromise of the authoritative server isn't an attack point.

2. With the client performing full recursive lookups themselves, DoH provides partial query privacy.

3. With the client delegating to a trusted third party (eg Mozilla), DoH provides full query privacy modulo that trusted third party (the TTP can break both privacy and integrity)

Comparing (2) to (1) I do see your argument much better now.

However, DNSSEC still has the property of E2E validation that can be used for more than what current software does. One could write a resolver that shipped records laterally between peers to extend its privacy properties beyond either DoH setup. Adopting this wouldn't require all the authoritative servers to get on board, just a community of end-users. This is where my argument is coming from, especially with DoH meaning (3) to most users and the general course of what happens to trusted third parties.

In reality, DNSSEC signers are online, and need to be, because the way you keep DNSSEC from revealing the full contents of your zone is to dynamically inject signed "whitelies" responses to queries. That, and then just the dramatic extra ops burden of keeping offline signing keys, means that the realistic real-world deployment of DNSSEC is, like that of TLS, based on online keys.

DNSSEC does not have E2E validation. DNSSEC is validated between the recursive cache server and the authority server. The protocol explicitly delegates validation away from endpoints with the AD bit. You can run a full resolver on your laptop the same way you can run a full-feed defaultless BGP4 peer on your laptop; it'll "work fine", but that's simply not how it's deployed in reality.

Another dramatic difference between DNSSEC and DoH is that DoH works whether or not zones sign (a tiny portion of the zones people actually make queries on are actually signed). Nobody needs "permission" to protect their queries with DoH, but everyone has to cooperate to make DNSSEC work.

Since the value DNSSEC provides is made more marginal with each passing quarter --- because of MTA-STS, because of multi-perspective CA DNS validation, because of DoH, because of LetsEncrypt making X.509 certificates free, because of certificate transparency --- the rationale for its continued deployment has become extremely thin. It's 1990s cryptography --- queries aren't even encrypted! --- that people are advocating we forklift into the Internet to solve... it's hard to see what problem?

A better plan would be to take DNSSEC back to the drawing board and come up with a modern alternative to it. DNSSEC itself is a failed protocol.

> DNSSEC does not have E2E validation

My comment compared the strengths of each protocol with the fairest interpretation of each. Your judgement here does not do this - it's obvious that a stub resolver relying on a third party to do verification is braindead, and clients doing a full recursive lookup is the correct answer. How clients are currently setup has little bearing on discussion of a protocol's properties.

> Nobody needs "permission" to protect their queries with DoH

This is also false if you compare the protocols on equal footing - if the authoritative servers are not speaking DoH/DoT, then queries are only partially protected. In order to do "DoH across the path" as you said above, cooperation is needed.

> A better plan would be to take DNSSEC back to the drawing board and come up with a modern alternative to it

Sure, but this becomes harder when things like DoH are touted as being a sufficient replacement...

It is not at all obvious that stub resolvers are "braindead" and the "correct answer" is full recursive lookups on the desktop. One way you know this is that no mainstream operating system works this way; another way you know it is that the DNSSEC designers explicitly took stub resolvers into account; yet another is that full recursive lookups eliminates caching, which the DNS depends thoroughly on.

I'm not interested in a debate about a fictitious version of DNS that you make up as the discussion progresses. I think we can probably just wrap up here.

You've written off the whole protocol because of 1990's cryptography. I think it's reasonable to just ignore the specific parts that don't require cooperation to change.

I would be interested in any stats that the DNS system actually "relies" on having clients share caches. Firing out UDP packets is a heck of a lot easier than a TCP/TLS session, and modern websites take the latter for granted for every single user.

If clients sharing a cache is actually important, that's actually a negative point for DoH/DoT as increased resource utilization means that major authoritative servers will be tempted to form a clique with major recursive resolvers, rather than everyone being able to query the zones directly.

DoH doesn’t even attempt to address DNS tree validation, which is what DNSSEC does.

The two are complimentary.

They're complimentary in that one of them does something useful and directly addresses a real ongoing threat, and the other doesn't. It's true, those aren't the same things.

I run my own DNSSEC-validating DNS server for my whole local network (also on the road with DNS over TLS on my phone) using Pi-Hole (DHCP service + blocking ads) and unbound (local recursive DNS resolver that validates DNSSEC). I do DoT using nginx. So basically any Unix can run a resolver and virutally any OS can profit from it !

Why is this supposed to be interesting? You could run defaultless BGP4 on your Linux box if you wanted to badly enough. The question is "what end-user operating system already does this?".

I think some people assumed you were being imprecise in wording your question, because when there's a comment talking about what OSes don't do but should do, "What OS is able to do this?" makes more sense than "What OS already does this by default?".

So "Why is this supposed to be interesting?" is a bit rude for someone that was trying to answer a reasonable interpretation of your question.

You might be right. I really am frustrated by the mindset that says that because a Linux system administrator could get some feature to work, that means it's available to mainstream users; that logic really does suggest that you can do virtually anything on a desktop computer, which is technically correct but kind of negates the whole premise of the question.

But if I came off as personally rude, I apologize and will try harder not to do that.

Remember when Dropbox was first announced? Reminds me of that.

Everyone conveniently forgets that the dropbox comment had some very legitimate points, and the part that gets quoted and mocked was specifically about the benefit to linux users.

It's better not to bring it up.

https://news.ycombinator.com/item?id=9224

That comment there? Never seen it quoted, always in context.

It’s being brought up because it’s very a propos. Someone, somewhere said x problem needs a better solution, and someone else replied that it can be done on one specific Linux distro with a specific kernel version or configuration.

What am I missing here?

> Someone, somewhere said x problem needs a better solution, and someone else replied that it can be done on one specific Linux distro with a specific kernel version or configuration.

Hang on, what is 'it' in this sentence?

Because the dropbox comment was skeptical of the need for a "better solution". In this interpretation, 'it' is an explanation of how to solve the problem the old-fashioned way.

But the comment we're replying to agrees that we need the "better solution" of DNSSEC, and is suggesting a way to deploy the "better solution". In this interpretation, 'it' is the "better solution".

Those two ways of interpreting 'it' are opposites. The two comments are doing very different things.

Opposites are black and white, something and nothing. In conversation, you'll find that analogies are never the exact event they're being compared to, but something parallel enough to evoke a familiar emotion or memory. This isn't math, but you're approaching it as such.

The overlap here is simple. The question "What end-user OS does this?" was answered with...well, gibberish...and tptacek's reply resonated with me and reminded me of the dropbox comment. I think that's about as well as I'll ever be able to explain it. The fact that the two agree that there's a better solution is one facet of the discussion taken out of context, that doesn't even factor into my response or this whole spiel.

What I'd really like to ask though is what your motivation is for mounting such a defense. I seriously doubt it has to do with it having "very legitimate points" or you would've brought them up by now. Also, re-reading that thread, the OP ends up agreeing with everything except that it shouldn't be marketed as a USB replacement.

I completely stand by my decision to reference that comment in jest and will bring it up again!

> In conversation, you'll find that analogies are never the exact event they're being compared to, but something parallel enough to evoke a familiar emotion or memory. This isn't math, but you're approaching it as such.

I'm saying that the comments are barely similar at all. Yes, they both suggest how to do something on linux. That's the only similarity.

> answered with...well, gibberish...and tptacek's reply resonated with me and reminded me of the dropbox comment

But the dropbox comment isn't gibberish...

> What I'd really like to ask though is what your motivation is for mounting such a defense.

Because it annoys me when people misrepresent the comment as a fool who couldn't see the value of Dropbox, too attached to some overly-complex system not applicable to normal users. He clearly did see the value of Dropbox. He said right there that it was "very good" for Windows users. And the mocked point was only one out of three.

> I seriously doubt it has to do with it having "very legitimate points" or you would've brought them up by now.

I didn't bring them up because I thought it was obvious, and it would be a waste of time to list them. But fine, I'll do it.

The post has three points:

The point about cobbling something yourself is a bad point. But it was very strictly limited in scope.

The point about not replacing USB drives is both correct and important.

The point about "not being viral" is agreed to be correct by dhouston, because the viral parts were secret at that time.

So that's two good points out of three.

I'm not an OpenBSD user and I'm not claiming OpenBSD represents any kind of end-user OS, but: my understanding is that the aim of their unwind[1] tool is to do this. And as far as I know, sending DNS UDP/TCP packets is more or less portable to anything with BSD sockets, although I don't know that anyone has tried running it anywhere except OpenBSD. So to the extent you consider Linux, MacOS, or even Windows an end-user OS, and to the extent that the tool is portable and could be configured... eh, there's the pieces of something there.

Anyway, I think DNSSEC is stupid, so I'm not advocating for using this tool or enabling it in OS's as default policy.

[1]: https://github.com/openbsd/src/tree/master/sbin/unwind

I wonder whether this attack also works when the VPN device and VPNed is put in a separate network namespace, which would have its own routing table.

I was thinking the same thing as I read the on the description. I see no reason that a separate VPN namespace would be vulnerable to this attack. The compromised device would be able to spoof packets with whatever IPs it wanted, but they would never be received in a context where the tunnel interface would be directly accessible and therefore the device would never see a response from that address, even if correctly guessed and probed.

This seems like the most popular solution, but there are concerns that it may disrupt routing, and also does not work on -some- systems.

Using DNSCrypt while using a VPN has become a common practice. It should prevent that specific issue.

Off-topic but "service.example.com" woukd have been appropriate to use in the example in your mail. It's RFC-complianr. :-)

> Encrypted DNS queries and replies can be profiled by traffic analysis, and the reply "paused", making it easier to ensure that a DNS spoofing attempt will succeed

"Encrypted" only in the sense of encrypted by the VPN which you've sidestepped. An increasing fraction of DNS traffic from real clients will have been encrypted under DPRIVE, and so sidestepping the VPN doesn't help you spoof that. In this case the connection to a "real" resolver has TLS guarantees about integrity / authenticity and so you can piggyback DNSSEC assurances on top of that if that's how you choose to do things.

By the way the error side channel is very of this moment - something that for example very much concerned developers of protocols like QUIC and TLS 1.3, the reaction reminds me of an anonymous criticism of Unix using a car analogy from the UNIX-HATERS handbook:

> ... If the driver makes a mistake, a giant “?” lights up in the center of the dashboard. “The experienced driver,” says Thompson, “will usually know what’s wrong.”

During TLS 1.3 development more than once contributors expressed a desire for a feature - maybe an optional feature - to get more verbose or detailed errors than those provided already by the protocol in hopes that it would ease debugging. Old hands correctly urged caution, the giant ? is one less weapon for bad guys.

If you send garbage to a WireGuard endpoint as I understand it nothing at all happens. QUIC is a little less tight-lipped, but still endeavours to ensure that a third party can't distinguish anything useful by injecting garbage and looking at what happens next.

The "?" story is about the error handling of the ed line editor. In the event of erroneous input, it just prints "?". The original reason for this, is that Ken Thompson had better things to do than add code for nice error messages – initially the only people using it was himself and immediate colleagues who could just ask for his help in person if they got stuck. Also, the minimalist error handling was valuable in an era of extremely constrained resources (very limited memory, 300 baud modem connections, etc) – a smaller editor could be used to edit larger files, and briefer error messages made editing over slow connections faster. As soon as there was a need for a more user-friendly editor, people built new ones and left ed as it was, and those new editors soon had much better error handling.

To associate its spartan error handling with security is a bit of a retcon. Interesting analogy, but the minimalistic error handling of ed was not motivated by security concerns and had basically zero security benefits (unless you count poor usability as a form of security through obscurity.)

Ah, I have never tried to use ed interactively (I've worked with sed and I mostly edit in vi or its relatives but even the oldest machine I own has a video terminal so I don't need ed) but that makes sense as the source of the '?' error.

I'm aware that the story isn't about security, maybe that didn't come through in my post, it's just that the story always comes to mind when talking about how verbose to make error messages. As you illustrate, there are real trade offs, just today's trade-offs are different than in Ken's PDP programming days.

> If you send garbage to a WireGuard endpoint as I understand it nothing at all happens.

As far as I understand, the idea is to send garbage not to the VPN endpoint, but to any interface on the machine that the VPN runs on, with VPN tunnel's IP there as destination.

The fact that the machine would even consider accepting that leaves me speechless.

You're correct in describing this attack, which is on the TCP/IP stack in various Unix-like operating systems.

I was describing the behaviour of rather newer systems like QUIC, TLS 1.3 and WireGuard which have decided that maybe discretion is the best option.

It seems so far I confused everybody who read what I wrote, (at least everybody who replied) so I apologise for that.

Reply on news.ycombinator.com