Hacker News Reader

Tangential, but since you mentioned Akamai.

I wrote a framework at work that maintains authoritative DNS configuration truth (static, and advanced GSLB variations) as text files going through git and regular PRs, auditing, etc.

The truth is then flushed out via APIs to multiple DNS providers (active-active), Akamai being one of them. Nobody's APIs are perfect, but I don't recall it being too difficult to push the state to their system.

For sure, you can totally make something like this happen, but again that means writing and securing a bespoke system. If you’re a team managing your AWS infrastructure with Terraform, it sucks to have to write a totally different system (and all the secrets management that goes along with it).

Additionally, for a government agency you could have dozens of contractor teams, and you need a method of securing access for all of them. It’d be much easier to just use Route53 and delegated subdomains, as all teams are largely deployed and managing infra (with MFA) there.

But it’s not allowed, because DNSSEC.

Is there anything that would make it especially challenging for Route53 to support DNSSEC, or have they just not chosen to implement it?

I don’t work for AWS, but I’ve talked to people that work on Route53. Amazon just isn’t willing to support it, none of their large customers care about it enough it seems like. Plus DNSSEC has all sorts of issues, some of which people in the thread have mentioned. Actually, tptacek has a blog post that summarizes a lot of what’s wrong with DNSSEC iirc.

This unwillingness may indicate a lack of competency, complacency as a market leader, or complicity with censoring regimes.

How f difficult is it to sign a zone?

DNSSEC isn't perfect (indeed it only provides assurances of record integrity and doesn't secure the channel); but it's certainly better than nothing, than no signature at all.

If route53 can't or won't or doesn't have to because they don't want to implement DNSSEC, route53 is not suitable for .gov and .mil domains.

It's really that simple.

I get that you want to use terraform; I don't see why you think route53 is the only DNS that terraform works with.

The article/directive specifically called out DNS account passwords on DNS hosting providers. If they were using support tickets and manually implemented changes by hand then there would not be an account. With no account for a control panel you don't have any risk that the email and password gets dumped on pastebin. In the directive it even speaks about password managers, which again implies that the issue they are talking about are DNS control panels like the one at Route53, likely with reused password, and not bespoke systems.

For high priority domain names I would go so far and recommend against having a DNS control panel accounts on things like Route53. It has access control management but unless the department has the experience and skills it will likely end up with a single account that has full access with little oversight when something goes wrong. Those departments are likely better served going through a domain management company that handles all the security and manage who can request changes and how. If they have the experience and skills then managing it fully themselves can be a better option, but then what they really need is a registrar which allows for DNSSEC keys to be forwarded (with for example a cds record).

> The article/directive specifically called out DNS account passwords on DNS hosting providers. If they were using support tickets and manually implemented changes by hand then there would not be an account.

I don't think it's difficult to imagine that, somewhere within the massive federal bureaucracy, there is a team that works on DNS record tickets filed by other parts of the bureaucracy, and logs into a web console with a DNS hosting provider in order to fulfill those tickets.

Yes, this is pretty close to how this agency does things. Your contracting team files a support request for a record change and a manager from within the agency approves. The contractor that owns DNS changes then goes and does it.

This makes all sorts of other things bad, like generating TLS certs with automation, as typically a validation for cert generation will be done via a DNS entry to prove ownership. But that requires a support ticket, so no automation will work here.

“If they were using support tickets and manually implemented changes by hand then there would not be an account.”

Your contracting team may not have a DNS account, but the contractor that deals with DNS changes certainly does.

The point is that all infrastructure really should be managed with code. If you managed DNS from Terraform you’d notice that your production DNS system was in conflict from what your code said it should be the next time a terraform plan/apply was run (which is happening all the time from dev or maybe CI). Without it you could go months or years without noticing a changed entry on a little used, but critical system. From what I remember this attack was ongoing for a couple years before it was noticed.

If they are using contractors that have DNS accounts with email and password as authentication, and who reuse that on other online websites which later get leaked, then that is major a failure in procurement when the bid was initially created.

> The point is that all infrastructure really should be managed with code

That I agree 100% with. If the agency itself has a IT department with experience and knowledge then the most optimal choice is that they do it themselves with either something like Terraform or their own physical hardware with authoritative DNS running on it. Agencies that is involved with either critical infrastructure, personal information, or state secrets should always have a security risk analyze done which describe what would happen if someone broke into DNS and where the vulnerabilities are. Sadly very few agencies does this.

Other agencies are likely better served by updating their procurement in respect to DNS management and make sure the requirements lists liability and high security. Make sure that rather than some cheap bulk registrar with a do-it-yourself control panel, where the registrar always have zero livability if the credentials are leaked, it is better to pay a more costly management company that manage the infrastructure with code.

I wonder why Route53, etc. don’t support DNSSEC.

Even without widespread client support for DNSSEC (and the AD bit doesn’t cut it), it would be quite nice if CA/B rules required that CAs enforce DNSSEC (if enabled for the domain) when checking CAA records. This would impose no burden at all on web browsers and only a minimal burden on CAs. Unfortunately, since Route53 and the like mostly don’t support DNSSEC, getting the benefit for domain owners is a pain in the neck.

RFC 6844 seems to recommend but not require this validation. I don’t know whether newer rules require it.

edit: Unsurprisingly, people are working on this: https://tools.ietf.org/html/draft-ietf-acme-caa-06, section 5.6, for example.

One of the problems with DNSSEC is that it causes the size of DNS responses to increase substantially, which becomes a DDoS vector. The response to this is nominally to use response rate limiting, but the more people who use DNSSEC the less effective that is because it means more DNS servers the attacker can use to reflect their attack through without reaching the rate limit for any one. (RRL is also not ideal because a large DNS cache can hit the rate limit by just making a large number of legitimate queries.) And the more servers that use DNSSEC the harder to block a DDoS attack because it comes from a larger number of unique addresses.

And now TXT records used for domain validation are becoming quite large as well. You add a ~hundred byte TXT record for each service you want to validate your domain against, but they all want the entry to go in the same place (at the root), so you do the DNS TXT query for the root and you get a big response. Adding DNSSEC on top of that might even blow the practical limit for UDP EDNS0 responses sometimes, never mind the DDoS potential.

There may be a solution for that one -- specify a location for each TXT record instead of everyone using the root, so Facebook uses _facebook-domain-verification.[example.com], Google uses _google-site-verification.[example.com], etc. Then you wouldn't get a dozen large TXT records in response to a single query because they're each at a different name. But that's not currently what companies are doing, and a purist wouldn't like it because in principle controlling _google-site-verification.example.com doesn't mean you control all of example.com.

For at least some of this, TCP could be a decent solution. For CAA queries, for example, no one really cares how fast the query is, so a server could plausibly refuse to answer over UDP.

That works for the thing expecting the large response, but requires an unusual custom configuration on the server (refuse UDP queries for specific record types but not others), and doesn't really help other clients that may make queries that yield large responses unexpectedly.

For example, the large TXT records can prevent mail delivery from some versions of qmail, including the one currently packaged with Debian stable, because it makes an ANY query for the domain (to avoid separate queries for MX, A and CNAME) but only supports UDP and only supports responses up to the 512 octets specified in RFC1035. Then it gets a truncated response when there are >512 octets of TXT records and considers it a name resolution failure.

Making the ANY query is an unusual quirk in qmail, but it could have just as easily been an actual TXT query looking for the SPF record etc.

Route53 was very slow to even enable CAA itself. They are not exactly at the bleeding edge of DNS features.

CA/B Baseline Requirement 3.2.2.8 says to do RFC 6844, and that CAs need to implement at least enough DNSSEC so that they can determine whether "the domain's zone does not have a DNSSEC validation chain to the ICANN root" because if there is no DNSSEC validation chain you can treat DNS failure as permission.

In practice most larger CAs just do DNSSEC entirely here, Let's Encrypt uses it everywhere. Deployability to ordinary end user clients remains poor due to middle boxes and local DNS implementations that are garbage, but if you actually control your network (as a competent CA should) then it isn't a problem.

The sort of CAs and CA-customer relationships where DNSSEC was argued to be problematic aren't going to be doing ACME CAA, or probably ACME at all.

[Edited to delete me confusing one ID for another]

I don't believe that this description of DNSSEC functionality inside CAs is accurate. Can you be specific about which CAs you believe do a good job of validating DNSSEC --- not just somehow "running" it, but making use of it operationally?

The largest CA by volume, Let's Encrypt, uses DNSSEC for all its validations, and hard fails issuance if it can't get either a DNSSEC validation or the signed denial saying this part of the hierarchy isn't signed, it has done this for years now. Works well.

Several other major CAs including DigiCert do DNSSEC validation but I haven't used them and so can't even tell you from my own experience how well they work, though it seems likely some of their other customers might have noticed on that end.

Now, are they all doing a "good job"? If you actually were paying any attention at all in this space you'd presumably have quoted my already published opinions about that. I think they should retain the DNS responses the same way they would keep the actual raw data from an HTTP validation. So third parties doing incident investigation can do an effective post mortem - I also think they use too many "short cuts" that are going to result in someone finding a nasty bug one of these days and for which there's no real evidence they're necessary.

But since it seems for you a "good job" is just using something operationally, then yeah, in that limited Thomas Ptacek sense of "good job" they're already doing a good job I guess.

Can you confirm another CA other than LetsEncrypt that will reliably deny issuance on a DNSSEC failure?

(Obviously, just to point something out for the thread that you already know, the vast, overwhelming majority of LetsEncrypt issuances are for zones without DNSSEC signatures).

Given that I already wrote that...

> I haven't used them and so can't even tell you from my own experience how well they work

... I'm not sure what my "confirmation" would tell you, beyond that I know how to read the paperwork from the CAs. But sure, both Sectigo and DigiCert say their systems should deny issuance on DNSSEC failure.

I'm asking because I want to know, not to make a point.

Just stopped by to mention that Google’s Cloud DNS supports DNSSEC. You should be able to configure it via Terraform.

Not an endorsement of DNSSEC, just that it’s not impossible to support it.