I know we’ve all collectively accepted that DNSSEC is a terrible, complicated blight on the world but I still find it incredible that that an organisation with Slacks resources and access to expertise can’t make it work.
I think another interesting question here is why Slack bothered in the first place. As was pointed out on the other DNSSEC thread today: practically nobody in the technology industry uses DNSSEC in the first place. Presumably, Slack did DNSSEC (they don't anymore!) in service of FedRAMP compliance. Why? Slack has one of the most popular products in all of computing. What bad thing was going to happen if they said "nah, we're going to go with Cloud.gov's recommendation and not this FedRAMP document"?
(Kenn White points out on Twitter that some of this may be due to grandfathering --- though, the FedRAMP DNSSEC requirement is pretty old.)
When the DOD tried to mandate Ada, lots of projects were bid as Ada, then switched to C++ at the very first sign of any trouble whatsoever. I would 100% believe it if someone told me that this horrible rollout could be leveraged into an exemption from needing DNSSEC
Was it a hard requirement? No, but the fat fingered audit companies really like to tick that "should" box green and would be more lenient with other debatable findings, so it was suddenly "in our best interests" to comply.
As just one example, it's tremendously difficult, if not impossible, to sell your cloud-based SaaS to Navy customers if you have open FedRAMP compliance issues that you aren't at least working to address.
I say "compliance" instead of "security" for a reason as well, as "compliance" truly runs the show in Navy cybersecurity. And if you want to sell to that market (and it's hardly just Navy who runs this way), it's easier to check the checkboxes than it is to argue about whether NIST is right or cloud.gov is right.
I'm pretty surprised that slack doesn't have a more robust testing network. Is it really that hard to set up another DNS on Route53 for staging these changes? Idk, but that type of thing is the least you can do if you want some FBI agents to discuss active investigations on your chat platform...
(There's a whole thread here, and more on Twitter, getting into the actual details of what FedRAMP and NIST require here, and engaging with the fact that Slack is the only large tech company in the past several years to have attempted to flip the DNSSEC switch on.)
Your blog post makes the supposition that DNSSEC is only being pushed as an alternative means of security to CA for TLS. While it makes a compelling case that this isn't realistic, there are other security concerns that occur from the compromise of DNS records. If the government is going to use a DNS record, it should be signed by a zone owner.
Slack is actually a good use case for this security enforcement, because they maintain a handful of domains that are extremely authoritative for their messaging service[1]. If you can't maintain a security protocol on four domains that are crucial to the operation of your service, you maybe aren't cut out to supply software for the government.
1: https://slack.com/help/articles/360001603387-Manage-Slack-co...
Unfortunately, the GSA product market is its own bubble, as is people who work in IT for the USG in any capacity, and so it's easy to see how people with limited exposure to modern industry practice --- experiences almost wholly gated through vendors that snake through the GSA acquisition process --- might believe themselves to be operating several levels above where they actually are.
I would take Slack's security practice --- their infrasec, their corpsec, their software security, the whole shebang --- over anything done in any USG agency. Slack is better at this than their USG clients are, full stop. And Slack, while strong, is far from the S tier of industry security teams.
I just want to hammer home the point that requiring service providers to get their DNS records signed by DNS zone owners is a reasonable ask for USG software service vendors. Even if DNSSEC isn't capable of securing the whole internet.
Either way, this argument is starting to become political. Is Facebook a role model for cybersecurity, and keeping data out of the wrong hands? Or do NIST researchers know better? Neither - the government outlines its security requirements, and private companies play ball to compete for their business. And if a federal agency wants to be able to prove a DNS record's authenticity, even if it is maintained by a vendor, even if that isn't sufficient to secure their infrastructure, that's their prerogative.
Complex systems can and will fail. Try to do better, of course, but let’s acknowledge that perfection will always exceed our grasp. The world will continue to turn regardless.
One day it might just be your turn to break production.
Perhaps I did not read the room appropriately. Mea culpa.
Slack's second attempt wasn't a DNSSEC problem. Slack depended on a permissive fallback of revolvers when encountering a plain DNS protocol error. It is similar to how some websites in the past relied on permissive browsers implementation when facing broken HTML/JS/CSS. Slack fixed their broken DNS as a result of this.
Slack's third attempt was not the fault of Slack but rather a software bug at Amazon. I would make the argument that Amazon's primary product isn't DNS services, but they did fixed their bug after this.
The general conclusion I get from the article is not that DNSSEC is broken, nor that is too complicated. It is that when doing changes with your core infrastructure to make it more secure, bugs that may have been laying dormant might pop up and bite. I am sure some people has had that experience in domains outside of DNS.
What one can't ignore is the underlying chicken-and-egg problem that DNSSEC must overcome: Not many DNSSEC deployments and hence not much of it has been tested in the real-world, which results in colossal outages despite the attention of some of the most qualified engs, including the ones running one of the largest nameserver deployments in the world.
TLS and WebPKI has had a similar, perhaps even more painful route to ubiquity. So, this problem isn't unique to DNSSEC. What isn't working in DNSSEC's favour is, the world has not just moved on, but it has built solutions atop DNS' weaknesses, like it once did with IPv4 and NAT. Internet's strong network-effects coupled with its heterogeneity, make battling "the System" an even harder proposition.
See also: System design explains the world: Vol 1, https://apenwarr.ca/log/20201227