Comodo fails to check CAA records | Hacker News Reader

Comodo fails to check CAA records

mail-archive.com

186 pointshannob69 comments

Comodo is communicating actively in the bug, in the email discussion, and proactively CC'd the original reporter on the bug that was filed about this without being asked to do so.

The general rule in Operations work is tolerance. You don't fire someone for making a mistake, you fire them for lying about the mistake, for refusing to avoid mistake-prone behaviors, and other problem of that sort.

Applying that same principle to CA operations, why on earth should Comodo be "fired" as a CA for this? They're actively working to solve it, they're responsive, and the agreements they operate under will compel a post-mortem into existence if only to meet the terms of their next CA audit.

If, after investigation, they are found to be willfully violating policy, knowingly lying about CAA support, fail to produce an acceptable post-mortem, and/or fail their next CA audit, then of course they should be at risk of direct sanctions - not just a verbal reprimand.

So if your initial response to this issue is "Comodo should be fired", remember that in Operations culture that specific response "X should be fired" is a harmful response to a zero-day report of an incident, and many Operations teams would fire anyone who called for a firing in response to a fresh incident, in order to protect others from that harmful posture.

I would bet money (maybe not a lot of money, but money) that the only outcome of this is that Comodo ends up within a month being the CA that most reliably checks CAA records. It would be shocking if Google penalized them for this.

As you say, that's based in part on how they handle it.

The thing I'm sad to see is that there's no third-party information stated on when Comodo's CAA verification started and stopped. When they first announced CAA support, who tested it? When did it stop working? So I'm hoping this will come out of the post-mortem and be shared with us all.

Comodo is required by Mozilla to publish a public incident report: https://wiki.mozilla.org/CA/Responding_To_A_Misissuance

CAA support was only required starting a few days ago, IIRC.

Let's Encrypt is pretty good about checking CAA records. At work I built a system that allows our customers to order certs from LE and we see lots of failures for CAA query timeout (customer using a DNS provider that doesn't support CAA queries).

I'm not saying you should use Comodo. I'd use LE too.

This reminds me of a parable:

An employee at a big company makes a mistake that costs the company $500,000. The client is livid, and demands he be fired. The owner's response? "Why would I fire him? I just spent half a million dollars on training!"

This tends to be over-used though. Unfortunately some employees would continue making $500,000 mistakes and never learning. It really depends on the employee's attitude and response to the incident, as well as the circumstances of the incident.

It depends also on the employer's attitude and response to the incident. If the employee behaves perfectly and writes an impeccable post-mortem and is then fired "at the request of a client", that's on the employer, not the employee.

Kudos on this. Coming off of the Symantec thing, I think we're all viewing businesses in the CA space a lot more skeptically.

I worked in a hybrid Infrastructure / Development job for almost 17 years. One of the reasons I could never shake the infrastructure part of things[0] was that I knew the environment from more angles than most of the staff and I tended to be key in high severity issues. We were a highly geographically distributed team and coordinated almost all activity via conference calls.

We had two major rules when a critical failure was discovered. The first was that anyone asking "who caused this" was admonished and often ejected from the call. Often, the person responsible for the failure was on the calls and everyone had a pretty good understanding of who was at fault (if it was, indeed, a self-inflicted wound). The person at fault was also usually critical to the fix and putting them on the defensive caused evasiveness out of self-preservation. To help the first rule, the second rule was barring of anyone at the Director level or higher on these calls[1]. Generally, they were concerned with two things: (1) When's it going to work again and (2) why did it fail in the first place -- both result in delays. They were given updates, separately, by someone from the call at regular intervals. Keeping the politics low and focusing on the problem meant much shorter outages and calmer, more focused, brains of our technical staff.

And to your point, despite working at a company that laid off 10% or so of it's staff about every 9 months, I can't think of a case where a member of technical staff was specifically let go because of an (infrequent) failure -- even situations that would fall under gross negligence of the technician were forgiven provided they were one-off. People get distracted and fail to follow testing procedures/forget to read architecture documentation and make mistakes[2].

After the situation is resolved, a post-mortem/"root cause analysis" was always performed. It was often done via documents rather than meetings, to avoid the confrontation associated with a bunch of managers pointing fingers, and we always learned from our mistakes and got better as a result.

[0] I was a developer of infrastructure services, doing similar work that a DevOps might do, today but more focused on providing internal services and automating support. I was a full-time developer for 3/4 of that time with regular interruptions of sev0 incident response.

[1] This didn't always happen, but I can count on one hand the number of times when a director/VP/C-Level joined and actually made his/her presence known.

[2] I'll never forget the time early in my career when I was on-call and had to restore our most critical/largest file server that was destroyed as a result of a technician getting frustrated with a rack and dislodging a 60(?)-pin SCSI cable...partly...where it attempted to run for several hours in that condition. The best part was the maintenance he was doing was on the backup server that had required his attention for a few weeks and hadn't functioned since then. When I saw the condition of the cables, I was so pissed off that while restoring the weeks' old backups at 4:00 AM, I had the operators in that room show me how to review the (VHS) security tapes. Watching him kick the door several times to get it to latch, then peering in -- directly at the bent-up cable -- just before walking out the door was the best part. He should have been fired for that, and eventually a similar behavior had that result.

Worth noting that the rule they broke has only been in effect for three days (https://cabforum.org/2017/03/08/ballot-187-make-caa-checking...). This might cause the CAB to go a bit easier on them.

They also voted "yes" to the proposal and had 6 months to implement it.

Exactly. For some contrast: I use tinydns, which doesn't natively support CAA records. But I wanted them so I read the CAA spec and wrote a little tool to add support for these records using the generic record format. And I'm not even a programmer! (I can write a little code, nothing serious, and nobody would hire me for it.)

And yet with all of Comodo's resources, they couldn't get this done? I can't take them seriously.

I'd assume that Comodo wrote the code, tested the code, deployed the code, and forgot to flip the switch to actually enable it in production.

According to the mail, they have claimed to check CAA much earlier, so it's not just CAB guidelines they have broken, but their own word, which can be seen as worse.

If you're not already familiar with his work, you should know that Hanno Böck is a machine, and someone worth following. If there's some mistake you can make with the web PKI that is so stupid nobody would ever bother to check for it, rest assured that Hanno will eventually check.

Couldn't agree more, this (among others) was brilliant: https://blog.hboeck.de/archives/888-How-I-tricked-Symantec-w...

Oh that's amazing. Imagine if he'd got, say, Google or Facebook's key revoked and the havoc it would wreak.

Yes!

Hanno also writes the Bulletproof TLS Newsletter (https://www.feistyduck.com/bulletproof-tls-newsletter/) which is highly informative. Suggest everyone interested in webPKI subscribe there as well.

If this is the first time you've heard of CAA records, note that the "dig" command isn't yet aware of these records types, so you need to tell it to use "type257" as the record type:

    $ dig empty.basic.caatestsuite.com type257

More resources here:

https://support.dnsimple.com/articles/caa-record/

https://caatestsuite.com/

The dig command is totally aware of it if you have updated bind within the past couple of years.

dig on Debian stable supports CAA just fine.

Wikipedia has a nice list of previous Comodo malfeasance: https://en.wikipedia.org/wiki/Comodo_Group#Controversies

Highlights include allowing fraudulent certs to be issued for Google, Yahoo, and Mozilla; working with a malware company; and trademark shenanigans with Let's Encrypt.

>working with a malware company

Why is this a bad thing? A cert says "Yes, person x REALLY IS person x, and I can prove it mathematically." It doesn't say "Person x is trustworthy enough for me to vouch for them."

Who do you imagine they verified was Person X? More on the malware cert issuance: https://blogs.msmvps.com/donna/2009/05/18/microsoft-mvp-mike...

For malware concerns there's also Comodo's relationship with PrivDog. I'm not clear on whether PrivDog is deliberate malware or just incompetent insecure software.

That is ridiculous, to expect a CA to google every cert request domain and check what they are doing? That is a ridiculous requirement - cert issuance is automatic.

Do you think we should stop allowing Let's Encrypt to issue certs? You know they don't google each domain to see what they are doing.

In my opinion, it would be worse if they denied certs to certain people.

> Who do you imagine they verified was Person X?

The legitimate owner of that domain or operator of the website. That's what certificates are supposed to attest to.

Malware authors might be bad people, but they're definitely people who exist in real life and have an identity that can be attested to.

In this particular case it's more likely negligence.

I can easily see how a missing integration test, and a few code changes later, the feature gets broken and nobody notices it. It doesn't prevent any normal codepath from executing properly.

The best one is their stupid Chromodo "Secure" Browser, that used --disable-web-security:

https://threatpost.com/chromodo-browser-disables-same-origin...

Curious to see how the CAB will handle this or if they're going to be "soft" as it's the first days of the CAA enforcement. Historically, they've been very accurate in enforcing their rules, which could mean a serious reprimand of Comodo.

If anyone is interested in testing their own CAA records, we built an online CAA validator specifically for this; https://dnsspy.io/labs/caa-validator

Between their decision to release a rebranded "more secure" version of Chrome where their changes had introduced security holes, and their decision to try and trademark "Let's Encrypt", I can't imagine they have many friends in the CAB.

On the other hand, they issue a lot of certs - if you've used Cloudflare's free SSL stuff, you've got a Comodo certificate - so they're unlikely to be shut down or anything that extreme.

> On the other hand, they issue a lot of certs - if you've used Cloudflare's free SSL stuff, you've got a Comodo certificate - so they're unlikely to be shut down or anything that extreme.

They're still smaller than Symantec, for whom the nuclear option was considered (and ultimately decided against, in favour of a phased distrusting).

To clarify, the CA/B Forum has no enforcement powers. This would only really be an issue if a root program takes issue with it or their auditor qualifies their audit.

CAA is fresh like IPv10. Done some tracking the last couple of months on the alexa top 1 million domains. In April 400(!) had CAA records. In August 800. 10% have errors (the issuer-critical flag's value is set to 128, 4 6 or 9) where it should be either 0 or 1. iodef set to totally unusable web addressed, you name it. As a matter of fact there are MANY large DNS service providers that do not even bother providing CAA options. If you want to name and shame, there is ample opportunity for the years to come. Getting it up and running is harder but more rewarding in the end.

> 10% have errors (the issuer-critical flag's value is set to 128, 4 6 or 9) where it should be either 0 or 1.

Are you certain?

https://datatracker.ietf.org/doc/rfc6844/?include_text=1

> The data fields are defined as follows:

> Flags: One octet containing the following fields:

> Bit 0, Issuer Critical Flag: If the value is set to '1', the

> critical flag is asserted and the property MUST be understood

> if the CAA record is to be correctly processed by a certificate

> issuer.

> A Certification Authority MUST NOT issue certificates for any

> Domain that contains a CAA critical property for an unknown or

> unsupported property tag that for which the issuer critical

> flag is set.

> Note that according to the conventions set out in [RFC1035], bit 0

> is the Most Significant Bit and bit 7 is the Least Significant

> Bit. Thus, the Flags value 1 means that bit 7 is set while a value

> of 128 means that bit 0 is set according to this convention.

EDIT: Fixed formatting? I hope.

That's quite incredible... It'd be interesting to see how up to date the CRL is as well.

This inspired me to add a CAA record for my domain - but https://www.namecheap.com doesn't support that record type. Has anyone had better luck other places?

You may find my CAA website helpful: https://sslmate.com/caa/support

I used your site - it was quick and easy. It soon went downhill from there. I don't run my own DNS or I'd be done pasting the line into my config. Instead I ran into a UI that doesn't support CAA as a type.

Unfortunately, if you want to use CAA your provider must support it. I recommend you use one of the DNS providers on the page I linked - there are quite a few to choose from.

Have you tried their support flow? They've usually been very responsive in my experience, it might just be around the corner.

maybe time to revoke Comodo's CA authority. Similar to what has happened before with Symantec (?).

Revocation for being three days late in implementing a new standard would be a massive overreaction. It would harm thousands of businesses in the process and would be extremely petty. Symantecs misdeeds are far worse than this.

Revocation can be done only for new certs. It's time to throw a CA to the dogs once in a while pour encourage les autres.

Not at all. Revoke trust in the signing authority and all it's certificates turn invalid the second that change hits. Update your OS's trust store to not trust Comodo anymore and you'll see the effects immediately in IE/Edge on Windows or Safari and Chrome on macOS (Firefox still ships its own CA bundle).

It's just that when trust has been revoked in CAs recently, it's been done in a more gradual fashion to not hurt customers too much and give them time to migrate. After all, it's not the customer's fault the CA went rogue. This usually starts with not accepting certificates past a certain date but still validating previously issued ones, until sometime 6 months to a year later they're completely distrusted.

Which is exactly what I said above could be done.

That's not at all what you said. What I replied to is where you literally said:

> Revocation can be done only for new certs.

That's not true. The "newness" of a cert or a CA has absolutely nothing to do with how, if and when it can be revoked.

Comodo doesn't have good security practices, it looks like, from my recent memory (confirmed with some google searches as well).

Is it expected from all CAs that they obey CAA records, or is it something just made up by the community to crush the big CAs? I see an RFC from just a few years ago, and I'm not sure how these things are standardised.

Standardization goes through the CA/B forum. There was a ballot voted to make CAA checking mandatory for CAs[1], and COMODO voted yes for it.

Any CA that issues certificates publicly need to check CAA from the 8th of September onward.

[1] https://cabforum.org/2017/03/08/ballot-187-make-caa-checking...

Ah, so they are three days late. That doesn't sound too serious.

Three days is quite a long time to be late, so I'd hope someone over there is getting a reprimand, but yeah, it's also not a disaster. They're response and time to remedy this will be more telling I think.

Three days over a weekend, though. Context matters. Even if it's the most critical incident, you can't force employees to work outside of business hours.

Except they claimed to support it a long time before this. It’s not that they were late, it’s that they lied.

That makes no difference as to when the three days where. I never made any claims as to why it's late or them lying. I merely clarified that the three days were over a period where people don't usually work.

Karunamon et. al. are not saying your point is wrong; they are saying it is irrelevant, and if their facts are correct then they are right.

It's not about them being late these 3 days. It's about them lying about having already implemented this months ago.

How do you know they lied? What if they implemented it but simply did not flip the switch?

'Lie' is indeed a strong term, informally suggesting an intent to deceive. Personally, I would suspect negligence and incompetence rather than deceit, but negligence is a serious matter here.

Given the general crappiness of the CA industry, my first instinct is to say they willfully said "yeah we support it" without actually doing it first, knowing that it wasn't actually done.

The ballot was in March, so they have 6 month to prepare for it, not only 3 days to implement a surprising change.

These 3 days don't matter when you have months of lead time.

On a requirement they voted for half a year ago for a standard specified 5 years ago. It's sloppy and sloppy is not a property you want in a certificate authority.

From experience working with them, sloppy is an excellent way to describe Comodo.

I wonder about what else is Comodo being "not too serious" while they promise to be "super serious" about them in their marketing campaigns?

> Is it expected from all CAs that they obey CAA records, or is it something just made up by the community to crush the big CAs?

CAA was made up by... drumroll.... Comodo.

Yes, check the authors on the RFC: https://tools.ietf.org/html/rfc6844

Reply on news.ycombinator.com