Also if you didn’t read the article, at least forHP: it was HP’s mistake and fully acknowledged by the company.
Apple’s anti-malware system was actually just working exactly as designed.
Apple’s anti-malware system was actually just working exactly as designed.
This is something that Stallman[1] and others[2] have talked about for a while now.
Would be fairer to say that the file is untrusted or potentially compromised.
That's an interesting take, considering that there was no malware involved, and many Mac users were left unable to use their printers.
Hence anything signed with an explicitly revoked cert must be blocked.
The private key was not compromised. The proof of this is that Apple un-revoked the very same certificate. The cert is valid again.
It's not a question of whether it would "work". If the private key was out in the wild, then Apple would never un-revoke the cert. That would be insane, because anyone could sign software with it.
It seems that some people want to claim that Apple has to act based on a single email and is incapable of replying to that email or calling on the phone, when Apple has the phone # of every registered developer.
Apple gets certificate revocation requests all the time, to disable test builds, disabled badly bugged software versions, to disable software for licensing reasons. It’s the developer’s choice when or if they revoke a cert. Do you have any evidence this request was different enough to require escalated investigation compared to any other routine request? HP doesn’t seem to think it was.
How do you know this? What is the evidence for this claim?
The strange thing about this claim is that certificate revocation does not disable specific builds, it disabled every piece of software ever signed with the cert, so this wouldn't even accomplish what you claim it accomplishes.
The reason we're even discussing this issue is that there was a huge public fallout. You don't "just" do anything. This was a nightmare for many HP users, who just couldn't print at all.
How do you "just" deliver new builds to users when the old builds won't even open?
One of the uses of certs is to sign test builds with a different cert from the production release, so that you can ask Apple to disable them after release. So Apple gets requests to revoke certs for this, and other reasons all the time. In this case HP had used a different cert for these drivers as against other drivers, showing that they used different certs for different purposes, that's all.
Where in the world are you getting this information? Citation please.
Apple Product Security does not talk. About anything. So any claims about what Apple receives are dubious at best.
> you can ask Apple to disable them after release
There are very strict limits on how many Developer ID certs you can generate. You can't generate them willy-nilly. So this claim is also dubious at best. I have a developer friend who ran into this limit, and it took him months to resolve this issue with Apple Product Security. He was stuck.
Again, you just seem to be inventing things in your own mind with no first hand knowledge. Are you even a Mac developer? Can I download one of your Developer ID signed apps?
No actual malware ot breach was involved in this case, but what is the system supposed to do if the org responsible for the software revokes the trust anchor? Continue just as if trust was still established? What if the certificate had actually been used to distribute malware?
The system worked as designed, but HP needs to figure out how that to revocation happened.
HP cannot revoke the cert. Only the CA, Apple, can revoke the cert. HP can only request that Apple revoke the cert, but there was no reason for Apple to grant this request.
See section 4.8 on certificate revocation: https://images.apple.com/certificateauthority/pdf/Apple_Deve...
This is a security-usability tradeoff. Do you eagerly grant revocations, in the process protecting more users in cases of disaster but exposing yourself to the pain of false positives (like this one)? Or do you verify such requests, thereby making accidental revocations unlikely but increasing the time it takes for compromised software to get pulled?
Given revocation is more easily reversed than fixing a compromised computer, I stand with Apple’s decision to revoke first and ask questions later. But this is a preference, nothing more. Based on how Apple has positioned itself with respect to privacy and security, I imagine this is not an uncommon preference among its customers.
There are only 2 reasons to revoke a code signing certificate:
1) The private key is compromised. 2) The owner of the certificate is signing malware.
Obviously HP is not a malware author, so unless HP emailed Apple and said their private key is compromised, there's no good reason for Apple to revoke the cert. There aren't questions, plural, there's only one question.
Apple revoked HP's cert, which almost immediately caused all HP software signed with the cert to be disabled on every Mac in the world, and then afterward Apple has absolutely nothing to say about it?
There's no web page for revoking Developer ID certificates. Only Apple can revoke them. I know because I have a Developer ID certificate myself, and also because Apple says so in section 4.8 of their document: https://images.apple.com/certificateauthority/pdf/Apple_Deve...
> Who says HP didn't tell Apple the key was compromised?
Their key wasn't compromised, so why would they say this?
> Why wouldn't Apple assume the private key was compromised if that's the only reason a developer would request revocation?
Why would Apple assume anything? Apple can reply to the email, or call HP on the phone. (Apple has the phone # of all registered developers.)
The request had to be intentional if they had to email Apple. Whatever reason they thought they had turned out to be incorrect.
[1] https://www.quora.com/How-do-I-revoke-my-Apple-Developer-cer...
That link is for an iOS Development certificate, which is under the WWDR CA, not the Developer ID CA. Compare section 4.8 of this document, which says there is a website for revoking WWDR issued certs: https://images.apple.com/certificateauthority/pdf/Apple_WWDR...
Genuinely curious -- what makes you so certain that Apple granting HP's request was a mistake? Do we have information pointing towards HP providing (or failing to provide) a reason along with their request?
Why? If a business asked for their code signing certificate to be revoked, why shouldn’t they be able to?
I can't emphasize enough what an extreme, last resort measure revocation is for Developer ID code signing certs.
Why? Based on what?
Why isn’t a third if the owner of the certificate requests revocation?
Almost all known Developer ID cert revocations have been for malware authors. These revocations are initiated by Apple, not requested by the (malware) developers.
There seems to be a lot of misconceptions about Developer ID, because people are comparing them to web site certs, but Developer ID is very different. When a Developer ID cert is revoked, all software ever signed with that cert is immediately disabled on every Mac in the world. It's a remote kill switch. This kill switch is only used in extreme circumstances, such as malware distribution.
The cert owner's desires are not really relevant to Apple. The cert is for malware prevention only. Some Mac developers would prefer to be able to distribute their software without Apple's permission, and balk at having to pay $99 per year to Apple for code signing, but we effectively have no choice.
If there's no security issue, then the cert should not be revoked.
This is unfortunately a complete misunderstanding of Developer ID. The program isn't a "service" for developers, it was imposed on developers as a requirement for security purposes. We can't opt out! The intent of Developer ID is to allow Apple to remotely disable malware. That's the one and only purpose. Almost all publicly known Developer ID cert revocations were for malware authors.
When a Developer ID cert is revoked, every piece of software ever signed with the cert is disabled on every Mac in the world. Revocation cannot be used to disable specific builds.
Moreover, Apple has never used Developer ID for the purposes you claim it is used. You're just making up stuff with no evidence. It's clear you're not a Mac developer yourself, and you're just speculating about what you believe might be true, but you don't have any first hand knowledge of the situation.
It's truly astonishing to me, as one of the few or maybe only people in this thread who is actually a Mac developer and has a Developer ID cert, that I'm getting systematically downvoted by people who have no direct knowledge of Developer ID. Getting tired of this, and HN entirely. It's incredibly discouraging, and makes me not want to be on HN when people with expertise are downvoted by people without.
I've been a professional Mac developer for over a dozen years. I was around before Developer ID was introduced, before code signing was introduced. I was around for the announcement of Developer ID. I've been distributing Mac software for all these years, I know what I'm talking about.
HP revoked its own signing certs rendering their existing signed code invalid. That it took them days to re-sign and published fixed binaries is HP’s fault not Apple’s
No, HP requested that Apple revoke the cert. Inexplicably, Apple complied with this unusual request. Apple later un-revoked the cert after Mac users started reporting problems.
Kind of curious: how closely do CAs look at requests to revoke certificates? Is there a general pattern? Or does it vary by CA?
If they want to remain a CA, very closely!
--
Per the (Server Certificate) Baseline Requirements [0]:
> 4.9.1.1 Reasons for Revoking a Subscriber Certificate
> The CA SHALL revoke a Certificate within 24 hours if one or more of the following occurs:
> 1. The Subscriber requests in writing that the CA revoke the Certificate;
> ...
The Code Signing BR's [1] have almost the same requirement. For code signing certificates, the CA "MUST*" revoke the certificate upon request "within one business day" (cf. 13.1.5, IIRC).
I'm not sure if this particular certificate falls under those BR's or not, though (see my other comment). My guess would be that it does not.
--
[0]: https://cabforum.org/wp-content/uploads/CA-Browser-Forum-BR-... (PDF)
[1]: https://cabforum.org/wp-content/uploads/baseline_requirement... (PDF)
Curious what the intent is behind Apple saying they'll evaluate the request. I do realize that this request falls under different guidelines, but I wonder whether it's merely a difference in wording rather than practice.
The "examination" for revocation is very simple for every CA - does the request include demonstrated control of the private key. If it does, honor the revocation request. They have at most one business day to revoke. For evidence of actual compromise (e.g. not a first party revocation request) they have 24 hours. Note that the the CARB rules don't allow the CA to delay revocation for compromised certs, even if the true owner asks for it.
Inexplicable!? Not at all.
Unusual? Nope.
While I certainly can't claim to be familiar with Apple's code signing program (it's not relevant to me at all), pretty much any CA will revoke a certificate upon a request by the entity that possesses the private key.
In fact, in the "Web PKI", a CA MUST do so (per the Baseline Requirements) -- within 24 hours! -- and, although I haven't read them, I suspect the "Code Signing" BR's have the same or a very similar requirement.
Now, you might say "the Web PKI BR's don't apply here" and you'd be correct, obviously. I'm not even sure if the "Code Signing" BR's are relevant in this instance -- I don't know whether this is some private, Apple-only program or something broader.
However, either way, I'll bet you a beer that the "rules" -- whichever are in effect -- explicitly state that the issuer will revoke a certificate upon request by the subject.
By the way, this doesn't even consider the case where a private key becomes compromised. In those instances, you're pretty much always obligated to notify the issuer ASAP. That's besides the point, though, as that's not what happened here.
(Note: The generally accepted way to request revocation is to send to the CA a "revocation request" signed with the private key.)
See section 4.8 in particular. "The Subscriber may initiate a revocation request by sending an email to product-security@apple.com. The request for revocation will then be evaluated by Apple."
If the issuing CA is subject to the CABF's Code Signing Baseline Requirements, then Apple "MUST" revoke the certificate "within one business day".
My point, again, was that such a request is NOT "unusual" and an issuing CA complying with such a request is NOT "inexplicable".
--
EDIT:
Per the CABF's "Code Signing Certificate Working Group Charter" [0]:
> Other functional models include those which allow developers to self-sign code and those in which the platform supplier manages the code signing or certificate issuance process, and these models are expressly excluded from the working group’s mandate. Common examples of these models that are expressly excluded from the scope of guidelines to be promulgated by the working group are Apple’s Developer ID program and Google’s Android.
[0]: https://cabforum.org/2019/03/26/code-signing-certificate-wg-...
That depends on what "unusual" means. The only good reason for a developer to request their cert be revoked by Apple is if the private key is compromised, in which case Apple should definitely grant the request. Otherwise, there's no reason.
Revocation is not unusual or uncommon for a CA. For an individual developer it might seem uncommon, but uncommon * large number of certs == common.
If a Developer ID cert is revoked, then all software ever signed with that cert is disabled on every Mac in the world, as soon as the Mac checks OCSP, which is likely to be the next time the software is launched. Revoking a Developer ID cert is a death sentence for signed software.
Thus, basic housekeeping is not a justified reason for revoking a certificate. The express purpose of the Developer ID program was for security, and security is the only reason to revoke a Developer ID certificate. It's an extreme measure, to be taken only in the worst case scenario.
Apple has a remote kill switch for all signed software on the Mac. They need to be extremely careful about when, if ever, they flip that switch. Almost all known cases of a Developer ID cert getting revoked is when the owner of the cert was a malware author.
How so? They can't unilaterally revoke a Developer ID cert, they can only request that Apple revoke it.
> Apple did nothing wrong, unless you’re of the opinion that Apple should be auditing other company’s internal policies and controls.
This is missing the point of Developer ID. It's a program created by Apple and imposed as a requirement on Mac developers. The purpose of the Developer ID cert is to prevent malware. That's why it has a remote kill switch, to stop malware from running. The purpose was not to give Mac developers a remote kill switch to use however they please. Why in the world would Apple do that?
Developer ID is not an "internal policy" for third party developers. There's nothing to audit. Developer ID is Apple's program, designed for Apple's purposes, and Apple makes the decision whether or not to revoke the certs.
Almost all known Developer ID certificate revocations are not the result of the developer's own request but rather the result of Apple discovering malware in the wild and revoking the certs of the malware authors.
That is literally how revocation works. There is no magic "make this key no longer work" function.
Revocation of a cert means telling the CA to revoke the cert. It is no different when the CA is Apple, than if it were Thawte, or Google, or LetsEncrypt. Revocation means telling the CA to add your certificate to the list they publish of no longer valid certificates.
Managing your certs if absolutely the responsibility of the developer. It is no different from any other cert - TLS certs, S/MIME, etc are all the same. The only difference is the usage flags included in the cert.
> Almost all known Developer ID certificate revocations are not the result of the developer's own request but rather the result of Apple discovering malware in the wild and revoking the certs of the malware authors.
That is, apple has found evidence of key compromise, and so revoked the cert. For WebPKI CAs there is a hard requirement that the moment they receive any evidence of a key being compromised they are required to revoke within 24 hours, or risk penalization.
Again, Apple is acting no differently from any other CA.
No, this is where the argument is fatally flawed. The usage of code signing certs and web certs is completely different. At any time, you can put a new web cert on your server, and web traffic can continue as if nothing happened. Revoking a web cert that's no longer in active use does not affect current web traffic. On the other hand, that's not the way it works with code signing. Users download and install signed software. That software is expected to run indefinitely, essentially forever as long as the software itself works. The expiration date for code signing certs only affects the ability to sign new software, it does not affect the ability of old signed software to continue running. Code signing certs expire, but the signed software does not. However, when a code signing cert is revoked, it immediately (on the next OCSP check) kills the old, installed software. That's a total disaster, unless that installed software happens to be malware. This is why revoking a code signing cert is so much worse than revoking a web cert.
> Revocation means telling the CA to add your certificate to the list they publish of no longer valid certificates.
To be pedantic, Developer ID doesn't actually use a CRL, only OCSP: https://images.apple.com/certificateauthority/pdf/Apple_Deve...
Where HP screwed up was that they didn’t re-sign their binaries, so people would (correctly) go and download the drivers again and get the same failure.
Again, it is not apples job to work out if you really wanted to revoke a cert, or you just screwed up. How are they meant to make that determination?
Let’s put it another way, say someone requests a revocation issue to malicious actor or some such. And Apple then futzes around going back and forth through email to make sure revocation is intended. The headline would be “Apple tried to stop me revoking my signing certificate”
> CRL vs OCSP OCSP is fed from a list the CA manages.
Whether you’re downloading a list of revocations ahead of time or getting individual responses is moot.
It is literally Apple's job. From the Certification Practice Statement: "The Subscriber may initiate a revocation request by sending an email to product-security@apple.com. The request for revocation will then be evaluated by Apple."
> Let’s put it another way, say someone requests a revocation issue to malicious actor
There's no evidence that this is what happened with HP.
There's one known case where a Developer ID cert may have been compromised: https://panic.com/blog/stolen-source-code/ Apple worked very closely with the developer, they didn't just receive an email and reflexively act without thinking. Moreover, in this case, Apple did not actually revoke the cert! Old versions of Panic apps signed with the cert still run fine. Apparently Apple has a way to use the secure timestamp of the code signature to selectively disable apps signed after a certain date.
Again, all non-accidental Developer ID cert revocations that I'm aware of were owned by malware developers, and revoked on Apple's initiation. I've never heard of a case, until HP, where the developer requested the revocation of their own Developer ID cert. And we would know if a legitimate developer's cert was revoked, because their software would stop working, just as it did with HP, and with the accidental Charlie Monroe revocation (which was entirely Apple's fault).
Perhaps there are cases where a cert was never used, and then revoked. But Apple needs to be extremely careful about these cases, because we've seen the terrible consequences when a cert is mistakenly revoked. And there's certainly no rush to revoke an unused cert.