Hacker News Reader

New Update, explaining the issue with shared infrastructure and crediting the discovery to Frans Rosén of Detectify: https://community.letsencrypt.org/t/2018-01-09-issue-with-tl...

Do we get points for speculation based on these hints?

My guess would be that some major public CDN (Cloudflare etc) will let the attacker deploy their TLS-SNI challenge certs, and thus validate for other victim domains using the same CDN service.

EDIT: Main reasoning being that I can't think why it wouldn't work - apart from the TLS-SNI challenge certs somehow being considered invalid by the CDN provider and refusing to deploy them, but I find it hard to trust that happening with 100% certainty.

As long as we're tossing hats into the ring...

I don't think so - _maybe_ a MITM that would affect all TLS-SNI, but I don't think the model you propose exists (I'm a bit dated on that tho), and if it did, I think it would be 'wider' than the CDN's.

"It's an interaction between the protocol and provider services" -- I will wager it's that some providers don't validate the CA or RootCA authentication method/mechinism, and so would accept a SHA1 or MD5 where a SHA256 or ROT13 would be preferred. (Think JWK/JWS 'null' issues)

I assume standard rules apply - one internet point to the winner, in the event of a draw, fisticuffs at dawn facing opposite coasts until we're both bored with it?

"and so would accept a SHA1 or MD5 where a SHA256 or ROT13 would be preferred"

I don't think ROT13 means what you think it means.

I suppose it's possible in theory but I'm not sure if Cloudflare would perform other checks.

It's certainly the case with the Business and above accounts you can upload custom certificates but I don't have one of those accounts to check if further validation is made regarding domain control.

If they didn't it would seem on the face of it that both the conditions in the article* are met.

* >Many users are hosted on the same IP address

>Users have the ability to upload certificates for arbitrary names without proving domain control.

I haven't seen anything on our internal security mailing lists about this. If it does somehow involve Cloudflare I'd be happy to receive a report directly via HackerOne (https://hackerone.com/cloudflare). Happy to assist.

Sorry, I just chose cloudflare as a random example when speculating, I don't have any information about what specific providers this would affect, and I'm not implying that cloudflare would be affected.

Seems like my guess was right, though!

No, need to apologize.

When you say "Seems like my guess was right, though!" are you saying that there is some way that Cloudflare is involved?

EDIT: I see (https://community.letsencrypt.org/t/2018-01-09-issue-with-tl...) now. I don't think LE has been in contact with us so don't think we're affected but happy to be told otherwise.

I would indeed like to explicitly apologize, because I regret mentioning "cloudflare etc" as an example. That's exactly the kind of bad speculation that leads to harmful rumors based on misunderstandings.

> When you say "Seems like my guess was right, though!" are you saying that there is some way that Cloudflare is involved?

No. It seems like I was right about the general nature of the vulnerability. It remains to be seen what providers are affected.

That being said, at this point, I'd personally be happier seeing a list of providers NOT affected, rather than a list of affected providers... It's probably also in "major public CDN provider's" interest to demonstrate that their user cert validation processes would have prevented this attack, and their customers were not at risk before LE pulled the plug...

One of the requirements listed [1] as being needed for exposure to this for the provider is:

Users have the ability to upload certificates for arbitrary names without proving domain control.

When you say you were right, are you saying Cloudflare allows that?

[1] https://community.letsencrypt.org/t/2018-01-09-issue-with-tl...

I don't think CF is affected because TLS-SNI-01 has never worked for cf-proxied sites (I've tried). CF doesn't allow clients to use invalid certificates for domains they don't own, which is key for the tls-sni-01 process.

Too late for me to edit this comment anymore, but in order to avoid spreading any false rumors, I'd like to explicitly state that I have no reason at all to believe that Cloudflare in particular would be affected by this, and I do not wish to imply that customers of cloudflare would be at risk from this vulnerability. That was just simply the first example of a public CDN deploying user-provided certificates that came to mind.

The big public CDNs would certainly have the most impact from this, but also the most likely to get their cert validation right. I'd reckon that the risk is probably in the larger number of smaller providers that have their own home-grown cert automation for deploying user-provided certificates.

In fact, it seems a little odd to me to hear LE talking about collecting a blacklist of vulnerable providers to prevent from using TLS-SNI... TBH, how can you tell what providers will be affected... should that be a whitelist instead?

Indeed, I just tried a CDN provider that I recalled allowed custom SNI certs, and was able to deploy a ".acme.invalid" certificate to the global network. It's a real issue for public CDNs, but we can rejoice that Cloudflare and (I assume) Cloudfront are not affected, at least.

I don't think Cloudflare/CDNs are affected since TLS-SNI-01 doesn't work for cf-proxied hosts. (I've tried.)

Reading the detailed report[1], it sounds like tls-sni-01 requires being able to serve an invalid certificate temporarily. As far as I know, CDNs won't let you do that. This problem is probably more related to shared webhosts (which is hinted at in the detailed report).

Thanks for sharing this! Please share it on your blog as well https://letsencrypt.org/blog/

I subscribed the RSS to keep me up-to-date

I just read the details of the tls-sni-01 challenge again.

I can certainly see a problem. With respect to the final validation stage, where the validator is communicating with the TLS endpoint, all the information needed to satisfy the validator is presented IN the SNI request data.

It looks like anyone could request a tls-sni-01 validation with any account key, for a host that they new would resolve to a TLS server infrastructure with a certain behavior and in turn be able to get the certificate issued.

The qualifying behavior would be a TLS server infrastructure which has some sort of opportunistic generation of a self-sign certificate for an unrecognized SNI name, such that the just-in-time minted certificate would have a single SAN dnsName being that exact value requested in the SNI. Unlike the other challenges where the final validating query from the VA does NOT contain the "answer" needed, it would appear that the in the final pass of the TLS-SNI-01 protocol exchange, the final "answer" is indeed in the request.

You know, I'm trying to recall the details, but I'm pretty sure I came across a consumer router platform once that, until and unless administratively configured with a certificate, would keep opportunistically recreating certificates to match the SNI name. This was to make the only validation failure be "signed by an untrusted..." instead of "name mismatch", etc, etc.... I wonder if a critical mass of things like that has arisen?

If tls-sni-02 is affected too (as Josh indicated it probably is below), I'd suspect something like a major CDN or hosting provider allowing deployment of arbitrary, attacker-controlled certificates on arbitrary domains under "acme.invalid". That would be the only thing I can think of that would bypass the "anti-parrot" mitigation introduced in -02.

Otherwise, it's probably what you're describing.

I can totally believe that may exist too.

After all, for the CDN / Hosting company, what's the real risk?

It wouldn't shock me if lots of infrastructure lets you board a new and novel host label that hasn't already been boarded and starts letting you configure things like a custom TLS certificate to be associated with that label.

As long as they watch out for uniqueness, what's the risk to the CDN? That a config context gets created for a DNS label that isn't yet pointed there? It's not an obvious risk from the CDN's perspective. Even for an invalid domain.

"Oh, sad, I just wasted mere kilobytes of storage on a configuration for a domain that you're never going to actually be able to get into the DNS and point to me?" That sounds kind of low cost, from the CDN's perspective.

If it's something like that, the fix would be to define a tls-sni-03 with a couple changes:

The SNI name indication from the validation MUST be a child element desired certificate domain label.

(If I want a cert for a.com, the SNI indication in the TLS-SNI-03 would need to be something like obvious-acme-challenge.a.com)

Further, the self-signed cerfificate needs to have the SAN dnsName as in the SNI -- AND ALSO -- another authentication token signed by the account key stuffed into the certificate in some other certificate field.

What is the reason to enforce a specific root domain?

tls-sni-02 requires the generated self-signed certificate to contain an additional subjectAltName (SAN B) that is not send as part of the request and thus only known to the actual client, not to any host that automatically generates self-signed certificates (if such hosts exist).

It eliminates one possible foot gun.

A web host or CDN might allow arbitrary domains to be added, and arbitrary certificates to be uploaded for said domains, all without any validation. That would ... not be my favourite implementation, but if you didn't know about how the Web PKI and ACME works, that's an implementation you might come up with and not expect a whole lot of issues.

However, it's unlikely that the web host would allow you to do this for a domain already associated with an account, or a subdomain of such a domain. Unlike the first case, this would effectively allow an attacker to fully control any subdomain, so even without any Web PKI involvement, that would be a vulnerability in and of itself.

It remains to be seen what the two affected providers were actually doing, and I don't really have enough data to make a call on whether it's actually worth changing this aspect of tls-sni-02, but it's something to consider.

Thanks for the update and good work. Are you able to share and/or clarify whether the TLS-SNI-02 challenge is affected as well, in addition to TLS-SNI-01 which is currently Production on LE?

Haven't put much thought into -02 since nobody actually runs it in production but I suspect the same issue applies.

Curious question: Why do you write those details here on HN and not on the Letsencrypt Blog or letsencrypt.status.io?

Are HN readers somehow more entitled to these information that those who "merely" subscribed to your blog's RSS feed?

I agree that a blog post would be more helpful, but the barrier to creating one is higher. This sort of update wouldn't fly there because its an off-the-cuff status report that will be buried in a day or two. The Let's Encrypt blog will be available and readable for all eternity (give or take a few years), so it requires an update that's more in-depth and vetted by all the people involved in solving this issue.

I'd imagine that once they fix this issue the first thing they'll do is to post a post-mortem on their blog.

I see. So this is more to be interpreted as a rushed preliminary report, forced upon them by appearing at the HN front page, to prevent uninformed rumor to appear and to spread. Or, did I miss something?

EDIT: Apparently the latter didn't work, as the very first response to their comment starts with "Do we get points for speculation based on these hints? ..."

It prevented _uninformed_ rumours :D

Almost all the speculation was about the exact scenario that Let's Encrypt have subsequently confirmed - you can trick some bulk hosting or CDN providers into letting you (a customer) answer a challenge for one of their other customers' systems.