Hacker News Reader

Comment by Karunamon | Hacker News Reader

And what exactly would that cost be? Can it even be quantified?

We're speaking of a 4x yearly ACME request of a few kilobytes going to and from the LE servers. It certainly isn't bandwidth.

The engineering work was already done, and doesn't change based on the number of instances that are making the cert request. It certainly isn't labor.

It certainly isn't storage. Certs are 2-4 kilobytes. Even if we assume Heroku has 5M active dynos and each one has its own unique cert, that's only 20 gigabytes worth of certs, which is miniscule.

So where's the cost coming from? Answer: It isn't. This is just tier differentiation, not cost recovery.

View on HN

Oh my - so, it's actually a lot more complicated than that. Let me run it through for you:

- You have to build enough of a retry algorithm so that you start renewing well in advance of the expiration date.

- You then have to build the mechanism for warning customers that there was an issue renewing for one of a variety of reasons

- You then have to deal with situations where LE has issues, which happens fairly often

- There's a queueing system, where you have to handle not sending too many certs at once

- You will end up in scenarios where users will migrate off of you, not tell you, attempt to issue another LE cert with another service, and fail, and then blame you

- Similarly, you will have users who connect and disconnect domains, and your system has to be smart enough to properly revoke certificates without locking out a domain from too many retries

- and then what happens when you can't renew a cert for whatever reason? Do you break the user's site? Do you fall back to http?

I'm not saying it's millions of dollars, but at scale, it's complicated. Here's a blog post about how Squarespace did this (disclosure, I work there):

https://engineering.squarespace.com/blog/2016/implementing-s...

Saying it's "just" a 4x acme request annually demonstrates a real lack of understanding of supporting this kind of system at scale.

I didn't say that it was easy. I enumerated three things that it can't be.

Given that Heroku is already generating certs on the fly for the (randomly named) dynos, offering that is even more engineering work than just sending a cert request for a custom domain, the numbers of which will be significantly smaller. [DISREGARD THIS - no they're not. Those are all on a single wildcard]

My whole gist here is that every conceivable practical excuse I can think of for not extending that feature out to custom domains resolves as a nonissue, which only leaves feature differentiation to drive sales as the remaining option.

Which, as mentioned before, is a legitimate business tactic, but morally evil in the security climate of 201x+.

They aren't generating certs on the fly for randomly named dynos:

  -bash-4.1$ curl -vkLs https://wind-river-5693.herokuapp.com 2>&1 | grep certificate
  * Server certificate: *.herokuapp.com
  * Server certificate: DigiCert SHA2 High Assurance Server CA
  * Server certificate: DigiCert High Assurance EV Root CA
  -bash-4.1$

Foo! Okay, I thought they were doing LE certs for every dyno's name. One wildcard makes a lot more sense.

FTA:

  ACM handles all aspects of SSL/TLS certificates for _custom domains_;
  you no longer have to purchase certificates, or worry about their
  expiration or renewal.

Emphasis mine.

But this is all engineering work that needs to be done for the system at all. None of these things have incremental costs that justify hiding encryption behind a paywall.

Saying that "that's only 20 gigabytes worth of certs" is roughly equivalent to saying "this database is only 1.8TB - that's miniscule, 2TB drives are only $70!". There's a whole lot more moving parts, especially at scale (developing a system to manage 5M certs? Pretty sure that's a whiteboard interview question somewhere).

Please read this conversation with the context in mind rather than replying to the one statement. When you're referring to costs as a category, as that statement was, the "moving parts" aren't factored into storage costs.

I have, and I believe that you're being extraordinarily reductionist about the true costs involved in developing and operating services like certificate management at scale.

> The engineering work was already done, and doesn't change based on the number of instances that are making the cert request. It certainly isn't labor.

And what pays for that engineering work? The money you make from having the feature.

Reply on news.ycombinator.com