Hacker News Reader

Comment by jeffbee | Hacker News Reader

Quick reminder from your friendly local SRE: never ever issue certificates that expire on weekends. Make certs expire in the middle of the afternoon on a business day wherever your operators live and work. The cert in question expires at May 30 10:48:38 2020 GMT, which smells suspiciously like a fixed time after the cert was generated, rather than at a well-chosen point in time.

View on HN

All my applications use a component that watches certs configured (everything in cert and trust store) and returns warning in telemetry from the application if any of the certificates is less than a week from expiration. This is checked periodically while the application runs.

This not only makes sure we don't miss expiration but also ensures we don't forget to configure any of the application.

We had a situation when the cert was replaced but the file was placed in incorrect path and was not actually used by the app. Having the app report on what is actually being in use is the best way to prevent this from ever happening.

Good old "cert replaced but apache/nginx failed to reload" has bitten me more than once...

Me too! Especially with the short expiration times of LetsEncrypt. But I really don't want to put `nginx -s reload` in the Cron, in case I'm tinkering with the configs and they're suddenly live (which only really happens at staging or at home of course, but still).

You can use `nginx -t && nginx -s reload` for that.

It will first check the configs/paths, and only then, if successful, signal nginx to reload.

That's what I usually do. My problem is that I might be adding a location and nginx reloads between that and adding access restrictions (i.e. because I took a break to google).

Certbot has deploy hooks which is where I'd put the nginx reload statement. The hooks are run automatically when a new certificate is issued.

Oh, that's a great idea! Thanks :)

I've used this https://manpages.debian.org/testing/nagios-plugins-contrib/c...

After one scrambling emergency with a cert expiring in the middle of the day, a constant check with warnings and alerts a couple of weeks before expiry made a matter of defensive organization into something trivial.

There is just no substitute for Reality!

If you get to the point where the exact expiration date on the certificate matters, you've already lost the game.

Engineering for failure is important, you should always set yourself up so that you have several lines of defence which can fail. Some lines of defence to make failing "impossible" others to make a fail softer, even when you think failing is impossible.

Defense in depth.

it’s more like blue m&m’s than an actual requirement

>it’s more like blue m&m’s than an actual requirement

Did you mean Van Halen's famous "WARNING: ABSOLUTELY NO BROWN M&Ms" clause?

https://www.snopes.com/fact-check/brown-out/

Probably. It’s an honest mistake. For Gen Xers and early millennials blue m&ms are memorable for being added in the 90s with much ludicrous publicity.

Great tip. Did you notice that cert in this case was issued 20 years ago? It’s crazy to me that it was still being used to sign certs as recently as last week (according to twitter)

Of course, but that doesn't really excuse them. My first experience with middle-of-Sunday-night SSL certificate expiration was in December 1998, and it was already a well-known doctrine by then. I'd expect a commercial certificate authority to have these kinds of things squared away.

My experience with commercial CA is that they set the expiry exactly 1 year from creation. Doesn't matter if it's a week end or a holiday.

Generally that’s for Server certs, Roots and Intermediates will be multiple years from what I’ve seen. Roots in particular 10+ years.

1 Year is a relatively recent thing. Previously you were able to buy server certs with a 5 year expiration.

It's actually worse. The new root (good I believe until 2038) uses the same key as the now expired certificate. It has to or it would not be possible to validate the certificates that were issued. And this new one is a root certificate installed in browsers!

What "should" happen is that no certificate should be issued with an expiration date later than the issuing certificate. Then as the issuing certificate gets closer to expiration, a new one, with a new key pair, should be created and this new certificate should sign subordinate certificates.

Sorry to reply to my own comment. But I want to clarify. Two certificates (at least) expired. The root named "AddTrust External CA Root" and a subordinate certificate with a subject of "USERTrust RSA Certification Authority." Both expired around the same time.

The "USERTrust RSA Certification Authority" certificate signed yet another layer of intermediate certificates.

The "USERTrust RSA Certification Authority" certificate was promoted to a self-signed certificate, now in the browser trust stores, using the same key pair as the original certificate that was signed by "AddTrust External CA Root." It has an expiration of 2038 (although that concept is a bit vague in a root certificate).

There's actually a third certificate for "USERTrust RSA Certification Authority", also using the same key pair, signed by a different root called "AAA Certificate Services". It looks like the intended replacement for the expiring one is this one, rather than the one where it's the root itself.

It is explicitly not a replacement, but some kind of legacy fallback that they don't want you to use, but exists for enterprise customers that absolutely can't get trust.

Are you sure? That's the path that InCommon has been providing me for new certificates since they switched away from the expiring one.

On anything with a modern TLS stack, I see this trust chain:

- Leaf cert (your cert)

- InCommon RSA Server CA

- USERTrust RSA Certification Authority (this is/should be the final point)

That's what my browser shows me too, but it's just because it's ignoring the cross-signed one that chains to AAA. The server is sending it, per InCommon's setup instructions.

That's correct

The old TLS (versions 1.0, 1.1, 1.2) specifications said that the certificates supplied are to form a chain, starting from a leaf and leading back towards a root.

Pretty much all clients assume that once they can see a way to a root they trust they'll give up following the provided chain and trust that - but sadly not all of them, so "over-specifying" the chain can cause problems.

Modern clients tend to go further, they still assume the first certificate is a leaf, but all other certificates are just potential hints that might be helpful in working out an acceptable trust path. TLS 1.3 actually specifies that clients must tolerate certificates supplied on this basis rather than a strict "chain".

I'm actually surprised at the number of claimed clients which don't have vaguely modern trust stores but do understand SHA256.

> I'm actually surprised at the number of claimed clients which don't have vaguely modern trust stores but do understand SHA256.

All the clients were limited to SHA-1 have already been forced off https; CAs in the CA/Browser forum weren't permitted to issue SHA-1 certs valid past Jan 1 2017, and you had to have gotten those issued before Jan 1 2016. Browsers were showing warnings on SHA-1 certs depending on expiration throughout 2015, so you had to either put up with a warning (and the customer service burden thereof), ditch your old clients and go sha-2 only, segregate traffic, or build custom software to send sha-1 certs to some people and sha-2 certs to others.

Microsoft added support for sha-2 certs in the OS system stack with XP Service Pack 3, released in 2008, and Microsoft was always pretty slow with support on things, other platforms may have supported this earlier. A CA bundle from like 2005-2008 is going to be fairly limiting today. A lot of CAs back then had a 20 year validity period, which may have started 5-10 years before the bundle date. Of course, a lot of bundles today end in 2038, so we'll be screwed then.

Certificates issued with this CA will have been cross-signed by the newer root certificate, but our CA (Sectigo) was sending the old chain in issuing emails as late as April this year, despite the cross-signed root being available for a long time.

> It’s crazy to me that it was still being used to sign certs as recently as last week (according to twitter)

It's likely because it was issued 20 years ago. People have been using it for 20 years and no-one realized it was about to stop working.

> rather than at a well-chosen point in time

So, you're saying that "I'm not going to be working here anymore by then... hahahaha" isn't well-chosen?

Also make a Calendar placeholder (like a fake meeting), invite a lot of folks or a distribution list, and turn on an alert for 24 hours ahead.

For long-lived certificates they will outlive your calendar tech. The bit rot leave maybe the events fine, but anything fancier like notifications will fad away.

Source: BTDT 3 times in 7 years, and it was all with "Big Enterprise" grade products.

> Make certs expire in the middle of the afternoon on a business day wherever your operators live and work.

If I can predict that 20 years into the future I wouldn't be in the SRE business.

I think it was foreseeable even in dark days of the year 2000 that this certificate expired after business hours globally.

But your statement is really pointing out that nobody should be making long-lived certificates.

If you were to issue certs for short durations, and also maintain a calendar of cert expiry, those certs could be renewed in a timely manner.

In other scenarios where one would want to issue fresh certificates (receiving Ops control from other orgs, credentials refresh for what ever reason), one can still do so without waiting for the current certificates to expire.

"Middle of the afternoon" .. for who?

If you're going to ask a question about someone's comment, at least finish reading the sentence to make sure it's not immediately answered.

Don't be part of the death of internet discourse.

I read the comment several times.

I am just of the opinion that "make it expire in the afternoon" doesn't apply to root certificates that are used across the entire world (i.e. - the topic of discussion)

Reply on news.ycombinator.com