GitHub's User Content certificate has expired | Hacker News Reader

GitHub's User Content certificate has expired

github.com

126 pointsGOATS-47 comments

The cert for objects.githubusercontent.com has also expired:

    $ openssl s_client -connect objects.githubusercontent.com:443

    CONNECTED(00000005)
    depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
    verify return:1
    depth=1 C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
    verify return:1
    depth=0 C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io
    verify error:num=10:certificate has expired
    notAfter=Mar 21 23:59:59 2023 GMT
    verify return:1
    depth=0 C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io
    notAfter=Mar 21 23:59:59 2023 GMT
    verify return:1
    ---
    Certificate chain
     0 s:C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io
       i:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
     1 s:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
       i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA

What are the odds this happens the same day they rotate their SSH keys?

It's also the domain used for releases and other artifacts (after a redirect from github.com). There's going to be a lot of broken builds today:

    $ curl -i -L https://github.com/kyleconroy/sqlc/releases/download/v1.17.0/sqlc_1.17.0_linux_amd64.tar.gz
    HTTP/2 302
    server: GitHub.com
    date: Fri, 24 Mar 2023 20:51:56 GMT
    content-type: text/html; charset=utf-8
    location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/193160679/09048595-c7f4-45b5-858a-7f55baa2fd7d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230324%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230324T205156Z&X-Amz-Expires=300&X-Amz-Signature=772d0aa8c5c19b0a5ef84d718d2faf0d81f24b224a4ef634d2410787e8f50bad&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=193160679&response-content-disposition=attachment%3B%20filename%3Dsqlc_1.17.0_linux_amd64.tar.gz&response-content-type=application%2Foctet-stream

    curl: (60) SSL certificate problem: certificate has expired
    More details here: https://curl.se/docs/sslcerts.html

    curl failed to verify the legitimacy of the server and therefore could not
    establish a secure connection to it. To learn more about this situation and
    how to fix it, please visit the web page mentioned above.

> What are the odds this happens the same day they rotate their SSH keys?

Definitely a bad for them. When it rains, it pours.

They laid off the wrong person.

This is so comical because it's so relevant.

Do you think people architect poorly designed systems more often than not as a means of job security or just a failure to put much forethought in whilst planning it?

Failure of forethought with the wrong deadlines in place, soup to nuts.

Or maybe just laziness.

I know someone who joined a company and found a dead-man's switch in the server.

He could have taken it out, but instead he just resets it every three months, just like the guy before him.

If the company ever gets rid of him and doesn't hire someone equally skilled and thorough, the production server will eat itself right about the time his unemployment benefits run out.

I’d more bet on “Warning for years that this is a point of failure but management wants to chase $shiny instead”

I guess copilot can't write monitoring rules.

> I’d more bet on “Warning for years that this is a point of failure but management wants to chase $shiny instead”

I'd almost guarantee you're right on the money with that line of thinking…

Or the team is on a new project and after ten attempts to get new owners have an outlook rule to delete any mails about the old project.

The only way to do cert renewal at an org level is one well organized team of not creative software types. yeah yeah the team will automate but in the meantime someone has to check all the dates carefully. And usually good public certs can't be fully automated, at least in the deploy bit.

I heard about a new cert once with a longer private key that cauaed all the terminating F5s to fall over due to out of CPU

cert-manager though amiright?

Like what's going on there?

This seems to now be fixed.

Could be a good chance. I'd venture to guess they failed to update the known_hosts file for one of their systems that handles certificate management. Strictly me taking a stab at the answer though.

Now they are owned by Microsoft, its a celebratory 10 year tradition. Cause a worldwide outage, by letting your certificate expire...

"Windows Azure Service Disruption from Expired Certificate" (2013) - https://azure.microsoft.com/en-us/blog/windows-azure-service...

They're serving the wrong cert on pkg-containers.githubusercontent.com (it's for *.githubassets.com) and their support site also expired 3/21... https://support.github.com/ What is going on over there?

Still some weird stuff around (* subject: CN=apistatus.chorus.co.nz).

    curl https://www.githubstatus.com/ -vvvv -I
    \*   Trying 52.215.192.131:443...
    \* Connected to www.githubstatus.com (52.215.192.131) port 443 (#0)
    \* ALPN: offers h2
    \* ALPN: offers http/1.1
    ...
    \* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
    \* ALPN: server accepted h2
    \* Server certificate:
    \*  subject: CN=apistatus.chorus.co.nz
    \*  start date: Mar  6 23:10:30 2023 GMT
    \*  expire date: Jun  4 23:10:29 2023 GMT
    \*  subjectAltName: host "www.githubstatus.com" matched cert's "www.githubstatus.com"
    \*  issuer: C=US; O=Let's Encrypt; CN=R3
    \*  SSL certificate verify ok.
    \* Using HTTP2, server supports multiplexing

Here is the report for this incident: https://www.githubstatus.com/incidents/x7njwb481j9b

EDIT: this specific issue is resolved

Failing for us in GitHub Actions

For SEO purposes:

  npm ERR! code ERR_TLS_CERT_ALTNAME_INVALID
  npm ERR! errno ERR_TLS_CERT_ALTNAME_INVALID
  npm ERR! request to https://pkg- 
 npm.githubusercontent.com/npmregistryv2prod/blobs/\*\* failed, reason: 
  Hostname/IP does not match certificate's altnames: Host: pkg-npm.githubusercontent.com. is not in the cert's altnames: DNS:\*.githubassets.com, DNS:githubassets.com

This also applies to their avatars subdomain, causing them not to load anymore.

Are you sure? avatars.githubusercontent.com works fine for me.

> Are you sure? avatars.githubusercontent.com works fine for me.

That’s because they already resolved the issue → https://www.githubstatus.com/incidents/x7njwb481j9b

This is what I saw an hour ago:

        $ echo | openssl s_client -connect avatars.githubusercontent.com:443
        CONNECTED(00000005)
        depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
        verify return:1
        depth=1 C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
        verify return:1
        depth=0 C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io
    >>> verify error:num=10:certificate has expired
        notAfter=Mar 21 23:59:59 2023 GMT
        verify return:1
        depth=0 C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io
        notAfter=Mar 21 23:59:59 2023 GMT
        verify return:1

I wonder if this has anything to do with the recent SNAFU from a Senior Security Engineer* there?

https://twitter.com/viibeeng/status/1639374358287118336

(*yeah we can all make mistakes, but it's 2023, if you've not build controls into your workflows by now you don't deserve to be a Senior anything)

I built a free monitoring service some years ago if anyone doesn't want to be the victim of this...

https://ismycertexpired.com/check?domain=objects.githubuserc...

It looks as though it's back for me now. Status page is now showing the problem: https://www.githubstatus.com/

… well, the status page is back to green now, but AFAICT, the domains are still serving the expired cert.

  » TIMEZONE=UTC date; openssl s_client -connect support.github.com:443 2>&1 | grep 'cert.*has.*ex'
  Fri Mar 24 17:40:28 EDT 2023
  verify error:num=10:certificate has expired
      Verify return code: 10 (certificate has expired)

The previous incident seems pretty clearly to be this … so it seems like they think they fixed it…

Also serving wrong certificates for a lot of content domains.

https://news.ycombinator.com/item?id=35295191

Sounds like whoever is in charge of certificates at GH must have come over from MSFT. Afterall, I think Microsoft has had 2-3 certificate expiry issues in the last several years.

Azure had several global outages because of issues with certificates. One outage was caused by an incorrect date computation: the certificates last for one year, and this was computed with: "new DateTime(now.Year+1,now.Month,now.Day)".

If you do that on Feb 29th of a leap year, it'll throw an exception because the next year doesn't have a Feb 29th! Oops.

They "fixed" it and promptly had another related outage the very next day.

That was one of them I recall.

And today of all days I have a moment to upgrade homebrew stuff.

got a "RequestError: certificate has expired" doing a release just now...as usual, not a good idea to release on a friday

Previously I had the same issue, but it works for me now, as well as for a friend in another EU country.

For us dumb dumbs what does this mean?

Seems to be resolved now. My `brew update` works again.

ChatGPT, rotate my certs

It needs a plugin for that.

Did we learn nothing from Son of Anton?

Why? Don't most cloud providers have auto-renewing certs now?

Well I'm kind of just waiting on PRs for the rest of the day today and it's a Friday, so I'll consider this a modern equivalent of https://xkcd.com/303/

It's back now!

didn’t they announce a bunch of layoffs recently?

Back up now, it looks like.

Not Before Fri, 18 Mar 2022 00:00:00 GMT

Not After Tue, 21 Mar 2023 23:59:59 GMT

3-day certs.

There's a whole year between those dates.

3 day plus 365 days ;) you're missing a full year

the year is different

Short lived certs are a thing, also

Reply on news.ycombinator.com