GitHub was down | Hacker News Reader

GitHub was down

githubstatus.com

117 pointsbombcar40 comments

The status page shows stuff about slack https://www.githubstatus.com but it seems authentication is down hard, can't pull, comment, or even view public repositories when logged in.

Edit: https://www.githubstatus.com is updated as of 19:08 UTC and everything is on fire

Open articleView on HN

Implementing that type of multi-account support likely had some big authz changes (I've done a couple of those projects before), and GitHub defaults to a 404 for authz errors instead of leaking information with a 403. Just a wild guess.

The fact that https://github.com/ (which requires no specific authz check) seems to be up, and that some users report logging out and in again at least temporarily restores access, seems to support that hypothesis.

Still seems odd that no CI/QA would have caught such a widespread bug.

Oh if you only knew the code base they are up against. Or in other words “what QA <laughing.emoji>”

spot on:

> Update - We’re in the process of rolling back an authorization-related change that is causing 404s and other errors.

Like I said, not my first rodeo for implementing a multi-account system on top of a single-account system lol.

No, it was an authorization internals upgrade we were testing, not the account switcher.

nailed it man

Nobody would deploy such a change on a Friday afternoon though

#famouslastwords

It kinda makes sense though to do it on Friday afternoon, as on the weekend enterprises and most people just go skiing or do other things so less of customer base is affected in case of (unfortunate) outage.

Even if that were a fact, then that's the one time github should not be updating. Their customers will be doing deployments on a Friday afternoon, thus require their repositories for their internal processes to work

What day is the best day to have downtime?

Depends on your business. For my workflow we are happiest with risk for weekdays 0930-1130 and 1400-1600, and Saturday from 0500-1100, but it varies depending on the affected systems and workflows. If I'm doing work on one systems used by Department A then a Saturday is great. If it's on Department B though Saturday is the worst possible time. Department C's systems are mainly used from 2000-0700. Department D's systems are used 24/7, which is why we have geographic resilience.

But you don't get to choose this with XaaS. It goes down when it goes down, it comes up when it comes up. Our connectivity supplier sends us a notice that "circuit from X to Y will be going down from 0200-0500" and we have no choice but to accept it.

Typically updates are best done on Tuesday morning given that most people are awake and working and thus if there are problems they can fix it, but you wouldn't go ahead while the resilience is down (say due to a fibre cut, or a power outage)

Getting 404 on all github.com repos, oh boy lol

I got a glimpse of some code on a reload of notifications, and then - poof.

+1 same here

same here!

We, uh, fixed the glitch^W^W all bugs

Yep, only 404 or 500.

They were experiencing issues with Slack at the beggining of the day.

If I had to guess, a fix had unexpected consequences or it's related to the release of multi-account login https://github.blog/changelog/2023-11-03-multi-account-suppo...

This is nature's way of telling us to stop working on Friday, y'all.

Good thing that Git is decentralized.

I mean yea, I'm still happy that I can commit locally. With a centralized VCS like Subversion, I wouldn't even be able to do that.

Yup, I'm also seeing very sporadic issues with 404 pages and some API actions not being able to complete, though it seems mostly related to private repositories for me.

Seems intermittent. Sometimes a refresh will work to display files and commit history.

Tell HN:

[dupe]

Status discussion over here: https://news.ycombinator.com/item?id=38133297

first cloudflare... now github... goodbye internet

I run my own git server and users clone from that.

smug ;-)

What we need is something like github but decentralised somehow, if such a thing is even conceivable.

Notice that there were issues with Slack notifications only before:

> The delayed Slack notifications should be fully processed in about 30 minutes > (Nov 03, 2023 - 18:25 UTC)

Exactly 30 min after:

Might be coincidence, might not be...

Logging out and back in seems to .... somewhat make it work? So maybe that could help.

Nope, still busted.

I've tried multiple times + incognito while trying to access a public repo and still randomly get 404.

I bet there is something wrong with caches.

Yep, can't push to repo. Can't view repo. Thanks guys.

chill dude, shit happens

Yep, seems to be down for me too.

Is there a major service that has a worse uptime than Github? I can't think of one that feels less reliable.

Microsoft 365

seems to be back now

I left that platform when M$ took over. Only have an account because of the many stubborn people still clinging to it and I have to tell them when I find a bug.

Especially the forced 2FA.

Whatever will you people do now, that you can't use co-pilot to code for you? It's the end of your career I'm sure. I'm sorry but I can't help being cynical on HN especially.

> Especially the forced 2FA.

How dare they protect their users?

Reply on news.ycombinator.com