GitHub was down | Hacker News Reader

GitHub was down

githubstatus.com

143 pointsFrostKiwi69 comments

Glad to see the status page being honest and reflective of reality for a change.

It was maybe 5 minutes late, but that is pretty decent seeing as most status pages are never updated ... ever.

They've had a lot of practice

Most status pages seem to be hosted on the same infra they're monitoring, so they tend to go down with the ship, while visitors are greeted by a conveniently cached "all green" version of the page.

Not surprising when one realizes that status pages, by necessity, become marketing tools.

I had the same feeling. When my PRs started failing, a few mins later I went to the status page and got immediate feedback that they were aware of it. That was useful.

Ohh, back up for me here in Japan. GitHub's incident team is on crack, that turn around time is quite something. Definitely some impressive stuff.

I haven't seen "on crack" used in a positive light, but apparently it is sometimes? Usually I'd say "they must have a crack team on incident response", meaning a highly functional, smooth-running team.

Its pretty unusual, I think it has to do with their local time coupled with a presumed enthusiastic and swift response on the part of github.

I'm more used to 'on crack' meaning like 'delirious, over confident'.

Yeah "on crack" usually means they're not right in the head.

In this context though, OP is referring to the drug where the Github engineers are being quick/energetic about fixing the issue because they're "boosted" by the drug's effects.

Yes, they obviously meant the more common, positive associations related to methamphetamines like, "great job on that quick PR, Kim. You must be on meth!" :)

Or “gee, Hunter Biden is a swell son, he sure got to Vegas really fast! He must be on crack!”

Huh, I swear I heard in a positive context before. But English in not my native language, so my feel on the nuance of this expression is probably a bit off.

I’ve heard “on crack” used similarly to “on steroids,” as an American English speaker.

For example, it would not be out of the question to say something like “when the iPhone launched it was amazing. Like a palm pilot on crack.”

So, when used positively, “on crack” is generally a modifier(in my experience). When used alone it does seem to have negative connotations.

There's a subtle distinction. "Like ... on crack/steroids" means a really good / fast version of something (I think steroids is probably the more common variation) e.g. "an e-bike is like a bicycle on crack". "They're on crack" means "They are crazy".

I think it's used like that for products rather than humans. "X is like Y on crack."

I love when people say "like crack" when they've obviously never done crack. - Ryan, The office.

To be crack -- as in "a crack shot" or "a crack team of commandos" -- is to be seen as expert.

To be on crack is entirely different -- definitely not expert or good.

It is sometimes used to refer to someone who has unusually high energy, speed, or productivity - to accuse them of using a stimulant.

Crack cocaine has a wide range of effects. Some people just get energetic, others become disorganized to the point of extreme dysfunction.

To be "on crack" is to display any of these qualities; sometimes good, sometimes bad.

I usually see it used to describe a positive trait in the form of high energy / confidence.

> To be crack -- as in "a crack shot" or "a crack team of commandos"

To explore the quirks of English grammar a bit further... While "the general expected this to be a crack team of commandos" flows fine, I've never seen "the general expected this team to be crack".

Perhaps it's similar to the rules for the adjective "top"? One can have a "top team of athletes", but "this team of athletes is top" doesn't work, it needs to be "topmost" or something.

IANAGrammarian, but it's almost as if "crack" isn't being used as adjective that modifies "team", but instead they've merged to become a single noun of "crack-team"... kind of like how "first-responder" is a category of emergency-related jobs rather than literally any person who happens to be the first to arrive.

I believe it is derived from "crackerjack".

I think the word you were looking for was “cracked”.

https://www.urbandictionary.com/define.php?term=Cracked

No its not. Cracked is a term which is usually used in gaming. The term "on crack" is used to denote someone who is on very high energy like someone on literally crack the drug.

Seems like it's been a common occurrence ever since Microsoft took it over

got any data on that, or are you just using this as an opportunity to take a stab at a company you don't like?

either is fine, just be open about it.

Nothing wrong with hating Microsoft, it's a multiple felon.

There was an outage yesterday too when the GitHub action “checkout@v3” broke when they released “checkout@v4”

Yes, they broke the ability for GitHub CI to checkout repos…

https://github.com/actions/checkout/issues/1448

this is offset by taking your job slightly seriously when implementing gh actions for your org by understanding how they're resolved and pinning them to commit SHAs, like how github themselves recommend under their basic security guide

https://docs.github.com/en/actions/security-guides/security-...

after hearing similar stories from frontend developers who haven't even heard of "npm ci" or use range pinned dependencies on production systems, im almost never going to be sympathetic on blaming the package maintainers over developer ignorance following basic security guidelines

Sure, but then again the link you sent says: `Pin actions to a tag only if you trust the creator`

If I don't trust GitHub, why am I storing my source code on it and running CI via GitHub actions?

There are also some security gains from ranged pinning. Suppose I pin my library to `1.1.x` instead of `1.1.0`, when a security patch comes along for some CVE, my service will automatically run the `1.1.1` patch release the next time we deploy it.

I would say the likelihood of a developer getting lazy and not bumping a dependency causing a security incident is higher than a supply chain attack if you're already using reputable libraries.

Let's be frank, how many developers care enough to go into the 10 repos they've ended up owning to go and change their `actions/checkout@v3.1.6` to `actions/checkout@v3.1.7`.

Didn't notice it this time! At work we use bitbucket, and for the last few weeks I've been hacking on a hobby project on a 37 year old Mac. It can't run Git, but it does have Amend[1]!

[1] https://amendhub.com/

Wtf is going on at Github? They're having outages almost every other day.

August had 11 incidents it seems. September already has 3.

https://www.githubstatus.com/history

They run on Azure, don’t they?

Only partially I believe, there quite famous for running their own hardware (metal cloud). I think some new features like Actions, Codespaces and of course Copilot run on Azure.

They are doing rapid deployment.

Lots of attrition of good engineers. Management is not held accountable for failures, only engineers are scapegoated. As a result, remaining engineers are disgruntled and don't care.

Source: https://www.teamblind.com/company/GitHub/reviews

Blind is an echo chamber, and attracts a certain type of person, for sure.

Vast majority of engineers I know aren't even on it and often haven't heard of it.

I think Blind is a data point, but its one of many, and certainly not definitive in my opinion

It's like a bitter GlassDoor.

> Blind is an echo chamber, and attracts a certain type of person, for sure.

It certainly is a misery-loves-company community. But I have found them to be the most honest representation of a company and its tech and management practices. It is also a reliable source of compensation data.

For example, I look at Meta's reviews there and that matches my experience. Nobody cares about their apps. All people care about is performance reviews with fake impact. That was exactly my experience in the company.

To add some nuance to what I'm saying:

I've found its accurate for Meta / Microsoft / Google / Apple / Netflix / Amazon. Having worked at Apple for half a decade, it seemed (mostly) accurate.

I have found it less accurate for other companies that are big and desirable but not nearly as sought after to work for in the last decade. For startups or smaller companies, its not the best data point

Don't know where I'd put GitHub, maybe the signal to noise ratio is closer to one of the big companies than not, however my current employer has a Blind channel and its not very accurate to my experience

You realise Github is Microsoft nowadays right?

I do, however for the purpose of this discussion, I'm treating them separately, as GitHub is a distinct employer from Microsoft on Blind.

Oh I see, I never checked out Blind, in LinkedIn Jobs tho the Github Offers seem very much Microsoft Offers to me.

Lol! If you're an engineer and you don't spend even a little time on Blind, you're signing up to get hustled. Learned so many interesting tidbits from Blind that I was able to use to get ahead in my career. Is it negative, yes; can you still benefit from it, yes.

Let's dont act as if the Blind was the only place where you can have discussion about stuff like this

Yet to see a more honest, transparent and non-posturing discussion board about tech.

This site feels like a terrible joke

Do really "serious" people use this?

Businesses that can turn profit from monkeys as employees are best businesses. Running such businesses on tightly standardized PhDs is best for its own survival. So that's what everyone does to various degrees, and it's also such a horrible thinking.

They are just honest about reporting these events. Others don't, so they look better. It's easy to lie without getting caught, because most people don't experience most outages personally.

No, they didn't have such a high incidence of events prior to the tech layoffs. something material has changed.

Eh, that's giving MS too much credit. They're as honest as most corporations are with status pages. I've experienced outages and error pages many times while the status page showed green across the board.

There's definitely been a noticeable decrease in reliability since MS took over. They've introduced many features, which is appreciated, but products like Actions are notoriously unreliable and flaky.

If this was Twitter / X having these outages every single week, the entirety of HN would come out and scream 'Twitter / X falling apart!', 'It is crumbling', 'Its grinding to a halt again!'

In comparison to GitHub which for a service to have tons of outages a week is quite atrocious for reliability.

But somehow it is acceptable to tolerate GitHub's ridiculous uptime record, but of course, on HN and in wider tech circles, a different standard of uptime is applied here to GitHub despite it being known that it has fallen over much more times than Twitter / X has done for years, week after week.

Until the next time GitHub goes down again. Probably won't be long anyway.

> If this was Twitter / X having these outages every single week, the entirety of HN would come out and scream 'Twitter / X falling apart!', 'It is crumbling', 'Its grinding to a halt again!'

Fail Whale is a thing tho, remarkable for frequent sightings in the seas of Twitter.

The 'Fail Whale' is not the point.

If Twitter / X went down every week like GitHub is currently doing for years, they would be called out for that, immediately. But here we have GitHub; a centralized service having over 100M+ active users and their record of uptime is far worse than Twitter / X's and it is given a pass each time it falls over every week.

By that standard, we should not be accepting GitHub's ridiculous uptime history and it has been found that GitHub is far more unreliable than most services including Twitter / X but somehow right here, GitHub is not falling apart or seen as unreliable, especially with having more than 5 incidents in one month for years?

I think at this point my hypothesis is definitely erring towards them being hopeless goons.

  This incident has been resolved.
  Posted 1 hour ago. Sep 05, 2023 - 17:01 UTC

Perhaps you should actually consider self-hosting? [0] As I said and predicted years ago? [1]

But as expected [2] every month, there is an incident and something at GitHub breaks and goes down. Maybe GitHub is gradually falling apart.

It is that unreliable.

[0] https://news.ycombinator.com/item?id=36526334

[1] https://news.ycombinator.com/item?id=22867803

[2] https://news.ycombinator.com/item?id=37320925

Github is down a few hours per month but it has never lost data. Work doesn't grind to a halt because your git host is down. So really, Github still provides a lot more value than self hosting.

But sure, keep evangelizing and talking down to people who prefer it to self-hosting, I'm sure it will accomplish your goal.

> Work doesn't grind to a halt because your git host is down.

When you cannot push your critical change to master on GitHub or run your GitHub Action, then it does grind work to a halt, as it has done with the many commenters here.

> So really, Github still provides a lot more value than self hosting.

Centralizing everything to GitHub is much more riskier than self-hosting given the number of times GitHub has fallen over or had an 'incident'.

> But sure, keep evangelizing and talking down to people who prefer it to self-hosting, I'm sure it will accomplish your goal.

My point is self-proving every month and week that something in GitHub goes down or has an incident. I'll be expecting you to complain again once it falls over in a month's time at the latest.

I've been an incident responder where the incident was, "We promised a customer that changes would go out today, but GitHub is down, and thus our CI/CD is also down."

It definitely does matter when GitHub goes down.

Github.com is back up for me in San Francisco. At least it let me login and create a new repo.

Funny to finally see this. I have been having actions issues/failures since 3am EST.

So that’s why profiles aren’t loading

A historic event, something not seen since last week!

earlier this year, a team i'm no longer a part of switched from using AWS CodeCommit to using GitHub. at the time, i questioned why switch to something that goes down a lot. the response was it never goes down. the next week, outage. there was no valid reason on why switching to github other than "some potential VC questionnaire asked if the company was using github". i'm much happier no longer being there when these kinds of red flags are replaced with klaxons

> the response was it never goes down.

Seems those folks maybe have never used GitHub in any larger capacity. They've been having issues constantly for a long time now.

September - 7 incidents

August - 17 incidents

July - 11 incidents

June - 13 incidents

And so on. Last time they had less than 10 "incidents" was December 2022.

I find github to be a much nicer experience than codecommit. Not surprised that people would want to switch.

> some potential VC questionnaire asked if the company was using github

Were the repositories even public?

no.

Must be that push I made!

Reply on news.ycombinator.com