GitHub is down | Hacker News Reader

GitHub is down

githubstatus.com

73 pointsenescakir77 comments

The status page says it's been resolved 8 minutes ago (Apr 05, 2024 - 08:48 UTC): https://www.githubstatus.com/incidents/bnkkbj90yhz6

But it definitely still happens now (500s on refresh on PRs and GitHub actions)

Edit: still ongoing

Edit 2: still ongoing at Apr 05, 2024 - 08:56 UTC (keeping updated for the record since their status page cannot be trusted apparently)

Edit 3: I see they have switched to a different (ongoing) incident ID now: https://www.githubstatus.com/incidents/5ly0psff2s5d

Status pages can never be trusted.

=> Third-party sites can never be trusted.

=> Nobody can never be trusted

=> I think, therefore I am. Everything else is speculation.

=> Certainly is.

This is a first-party site.

This is the other-party site. Could be first in a sense but inevitably the other.

Huh? This site is part of Github. It is on a different domain (presumably) so that if there is a DNS outage it will not be affected.

I always think this should be an incident in itself - why did our status page not reflect the reality of a degraded service? It's so common that they don't, and something user-driven like DownDetector is often more reliable

I don't think an accurate and automated public status page is something any management would want. If it was accurate they wouldn't be able to lie to customers about the uptime. So I always suspect status pages are adjusted manually.

That's exactly what happens. How we need to respond though is by not linking to status pages hosted by that party, instead we should be linking to a StatusGator or DownDetector page as a 'source of truth'.

> why did our status page not reflect the reality of a degraded service?

There was a conflict with marketing, market movement, sla contracts, and our image.

I recently set up a status page for the services I run on my pi. The idea was to get some insights and apply experience at work.

My experience now tells me what we really need first is a solid alerting system, the status page can't be trusted. It's a PR tool (a useful one), not a sysadmin tool.

If the status page officially authorized by the company’s management team cannot be trusted, then why trust the company in the first place?

Is the company in the business of selling status pages ?

If they market it as a feature，yes?

Were you expecting me to say something else?

> If they market it as a feature，yes?

Well, the status page is not the main feature of github/gitlab but I agree that if customers decide to rely on it then it's a problem.

> Were you expecting me to say something else?

Yeah, something like "their status page is not their core business" or "status pages and SLA are two distinct things".

What about something like https://heiioncall.com/status (disclosure: helped build it) which gives a real-time view into what our monitoring & alerting system sees, both from various HTTP endpoint checks, and cronjob checkins?

But I don’t think most orgs would want a public version of this (too much transparency), which is why we haven’t built that.

Initially, perhaps by different teams, three separate incidents were created. Since then, two have been resolved, and one continues to be updated.

Since last few months, GitHub has become very unreliable. For example, last month, they reported six official incidents in just one week. I've also noticed many shorter, non-official incidents (i.e. the ones where there is no active incident on their status page, but many API calls return a 500 error). I learned about this the hard way (we offer affordable GitHub Runners at Ubicloud, which gives us the opportunity to observe all these failures firsthand)

I’d say years though it might depend on both your physical location and te services you use.

We’ve had to harden a bunch of automation to be more reliable in case of github going tits up, and while github is still currently the broker (as it’s where users get statuses / information from it made sense at the time to also have it handle broadcasts) we’re definitely planning on direct communication between services eventually.

Curious if this is related to them adopting azure

Did they officially announced they are adopting Azure?

What does 'adopting azure' mean?

In the past two months, they've had over 10 incidents. Are they so hyped by AI features that they're neglecting their core services?

Please stop doing catsstrophising tech journalist cosplay.

New features are sexy. Core stability is boring. AI is just the latest new feature.

True, and SRE teams get smaller and smaller and/or tasked to do other non-related things. So when users start leaving for some competitor, the same SRE team suddenly gets more heat, pressure, more belt tightening, more meetings "for transparency and coordination", and directly and indirectly more blame. Ask me how I know... I bet its a story as old as time though.

There is little incentive for them to invest in stability. Customers will eat the pain because they are so invested in the platform. They will never move away.

Meanwhile all the Staff engineers trying to get promotions are building AI shit.

My on-prem gitlab has now survived 6 cloud-based github outages.

how long do you spend upgrading it?

Not OP, but for mine I spend about 10 minutes every 2 weeks.

Once: an upgrade went wrong and I had to roll-back, that made me spend 30 minutes instead of 10.

Gitlab instance stats:

Projects 256

Users 216

Groups 17

Disk usage: 877 GiB / 3.38 TiB

They marked this one as resolved. They keep providing updates on another active incident: https://www.githubstatus.com/incidents/5ly0psff2s5d

Maybe what is happening in this thread should give everyone cause for concern and a clear heads-up that more serious problems are afoot.

Very emotional reactions here, whether fuming at the unreliability of the service, or zealously defending and identifying with it, indicate people have far too much invested in this single point of service.

People are saying - "That's me done for the day, have a good weekend!" What is the economic cost of this? On a global scale? The pattern is the same as we've seen recently with Office365 and other cloud services.

People talk about setting up elaborate monitoring services, yet with a similar effort one could set up one's own development repositories and tools.

It's not like this crowd is typically "helpless", or unaware of what is being done, yet people remain beholden; Github has diversified into a social network and AI tool, far beyond its initial remit as a code repository. People seem genuinely terrified at the prospect of it going. And rightfully so.

In the wake of the xz backdoor attempt Microsoft will soon move to offer "developer protection" and a system of validation, testament and supply-chain protections. At that point Microsoft will have all of you absolutely in the palm of its hand and at its mercy.

Now would be a good time to seriously invest in long term thinking and strategic protection of your development future, indeed of your fundamental ability and rights to develop software free from disruption, pressure and encumbrance.

I saw a post on LinkedIn, from GitHub, boasting about how many changes they deploy in a day. I couldn’t believe it. GitHub actions have been broken so many times for me this past month, not able to deploy my code when I want to, so it felt like a slap in the face.

In London it's dead. I guess I'm taking the day off! Happy Friday everyone!

Thank god (Linus) that git is a distributed "protocol" letting you pull and push to any location on your file system or anywhere you can access over ssh.

99% of git users have no concept of this. It's very much a client/server relationship with github.

Git itself is only a piece of the puzzle, you still need synchronisation points for collaboration e.g. bug reports, code reviews, discussions, CI, synchronisation, etc…

Those are the services forges provide.

Also git’s support for point to point distribution is bad. I miss `hg serve`.

This comes up pretty much everytime Github is down. Github is not a replacement for git. Github is an issue tracker and CI platform just as much as it is a git hosting site, none of which git can provide by itself. It's complementary. The comment really doesn't make any sense.

It's like being snarky about the distributiveness of git when the Linux kernel mailing list goes down.

Apr 05, 2024 - 08:51 UTC: Update > We're seeing connection failures to some databases in two of three sites and are investigating.

This should probably link to the main status page as it seems there's more than one incident ongoing.

Already listed as being resolved. A 15 minute outage at 4:30 am EST.

Or 9:30am GMT, so just as europeans are starting to work. Or middle of the working day in India.

This happens fairly regularly. Do HNers think that Microsoft azure and/or Ruby has a role in it?

Why would Ruby play a role in it? Azure definitely does though. (ex azure employee here )

GitHub itself is written in Ruby.

[0] is a my favorite demonstration of it.

[0]: https://github.com/rails/rails/commit/b83965785db1eec019edf1...

Don't underestimate Ruby's performance https://x.com/tobi/status/1728524453854756883?s=20

Well, when we say GitHub, we are talking about thousands of services. I don’t think we can pinpoint the overall operational failures to programming language choice. In my experience, these kind of issues are mostly caused by poor organizational architecture + poor project management. In GitHub’s case, I’m also confident to say the choice of infrastructure plays a much bigger role than Ruby.

> Why would Ruby play a role in it? Azure definitely does though. (ex azure employee here )

Oh yeah. If someone says C++ has poor memory safety than Rust then no one is triggered. But say something about Ruby (the language on a secular decline - https://www.dice.com/binaries/content/gallery/dice/insights/...) and the pitchforks are out.

Azure is such dogwater - Even a narcoleptic intern could run a more effective cloud service using raspberry pis and a tin can.

A better question would be "is there anyone who doesn't think Azure played a role here?"

homepage gives me a 500

Working for me in NZ, logged in.

its back

Up in Europe still.

Not for me, in Europe.

Not working for me in Europe

Yea, in Germany it works.

The failure has yet to propagate to Germany, given how slow internet is here.

Which country? There are multiple data centers across the continent.

"Degraded performance"

Yeah, experiencing issues with Pull Requests and commits

Classic Github for the past few years we are used to it / resilient now.

Sorry everyone. This was me.

What did you do?

Bounder

Reply on news.ycombinator.com