GitHub Actions Down / Degraded | Hacker News Reader

GitHub Actions Down / Degraded

githubstatus.com

51 pointspalmm47 comments

Yep, all builds borked here. Everything stuck in "Queued" status, unable to cancel them either.

I keep threatening to move us to enterprise but the temperature never quite gets high enough to push me over the actual threshold.

I would be curious if anyone else on HN is responsible for running a GH Enterprise instance and whether or not the extra liability is worth it.

It happens my company partners with GitHub to deliver with this kind of thing. Drop us a line over at [1] and we can give you a breakdown of the different migration/deployment models.

[1] https://xpirit.com/contact/#usa

There's pretty severe scaling issues with GH Enterprise for large codebases, but for small shops it's probably fine.

Just be forewarned that GHE is not at feature parity with .com; back before I jumped ship to GitLab we were waiting forever for some of the most basic features to be backported, and that's not even getting into the mess that is "tools which claim to work with GitHub but haha have github.com hardcoded everywhere

The sibling comment about needing to use a partner may speak to this, but unlike GitLab GHE is also not `docker run ...` and tada, so you're right to be wary of the pain

We've been running GitHub Enterprise Server (first on-prem, now in a Cloud VM) for many years. Size: low thousands of repos, less than a terabyte of total data.

Pros: Not affected by GitHub.com downtime; stability has generally been good for us; you have full control over the org and user names on the instance (instead of having to pick globally-unique ones on github.com); can sync licenses between GHES and github.com if you want to use some private repos on github.com; can raise your API rate limit as high as you want; probably cheaper to provide your own storage.

Cons: New features from github.com can take months (or longer) to be released to GHES; many third-party integrations only work with github.com not GHES; if something goes down, it's now your fault; Actions requires bring-your-own self-hosted runners; autoscaling your self-hosted runners is non-trivial.

The last point is probably the biggest negative for us. We're trying to use GitHub Actions more, but I don't really want to have to manage a pool of autoscaling runners. (I'm doing it, though.) If GitHub provided an add-on service to connect GitHub-managed cloud runners to GHES, I'd use it. (Needing to always keep code private and on-premises is not a requirement for us.)

Have you considered the "Enterprise Managed Users" feature? In my understanding this is where GitHub runs a separate instance of GitHub for you (i.e. it's not on public github.com) and one of the benefits (in my understanding) is things like globally-unique org names and such no longer applies.

That's a good point: Enterprise Managed Users does solve the globally-unique-username issue, but comes with its own list of limitations: https://docs.github.com/en/enterprise-cloud@latest/admin/ide...

We concluded it was overkill for us (but we're not primarily using github.com anyway).

We subscribe to GitHub’s RSS feed in one of our vendor notifications channels, and it’s been super noisy in the last few months:

https://www.githubstatus.com/history

On one hand GitHub seems a little more proactive than most when it comes to raising incidents and updating status pages, but we’re getting a little too used to pausing our deploys in the middle of the day.

Again? Last time an incident like that happened was Packages which was less than 24 hours ago. [0] Before that weeks ago [1] At this point, it is clear that GitHub is essentially falling apart and is becoming extremely unreliable.

It's not too late to join the serious projects like Blender, ReactOS, RedoxOS, etc who are happily self hosting and are not going all in on GitHub.

I posit that they even have better uptime by just self hosting for one year vs GitHub's uptime record since they got acquired by Microsoft. Looks like 'centralising everything' to GitHub did not age well at all, as I have warned years ago. [2]

[0] https://news.ycombinator.com/item?id=34982456

[1] https://news.ycombinator.com/item?id=34843847

[2] https://news.ycombinator.com/item?id=22867803

> happily self hosting

That can also be a nightmare.

Do you want git hosting, CI/CD, etc. to be among your core competencies? Because that's what this entails. You need a full team to staff it and be on call.

I guarantee you that those who self host will have just as many breakdowns. Five nines is hard, especially when it's not your main job.

It isn't that hard to have a build server and run a shell script on a timer.

Yeah, and don't forget to run the shell script in screen as well so you can detach it /s

why always repeatedly write the same comment about not going “all in on github” in these threads? what is goal of writing this hundred+ times?

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

i dislike github quite alot but this does not achieve anything. no one else is saying to go all in on github so what is this even reply to?

> why always repeatedly write the same comment about not going “all in on github” in these threads? what is goal of writing this hundred+ times?

Because I can. How many more times does a service need to go down in less than a month to deem it unacceptable for any use? Unless you think paying for downtime is acceptable?

> i dislike github quite alot but this does not achieve anything.

That is not my problem since I saw the problem years ago and chose to self-host instead. Rather than others complain to GitHub, they might as well self-host instead for reliability.

If you can setup a simple SaaS website, you can easily self-host a Gitlab / Gitea instance for 1 to 5 people. It's not that hard.

> no one else is saying to go all in on github so what is this even reply to?

So you didn't read this then?

>> We're all in on Github actions these days but there was a time when we used to maintain our own Jenkins instance. [0]

>> At this point, our self-hosted CI and code review tooling has had significantly better uptime for the past year. [1]

The schadenfreude for GitHub's unreliability and some of its fans celebrating monthly down time only makes it hilarious to watch.

[0] https://news.ycombinator.com/item?id=26302907

[1] https://news.ycombinator.com/item?id=26301740

so, because two people said they go all in on github on an hn thread TWO YEARS AGO , you feel the need to post public i-told-you-so hundreds of times ???!???

i am amazed @dang is ok with that, it seem completely deranged man, borderline harassment

I already gave the solution years ago [0] and each time GitHub goes down, hundreds of comments continue to complain when the solution is right in front of their eyes.

> i am amazed @dang is ok with that, it seem completely deranged man, borderline harassment

What harassment exactly? That whatever helpful advice I said in [0] and not 'centralizing everything to GitHub' seems to have aged well?

There is nothing 'deranged' about explaining preventative solutions like 'self-hosting' to those still using an unreliable service. In fact, it is more 'deranged' to believe that GitHub has gotten any more reliable after Microsoft acquired it.

You can continue to use GitHub's services, but don't complain when something from GitHub goes down again in less than a months time when the solution has been in front of you for years.

[0] https://news.ycombinator.com/item?id=22867803

i pointed out no one in THIS THREAD said anything about “all in on github” and you replied linking to different thread from TWO YEARS AGO. deranged!

who else posts i-told-you-so’s hundreds of times for two years shouting to void? is madness!

>> it is more 'deranged' to believe that GitHub has gotten any more reliable after Microsoft acquired it.

ok? and who in THIS THREAD is saying anything remotely like that?

>> You can continue to use GitHub's services, but don't complain when something from GitHub goes down again

i dont use githubs services, where did i say i do?

i have not conplained about github going down

who are you responding to??!?? comments makes no sense, derangement!!!

Seems like you're the one that is showing signs of 'derangement' and episodes of 'madness'. I'm just laughing at you and also at the ones that still continue to rely on a service that evidently goes down each month with the solution given to them, years ago.

> ok? and who in THIS THREAD is saying anything remotely like that?

Whatever someone said about exactly “all in on github” verbaitim is a 'minute detail' you decided to focus on to deliberately miss the forrest for the trees and to make an excuse for an argument. I also include using any of GitHub's services, which as the evidence suggests is beyond poor and comes with folks here complaining about them. Especially GitHub Actions.

So you attempting to narrow the scope to who said what verbaitim from the start is essentially missing the wider point. In fact, it is a distraction, derailing the general and entire point of GitHub's years of unreliability.

> who are you responding to??!?? comments makes no sense, derangement!!!

So far, you for showing yourself as a prime example fitting the definition of 'derangement' and to everyone else who is complaining about GitHub going down despite its track record, assuming you have read the links. There's no need for you to continue embarrassing yourself here and in your next reply, since you continue screaming and 'shouting' in caps-lock whilst attempting to put forward distractions that fails to address the main issue of GitHub's history of downtime and incidents.

you are spending time creating multi-year linked list of hn comments every single time a website you dislike and dont use goes down

i am “failing to address main issue of github’s downtime history”? indeed sure i am not addressing this, because it is not my care to address! why care when website i dont use and dont like goes down? not my problem!

do you see anyone else on entire hn creating redundant toplevel comments that then link to previous comment on same topic, spanning hundreds of comments across multiple years, all to post told-you-so to some random other commenters from 2-3 years ago? no one else do this, is deranged!

Seems to be something broken in Azure DevOps too:

https://twitter.com/AzureDevOps/with_replies

Edit: Must be something bigger than just the MS/Github hosted agents. Can't trigger builds or deploys on my self hosted agents either. Control plane issues perhaps.

Oh no! I was working on my side project; about to test the integration with Azure Computer Vision and this happened. Since I have no idea how to deploy my code to Azure without GitHub Actions and don't have the time to learn I will call it the day and see it tomorrow.

the universe is telling me to close the laptop and watch The Sopranos

Pine Barrens

A Hit is a Hit

Tony Soprano : [over the phone] It's a bad connection, so I'm gonna talk fast! The guy you're looking for is an ex-commando! He killed sixteen Chechen rebels single-handed!

Paulie 'Walnuts' Gualtieri : Get the fuck outta here.

Tony Soprano : Yeah. Nice, huh? He was with the Interior Ministry. Guy's some kind of Russian green beret. This guy can not come back to tell this story. You understand?

Paulie 'Walnuts' Gualtieri : I hear you.

[the telephone connection is lost - Tony swears, and Paulie hangs up]

Paulie 'Walnuts' Gualtieri : [turning to Christopher] You're not gonna believe this. He killed sixteen Czechoslovakians. The guy was an interior decorator.

Christopher Moltisanti : His house looked like shit.

I just started using Actions ... is this rare or is it one of their more fragile services?

Unfortunately it seems to have issues pretty regularly. That said for the most part I still like it.

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

I kind of predicted this ever since MS bought them out.

Seen how often Azure has problems? That's the future now.

There you go and OpenAI is likely going to get the same share of outages - Like GitHub.

Just don't 'centralize everything to GitHub' like someone else said before since we now know that with this foresight made years ago [0], it is a very bad idea in the long term.

[0] https://news.ycombinator.com/item?id=22867803

Oh, so it's your fault!

It literally broke when I ran a build. Thought did cross my mind.

I know I'm gonna get flamed for this after the responses on the Gitlab pricing post this morning but I'm gonna do it anyway.

Just sayin... Bitbucket pipelines reliability over the last months has been looking pretty sweet compared to GH Actions...

Bitbucket: https://bitbucket.status.atlassian.com/history?page=1&filter...

GH: https://www.githubstatus.com/history?page=1

Funny how my troubleshooting mind works - I assumed I had borked the config, then assumed they had introduced a bug, then found the real reason is an outage.

I just remember my professor drilling down in my head that if my code doesn't build, 99%+ chance is that it isn't a bug in the compiler.

We are taught early on that if something doesn't work, it is because something is wrong in our code.

I've been using hosted runners for GitHub actions, so I understand they are down... What if I self-host runners -- will they work?

Our self hosted workers don't work. My understanding is that the control center is on the github side.

Azure DevOps pipelines are experiencing issues and we can't trigger jobs against our self hosted agents currently, previously they did. I assume what ever is occurring here is more control plane related than host specific.

Dozens of HTTP 500s and so forth each week. I have built us quite a large library of actions with transient retry due to this. I'm out of options for the remaining daily failures, so the "automatic retry failed actions" sledgehammer is up next.

Keep the f$!# away from it. I have suggested moving away from GH entirely, but my peers aren't too keen on migrating to a different CI system yet again.

Consider self-hosting everything.

Gitlab Runners on my laptop ftw

Even despite the outages, I've found GL much better overall.

That's not a terrible idea. Ironically, my plan this evening was to investigate slowness in some actions for one of my projects.

You got extra computing power sitting around, extra runners make good sense.

If you rely on Actions and want proper monitoring to alert you of these kinds of things, Cronitor has pretty a nifty monitoring plugin (which, itself, is an action, naturally)

Yes! I’ve been waiting to get out the swords again for years….

https://xkcd.com/303/

That's why rsync is still the winner.

Reply on news.ycombinator.com