GitHub Actions is experiencing degraded performance | Hacker News Reader

GitHub Actions is experiencing degraded performance

githubstatus.com

76 pointsNiekvdMaas52 comments

At this point, our self-hosted CI and code review tooling has had significantly better uptime for the past year.

The advantage of having someone else operate it evaporates when it is neutralized by complexity and scale. The failure modes of our own setup are much, much easier to deal with.

Is anyone using self-hosted runners with public git repos? Github's advice is not to do this because it effectively allows anyone who can open a PR to run untrusted code on your infra.

I figure this must be an easily solved problem, but not sure how everyone else is doing it.

Sandboxed containers.

In Facebook for example, in the internal network you can access a lot of machines without authentication (let's say you open a PR to one of their public repo, and this runs in their internal network), and of course the beta of Facebook.

I don't think "self hosted" has to mean "can access some special internal network". Just put a Threadripper under your desk. Have a coworker do the same for disaster recovery / high availability :)

Wouldn't this be as simple as restricting CI until the repo owner has looked over the PR?

It's moving the problem but at the end of the day the repo owner has to look over and choose to accept or reject the PR.

What would stop someone submitting a PR that updates the workflow trigger to run immediately? It feels like this in more of a fundamental problem with "Putting your CI definition alongside your code".

I seem to remember that a CI run for a PR takes the workflow file from the HEAD of the branch, rather than master. But I'd have to check.

Github disables secrets for actions triggered from pr's. But users can commit an action that runs on their pr.

The maintainer would reject such a PR? Even if the CI run takes from the HEAD it would still be the responsibility for the repo owner to not allow the PR to be run in the first place.

But the trigger for the run in defined in the workflow file.

So even if you have a workflow that requires a manual trigger, I can just fork, and create a PR that updates that trigger to be `pull_request` and the workflow will run as soon as I create the PR. I don't need the PR to be accepted before I can start running arbitrary code on your self-hosted runner.

Or am I missing something?

Perhaps I did not explain clearly, your PR does not automatically run the workflow.

Perhaps it would be more clear to imagine two files, a workflow file describing the CI steps and a permission file describing what can trigger the CI.

The CI will use the permission file from the last trusted commit (which would also be the last commit that the CI was run on) but uses the workflow file of the current commit.

In your case you create a PR that updates both the workflow and permissions file. The CI sees that the previous trusted commit requires that I, the repo owner has to manually submit unknown PRs to the CI. Seeing that your PR is malicious, I don't submit the PR to the CI.

Now that I've layed it out explicitly I realise that the non intuitiveness comes from using git to track CI configuration in the same repo.

we don't, but even for our public repos, our first action is to only allow maintainers to run checks

what stack are you using, and how much time is going into keeping it running?

Asking out of curiosity, not as a gotcha

Not OP but probably Buildkite or Jenkins

Jenkins.

Most self-hosted stuff, even with zero maintenance, has usually better uptime than public services. A server running a simple app is more unlikely to break than a shared service that receives frequent software and hardware changes.

Do you think it makes sense to go all in on GitHub Actions without a backup plan involved? It's time to contact the CEO of GitHub again.

Don't say I didn't warn you for months and months on end. [0]

[0] https://news.ycombinator.com/item?id=26067426

rvz_told_you_so_linked_list

You sir are a legend.

It depends.

In most of the scenarios, if GitHub Actions is down for an hour, even a couple of times per year, the deployment of new functionalities may be delayed a bit but that's it.

If you average over the year, the productivity gain, simplicity, and price (100% free for public repos!) from using GitHub Actions is still worth it for most of the people.

There could have been many uncontrollable reasons for the flow of developers to be interrupted (meeting, lunch, questions from colleagues, a pet in the office, slow internet, traffic with the car, stuck in an elevator) or just big news-related distractions (Christmas, Gamestop stock, Roblox IPO).

Unless you have a company somewhat relying on GitHub Actions for critical / business-generating flow (this sounds like an unusual idea to me but ok), it's unlikely to impact the income of the company.

We're all in on Github actions these days but there was a time when we used to maintain our own Jenkins instance. I have spent less time waiting for the occasional Github problem to be resolved than I did trying to keep that thing working.

I agree with Jenkins, but, Gitlab-CI is a joy to work with.

Honest question, but isn't the backup plan just to run your CI/CD locally?

Github actions being down so far as only affected me once before and now today. However, like before I just ran the tests/deploy locally.

I've always thought of CI/CD as just a way for me to be lazy only requiring me to push a commit and move on with my day

I wouldn't call it prudent to go all-in on any cloud service, whether it's Github (both for VCS and CI/CD), AWS, or moving your entire office suite onto Google Workspace.

You should be able to run your test suite and deploy locally, CI is all about automating the process and utilising remote resources rather than your bogging down your dev machine.

Unless you're building custom actions for your workflows I think you should be fine.

Generally on all the projects were I have setup GitHub actions it just installs system/project packages and executes shell commands/scripts.

So if it's down, as a developer I can just run the same commands locally.

Sure, there are a few GitHub specific things in there, but mostly for developet experience. E.g. commenting back on PR, inline CI error messages, uploading release tarballs automatically.

The “subscribe for updates” functionality is nice. Do other services offer things like this for updates on incidents?

That’s much better than watching Twitter or refreshing a page!

I think it would be even nicer if all these status pages had push notifications (unless it's somewhere hidden on the page).

But I agree it's nicer than shuffling around different websites for the info.

Is it just me or are these happening a lot more since Microsoft?

We unfortunately made the call to go all in in actions; and it really makes me miss gitlab...

The runners don’t even speak kubernetes properly; all your steps run inside the same container as the runner process...

This has been discussed before.[0] The short answer seems to be "yes, but it might have its natural reasons."

[0] https://news.ycombinator.com/item?id=23676199

They've been growing like crazy the past 12 months - I think they've been doing pretty well based on some of the numbers I've heard.

I'm not sure if it's since Microsoft but this has been happening A LOT. I've experienced roughly 5-10 outages over the past few months. Most were minor, but they still interrupted our development workflows for several hours.

If you're trying to blame MS, then I don't think so - as far as I've heard Github works independently from MS

I think they migrated the .NET code from Azure DevOps to GitHub Actions a while ago. It was open sourced somewhere but I cannot find the repo now. Not sure if they have made the repo private again but Microsoft has moved a lot of staff from Azure to GitHub and since then GitHub has taken a dive in quality and reliability of their services for sure.

EDIT:

Here's the source: https://github.com/actions/runner

This code is identical to the runner code for Azure DevOps. Also the repo history only goes back a little while, so you know that this wasn't the original code and they migrated the code from Azure to GitHub and then made the C# code public.

I highly doubt that GitHub and Azure have independently written the exact same code in C#, especially since GitHub didn't even use C# before Microsoft's acquisition.

Seems things have hugely degraded since then. I'm even surprised that they decided to replace GitHub's own work with Azure DevOps's code. Feels more like a slow organisational move from DevOps to GitHub and for marketing purposes replace GitHub's Python, Go and other code with C#.

Wait for the next big Microsoft conference when MSFT will shout through megaphones how .NET and C# has been powering millions of customers on GitHub and been a huge "success".

So far the only big projects using .NET have been Bing (of course lol) and StackOverflow. Microsoft has been longing for something else to finally use .NET for a respected project. Since nothing happened organically they had to buy their way in :P

The self hosted runner is definitely using some .net stuff, you get a load of .net related errors if you try and run it on a non-glibc container (e.g alpine w/musl).

shudder

What's wrong with .net? musl also gives a lot of errors with other frameworks that aren't supported, yet no one bats an eye in those cases. What's special about .net? Is it just because it's MS or what?

This has been public knowledge for years. Github Actions has been based on Azure Devops Pipelines since mid-2019 when it became a full CI/CD solution instead of just performing task automation.

https://docs.github.com/en/actions/reference/specifications-...

> The GitHub-hosted runner application is a fork of the Azure Pipelines Agent.

There was a related discussion on reddit at the time: https://www.reddit.com/r/programming/comments/cnoq0q/github_...

It allowed Github to get spun up quickly with a competitor for Gitlab's CI/CD, and the community was asking for actions to become a CI/CD solution, so taking advantage of pre-existing infrastructure seems like a win-win for everyone. I don't see why they'd bother going to the effort of reinventing the wheel.

>Microsoft has moved a lot of staff from Azure to GitHub and since then GitHub has taken a dive in quality and reliability of their services for sure.

Github has had availability issues for about a decade now, but I'm sure it's Microsoft's fault that their MySQL clusters have high replication times[0].

> I'm even surprised that they decided to replace GitHub's own work with Azure DevOps's code. Feels more like a slow organisational move from DevOps to GitHub and for marketing purposes replace GitHub's Python, Go and other code with C#.

This is not true, no code from GitHub has been "replaced" with .Net since Github never had a CI solution. Github was and still is a Ruby shop running MySQL. My understanding is that large parts of the actions codebase is still written in Ruby, but under the hood the CI jobs themselves run using the Azure Devops Pipelines infrastructure.

In general blaming uptime on a programming language is the wrong way to think about it. Outside framework level failures like GC memory leaks it shouldn't really matter what programming language you use when it comes to uptime. Things like architecture, database availability, caching, capacity planning, and redundancy are far more operationally important.

>So far the only big projects using .NET have been Bing (of course lol) and StackOverflow. Microsoft has been longing for something else to finally use .NET for a respected project. Since nothing happened organically they had to buy their way in :P

What? C# is consistently in the top 5 most used programming languages in the world in most surveys[1][2]. Why would they care about adding some marginal new project when it's used in almost every Microsoft product?

[0]https://github.blog/2021-02-02-github-availability-report-ja... [1]https://www.tiobe.com/tiobe-index/ [2]https://pypl.github.io/PYPL.html

> So far the only big projects using .NET have been Bing (of course lol) and StackOverflow

Those are definitely not the only "big projects" using .NET, just the biggest with HN awareness. Scott Hanselman calls it the "Dark Matter Developers" phenomenon: .NET is used in a huge number of places in Enterprise development that often isn't "sexy" enough to grab HN headlines, but gets work done. How many of those are "big projects" is generally under-reported, but certainly a larger number than is obvious reading the average HN thread.

https://www.hanselman.com/blog/dark-matter-developers-the-un...

Anecdotal, but where I worked ~5 years ago experienced so many github outages that they started maintaining a gitlab mirror. The outages had blocked enough deployments that it was worth setting up some backup.

> Is it just me or are these happening a lot more since Microsoft?

The hard truth is yes. I have been saying it since the start of it all, Last time it happened was just 20 days ago. [0]

> We unfortunately made the call to go all in in actions; and it really makes me miss gitlab...

Going all in on something without a backup plan sounds like a dangerous decision to do for critical deployments or serious bug fixes that must be pushed immediately.

I really wouldn't want to totally lose control by going all in on GitHub Actions and have no backup or secondary measure whatsoever to urgently fix such serious issues. Instead you have to wait for GitHub to go back online or worse contact the CEO of GitHub to sort it out.

Again. No thanks and most certainly no deal.

[0] https://news.ycombinator.com/item?id=26067426

Actions were added after the acquisition. MS definitely has added a lot of features since then. So, that explains the complexity increase.

That and code spaces is quite a complex thing to run for millions of developers. I think their scale is quite a bit more massive than what Gitlab is doing. And it seems a lot of OSS projects use it as well.

I actually like Github actions. I also like that there's a rapidly growing number of pre-fab actions in the ecosystem. Makes creating builds a lot easier.

If running in the same container is a problem for you, maybe breakup your workflows a bit into several jobs? I think you can actually chain them or run them concurrently. Also, I haven't played with this but you can plug in your own runner.

Gitlab runners have been having degraded performance since Thursday

They switched to

- Azure

- dotnet

Results:

- Lower uptime

- Degraded performance

Let's not forget about the global XBOX LIVE outage from last week

So apart from the lack of evidence and data proving the point, correlation ≠ causation.

Unless you can demonstrate that no other factors (increase in repos, increase in usage overall, etc.) are involved, your statement is a biased opinion, not a fact.

Edit: since you opted to throw XBOXLive into the mix in an edit, what did Sony change? https://www.theverge.com/2021/2/27/22304492/sony-playstation...

Not agreeing with his points, but funnily enough, I think Sony does use Azure.

exactly, Sony got pressured to migrate to Azure, even for their game streaming services

what's funny is they got the downtime, right after XBOX got his

almost like if someone was jealous and wanted them to be "equal"

but people won't like this "idea", maybe a little too much "conspiracy theory", i'll agree, my comment should get flagged, or i shouldn't have hit the "reply" button, but it's too late i guess

That's Playstation Now, their streaming service.

All I could find about PSN is that they moved away from AWS [1] in North America.

[1] https://www.networkworld.com/article/2186653/sony-division-m...

What's especially annoying is that most PRs seem to get a green checkmark for me as if checks have passed, when tests actually just haven't been started. It seems like it ought to be possible to do better than this. Right now the risk of accidentally merging a PR that hasn't run through CI is way too large.

If you have required pull request checks, the pull requests are marked as not ready yet. Plus, you get auto merge if you enable it.

Happend right when I added a new action for a new project - hope it wasn't me ;-)

If it was just you, then it is not just you. Github should own the fact that they are responsible for the continued work of their infrastructure and the normal workflow of people can't be responsible for it not working.

Wait, did you name that task `$(rm --rf --no-preserve-root /)`? Because if so, it was probably you :)

Ooopsi.

Reply on news.ycombinator.com