Hacker News Reader

Comment by abidlabs | Hacker News Reader

It's not too hard to switch over from GitHub actions to other runners. For example, I wrote up the steps needed in order to use Hugging Face Jobs instead, which also enables GPU runners and other flavors. I use this for several of the repos I manage (such as Trackio):

https://huggingface.co/blog/github-ci-hf-jobs

View on HN

We use other runners (Blacksmith), and it hasn't helped, still hard down all day long. Brutal outage.

As a side note, I started getting these symptoms (actions staying queued forever or not running at all) at 5pm PDT yesterday, intermittently. And definitely full outage by 8:30pm PDT. So for sure full outage for 7 hours and counting, even if you use other runners, and I strongly believe partial outage for ~15.5+ hrs before that, even if it hasn't been acknowledged by GitHub yet.

Yeah, that’s brutal. And really interesting that you were seeing the symptoms intermittently for 15+ hours before the full outage.

You should check out Avrea — they’re building around exactly this problem, with a lot more control and reliability than relying entirely on GitHub Actions. New player, but insanely fast. You can even move your existing runners over with one line, so pretty easy to try.

Is that not still down though because it still relies on GH actions to orchestrate?

It most likely is. We are using Ubicloud (which seems to work the same way hf jobs does) and it's also down. GitHub is not sending out Webhooks and is also not able to queue external jobs. If we had to emergency deploy anything now I'd have no idea how to really do that...

Wait, were Webhooks also down, not notifying about commit pushes?

I am not sure if all Webhooks were affected or only those regarding actions. Theres not much information given besides "Webhook deliveries may be delayed."

https://www.githubstatus.com/incidents/qcvjkzcs7j74

Also use Blacksmith for arm64 runners but this outage apparently affected webhooks so their runners are not picking up jobs either.

Reply on news.ycombinator.com