Hacker News Reader

Comment by kjuulh | Hacker News Reader

We've self-hosted github actions in the past, and self-hosting it doesn't help all that much with the fragile part. For github it is just as much triggering the actions as it is running them. ;) I hope the product gets some investment, because it has been unstable for such a long time, that on the inside it must be just the usual right now. GitHub has by far the worst uptime of any SaaS tools we use at the moment, and it isn't even close.

> Actions is down again, call Brent so he can fix it again...

View on HN

We self host the runners in our infrastructure and the builds are over 10x faster than relying on their cloud runners. It's crazy the performance you get from running runners on your own hardware instead of their shared CPU. Literally from ~8m build time on Gradle + Docker down to mere 15s of Gradle + Docker on self hosted CPUs.

This! We went from 20!! minutes and 1.2k monthly spend on very, very brittle action runs to a full CI run in 4 minutes, always passing, by just by going to Hetzner's server auction page and bid on a 100 euro Ryzen machine.

After self hosting our builds ended up so fast, that we were actually waiting for was GitHub scheduling our agents, rather than it being the job running. It sucked a bit, because we'd optimized it so much, but we on 90th percentile saw that it took 20-30 seconds for github to schedule the jobs as they should. Measured from when the commit hit the branch, to the webhook begin sent.

My company uses GitHub, GitLab, and Jenkins. We'll soon™ be migrating off of GitLab in favor of GitHub because it's a Microsoft shop and we get some kind of discount on GitHub for spending so much other money with Microsoft.

Scheduling jobs, actually getting them running, is virtually instant with GitLab but it's slow AF for GitHub for no discernable reason.

lmao I just realized on this forum writing this way might sound like I own something. To be clear, I don't own shit. I typically write "my employer", and should have here.

> call Brent so he can fix it again

Not sure if a Phoenix Project reference, but if it is, it's certainly in keeping with Github being as fragile as the company in the book!

It is xD On the outside it feels like a product held together with duct tape, wood glue and prayers.

Hey, don't insult wood glue like that.

Indeed, wood glue is amazing. Such slander is totally uncalled for.

I don't know, maybe it's a compliment. Wood glue can form bonds stronger than the material it's bonding. So, the wood glue in this case is better than the service it's holding together :)

or prayers

I tend to just rely on the platform installers, then write my own process scripts to handle the work beyond the runners. Lets me exercise most of the process without having to (re)run the ci/cd processes over and over, which can be cumbersome, and a pain when they do break.

The only self-hosted runners I've used have been for internalized deployments separate from the build or (pre)test processes.

Aside: I've come to rely on Deno heavily for a lot of my scripting needs since it lets me reference repository modules directly and not require a build/install step head of time... just write TypeScript and run.

We choose github actions because it was tied directly to github providing the best pull-request experience etc. We actually didn't really use github actions templating as we'd got our own stuff for that, so the only thing github actions actually had to do was start, run a few light jobs as the CI was technically run elsewhere and then report the final status.

When you've got many 100s of services managing these in actions yaml itself is no bueno. As you mentioned having the option to actually be able to run the CI/CD yourself is a must. Having to wait 5 minutes plus many commits just to test an action drains you very fast.

Granted we did end up making the CI so fast (~ 1 minute with dependency cache, ~4 minutes without), that we saw devs running their setup less and less on their personal workstations for development. Except when github actions went down... ;) We used Jenkins self-hosted before and it was far more stable, but a pain to maintain and understand.

In 2023 I quoted a customer some 30 hours to deploy a Kubernetes cluster to Hetzner specifically to run self-hosted GitHub Actions Runners.

After 10-ish hours the cluster was operational. The remaining 18 (plus 30-something unbillable to satisfy my conscience) were spent trying and failing to diagnose an issue which is still unsolved to this day[1].

[1]: https://github.com/actions/runner-container-hooks/issues/113

Reply on news.ycombinator.com