Comment by golovast | Hacker News Reader

Comment by golovast | Hacker News Reader

I got contacted by our rep a couple weeks ago, who informed me of this news. I thought it was a disaster and it really pissed me off. The rep couldn't even explain the reasoning well. It basically summed up to "because we can" and "where are you going to go?". He was shocked to find out that I didn't like it.

We currently self-host on kubernets/aws. The thing that really got to me isn't the new charge per se. It's the fact that GHA has a ton of problems. I can hold my nose and deal with them when it's free. But now that you're squeezing me, at least you could have created something like GHA 2.0 and added a charge for that. Instead, there are vague roadmap promises which don't even include things that I care about. Specifically:

- Jenkins had better kubernetes integration years ago. It's crazy that GHA can't beat that.

- "Reintroducing multi-label functionality" - yeah, so they first broke it. They did supply "reasons", which looked like they never talked to a customer. [1]

- Still no SDK of any kind.

- "Actions Data Stream" - or you can just fix your logging.

There are dozens more complains, which are easy enough to find. This kind of an approach just makes me want to make sure that I don't use GHA again. Even if I end up paying another vendor, at least I'll be treated as a customer.

[1] - https://github.com/orgs/community/discussions/160682#discuss...

Any official Github action today:

"Thank you for your interest in this GitHub action, however, right now we are not taking contributions.

We continue to focus our resources on strategic areas that help our customers be successful while making developers' lives easier. While GitHub Actions remains a key part of this vision, we are allocating resources towards other areas of Actions and are not taking contributions to this repository at this time. The GitHub public roadmap is the best place to follow along for any updates on features we’re working on and what stage they’re in."

Reference: https://github.com/actions/runner?tab=readme-ov-file#note

They are focusing on an Azure migration for then next 2 years...

GitHub Actions runners are hard to self-host.

The runner configuration and registration process is unnecessarily byzantine. [1]

They can't cancel jobs cleanly. [2]

There are consistency problems everywhere. [3]

Their own documentations describes horrible things unless you use runners in JIT mode. Though JIT runners are not always removed after exit.

If there is a worse self-hosted CI runner, I haven't yet met it.

[1] https://docs.github.com/en/actions/how-tos/manage-runners/se...

[2] https://github.com/orgs/community/discussions/26311

[3] https://github.com/orgs/community/discussions/62365

And if you want any concurrency at all, you need 1 runner registration per concurrent job. And each runner needs its own user. And each runner requires a full and separate copy of the runner software, which is large (hundreds of megs) and self-updates.

You don't need your own user.

The rest is correct. (Though you can hardlink the installation.) And you can disable self-update, though it does it by default.

Hard-linking and running concurrent self-updates, sounds like a recipe for disaster.

Ah right, I've forgotten because I'm using a multi-user strategy and a patched version of the runner at this point anyway. The config directory for each runner is normally based on its install path (insane), something like that?

The config directory based on the install directory, yes.

To be fair, running multiple copies of a service isn't that common.

E.g. if you wanted to run multiple copies of PostgreSQL you can't just do a vanilla install.

I am developing a self-hosted solution for this [1]. It’s true that it’s somewhat of a pain but JIT runners allow a lot of flexibility that we don’t find elsewhere.

[1] https://runs-on.com

I did something very similar.

Create/start/stop/terminate EC2.

https://github.com/redoapp/fast-actions-runner-ec2

In my experience even with runs-on it is a bit painful with runner cancellations.

And they are even on their third (fourth?) from-scratch rewrite of their agent server. When will they get it right? (Rhetorical question)

Yet it works perfectly which forgejo/gitea (uses the act runner)

Nearly 20 years ago, some VP at a security products company now owned by Broadcom threatened us during contract renewal with, "The price is what it is. Your contract is up in two weeks. What are you going to do? Move to a competing product?"

We had it done with a week to spare.

Never underestimate the power of spite! :)

This kinda change also has some different gears turning in my head. At $0.002 / build-minute, some of our large software integration tests would cost us around 15 - 20 cents. Some of our ansible integration tests would be 5 - 10 cents - and we run like 50 - 100 of those per day. Some deployments might cost us a cent or two.

Apples to oranges, naturally, but like this, our infra-jenkins master would pay for itself in hosting in a week of ansible integration testing compared to what GHA would cost. Sure, maintenance is a thing, but honestly, flinging java, docker and a few other things onto a build agent isn't the time-consuming part of maintaining CI infrastructure.

And I mean sure, everything is kinda janky on Jenkins, but everything falls into an expectable corridor of jank you get used to.

Last place I worked had long running end to end tests that would take 30 minutes on GHA (compared to maybe 5 locally) on every PR. This is going to make that a very expensive endeavour

We host a fair bit of Terraform code in a repos on GitHub, including the project that bootstraps and manages our GH org’s config: permissions, repos, etc.

Hilariously, the official Terraform provider for GitHub is full of N+1 API call patterns — aka exponential scaling hotspots — so even generating a plan requires making a separate (remote, rate-limited) API call to check things like the branch protection status of every “main” branch, every action and PR policy, etc. As of today it takes roughly 30 minutes to do a full plan, which has to run as part of CI to make sure the pushed TF code is valid.

With this change, we’ll be paying once to host our projects and again for the privilege of running our own code on our own machines when we push changes…and the bill will continue to grow exponentially b/c the speed of their API serves to set an artificial lower bound on the runtime of our basic tests.

(To be fair, “slow” and “Terraform” often show up and leave parties at suspiciously similar times, and GitHub is far from the only SaaS vendor whose revenue goes up when their systems get slower.)

> Sure, maintenance is a thing, but honestly, flinging java, docker and a few other things onto a build agent isn't the time-consuming part of maintaining CI infrastructure.

Depending on your workplace, there's a whole extra layer of bureaucracy and compliance involved if you self-host things. I aggressively avoid managing any VMs for that reason alone.

Luckily, at work we are this layer of bureaucracy and compliance. I'm very much pushing the agenda and idea that managing a stateful, mutable linux VM is a complex skill on it's own and incurs toil that's both recurring and hard to automate. The best case to handle that is to place your use case into our config management and let us manage it.

Most modern development workflows should just pickup a host with some container engine and do their work in stateless containers with some external state mapped in, like package caches. It's much easier for both sides in a majority of cases.

> And I mean sure, everything is kinda janky on Jenkins, but everything falls into an expectable corridor of jank you get used to.

Self-hosting Jenkins on an EC2 instance is probably going to result in a _better_ experience at this point. Github Cache is barely better than just downloading assets directly, and with Jenkins you can trivially use a local disk for caching.

Or if you're feeling fancy and want more isolation, host a local RustFS installation and use S3 caching in your favorite tools.

Self-hosting on a host whose data actually persists is an even better experience, as it removes a lot of the tedium and workarounds such as extracting/down-/up-loading caches and so on. Get another host for redundancy and call it a day.

Hardware is getting cheaper and cheaper, but the fear-mongering around running a Linux machine has successfully prevented most businesses from reaping those cost reductions.

Complete persistence has its downsides, as you can start getting "path dependency". E.g. a build succeeds only because some images were pre-cached by a previous build.

But having an _option_ to not download everything every time is great. You can add a periodic cache flushing, after all.

> Hardware is getting cheaper and cheaper

Unfortunately not anymore and not in the foreseen future if we don't see some AI investment corrections.

I repurposed old M1/M4 Mac Mini's at my workplace into GitHub action runners. Works like a charm, and made our workflows simpler and faster. Persisting the working directory between runs was a big performance boost.

> And I mean sure, everything is kinda janky on Jenkins, but everything falls into an expectable corridor of jank you get used to.

This is kinda where I am. No one really feels like they are selling a premium "just works" product. Its all jank. So why it the jank I chose at the price I chose?

At the moment I'm self hosting gitlab runners. Its jank. But it's free.

A while ago I set out to find a replacement for Jenkins. On prem with a comparable feature set. What I found out is that Jenkins is the worst apart from all the others.

It seems clear to me this is in response to all the third party GHA runners who were undercutting GitHub by just reselling cloud instances for cheaper.

They’ve lowered their runner costs to compete, and introduce minimum charge to discourage abd make sure they still get paid.

100% this.

They could have made their service better or more competitive like with price but instead they chose this route. SMH.

Reply on news.ycombinator.com