GitHub Actions Are a Problem | Hacker News Reader

GitHub Actions Are a Problem

felix-knorr.net

158 pointsbenrutter187 comments

The runner is open source, but once you read the code you'd wish it wasn't.

For anyone in the audience who didn't know: GitHub Actions is based on Visual Studio Team Foundation Server's CI, and later Azure DevOps. And nobody, including the current maintainers, seem to know exactly how it all works (or how it doesn't).[1] The random exit codes is just the cherry on top!

[1]: https://github.com/actions/runner/issues/510

> And nobody, including the current maintainers, seem to know exactly how it all works (or how it doesn't).[1]

I read through this whole issue, and I cannot find exactly what you're referring to. What message gives you the impression they don't know how it all works? Seems it's mostly people asking for updates, and eventually something similar but different got implemented.

The fact that the --once flag was implemented and just didn't work. And it didn't work for several reasons. One, the control flow in the runner is extremely convoluted and exiting out is not trivial, and two, the scheduler doesn't handle assigned jobs not being picked up.

And reason 0: it was never completely implemented, wasn't documented anywhere and wasn't supposed to be used by anyone at all.

It's all the linked GitHub issue, begin from the top.

"Wasn't supposed to be used" is all fine and good until your GitHub rep uses its existence to sell you on actions...

Is that something that actually happened, or are you talking about a theoretical case? It wouldn't surprise me if a Microsoft salesperson would do something like that, but still

I attempted to read that code in order to glean information about how it communicates back to GitHub about progress etc. The goal was to make a new runner. I ran for the hills, fast.

It’s easy to look at the endgame. Just look at the hurdles https://github.com/nektos/act has to work with

Sadly that's an emulator of the entire stack, I wanted to do the runner part only (the goal being to end up with a sane self-hosted runner replacement).

I’m not sure if it’s open source but the actions-runner-controller project maintains a slimmed down version of the runner image that might be worth looking at

Sounds like most Microsoft projects to be honest. Not a lot of them can be highlighted as the pinnacle of software engineering, including their GitHub Actions implementation.

No kidding. I have been in the bowels of EF at work, and it has led me to (likely not independently) coin "encapsulation infatuation". Bloody hell, what a beautifully decoupled mess. The only way to make sense of a single thing is to step through/into each line of code.

Looks like GHA was announced[1] around the same time as the acquisition by MS. The first commit in that repo is a year later when they opensourced it, so we can't see how it evolved before then.

For anyone who was at Github at the time, was it always written in C# or rewritten/replaced after acquisition? If "GitHub Actions is based on Visual Studio Team Foundation Server's CI" is the case, then it sounds like the latter.

[1]: https://github.blog/2018-10-16-future-of-software/

This probably answers your question:

https://github.com/actions/runner/blob/a4c57f27477077e57545a...

I don't recall the exact context anymore but during the Microsoft Build this year it was said by an employee at Github that the service came with the acquisition which is the reason it was and stays C#.

You can also find many leftovers with the "VSS" acronym, like https://github.com/actions/runner/blob/main/src/Sdk/Common/C... or https://github.com/actions/runner/blob/main/src/Sdk/Common/C... - which also mentions TFS (which is yet another acronym that used to refer to the Microsoft team-development thing).

I searched for "github actions random exit codes" and didn't find anything.

Personal experience building infrastructure and an autoscaler for these things (which performed horribly due to API lag) while having weekly meetings with our GitHub rep to get the --ephemeral flag in (took about a year longer than promised). Sometimes the exit code would be 0 when using `--once`, and sometimes it would not be. Sometimes it'd also be 0 if the token somehow didn't work and the worker couldn't even register no matter how often you restarted it (of course with a cryptic Azure IAM error code). Either way, we eventually just decided that throwing away the machine if the runner exists for any reason was safest.

> GitHub Actions is based on Visual Studio Team Foundation Server's CI, and later Azure DevOps

Yes and no, ADO Agent (https://github.com/microsoft/azure-pipelines-agent) is far more secretive and "black-box" alike. Like stuck in old version of NodeJS, Powershell, API without documentation or even enough tests/samples...

I read one file, src/BrokenSomething.cs, and as an opinionated SWE I would say they could do some Separation of Concerns refactoring to clean it up.

I could do that refactoring, I suppose I could make it better, in addition to piling on about it on hacker news. ;-)

If you want the ability to run as much of your GitHub Actions locally as possible (which you should! Testing via pushing commits is painful) you can arrange your actions so the majority of the logic lives in shell scripts (or Python scripts or whatever) that are called by the workflow.

Then you can run those scripts locally and iterate in them before you try them on GitHub Actions.

I agree: Actions would be more useful if I could run the entire stack locally (an official implementation, not a community-maintained clone). But it's not a big enough inconvenience for me to care very much.

The problem is that GitHub Action offers a large ecosystem of reusable actions that you can't easily call from a script.

I mainly ignore those - I stick to the actions that install things (setup-python) and run scripts.

You can get around the setup ones by using Docker images (e.g. run in python:3.11 rather than using setup-python).

Can you provide a link to an example?

The docs have examples: https://docs.github.com/en/actions/using-jobs/running-jobs-i...

Basically you set `container.image`.

You should check out: https://github.com/nektos/act

Works pretty well for the actions I have used it on.

That's not a problem. You just don't use them if you don't want to suffer vendor lock.

> That's not a problem. You just don't use them if you don't want to suffer vendor lock.

How do you use GitHub Actions without using actions such as actions/checkout, actions/upload-artifact, actions/download-artifact?

You, you sacrifice that convenience for something purely in shell that is more portable.

you you?

It's like a yo-yo but faster

Like yo-yo, but slower due to the extra W.

This is the way. It also lets you pretty much seamlessly switch CI tools at any time.

I third this opinion: another reason it’s important is it allows you to do your build in an emergency when CI is down for whatever reason.

CI automation is for me the thing that replaces me running scripts one by one (and reporting/deploying results)

It is not the thing that does the building/testing/deploying. That’s the scripts that are hopefully written in a debuggable, portable, language.

This is a really good point. Most of my deploys go via GitHub Actions these days and it's not easy for me to deploy manually should I need to when Actions is unavailable - I should fix that!

Like aviation, every regulation/rule is written in blood ;)

This is why the three of us should record a podcast episode talking about this in the context of the points made in the Ironies of Automation

https://en.wikipedia.org/wiki/Ironies_of_Automation

Agreed - I had to switch from TeamCity to GitLab earlier this year and I was very glad that all of pipelines were single-line calls to shell scripts.

Indeed, to rely on third party actions in a closed runner ecosystem, is simply unwise.

Isn't that exactly what the self hosted runners are?

https://docs.github.com/en/actions/hosting-your-own-runners/...

Then you configure the repo settings to only use the self hosted ones.

If you're suggesting the self-hosted runners are equivalent to running locally, then no, self-hosted runners still require a commit + push and runs remotely with the added overhead of the job being queued, reported, etc.

The dream is to have something like a container _locally_ and be able to run something like:

    > gh actions run

I wish I didn't have to circumvent the UI with a strategy like this. Part of the reason I like CI/CD tools is the visualization factor - I can create specific steps, see which step is currently active during execution, and only care for the output of a specific step during debugging.

A platform with support for visual control from the scripts (implemented as no-ops during local execution) would be perfect.

If you split your scripts into small tasks (i.e. one task per script), and call each one from your GA yaml, you would seem to have the best of both?

That still makes running it a bit annoying, since you have to call each script in order. You could of course create a script that covers this, but that's duplication that can drift apart. I guess you could generate the script from your YAML or use a linter or something, but it all feels a bit annoying. It would be awesome to e.g. just call a command like "gh-success" or "gh-failure" to manually indicate the success/failure, maybe with an optional ID to prevent accidental double calls etc.

Doing this also means that your build scripts are version controlled. You can iterate on the build scripts in different branches, for example.

GitHub actions are configured by version-controlled files too though. I have an actions-dev branch on my project, so I can iterate on actions config without having to clutter up master.

Would still be better if I could run locally to iterate. But the local run would have to have very high fidelity to what happens on GitHub to be useful.

Running actions triggered from a non-main branch does it for me, either on pushes to the branch, or 'workflow_dispatch'. You'll otherwise never get that level of fidelity to what happens in production.

Keep the logic in your YAML "dumb". Avoid variables and subroutines in the file, if you want to DRY something, create your own custom action. You can have unit tests on an action, you can't have unit tests on a GHA workflow.

That cuts both ways, in my experience: a bug fix needs a metric boatload of cherry-pick, possibly across a bunch of repos, to fix any bugs. Centralization is great until it's not

We have gotten a lot of good mileage out of GitLab's include: feature <https://docs.gitlab.com/ee/ci/yaml/#include>, which for clarity absolutely carries this same hazard, but can mean centralized fixes are less onoz than trying to patch every active branch of every active repo's build scripts

I am also aware this cuts against the thrust of the thread about "build scripts should be locally executable," but until they (they: GitHub, they: GitLab) straighten out their runner binary to be sane, it is currently a better use of our engineering effort maintaining CI yaml than teaching devs how to run CI locally

Incidentally, this is sort of how the Earthly runner serves to provide you local actions that also work in CI.

I'm not quite familiar with GitHub Actions, but Azure DevOps Pipeline has a nice Preview API: https://learn.microsoft.com/en-us/rest/api/azure/devops/pipe... which runs the preprocessing steps(like running the C preprocessor on a C source file) then gives you the processed yaml file. Then if you'd like, you can write your own local runner based on that if your yaml files do not use too many different kinds Azure DevOps pipeline steps.

yes! what we did was use a task runner to encapsulate the work in a “ci” task. you can run it locally, and then the GHA runner does little more than clone, install a couple depa and run the task.

This is the way.

Building Java and C# on the command line nowadays is a hard undertaking, requiring weeks of learning and debugging, just to have your code change in a way that makes the scripts outdated at any time (oh, and it's never portable).

I really don't get why people use those complex CI/CD tools for other languages, but at least on the enterprise ones, I can understand people moving from the IDE into some huge centralized mess that can import it.

Why does Java take weeks?

For a straightforward application or library project, you can fill in this form and get a shovel-ready build: https://gradle-initializr.cleverapps.io/

I usually tweak it a bit, but that's minutes, not weeks.

There's often more stuff to add as a project grows, but again, minutes to hours for the common stuff. Setting up publishing to a Maven repository is more work than I'd like, but it's still not weeks.

So what are you doing that it takes so long? And how long would that take if done any other way?

Building C# is quite literally 'dotnet build -c release'.

Exactly. I don't know where the newer .NET versions get the bad rap from.

You can criticize Microsoft for other things but .NET is not one of them. They've done a great job with it.

(And building code that targets the classic .NET Framework can usually be accomplished by a single call to msbuild.exe.)

I would respectfully suggest that the author is misusing CI. If you have trouble running your tests locally, you have a problem. If you have trouble deploying from your local code, you have a problem. All of those capabilities should exist as simple scripts in your project already. Once you have that done, the CI yaml is a simple glue layer that defines an order of operations, e.g.:

1. Run static tests

2. If those pass, run unit/integration tests

3. If those pass, deploy

If you find yourself screaming about YAML, you're leaning too heavily on it and need to refactor your project's scripts.

Maybe a good question to ask would be "if I had to switch to another CI system today, how hard would it be?" If the answer is "hard", perhaps you're leaning too heavily on it and need to refactor your project's scripts.

90% of the tricky parts of CI are secret management. As long as you write scripts to pick up credentials in a sane way (.awsprofile or similar) you should be able to configure your CI to provide the credentials just as well as you can locally - but in practice, the various different ways that things like artifact repositories, integration test databases, and cloud deployment tools want to manage auth is the cause of most of the complexity in getting your build/test/deploy pipeline working on the runner.

I would say the secret management part typically isn't the most insane or annoying part.

For me it's external system state management. Like making sure the integration test db is cleaned up correctly.

This is all fine and good if you are the principal developer of the project. However, the author makes it clear that he is migrating other people's CI pipelines. He is a DevOps engineer working across several teams.

This is why he makes the important point that discipline is not enough. The reason is most teams simply don't care. I find one in five teams where everyone on the team cares about the build (when I'm lucky!), most teams have one person who cares, and some teams have no one that cares.

When I am tasked with the proper care and feeding of the pipelines of others, I want tools that can work and help me out even when the developers who created the software are Holding it Wrong.

With these requirements in mind -- managing and migrating the many different CI piplines across an organization -- it would be a major breakthrough to have a tool that 1) transpiles to the workflows of all the CI tools and 2) allows for local testing. So many orgs have different teams using different CI stacks, and the local testing problem is always a struggle. I would use a tool like that into the ground.

So I would qualify your original statement: The author isn't misusing CI. Rather, the author is attempting to survive in a world where others are misusing it, and where the author is tasked with managing all the CI pipelines.

Testing locally, I agree with.

Deployments, I think it'd be fair to consider the requirements. At work, our softare can be tested locally but deployments are all registered against a central authority, and after a point of composing enough access requirements, only then does a role (cicd in this case) have enough policy allowance to perform a deployment.

The entire transaction is auditable. And I think that with a deployment, that's how it should be; allowing that trust down to a local environment strikes me that too much permission is accured with a single entity.

I guess that we could better define what a deployment is; to some nonprod environments I'd agree, but I'd still probably insist on the heavy machinery up at the test, perf, qa, areas, and then getting into staging and prod, there'd be no wiggle room.

Fair critique, totally depends on what kind of software we're talking about and where the deployment is happening. In general though, how screwed are you if your CI environment goes down? Can you not deploy anything? That would be scary.

My point, however, was mostly that the logic necessary to deploy should live as part of your codebase, not written out in YAML. The privs necessary to deploy are a separate discussion.

This sounds fine and well, but it's not how Github Actions work (or at least, not the encouraged workflow). Let's have a look at the snippet from one of the projects I work on:

  - name: Set up Docker Buildx
    uses: docker/setup-buildx-action@v2
  - name: Build and push mwdb-core image
    uses: docker/build-push-action@v4
    with:
      file: ./deploy/docker/Dockerfile
      tags: |
        certpl/mwdb:${{ github.sha }}
        certpl/mwdb:master
      cache-from: |
        type=registry,ref=certpl/mwdb:buildcache
      outputs: type=docker,dest=./mwdb-image
  - name: Upload mwdb-core image
    uses: actions/upload-artifact@v3
    with:
      name: mwdb-image
      path: mwdb-image

Good luck running this locally. There's no script code to speak of, just references to external "actions" and parameters (for example, https://github.com/docker/setup-buildx-action).

Some CI platforms are just a simple glue layer (Gitlab CI - which I prefer - is one of them), but in most cases Github CI is not. Maybe it adds to the author frustration?

Building it that way is a choice. It's not mandatory.

You can use gitlab CI with special-purpose docker images for all your steps and magic parameters driving everything too (Gitlab AutoDevops works that way).

But if you just run your steps in shell scripts in vanilla docker images containing your build-time dependencies, you should be able to produce something that works the same in any CI pipeline, or locally.

The most annoying thing for me is that a lot of CI engines make docker-in-docker complicated. I love using compose to set up integration test environments, but doing that in CI is often a fight.

> Building it that way is a choice. It's not mandatory.

This ^ . In GitHub Actions, I personally try to use pre-baked actions as little as possible, for exactly the reasons I outlined.

I prefer GitLab CI, but you can make a mess of that just as easily. In general, if you approach CI as I suggested, you end up with something maintainable regardless of the CI engine in use.

Yeah, Github Actions has its cool parts, but I still prefer Gitlab CI since its just 'dumber', and doing custom things is simpler to achieve.

As another nod to GitLab's CI, it has sane autocomplete in the editor since the schema is mostly static. I'm not aware of any GitHub yaml tooling that downloads the descriptors(?) for any actions in use and with then allow autocomplete of its uses: or with: blocks

> I would respectfully suggest that the author is misusing CI. If you have trouble running your tests locally, you have a problem.

I would respectfully suggest that you misread the author. The issue isn't running tests locally, it's running the CI config locally.

I experienced the same problem with gitlab CI years ago, where, basically, you can lint the file and not much more. Past that, you need to run it through your CI and debug if you get slightly different results compared to running a script locally.

Yeah, it's a horrible experience all around. CircleCI solved this problem like a decade ago, enabling SSH builds so you can troubleshoot straight up in the build itself, and once you've figured it out, just copy-paste the steps to your Makefile/CI config.

I don't understand how one could build a CI service so long time after CircleCI launched, and still not have that very same feature (or something similar).

The authors speak about how hard it is nowadays to provide a CI extension to the major platform including GitHub, Azure DevOps and Gitlab. (and others)

Wanting to run locally a developed extension is totally legit as some can be really tricky and depends on the behavior of [runner & OS].

My strategy is just to keep the yaml to a minimum and call python scripts within the action to do anything complicated. The GHA yaml is just used for defining when the action runs, and on what architectures, etc. It's worked well for me.

Same. My local astk runner does the heavy lifting.

Was that a typo for task or is 'astk' a punnily named task runner in its own right?

(sorry for the dumbass question but it could go either way)

Github Actions gives me literally free server time across an extremely wide range of OS'es that I dont have to worry about at all, including Windows and OSX, which I therefore dont have to deal with in any way, buy any license keys, none of that. It's nothing short of miraculous for us as it's how Python projects can have binary wheel files for dozens of OSes and Python versions: https://pypi.org/project/SQLAlchemy/#files . This task remained impossible for years (to be clear: because I don't have a server farm, or Windows licenses, or however you'd run OSX on a headless server, or any kind of resources to fund / manage dozens of images and keep them running, or any of that) until GH Actions made it possible.

Now is this all part of Microsoft's evil plan ? It probably is! But unless someone else wants to give me a free server farm that includes Windows / OSX up and running without me paying anything / writing containers / storing images / etc. I dont see this aspect of Github actions losing any popularity.

> This task remained impossible for years until GH Actions made it possible.

Every major distribution used build farms long before GitHub (and git) existed...

Yes, but I don’t have a build farm.

Vendor lock-in hence the evil plan:

https://dev.to/pwd9000/how-github-actions-can-improve-cicd-a...

"While GitHub Actions is an open platform, it is still tied to the GitHub ecosystem. This can be a concern for teams who want to maintain flexibility and avoid vendor lock-in."

https://gsilvapt.me/posts/moving-to-github-actions/

"Actions are a Vendor Lock-in Mechanism"

"But are you really sparing that many lines of code? Ultimately, you’re going to have to declare what your environment needs so other developers can contribute to the project anyway, so why not simply write a bash script to take care of that for you?"

For Windows and OS X?

That's still the main reason I use Github (although Gitlab has them now in beta(?) https://docs.gitlab.com/ee/ci/runners/index.html).

Local execution of GitHub actions for testing will become possible some time after pigs learn to fly.

Everyone here is asking for it as if it’s some minor oversight, soon to be rectified.

Unfortunately, this tech stack is a significant revenue source. Microsoft charges for pipeline minutes, concurrent runs, etc… This is especially true in Azure DevOps which shares much of same underlying pipeline software.

Letting anyone run this locally for any reason would let them bypass the monetisation.

It’s the same reason that ad-supported YouTube is “missing” a download offline feature.

It’s not an oversight. It’s not happening. Stop asking.

The only thing we the dev community can do about this is to develop our own open-source CD platform with blackjack and hookers.

In other words, we need a Kubernetes of CI/CD.

Yes Github actions is (are?) a mess lacking conceptual integrity and widely misused. But we do the following with success:

1. Jobs should always (just) execute a script or program. This allows running outside the CI system.

2. To test/debug CI jobs, use act.

3. To test/debug more complex scenarios spin up a Gitea instance (which provides a Github clone wrapping act).

> 2. To test/debug CI jobs, use act.

I can't wait for this meme to die, or for act (or gitea's fork thereof) to catch up to the hype train. Then again, I guess this fantasy is being promoted by folks who are all "just use run: and that's it" because any moderately complex one <https://github.com/VSCodium/vscodium/blob/1.84.2.23314/.gith...> for sure fails with spectacularly illegible error messages under both act and gitea's fork

> I can't wait for this meme to die

This is not helpful. You, the author, and random commenters scattered around keep teasing that act is bad, but I can't seem to find our what any of you mean.

One comment said github APIs fail, which would make sense to me. Is that the primary reason for act being a pain? Do you have output for the linked build yml or an explanation of where it goes wrong?

I feel I'm being trolled, but I'll bite and accept the resulting downvotes

I don't think treating every mention of act as an opportunity for airing of personal grievances is helpful in a discussion when there's already ample reports of people's concrete issues with it, had one looked at the 800 issues in its repo https://github.com/nektos/act/issues?q=is%3Aissue or the 239 from gitea's for https://gitea.com/gitea/act_runner/issues or whatever is going on with Forgejo's fork https://code.forgejo.org/forgejo/act .

But, as for me specifically, there are two and a half answers: I wanted to run VSCodium's build locally, which act for sure puked about. Then, while trying to troubleshoot that, I thought I'd try something simpler and have it run the lint job from act's own repo <https://github.com/nektos/act/blob/1252e551b8672b1e16dc8835d...> to rule out "you're holding it wrong" type junk. It died with

  [checks/lint]     Failure - Main actions/setup-go@v3
  [checks/lint] failed to create exec: Error response from daemon: Container b9059f831d3a1549c9902cabc7e5258231b2a7291b2692eaa33f976794059738 is not running

and no amount of --verbose would tell me what, specifically, "Failure" means or what action I'm supposed to take about it

Finally, I believe I am pretty self-service when projects are written in sane programming languages and publish under a permissive license, so the "and a half" is that I tried to introduce some better logging or fix up some of the more egregious silliness that I found in the codebase, but ultimately it felt like I was pushing a boulder uphill so I let that go. I haven't been following the progress of any of the 1100 forks on GH, or the hundreds of Gitea forks, or whatever the hell is going on with Forgejo nowadays

But my experience is that if the thing can't run its own workflows, or at least emit some constructive action that I, the user, can take about that situation, then the VSCodium one that I linked to has no prayer

> I feel I'm being trolled

Fair, but I definitely wasn't trolling, and I really appreciate your comment. 2 days ago I told myself I'd set up gitea actions after experimenting with drone, buildbot, earthly, and Jenkins and not being fully satisfied with any (Jenkins feels old and runners were annoying to set up, buildbot was too involved and didnt have a ui, and I couldn't find an execution environment for earthly to run in on git push).

Personally, I put more weight in real-world complaints than issue trackers. For example, everyone I know that uses OBS has never had a single complaint, yet there are 400 open issues. I was looking for an example of what real users run into and I just couldn't find that looking at the 11-month old hn article or reddit. Based on your comment, I will keep that in mind and bail at the first sign of frustrating behavior.

I guess I'll give my opinion here for anyone who's looking for CI or CD. Earthly is a really cool CI tool, but as far as I know, it doesn't have an actual run component. Think of it like make for the docker era, and it does that very well. Drone is cool, but it runs in docker, so trying to build docker images with earthly (also runs in docker) is a pain unless you mount your docker sock, which I don't want to do. Jenkins is the most functional and handles all use cases. There's a reason it's still so huge in the self-hosted world.

> Hosting a git repo is hardly more than providing a file system and SSH access. The actual mechanism they use to keep you on their platform is the CI-pipelines

Then why was GitHub so popular for the 10+ years it had no built in CI system?

I‘d also rather say that it was about hosting the code for free in the first place and then the pull requests, including the possibility to comment on code in PRs. I use the git cli (instead of some UI / IDE extension) for all interactions with the repo locally. But as soon as it‘s about collaboration, these platforms come into play.

I just quickly scanned, to find that there is the `git request-pull` command, before I wasn’t even sure whether pull requests are a git built-in feature at all.

Side question: does any code hosting platform allow to comment on lines of code outside of pull requests? I‘ve had several occasions where I wanted to ask, why something was written the way it was.

a github pull request isn't a pull request; a pull request is an email from one of linus torvalds' direct underlings (subsystem maintainers) to linus torvalds "requesting" that he "pull" (hence the name) some tag. an arbitrary example: https://lkml.org/lkml/2017/11/13/229

git request-pull generates these emails.

note that a "pull" is just "merge from a URL", and requires some preexisting trust, hence why it's only for the subsystem maintainers.

github stole this term for their signature misfeature and we've all been suffering since. some of its clones walk back this poor naming by saying "merge request" instead, but the damage to the name is done.

Before GitHub, even before git, we just emailed or irc'ed a patch request to the team. Somehow, "pull request" caught on as a term.

Yeah, utter nonsense. The mechanisms they use to keep people on GitHub is a) network effects (easy to create issues, PRs etc because you already have an account), and b) GitHub is actually really good!

But aside from enterprise SSO, far better permissions management than you get with SSH and Linux filesystem permissions, a unified open-source project discovery and vetting-assistance system, secrets management, integrated CI, lfs support, issue tracking, a billion integrations for-free, automatic dependency vulnerability detection, et c…

What have the Romans done for us?

I've always been opposing push-to-deploy, unless you actually have per-branch, ephemeral test environments, and take the required care to set up your data fixtures - which is a major technical challenge on any non-trivial project.

The trick is to keep the deployment code simple and stupid, while making the process transparent and dependable for the team.

For production environments, rather than thinking in terms of pushing a commit, you want to reframe the problem as promoting a version from a staging environment - preferably subject to approval from QA and/or project owner. ChatOps is great for that, and it doesn't have to be painfully complicated, or full of hacky Hubot code - you can get huge ROI from a simple script[1] that posts a message at the start&end of a deployment, which includes a version string (autogenerated from a git tag[2]), environment, maybe your login/host name ("$(id -un)@$(hostname)"). CI can build the artifacts, then your job is just "./deploy prod v3.14", which is also easily delegated.

[1]:

    def slack_post(data):
        req = urllib.request.Request(
            "https://hooks.slack.com/services/xxx/xxx/xxx",
            headers={"content-type": "application/json"},
            data=json.dumps(data).encode("utf8"),
            method="POST",
        )
        res = urllib.request.urlopen(req)
        assert res.getcode() in range(200, 300)
        return res

[2]:

    #!/bin/sh
    set -eu
    slugify() {
        sed -E -e 's/[^A-Za-z0-9]/-/g' -e 's/-+/-/g' -e 's/(^-|-$)//g'
    }
    git_tag=$(git describe --tags --dirty --always)
    git_branch=$(git rev-parse --abbrev-ref HEAD | slugify)
    case "${git_branch}" in
        master|HEAD) echo "${git_tag}"; ;;
        *) echo "${git_tag}-${git_branch}"; ;;
    esac

push to deploy can be made okay by giving it promoting-a-version semantics, with the advantage of having a deploy-history graph, but the concept as it's typically described and the way it's used in practice push people towards being not very okay at all.

your approach is less likely to be misused, which often outweighs having a deploy-history graph

The chat log is the deploy history ;)

Sometimes stupid really does beat clever. Spend your tech debt tokens on the product instead!

there's zero (or negative?) technical debt, it's all social debt

Hey there, Dagger employee here.

We've heard this feedback from a bunch of our users, and even though the awesome work of nektos/act has helped some of them, we feel like there might be more to just try to emulate GHA locally. After all, GitHub's doesn't have any incentives to allow their users escape their platform.

Having said that, our community has pushed the boundaries of Dagger and has made awesome progress on streamlining GHA into our platform. Here's two in-progress demos about some of that work

Source to action: https://youtu.be/EgOvSbwKVts Running GHA in Dagger (project gale): https://youtu.be/rHk3spfCHFQ

We could use some feedback and help to make this even better. If you feel like that might be interesting, we're very active in our discord server: https://discord.com/invite/dagger-io

The non-video TruStacks link: https://github.com/TruStacks/trustacks#trustacks-software-de... (Golang; GPLv3)

The non-video Gale link: https://github.com/aweris/gale#github-action-local-executor (Golang; Apache 2)

We have built an open-source generic workflow engine to run arbitrary scripts (https://windmill.dev) with a vscode extension to build the yaml using a low-code builder and each individual script in their dedicated python/ts files so you get your full editor assistants https://youtu.be/aSOF6AzyDr8?t=116

One of the area we are expanding next is a github app so you get exactly the same UX as github actions but running windmill workflows on your windmill workers.

No pwsh support?

We do support pwsh :)

Great! Will give it a try.

I had to Google this, apparently pwsh is the incarnation for PowerCrap I mean PowerShell.

You had to Google that? It’s been widely used since 2018 to refer to PowerShell.

What I always guessed: if you don't want to bite the bullet of trusting SRE scripters (with an s) to the role of building the CD that actually makes sense for your use case, why do you expect some random company will have more of your interest at heart? At least with Joe's python deployment scripts, Jane can do a code review. So much of cloud CI/CD lingo translates to "believe in magic, and also our bigness-based authority" when actually proper software deploys itself properly without making a fuss or glueing your fingers with sticky Yaml...

I've been migrating some Workflows to dagger.io. So far it's amazing.

Thanks for sharing, I was unaware of dagger.io!

I'm curious now -- are there any other alternatives with similar philosophy/benefits?

Yes, there's us over at https://garden.io! We're big believers in pipelines that run anywhere. I even made a short little video that should give you the gist. [1]

Some of the short-list of differences: we use YAML for our configuration language, Dagger can use full-fat languages to define its pipelines. Our feature scope is broader: you can use us to vend IDP-like stacks to your developers if you're a Platform Team; we make development with remote Kubernetes clusters very easy, including all the remote image builds; and we have a number of integrations so you can bring your IaC tool of choice (Pulumi, Terraform) into your pipeline and set up service -> infra dependencies.

[1] https://www.youtube.com/watch?v=JFnan6s2cDg

Thanks for sharing, will definitely check this out!

There's earthly.dev but like previous commenters, I went with dagger because i prefer its Apache 2.0 license over earthly's BSL.

Earthly is MPL 2.0 now. I work for earthly and people didn't like BSL so we listened and changed.

Earthly is legitimately one of the coolest projects I wish I could use. But entrenched velocity in existing systems makes it a difficult pitch

Hey, Dagger employee here. This exactly is one of the the main design decisions of Dagger. We're not expecting companies and/or projects to perform a full migration to it, that's why we generally recommend starting by wrapping different parts of your pipeline and move on from there. Unlike other solutions, Dagger allows you to keep using your existing tooling and reuse Dockerfiles (https://docs.dagger.io/quickstart/429462/build-dockerfile/#r...) while slowly transition your pipelines to code.

That's great to here!

Sometimes people find just starting with a messy part of the build process, or starting to replace dockerfiles here and there can add value, without having to change the world.

Thanks for sharing, will definitely check this out!

Same here, it's so liberating! I'm also using a Magefile to run the pipelines (written in Go) locally.

Crazy thing about this and is frustratingly obvious, as I become a little older in IT, is that this is just a workflow problem.

CI build pipelines aren't even really in the class of workflow problems that are even difficult to implement... workflow engines start you down a rabbit hole towards different models of computation that eventually ends up with a turing machine anyway, but I digress.

It's just really kind of ridiculous that the same basic problem gets re-implemented literally thousands and thousands of times across different platforms and use cases. The fact that the current implementation is based on some Frankenstein of former visual studio code and other stuff really, isn't that surprising.

Of course! Standardizing workflows leads precisely to the XkCD referenced in the article, in standards produced by the workflow vendors of of course been horrendous and focused on preserving or enhancing market share, and we're done in the era of XML.

The issue of course is that workflows are an "Enterprise" feature and solution.

Perhaps someday in the halcyon future someone will actually address a useful cross platform workflow model I could conserve algorithms, processes, techniques, and maybe even plugins. Because right now the workflow engine is balkanized across every single CI platform.

> It's just really kind of ridiculous that the same basic problem gets re-implemented literally thousands and thousands of times across different platforms and use cases

Why not? It's incredibly easy and quick to implement one, why not just make one perfectly fit for your use case every time?

The actual issue here is people insisting on offshoring that implementation to GitHub, so that they deal with a brittle, badly defined interface that is not aligned with their needs, instead of just starting with a simple engine and reimplementing whatever they need. It's actually the opposite problem that standards were meant to solve.

You THINK it's easy. A nice clean set of boxes in a line with arrows.

... until you get branching. Exceptions/error branching. Subflows.

Suddenly, you have a fully turing complete engine. The modern compromise is a DAG that hopefully guides a workflow to completion a la the halting problem.

Anyway, there are a ton of piss poor ones with too many bespoke interfaces.

> It's incredibly easy and quick to implement one, why not just make one perfectly fit for your use case every time?

Look around, it’s so easy there are corpses of companies and projects all around the industry.

You mean companies that failed because they implemented their own build and deploy scripts? Their own CI/CD workflow engine? Or for trying to sell a workflow engine?

Because I can imagine why almost nobody could be able to do the third one.

I actually really like GitHub actions, I read the article and while I get some of the concerns, others I don't understand. In any case the author's situation doesn't apply to me, and I wanted to share something I really liked about GHA.

So, I recently figured out a way to host a remote browser on them by using an Ngrok tunnel. It's really cool to see BrowserBox running from inside a GitHub action container. I literally couldn't believe it actually worked when I first figured it out!

I was so excited. It started as just this tech prototype in my mind (could this be possible? Probably not but I Feel like it could be). And to see it actually achieved so cool! :)

It has made CI integration testing SO much easier and more repeatable. I love that it can just run it up on Ubuntu and I can verify.

Anyhow, I thought this was so cool, and such a useful way for people to either just get started with BrowserBox trying it out, or even run a quick little VPN-like/proxy browser from another region. I've even logged into HN from it on the sly. I liked this whoel concept so much, that I even wrote an action that integrates with issues to make the process as easy as possible for people.

Basically you can just clone or fork the repo: https://github.com/BrowserBox/BrowserBox and then open an issue and pick the template that is like "Make VPN". The login link will get published in the repo. The link is not private (unless you make your fork or template private) and there's a bit of setup with your ngrok API key (free is OK) but the issue conversation automatically guides you through all that.

I thought this was so cool (free server time, actually working app), that I even created another version that uses MS Edge under the hood instead of Chrome in the original, just to show how easy it is: https://github.com/MSEdgeLord/EdgeLord

Just a niggle is that the other services we normally have (secure doc viewer, audio, remote devtools) do not work as ngrok only maps 1 port. I could use an ngrok config file I think to fix that but somehow, easy as that is, I have not gotten around to it! Another niggle is I noticed the auto-tab opening used in the GHA demo seems a little funky lately, and you may need to manually reload or resize them to un-wonkify it. Probably a little regression!

Anyway! :)

You might also be interested in https://github.com/mxschmitt/action-tmate, which enables SSH to workers for debugging. If you have trouble reproducing issues on CI, this can be a life saver (or at least save a few hours of commit-push-wait rounds).

Cool, thank you! Yes, I haven't so far but I might in future need that. Anything that increase the speed of those feedback loops! :) I wonder how it works? Looks cool

Super cool. Have you considered using alternative technology that allows more mapping than 1 port, e.g., open source OpenZiti? This is an example of embedding their SDKs into a webhook action to connect to a server in a completely private network - https://netfoundry.io/this-is-the-way-invisible-jenkins/

I didn’t know about that but thank you for letting me know. I will take a look! :-)

> Defining a language to run scripts in parallel with some constraints is actually quite easy if you just pass the script to an existing interpreter and let the language mainly deal with the constraints around order and parallelism (and code reuse)

Isn't this Make?

Right I like doing everything in Bazel instead of make now but still the same idea to keep it independent of CICD.

Bazel? How many hours per year are you allowed to spend on CI? :D

Lol our company uses it a lot so I use it a bit. I never learned make properly so I figure I might as well ignore it for now.

The idea of a DSL over YAML is also a good one. Jenkins got this right, using a DSL on top of Groovy. This allows for Jenkins build script libraries[1]. I have used them in the past, and they were a joy. They allowed for standardizing builds across many different small git repos, each with their own terraform or helm chart. I miss this feature in GHA.

1: https://www.jenkins.io/doc/book/pipeline/shared-libraries/

This is a topic of debate in my current job. We use Jenkins Multibranch Pipelines and don't want to discard all that work, but we also don't want developers doing release management based on the Jenkins UI/UX. It seems logical to use GHA with local runners to call Jenkins via bash scripts running curl commands, except for how "Rube Goldberg" that sounds. But it seems like everything is like that these days, and all you get to do is pick your poison.

There's also the problem of Jenkins in general, that it's a mature product and you never know if the one plugin you depend on is going to stop being maintained. Or you know you have CVE's but can't upgrade without running two environments, one production and one with all the plugins updated. Or you're chasing down an issue only to find bugs that have been open issues for several years.

https://issues.jenkins.io/browse/JENKINS-52362 https://issues.jenkins.io/browse/JENKINS-52966

It honestly sounds like you've simply outgrown Jenkins in terms of scale. Jenkins is an excellent product but it's not very good at dealing with more than a pizza sized team's worth of CI pipelines. Once the number of teams grows past four or five, you either split them off into their own Jenkins server (which I unfortunately recommend) or you face the pain that follows.

It's scary how closed-source our development platforms are.

That's why building a fully-self-contained VCS, one that includes CI, is on my roadmap, but I fear the open part of our industry will fail before I finish.

Github is pretty much the only major one that's closed afaik. Gitlab, Gitea/Forgejo, and SourceHut are all great existing alternatives that are open source. It would probably be better to volunteer your time to one of those than to build another one.

Isn't GitLab partially closed?

Didn't Gitea suffer a hostile takeover recently?

SourceHut seems to be alright, but the founder hates me.

* Gitlab EE (enterprise edition) is closed, but Gitlab CE (community edition) is open source (https://gitlab.com/gitlab-org/gitlab-foss/)

* I didn't follow the Gitea drama too closely, but my understanding is that Forgejo was a fork born out of that situation

* I've heard the SourceHut guy is a controversial figure, so avoiding it because of that isn't unreasonable. I will just say that "spite forks" tend not to last very long

EDIT: actually, ignore my pointless negativity. I realize now that there really aren't that many open source code forges out there, so if you're willing to make one, then go for it!

Thank you for your encouragement. :)

You're very right about "spite forks"; I'm explicitly avoiding a fork of SourceHut because of that.

I'm also going to be building the VCS from the ground up to solve some problems with Git; otherwise, why would anyone switch to my CI?

Maybe validate the idea by prototyping whatever features you want atop Fossil first?

Might want to see if you can catch rkeene on libera#tcl for a few minutes at some point - IIRC he has his own Fossil+CI type setup with custom BuildKit integration to produce binary artifacts and enjoys talking about it if he has time.

Also the tcl/fossil/etc. crew tend to not care about forks/"NIH"/custom versions/etc. because the whole hardware background thing means those are just something you gotta do sometimes.

(this is -not- me trying to sell you on using fossil in your final approach or anything, mind, just musing on what might make your experimental phase more tractable and mentioning a community I like who might be amenable to chatting about it :)

Your comment brings up good points.

I've thought about prototyping with Fossil; not sure if I'll do it because I'm also building a sandboxing build system that should slot into CI easily. We'll see.

I haven't dug into CI design much, so I could be wrong, but CI seems pretty easy to do on top of a sandboxing build system and a VCS with hooks. One thing I know for sure is that I'm not going to use YAML. :)

But I do take inspiration from the Fossil guys; for example, my VCS will have a bug tracker, wiki, and other project management stuff built in. It will also allow multiple checkouts and be one self-contained binary. It will also use SQLite for storing information about commits and other small things.

It will, however, depart from Fossil's format to handle large files. And binary files too.

So rest assured that I'm in touch with the Fossil guys. :)

Also worth noting even the enterprise edition of GitLab is source-available, it's just licensed differently and is found in the 'ee' folder of the GitLab source.

Other people must have very different setups to warrant this kind of additional overhead and tooling.

In our repo local testing is first. You check out the code and can run the tests. The github runners do the same steps the readme encourages humans to do when running tests locally.

Also I'm not sure what duplication the author is avoiding with their DSL as Github actions can be broken into components and reused and parameterized. Of course if you don't like YAML you're never going to be happy until you've wrapped it in your own layer of tech debt.

The times I've seen more complex testing set-ups that use actions is normally some combination of: 1. Testing involves some kind of complex environment 2. IT infrastructure in some corporations makes setting up that environment impossible.

I worked once in a role where I had to write a library aimed for working with spark, but getting spark (i.e. java etc) installed on my machine involved weeks worth of requests and escalations to IT.

A lot of libraries (I'm thinking of adlfs as a good example which interacts with azure data lake file systrmd) have to have relatively complex testing involving docker by nature of what they are.

None if those points make "put in a PR to run tests" a good or justifiable workflow. But I understand how people wind up falling into that trap.

This is my approach too.

I don't use any of all-in-one actions.

I write the things that I need to have locally, in bash, or a build.rs script or via package.json.

Then I invoke those things in GitHub actions. I split them up so they can run concurrently.

> the runner is closed source

My team runs our GHA runners on our own infrastructure using Actions Runner Controller (ARC).

> The fact that you have to push a commit to the repo to test a change, and then wait for a runner delays everything indefinitely, and makes iterating really painful.

Sounds like a mistake. Steps in jobs can often be run locally, and you could make choices in your jobs to make that more or less true. While there are certainly some things that you can only realistically test by triggering a CI job, many things can be tested locally.

> writing pipelines in YAML is just painful... I invested 3 days to write a small DSL that compiles to GHA

Yes, that can definitely be the case. But the thing about YAML is that it can easily be made a target for DSLs, as you demonstrated. I'd rather GHA use something common like YAML than force another DSL on to the world. It is better for users to decide if/what abstractions to use in front of YAML.

In addition to the suggestions others have made for locally testing workflows, there are also reverse shell actions[0] that can be used for troubleshooting CI failures on the GH runners themselves.

[0] https://github.com/mxschmitt/action-tmate

> The fact that you have to push a commit to the repo to test a change, and then wait for a runner delays everything indefinitely, and makes iterating really painful

This isn't correct. The workflow_dispatch trigger specifically exists to allow for manual runs and you can also re-run existing failed (and successful) jobs.

The over-complication of software deployments is just mind-blowing. I hope the end of cheap money reigns in the insane bloat that is DevOps today. It doesn’t have to be this way, but there continues to be too few rational adults in the room.

It's not a zero sum game. All the enterprise companies hiring 3,000 engineers only need a tiny fraction of their software to run their core businesses, and most of it is legacy.

When a tech recession hits and clears out the clown car, everyone stops paying for software. You might be working on a team of 3 experts instead of 30 morons, but you'll work twice as hard for half the pay, with the same dickhead managers that treat you as replaceable cog, except this time they'll be right.

If I were you I'd stick with the easy money where you get to look like a genius for being able to write basic simple code.

Last year George was a football star. George was injured in a big game. George learned to code YAML+Python scripts. George is now a DevOps rock star.

I’ve been using a Makefile in front end projects for years- use it for all dev, build, test, and deploy scripts. Local users and GH Actions use the same Make targets. Our GH Actions are very lean.

This article sounds like an incoherent rant against CI/CD in general.

Coming from Jenkins, App Veyor, Circle CI, and GitLab CI, GH Actions was such a fresh breath and is a very cohesive solution to the cluttered CI space.

Most of the "points" mentioned in the article are merely code smells of not using a decent deployment/testing setup.

Having centralized build setups is not new either. Ubuntu PPAs and Arch's equivalent of package builds share similar pain points, but it's been serving us for decades at this point.

GHA is "good enough" (GitLab CI was/is much better IMO), and it'll likely never go away because of how dominant GitHub is.

Also, a simpler solution that takes a bit more discipline is writing all your CI tasks as scripts/makefile|justfile targets/etc and using containers to run them in GHA.

IIRC containers are a little slower than running scripts in the VM directly (more than it must necessarily be), and also cross-platform builds might be slightly more complicated...

Gitlab CI desperately needs backwards dependencies. A huge number of my remaining problems with it would be solved if I could subscribe to upstream outputs from my own pipelines.

I'm aware it says Premium but FWIW it does seem that exists since 12.8: https://docs.gitlab.com/ee/ci/pipelines/#trigger-a-pipeline-...

I can also appreciate that the devil's in the details, so all its current constraints may not cover your use case

So this seems to be enabled on the edition I'm using at work, but the limit is bizarrely "two pipelines". Which I already exceed since I have projects using git-dependencies in Python with 3+ repo dependencies.

But it's also a weird project setting not a .gitlab-ci one.

> The fact that you have to push a commit to the repo to test a change, and then wait for a runner delays everything indefinitely, and makes iterating really painful.

Iterating fast is key. As mentioned `act` helps a lot, getting as much as possible to run locally really cuts cycle times.

GitHub Actions feel like bash to me. They are not great languages, but they are available so often that knowing them is a net productivity boost.

I agree and use act as frequently I can, but it has some pretty hard limitations. For example your runners better be using Linux as act doesn’t support Windows or Mac runners.

Cirrus CI had an attempt to make CI-agnostic tool for running Dockerized tasks, see [1]. (I'm not affiliated wtih Cirrus CI)

1. https://cirrus-ci.org/blog/2020/10/07/cirrus-cli-ci-agnostic...

Not sure if related, but recently read about https://github.com/Cicada-Software/cicada which looks like to abstract GithubCI and GitlabCI.

I’d like to see better centralized support generally from CI systems. Six months of steady migration and a custom DSL seems alarmingly common.

CI Wishlist:

* testable outside of CI

* required steps based on repo characteristics (ex: security scans)

* central step and workflow definitions

* controlled rollouts/rollbacks to simplify migrations

We run CI in an image generated by our devcontainer Dockerfile. All the commands in the ‘run:’ steps of GHA can be run locally in the devcontainer. Most commands work outside in random Linux environments, but a few things will break.

This was my first startup idea, and the first one we executed on: an open-source CI that runs on your machine exactly like it does in the cloud and doesn’t use YAML as the main configuration format.

Others have tried the same, including Cicada, which was built by the same folks as Fig.

Here are the failure modes for this idea:

- CI succeeds most of the time. It’s only in the minority of cases that running CI locally is useful, as you don’t need to debug that often.

- The only case where CI fails more often is when building pipelines. If you’re building pipelines then GitHub actions already has bazillions of actions and recipes available. Their “whole product” is a huge advantage (shout-out to readers of Crossing the Chasm).

- The friction to adopt GHA is much smaller as it’s one click away on GitHub and all your other stuff is already there.

- Making an MVP of CI is really difficult because it requires lots of features. You can’t even start if you don’t have things like permissions, outputs for different workflows, and a significant amount of infra for orchestration.

- Even when you have an MVP the switching costs of CI are ridiculously high, and it’s a critical piece of infrastructure, so it’s difficult to get people to move over if you’re only offering marginal gains.

- GitHub is owned by Microsoft, which owns Azure. Their compute will be way cheaper 99% of the time. Your product advantage must be significant enough to justify higher costs. In any case, their margin will be ridiculously higher.

It’s funny how many people get to the same conclusion, yet this is a really difficult problem to solve.

TL;DR: the author idea is great and many people thought about it before. Executing such an idea is extremely difficult though.

> GitHub is owned by Microsoft, which owns Azure. Their compute will be way cheaper 99% of the time.

I hope that regulators in Europe will start to notice and respond to these monopolistic practices of IaaS players who unfairly compete on the managed services market by providing services below cost. Previously, such actions were taken against e.g. Microsoft to ensure that Windows licenses are offered on equal terms to all cloud providers (without preferences for Azure).

> - GitHub is owned by Microsoft, which owns Azure. Their compute will be way cheaper 99% of the time. Your product advantage must be significant enough to justify higher costs.

This definitely isn't true. GitHub Actions are extremely expensive for compute.

> CI succeeds most of the time. It’s only in the minority of cases that running CI locally is useful, as you don’t need to debug that often.

It's also greatly impacts migration time which would be a leading signal for any team considering a transition.

This are the reasons I don't use GitHub actions. I don't want to add another hard dependency into my development environment and testing actions looks like a very slow feedback loop, very time consuming.

Local actions can be defined with a dockerfile. I haven’t used this approach extensively, but it seems like a simple way to use actions while also removing significant dependence on the platform.

This is why you should write your business logic in external scripts and reference them from your CI/CD. So the YAML is just an interface while your pipeline logic is CI/CD agnostic.

Yaml shouldn’t be used for anything except configuration files. It’s just lazy for services to use it as a kind of DSL

These kinds of systems should use an actual programming language or their own DSL

I love GitHub Actions for basic CI/CD use cases, but I agree with the criticisms that testing locally is a major pain point and coding as a YAML DSL is a minor pain point.

I mean to me it's no different than a video game that's "online only". And I know how much I love that.

Why not just use act? It was made for this exact set of problems?

https://github.com/nektos/act

Because it’s incomplete and unstable

I use it when I can, but in my experience it’s only useful about 50% of the time.

This applies to just about any CI/CD program, so I guess they just don't like them in general.

I unironically said GitHub Actions was the worse thing they made when it came to CI/CD. I tried setting up a local runner for Forgejo Actions (compatible with GHA) on my k8s cluster and it was a pain to properly set it up, 1) because Forgejo/Gitea Actions is in beta and k8s support is experimental and 2) because GHA has a very dumb syntax.

The thing others mentioned: you just add an external action and you have absolutely no idea what it's doing and you barely have useful output from it.

I'd rather continue using CI like Woodpecker or any other one because at least you can understand what's going on behind the scenes, but GHA is inherently bad.

EDIT: author makes a point about confusing terminology from GHA:

> I think some disambiguation is in order: The whole thing is advertised as "GitHub Actions", however doing something is defined in a "workflow", and workflows can be packaged, parameterized, and reused in other workflows, and this is then called an "action" again.

I noticed this when I was setting up my local runner. What is an action, is Action the CI program itself or the job or a stage of the pipeline? What is the workflow, is it a job or a stage?

Now I'm more convinced that this is just something stupid made by Microsoft, instead of relying on existing infra they wanted to create something "entirely new" just to force people to use GitHub. Traditional big corpo fuck you right there.

Honestly the fact that you can’t run this locally is the main problem here

A big problem with making an abstraction over CI providers is that they are, on the surface, very little more than wrappers around steps made up of small scripts, but that abstraction leaks like hell. So I think you would wind up needing to go quite high-level to make something that is useful here, but even if you do do this, the variance in runtime environment will make it very difficult to make sure things are relatively portable, and attempts to work around this will lead to problems (e.g. you could just wrap everything in Docker, but fetching large images on each CI invocation isn't really a good use of resources.)

Let's say you have a project using CMake with vcpkg. Vcpkg has support for the GitHub Actions Cache API, so it can integrate very well into GHA and provide good performance for caching dependencies so they do not need to be built on each invocation. Doesn't seem too unreasonable, you could have a vcpkg+CMake plugin of some kind that tries to do the best thing for each CI provider. But, what version of vcpkg do you use? what version of CMake do you use? Vcpkg is essentially just a rolling-release Git repo, there's no specific version to use. But it would save a lot of time to make use of the vcpkg installation that already exists on GitHub Actions images. And obviously, a tool like CMake depends super heavily on the environment around it. A lot of people use the Ninja provider for better performance... but if you use it on Windows, you need to be careful: CMake with the Ninja provider on Windows can not automatically pick up the Visual Studio installation, unlike the default Windows provider (which is, well, Visual Studio.) So instead it might pick up MinGW, which is often unexpected, and does not always work correctly. Does Microsoft have an answer for how to deal with this? Not really. You might think to use the official setup-msbuild step, which has been mistakenly recommended by Microsoft, but it only sets up enough environment to do .NET development.

So actually your CI abstraction needs plugins that deal with:

- Every CI provider

- Using every operating system

- For each build step

And, also, it needs to be updated fairly frequently, especially if you want to keep up to date on e.g. macOS versions.

Personally, I think this is an unwinnable situation, though I'd love to be proven wrong. I think it would take something fairly dramatic to actually solve this problem most of the way, though. Something that might need a lot of collaboration between stakeholders, possibly the CI providers themselves, and probably even build/packaging tools at some level.

One thing I thought was interesting was the approach of using Nix[1]. I've tried rudimentary implementations of Nix with GitHub Actions though, and it does definitely have some issues still. The most obvious issue is that it doesn't really support Windows, followed by the fact that binaries made using Nix are generally not going to be very easy to distribute (unless you're building an OCI or AppImage or something like that.) But for just running tests, it's enticing at least, and it can do a whole lot of caching in theory (for dependencies; you'll still need to handle caching of intermediate files for local tests/builds in your build system underneath Nix.)

The more I think about this problem, the more I wonder which angle is the best one to come at it from. Is this ultimately a problem about software SDKs in disguise?

[1]: https://determinate.systems/posts/nix-github-actions

What? You hopefully do run your tests locally. The premise that you have to run them via the GHA configuration is only an annoyance. Same as I've never seen people running the Jenkins pipelines locally. In the end it's up to you to develop your tests "local first".

After two forced CICD migrations, (shoulda been the first), I learned a pattern I think must be very common: decouple absolutely everything you can from your CICD. Small Jenkinsfiles, no groovy libraries, or Bamboo specs; Keep all of it in portable shell script or some other easily movable environment, and call it from the tiniest pinning you can in your Jenkinsfile.

Then have your team use the scripts locally, and make sure that Jenkins is basically just a person on your dev team calling it the same exact way. Jenkins will end up testing your build/test/deploy solution for free and you can bring bugs it catches back to your team so that they are wasting less time on those bugs and more on product work.

This is very wise and I recommend this to everyone, but is not without problems; notably if you want to parallelize the build in a stateless build system with ephemeral runners (like GHA) you have to be very careful about caching your intermediate artifacts. This can be a lot of complexity which doesn’t exist on a single build box.

patiently waiting for the "github actively considered useless" post on medium.com at some point.

actions are a howling trainwreck. Copilot is a dancing bear of potential litigation and code exfiltration. neither are immediately very dangerous though as Github.com itself, and its ecosystems, have had more than sixty outages this year alone. if you can get over those, you'd better hope members of your team dont live in --or have been actively sanctioned by-- the United States because Microsoft will just delete your work and lock you out.

Look...when you drop 7.5 billion dollars on a code ecosystem in order to capture the worlds developers, its incumbent upon you not to squander the braintrust youve just aligned yourself with. Microsoft feels like its done nearly everything it could to erase whatever goodwill github had as a standalone company in an effort to do the same pathetic whitewash its done with every other acquisition. Shoehorn its own garbage languages into the product, migrate its services to Azures low-performance dumpster fire, and lock in whatever users ya got in order to staunch the potential bleed to other much more meaningful programs like gitea or github.

I give this github.com another 7 years before Microsoft quietly packs it up, sells it to a capital management firm, and sweeps it under the rug.

I wouldn’t deny that Microsoft brings a lot of friction and problems to things within GitHub, but you also shouldn’t underestimate that GitHub itself is doing a lot of this on their own merit.

Regardless of the parent company, the GitHub org is huge, with a lot of people and execs involved. Just wait until the next year or two of reimplementing all these existing features in React- that’s really going to be fun!

Reply on news.ycombinator.com