GitHub Actions could be so much better
blog.yossarian.net
blog.yossarian.net
1) Program with github actions. Google "how can I send an email with github actions?" and then plug in some marketplace tool to do it. Your workflows grow to 500-1000 lines and start having all sorts of nonsense like conditionals and the YAML becomes disgusting and hard to understand. Github actions becomes a nightmare and you've invited vendor lock in.
2) Configure with github actions. Always ask yourself "can I push this YAML complexity into a script?" and do it if you can. Send an email? Yes, that can go in a script. Your workflow ends up being about 50-60 lines as a result and very rarely needs to be changed once you've set up. Github actions is suddenly fine and you rarely have to do that stupid push-debug-commit loop because you can debug the script locally.
Every time I join a new team I tell them that 1 is the way to madness and 2 is the sensible approach and they always tepidly agree with me and yet about half of the time they still do 1.
The thing is, the lack of debugging tools provided by Microsoft is also really not much of a problem if you do 2, vendor lock in is lower if you do 2, debugging is easier if you do 2 but still nobody does 2.
At the same time, I've found that it often isn't sufficient to push everything into a proper programming language: I do sometimes (even frequently) need to use vendor-specific functionality in GHA, mark dependencies between jobs, invoke REST APIs that are already well abstracted as actions, etc. Re-implementing those things in a programming language of my choice is possible, but doesn't break the vendor dependency and is (IME) still brittle.
Essentially: the vendor lock-in value proposition for GHA is very, very strong. Convincing people that they should take option (2) means making a stronger value proposition, which is pretty hard!
Another thing I noticed is that if you do 1 sophisticated features like build caching and parallelization often becomes completely impractical whereas if you default to 2 you can probably do it with only a moderate amount of commit-push-debug.
A CI "special" language is almost by definition something that can't be run locally, which is really inconvenient for debugging.
It's not just true for GHA, but for any build server really: The build server should be a script runner that adds history, artifact management, and permissions/auditing, but should delegate the actual build process to the repository it's building.
Ideally you want your scripting to handle of the weird gotchas of different versions of host OSes, etc. Granted my work is cross-platform so it is compounded.
So far I’ve found relying on extensive custom tooling has allowed me to handle transitions from local, to Travis, to AppVeyor, to CircleCI and now also GitHub Actions.
You really want your CI config to specify the host platform and possibly set some env vars. Then it should invoke a single CI wrapper script. Ideally this can also be run locally.
FWIW, there's nektos/act[^1], which aims to duplicate GHA behavior locally, but I haven't tried it yet.
Can't you just use the Github API for that? The script would be triggered by the YAML, but all logic is inside the script.
But `act` is cool, I've used it for local debugging. Thing is its output is impossibly verbose, and they don't aim to support everything an action does (which is fine if you stick to (2)).
The main thing that you can't decouple from GHA is pushing and pulling intermediate artifacts, which for some build pipelines is going to be a pretty big chunk of the logic.
> can I push this YAML complexity into a script?
- what language is the script written in?
- will developers use the same language for all those scripts?
- does it need dependencies?
- where are we going to host scripts used by multiple github actions?
- if we ended up putting those scripts in repositories, how do we update the actions once we release new version of the scripts?
- how do you track those versions?
- how much does it cost to write a separate script and maintain it versus locking us in with an external github action?
These are just the first questions that pop in my mind, but there is more. And some answers may not be that difficult, yet is still something to think about.
And I agree with the core idea (move logic outside pipeline configuration), but I can understand the tepid reaction you may get. Is not free and you compromise on some things
#!/usr/bin/env bash
set -ex
aws send-email ...I think the point about maintenance cost is valid, but the thesis of the comment that you are responding to is that the prebuilt actions are a complexity trap.
I think of it like this:
I write scripts (as portable as possible) to be able to build/test/sign/deploy/etc They should work locally always.
GitHub is for automating me setting up the environments where I can run those scripts and then actually running them.
I always opt for option 2 where possible though.
May have misread this but you know you can push to one branch and then run the action against it? Would reduce PRs if you're doing that to then check the action in master. You have to add a workflow_dispatch to the action: https://docs.github.com/en/actions/using-workflows/manually-...
git commit --amend --no-edit && git push -fThe devil's in the details, etc, etc, but I think it's a _much_ more sane default, even for single-user setups/branches because accidents can happen and git DGAF
I recommend making these scripts cross-platform for flexibility. Use matrix: and env: to handle it. Go for Perl, JavaScript, or Python over OS shells and put file tasks in scripts to dodge path issues.
I've tried boxing these scripts into steps, but unless they're super generic for everyone, it doesn't seem worth it.
https://lobste.rs/s/veoan6/github_actions_could_be_so_much_b...
Also, seamlessly interleaving shell and declarative JSON-like data -- without YAML -- is a main point of http://www.oilshell.org, and Hay
Hay Ain't YAML - https://www.oilshell.org/release/0.18.0/doc/hay.html
I won’t go into the details on why it’s this way (build chain madness). It’s stupid and necessary.
Generally, I would use the same tools (e.g. ./gradlew build or docker build) to build stuff locally as on CI, and config params are typically enough to distinguish what needs to be different.
My CI scripts still tend up to be more complicated than I'd like to (due to things like caching, artifacts, code insights, triggers, etc.), but the main build logic at least is extracted.
Maybe this means I should encapsulate this into an action, and check it in somewhere else. But I don't really feel like that; an action is a lot of overhead for a 20-line bash script. Not to mention that erases the lack of lock-in that the script alone gives me.
I guess I could check the script into a separate utility repo, and pull it into my other repos via git submodules? That's probably the least-bad solution. I'd still have to update the submodule refs when I make changes, but that's better than copy-pasting the scripts everywhere.
As in, use GitHub Actions as a YAML wrapper around bash/zsh/sh scripts?
Python is actually preinstalled on Github CI runners.
Is it possible to write Python package that based on YAML specification produces Python API? User will code in Python and YAML will be the output.
I was working on YAML syntax for creating UI. I converted it to Python API and Im happy. For exmple, dynamic widgets in YAML were hard, in Python they are strightforward.
I think you explained quite well what I couldn't put my finger on last time: Building every simple workflow out of a pile of 3rd party apps creates a lot of unnecessary complexity.
Since then, I have used GitHub actions for a few projects, but mostly stayed away from re-using and combining actions (except for the obvious use cases of "check out this branch").
Compared to Gitlab CI, GH Actions still feels like a toy unfortuantely.
They don't seem to grasp how bad their setup is, and consequently are willing to understand awful programming conditions. Even punch cards were better as these people had the advantage of working with a real programming language with defined behaviour. "when exactly is this string interpolation step executed? in the anchor or when referenced? (well, it depends)". No it's black box tinkering (you might as well be prompt engineering)
the C in IaC is supposed to stand for code. Well, if you're supposed to code something you need to
- be able to assert correctness before you commit,
- be able to step through the code
(sorry for the rant)
Adding state? Adding conditionals? Adding (more than a couple) procedure calls?
These concepts perform poorly without common programming tools: testing (via compilation or development runtime), static analysis, intellisense, etc etc
Imagine the curve:
X axis is (vaguely) LinesOfYaml (lines of dsl, really) Y axis is tool selection. Positive region of axis is “use a DSL”, lower region is “use a GeneralPurposeProgrammingLanguage”
The line starts at the origin, has a SMALL positive bump, than plummets downwards near vertically.
Gets it right? Tools like ocurrent (contrasted against GH actions) [1], cdk (contrasted against TF yaml) [2]
Gets it wrong? Well, see parent post. This made me so crazy at work (where seemingly everyone has been drinking the yaml dsl koolaide) that i built a local product simulator and yaml generator for their systems because “coding” against the product was so untenable.
[1] https://github.com/ocurrent/ocurrent/blob/master/doc/example... [2] https://docs.aws.amazon.com/cdk/v2/guide/getting_started.htm...
Independent tutorials unfortunately fall in the same bucket as they first look at official documentation to try to follow so-called best practices or just try to get their things working, and I would say also because shell scripts will seem more hacky for many people -unfairly-.
There are many pipelines you can't run locally, because they're production, for example, but there's no reason why we can't capture these workflows to run them locally at less-critical stages of development. Garden offers portable pipelines and then adds caching across your entire web of dependencies. Some of our customers see 80% or higher reductions in run times plus devs get that immediate feedback on what tests are failing or passing without pushing to git first using our Garden Workflows.
We're OSS. [2]
Caching, I can mostly understand as unique there. Though, I think I'm living with whatever the default stuff in actions is. Slow for builds that don't happen often, of course, but not so slow that I care.
I'm still not entirely sure why it's the case, but the connection to the package registry is incredibly slow, so downloading all dependencies takes forever.
That is, I am assuming that a CI build is not on the immediate dev loop, such that the person pushing it doesn't have to wait for the build before they prepare a review on it.
As much as I like make it just doesn't work with the typical cloud stateless by default configs. If it works for you, your project is small enough and try to keep it this way.
That said, I'm still not clear on what difficulties folks are worried about. I'm also not clear I care on the mess of commits getting things working. The initial commits of getting anything working are almost always a mess. Such that worrying about that seems excessive.
There's https://github.com/actions/cache though?
My GitHub actions workflow consist of calls to make lint, make test, make build, etc. Everything is useable in local.
There’s just some specificities when it comes to boot the dependencies (I use a compose file in local and GitHub action services in CI, I have caching in CI, etc.) but all the flows use make.
This is not a technical problem, you’re just doing it wrong if you don’t have a Makefile or equivalent.
Confusion arises when developers don't realise they are using something in their local environment, though. It could be some build output that is gitignored, or some system interpreter like Python (especially needing a particular version of Python).
Luckily with something like Gitlab CI it's easi to run stuff locally in the same container as it will be run in CI.
Very basic things, like having reusable logic, is needlessly complex or poorly documented. Once I figured out how to do it it was fairly easy but GitHub's docs were terrible for this. They made it seem like I had to publish an action to get any reusability or put it in a different repo. Turns out you can create new yaml files with reusable logic but make sure you put them in the root of the workflows folder or they won't work, go figure.
It's just incredibly painful to work on GH Actions but once you have them working they are such a joy. I really wish there was some kind of local runner or way to test your actions before committing and pushing.
And it's not just Github. The others big CI platform are not really better in terms of workflow and integration.
Now I just script everything to the maximum.
One tool is to use draft PRs for this - you can run changes to your action YAML from the draft PR. When you are happy just squash the commits as you see fit on a "real" PR to merge the changes in without the mess.
I've found draft PRs for debugging/developing GH action logic to be pretty reasonable.
You don't want to experiment too much on main because it dirties your commit history with 20 "Fix typo"-esque commits.
Or, if you try to emulate the main branch with a fake main branch (so you can squash it later), you're still going to have some test commits when do the find-replace back to main.
Neither are great.
It gets painful if there are things you've only got on the main repo (e.g. custom runners, credentials, etc.) though.
The problem is GA also does not allow you to commit a new workflow in a branch. It must first exist on your primary branch and then you may tweak it in another.
git push —force-with-lease origin mainBut if you're working alone or in a smaller team, that's perfectly fine.
No, in GitHub Actions, you will attempt to repopulate from cache on every single run. Of course, sometimes the cache isn't found, particularly because there's a 5 GB cache size limit (which cannot be enlarged, not even for payment) which cycles out FIFO. So if you go over the 5 GB cache, you might as well not have one.
I deeply miss Concourse. Too bad Pivotal dismantled the team and it barely gets even maintenance work. Parallel tasks. Custom pipeline triggers. SSH into the CI environment to debug. Run one-off tasks in CI without needing to set up a pipeline. Transfer directory contents from one job to another without needing to create artifacts (which cost money to store if you're not careful about how long they should stick around for).
GitHub Actions is a bastardized toy CI/CD system that only got popular because GitHub make it as simple as uploading a file to .github/workflows in any repository - no additional signup, no in-house ops required, everything you could want is just there. So let's be very clear about what GitHub Actions is good and what it's bad at - it's good at getting people to sign up, but it's not nearly powerful enough to be the "best" system once you start to outgrow the early features.
Looks like I can move on that "build caching mysteriously broken" issue now. Thanks for the heads up!
(a) Consider Dagger's integration with GitHub Actions: https://docs.dagger.io/cookbook#github-actions where you anyway need to run setup-node, npm ci, etc. just to start the Dagger pipeline. So Dagger isn't saving you from the having to deal with GitHub Actions' caching layer and always-blank starting point - it's unavoidable. Well, if I can't avoid it, why should I use Dagger in the first place - why not embrace it?
(b) Consider a usecase where I want to parallelize the computation onto a number of machines dynamically chosen at run-time. Maybe I want to allow a test suite to run on an increasing number of machines without needing to periodically manually increase the number of machines in a configuration file, or maybe I'm using Terraform workspaces where I want to run terraform apply for each workspace on a different VM to let the number of workspaces scale horizontally. This is fundamentally impossible with something like Dagger (also impossible in GitHub Actions) because it would require Dagger to communicate with the infrastructure provider to tell it to scale up compute to handle the parallel jobs, and then scale down once those jobs finish.
This was achievable with Concourse by having Concourse pipelines generate other Concourse pipelines, and running the underlying Concourse workers as an autoscaling Kubernetes statefulset/deployment, combined with other Kubernetes implements like cluster autoscaler.
> you anyway need to run setup-node, npm ci, etc. just to start the Dagger pipeline
You do need to write CI configuration to run a dagger pipeline, but it’s a very small and standardized snippet (“install dagger, the run this command”) and it typically replaces a much larger custom mess of yaml and shell scripts.
The main benefit though is that your pipeline logic is decoupled from CI altogether. You can run the same Dagger pipeline post-push (in CI) but also pre-push. Similar to running a makefile or shell script, except it’s real code.
> * Dagger isn't saving you from the having to deal with GitHub Actions' caching layer and always-blank starting point*
Dagger most definitely does do that :) We use Dagger and Github Actions ourselves, and have completely stopped using GHA’s caching system. Why bother, when Dagger caches everything automatically?
> Well, if I can't avoid it, why should I use Dagger in the first place - why not embrace it?
I think that’s your Stockholm syndrome talking. The terrible experience that is CI - the “push and pray” development loop; the drift between post-push yaml and pre-push scripts; the primitive caching system; the lack of good composition system; the total impossibility of testing your pipelines - that pain is avoidable, and you shouldn’t have to embrace it. You deserve better!
> This is fundamentally impossible with something like Dagger (also impossible in GitHub Actions) because it would require Dagger to communicate with the infrastructure provider to tell it to scale up compute to handle the parallel jobs, and then scale down once those jobs finish.
This is possible with Dagger. It certainly shouldn’t be a core feature of the engine, but the beauty of a programmable system is that you can build infinite capabilities on top of it. You do need a truly programmable system though, which Github Actions is not.
> This was achievable with Concourse by having Concourse pipelines generate other Concourse pipelines
Dagger pipelines can dynamically run new pipelines, at arbitrary depth. In other words nodes in the DAG can add more nodes at runtime.
Give vito some credit, he is an incredibly talented engineer who built a product that you love. Maybe he saw in Dagger the potential to build something that you will love too :) He blogged about his thought process here: https://dev.to/vito/why-i-joined-dagger-43gb
I will concede that Dagger’s clustering capabilities are not great yet. Which is why piggyback on CI infrastructure for that part… for now!
The fear with needing to run `npm ci` (or better, `pnpm install`) before running dagger is on the amount of time required to get this step to run. Sure, in the early days, trying out toy examples, when the only dependencies are from dagger upstream, very little time at all. But what happens when I start pulling more and more dependencies from the Node ecosystem to build the Dagger pipeline? Your documentation includes examples like pulling in `@google-cloud/run` as a dependency: https://docs.dagger.io/620941/github-google-cloud#step-3-cre... and similar for Azure: https://docs.dagger.io/620301/azure-pipelines-container-inst... . The more dependencies brought in - the longer `npm ci` is going to take on GitHub Actions. And it's pretty predictable that, in a complicated pipeline, the list of dependencies is going to get pretty big - at least a dependency per infrastructure provider we use, plus inevitably all the random Node dependencies that work their way into any Node project, like eslint, dotenv, prettier, testing dependencies... I think I have a reasonable fear that `npm ci` just for the Dagger pipeline will hit multiple minutes, and then developers who expect linting and similar short-run jobs to finish within 30 seconds are going to wonder why they're dealing with this overhead.
It's worth noting that one of Concourse's problems was, even with webhooks setup for GitHub to notify Concourse to begin a build, Concourse's design required it to dump the contents of the webhook and query the GitHub API for the same information (whether there were new commits) before starting a pipeline and cloning the repository (see: https://github.com/concourse/concourse/issues/2240 ). And that was for a CI/CD system where, for all YAML's faults, for sure one of its strengths is that it doesn't require running `npm ci`, with all its associated slowness. So please take it on faith that, if even a relatively small source of latency like that was felt in Concourse, for sure the latency from running `npm ci` will be felt, and Dagger's users (DevOps) will be put in an uncomfortable place where they need to defend the choice of Dagger from their users (developers) who go home and build a toy example on AlternateCI which runs what they need much faster.
> I will concede that Dagger’s clustering capabilities are not great yet
Herein my argument. It's not that I'm not convinced that building pipelines in a general-purpose programming language is a better approach compared to YAML, it's that building pipelines is tightly coupled with the infrastructure that runs the pipelines. One aspect of that is scaling up compute to meet the requirements dictated by the pipeline. But another aspect is that `npm ci` should not be run before submitting the pipeline code to Dagger, but after submitting the pipeline code to Dagger. Dagger should be responsible for running `npm ci`, just like Concourse was responsible for doing all the interpolation of the `((var))` syntax (i.e. you didn't need to run some kind of templating before submitting the YAML to Concourse). If Dagger is responsible for running `npm ci` (really, `pnpm install`), then it can maintain its own local pnpm store / pipeline dependency caching, which would be much faster, and overcome any shortcomings in the caching system of GitHub Actions or whatever else is triggering it.
I was going to reply that you misunderstand how Dagger works: that your pipeline logic typically requires very few dependencies, if any, since it can already have dagger build, download and execute any container it needs - with free caching. So your “npm ci”, although technically not free, is a drop in the bucket and easily offset by the 2x or even 5x speedup that is typical when daggerizing an existing pipeline.
All the above is true… But I realize that the documentation you link to contradicts it. Reading these guides I understand your fear. Just know that these guides are, in that respect, giving you the wrong idea. In a more representative example, instead of importing a GCP client library, the script would have dagger run a GCP client in a container, with the appropriate input flags, config files and, if needed, artifacts. We will fix those guides accordingly, thank you for bringing this to my attention.
Also, soon Dagger will do exactly what you were expecting: it will execute the “npm ci” itself, in a container. So whether that script’s overhead is 3 seconds in the typical scenario I described to you, or a few minutes in the scary scenario you described to me: either way it will get cached, and either way it will have no host dependency on npm or any other language tooling: just the dagger CLI.
Great :D
I read a lot more of the documentation this morning, in general I really like what I see, but the clustering bit is the key missing point for me at the moment. My current employer produces an Electron desktop application - we need to run 90% of the tests on Linux (most cost-effective), and 5% each on a Windows and macOS machine. I see the Dagger CLI runs on both macOS and Windows, but the architecture as it stands today expects that the Dagger pipeline will run all its workloads on the same VM. If I want to run pipeline tasks on other VMs, I can do it from Dagger, treating the Dagger Engine as the orchestration layer which makes some kind of an IaaS or PaaS call to schedule the workload on a separate macOS or Windows VM, outside the Dagger architecture. But... today I expect that workload scheduling to happen within the CI/CD architecture (it's trivially expressible within GitHub Actions, particularly as GitHub Actions hosts both Windows and macOS runners), and needing to hoist it outside the CI/CD architecture in order to use Dagger is a needless complication.
SSH debugging and one off tasks absolutely dreamy.
How is this not resolved?
Easily bypassing security controls is a major security issue.
Yes, you need to convince someone to use your SHA, but social engineering is usually the easy part.
- (unrelated) build failures just randomly notifies the latest maintainer who happened to merge something? (Imagine you finding this out when your newly added maintainer pings you on Matrix and tells you 1: about this behavior, and 2: that your update/builds have been failing for a week without you knowing?!?!)
- The cache action is horribly, trivially observably broken with seemingly no maintainer?
- Can't tail the build log of a step if their UI poops or your tab unloaded after it started?
- The complete lack of abstraction that might actually make workflows portable across worker nodes? pfft.
- the default image is a travesty. I thought it was obnoxious how bloated it was and then I started digging in and realizing "Oh, some Microsoftie that didn't know Linux was put in charge of this". (saying this as a former FTE that knows). And there's no effort to allow a more slimmed down image for folks that, you know, use Nix? Or even just Docker?
I'm in the process of migrating off GitHub and it's mostly because Actions shocked me to my senses. Too bad MS can't figure out how to retain virtually any Linux talent, and not just their cuffed-acqui-hires or Windows-devs-cosplaying. Even the well compensated ones head for the door.
And I'll just say, I don't program in YAML because YAML is a disgrace wrought upon us by Go+Yaml enthusiasts that don't know any better fueled by senseless VC money shoveled at an overall ecosystem incognizant of modern, actually useful technology.
edit: removing some of the blatantly identifying former-FTE trauma. Knowing what I know I should sell all my MSFT, but the market thinks differently.
The idea is that you write hooks (or use pre-existing ones) that check (and fix, where possible) your code before you commit, and then once you've pushed, the CI will check again, including automatically committing fixes if necessary.
Anyway, it works brilliantly - unbelievably fast thanks to good automatic caching, and using the exact same setup on your local development machine to in CI negates a lot of these debugging problems. It's free for open source.
It only does checks, not things like automatic releases, but it does those them well.
Here's some boilerplate to do all that:
[tox]
envlist = py{310,311}
[testenv]
passenv =
PIP_CACHE_DIR
deps = coverage # deps only for tox
extras = testing # testing extras include pytest
commands = pytest ...
[testenv:check]
passenv =
PIP_CACHE_DIR
PRE_COMMIT_HOME
skip_install = True
deps = pre-commit
commands = pre-commit run --all-files --show-diff-on-failureParticularly for test builds, I very often just want to quickly view the output HTML report. The current approaches I am familiar with are the aforementioned upload-artifact, or using GH pages, but GH pages is not great when you have multiple different reporting output for the same repo, or you wanna quickly view historical reporting rather than latest.
I'd love a simple "attach-report" action that just put a link to an HTML report on the job summary, and clicking on it renders the html.
Other automated CI/CD style systems have much richer support out of the box for dealing with HTML report capturing and viewing.
Similar too if the reporting captures video/screenshots of a given bug - the original report UI is far easier to deal with. Many test frameworks already natively put out an HTML report UI etc, it's just harder to get to than it should be with GH actions today.
Not sure how GHA treats it, but markdown is a subset of HTML so by default all HTML pages are valid Markdown documents. No conversion required.
There was a project to allow you to run the pipelines locally so you could do the edit-run-debug loop on your own private environment without committing. It was, of course, canned.
https://github.com/microsoft/azure-pipelines-agent/pull/2687...
However, there are tools to improve QOL. For example:
https://marketplace.visualstudio.com/items?itemName=ms-azure...
A vscode extension that's syntax-aware.
Now, I'll be a bit controversial: if they'd used XML instead of YAML, you could have an xmlns declaration up-top that would give you validation in most decent code editors without user intervention. XML is awful, but it has a lot of useful features that we gave up when we threw the baby out with the bathwater.
I was under the impression (which might be wrong!) that GHA was an independent project within GitHub that was well underway before the acquisition. Are you saying that GHA was rebuilt on top of AzP, that it's just a relabeling of AzP, or something else?
(I have no particular dog in it being one way or the other, but I'm curious about the history here.)
> GitHub Workflows execute on runners. The runner code is essentially a fork of the Azure Pipelines code, so it's very similar. It's also cross-platform and you can also use hosted or self-hosted runners.
https://learn.microsoft.com/en-us/dotnet/architecture/devops...
Probably even moreso than most people think - a large portion of the AzDo team got moved over after the acquisition to work on GitHub Actions/Projects.
Call me crazy but xml just hurts my eyes. I'll always take a nicely formatted yaml doc with all the pains that come with it over the horrors of angle brackets and camel case.
It's a pretty big frustration in a language that is so slow to test -- commit push run wait find the error oh I made a typo. Better code-time validation fixes this but the tooling for that in yaml is weak.
I want to be able to run my all of CI workflows on my local machine.
I agree with other commenters that most of what CI does should be abstracted out into scripts or other non-CI tools. Unfortunately it's not easy to do that for large pre-existing CI setups, especially if a different team is the one maintaining your CI workflows.
Unfortunately, it can't lint actions themselves, only workflows that call actions[1]. This is a substantial deficiency, especially for users (like me) who write and maintain a decent number of actions.
As a result, few GitHub workflows benefit from the immutability or reproducibility guarantees that can be provided by containers, and most workflows I interact with spend more than half their time running ridiculous installer scripts.
https://docs.github.com/en/actions/using-jobs/running-jobs-i...
We built Earthly [1] to tackle these two problems specifically. We're open-source (10k stars).
[1]: https://earthly.dev
I’m spending more time finding bugs in the build pipeline than the time I spend doing other things. And it’s all GitHub actions fault. I hate it with a passion.
And what the earth is an "actions" anyway? How on earth is simple bash functions not just as suitable here? Instead you have some weird YAML scripting language. It's so bad. Why. Somebody please tell me. I'm losing my mind. It is a good reflection of the rest of the world though and why the worlds infrastructure is crumbling in many places.
So exporting an environment variable doesn’t work for example.
I'd like to know if my builds are getting slower over time. I'd be able to detect flaky tests automatically.
It seems basic, but I know third-party solutions exist for this. It's out of the box in Circle CI and Buildkite and feels like it should be here.
After the team migrating to actions swept though, we ran without this basic stuff for years because nobody had the time to figure out what 3rd party tool to use, money to pay for it, or capacity to re-implement the functionality. It made dealing with test failures or flake that crept in awful.
So the CI pipeline would only call the build tool, like "./build containers push-to-registry release:1.0.0 run-tests"
Those scripts can be tested and debugged everywhere. Also migrating to a different CI platform would be really easy.
Also, why would a large commercial CI want to have their environment too open?
Those enterprise tools i know (like sonarqube for example) are mostly cli tools in their core. Can be integrated everywhere.
may be an alternative. You can run it in GitHub actions somehow as well.
I was using https://nuke.build for one or two projects. It's in .NET and probably not so useful if you need to build non .net code bases. Stopped using it because some of their code is proprietary, and it's not as useful as I thought.
Seeing people push hundreds of commits to their CI pipeline trying to fix it by trial and error makes me cringe. I do not understand how some people accept such a workflow without thinking it could be better.
When I was planning the CI set up at my job I made sure it could all be run locally and hammered this point home to developers. CI isn't magic, it's just some other computer running your shit.
We use Gitlab CI which runs your script in a container. It's trivial to run the container locally with docker to test exactly what will happen in CI (although you do have to make a fresh clone of your repo for the mount, I thought about scripting this but in practice it was never needed that much).
GitHub Actions seems much harder to test locally. There seems to be some projects to do it but they are too heavy for me to want to install. For this reason alone I don't like Actions.
Some real issues with github actions:
1. MacOS runners have ridiculously inconsistent IO performance causing 200x slowdown in some cases
2. Getting charged to the nearest minute is asinine and punishes highly parallelised (fast) workloads
3. GitHub OIDC endpoints constantly timing out
Unfortunately most of the work of GitHub actions is configuring the runners and setup data, which requires you to use lots of yaml to invoke unvetted public repositories (actions) written in JavaScript to do what is easier done as shell scripts, and then use shell scripts to workaround the limitations of actions.
State is impossible, data is even more impossible, and caching is completely broken. And if you use your own runners, GitHub actions don’t even clean up their own garbage. I suspect it’s the same on public runners, but just statistically less likely to affect you, but a potential security vulnerability.
I’ve been working 2 things:
1. A front end that crafts GHA workflow yamls from a sane configuration with common tasks baked in.
2. An event and reporting system so you can see what is happening during a workflow — not after — without having to scroll through GitHub’s hideous log output in html or download and unzip the full log.
shell: run_workflow name=MyJob in=MyWorkflow.yml params={}
Obviously you could put the whole CI/CD into a bash script and not use any features or functionality provided by github actions but there are plenty of nice things that it does and it would be a shame to not use any of them.
> Obviously you could mock up the entire github actions scaffolding locally
Not sure if anyone's advocating for implementing GA entirely. You can at least automate project-specific bits as much as possible using scripts, and then have the CI environment use the same automation. That allows for more local debugging than overusing GA.
This part of the article half triggered me and half made me laugh. I was recently on a GitHub webinar to listen to the pitch for CoPilot, but of course they also talked about Actions and GHAS. Anyway, during the overview of Actions, the presenters made a comment about how Actions was "backed by the community." It felt to me like a glaring admission that the product isn't mature. But in true Microsoft fashion, they charge you like it is.
So your GHA workflow is: set up secrets (AWS and co.), set up Python, download cache to path X if possible, run the build in X and send reports to Y, publish reports in Y. And put as much work as possible inside the build script. We have a whole Python project for that.
This is truly painless after the first development effort.
I really, really wish there was a way to clone the GitHub Actions or GitLab runner environment, spin up a runner locally and test. That would shave off 95% of the wait time
When I create a new Function app, it gives me the opportunity to enable GitHub integration. The experience for this is flawless, IMO. You provide your GH credentials, select the org/repo/branch, and then it will create the workflow file for you and push it automatically. It will also update the secrets in your repository settings to match what Azure expects on its end for deployment.
By the time you get to look at your GitHub repo, the action is already running and will 100% complete successfully if you followed a standard/default project structure. The automatically-generated workflow files aren't perfect, but they're so close that it becomes trivial to tweak for additional build args or project arrangements. Just getting the secrets & related boilerplate configured makes the difference between me doing it right now vs maybe never. The consequence of always having proper CI/CD from day zero, even for the most trivial projects, seems profound to me.
There exist some really happy paths now wrt GH actions, but you gotta be willing to get pretty hammered on the Microsoft koolaid to explore them.
At least it's deprecated now, but they apparently decided to postpone the removal because too many people are still using it. Maybe if we all do our part in migrating to the new way [1], they will eventually close this giant security hole.
[0] https://github.blog/changelog/2022-10-11-github-actions-depr...
[1] https://docs.github.com/en/actions/using-workflows/workflow-...
1. Github actions embodies the wrong abstractions. I'm not 100% sure what the right ones would be but it feels all wrong.
2. It has the feel of an unfinished/undocumented project. Someone's project to get a promotion perhaps that they handed off to interns after securing said promotion. A product that forces me to expend brain cycles reverse engineering it's behavior just makes me want to design it out of my life. I may as well develop my own thing : at least I'll understand that.
3. A build system that can only be run in a proprietary SaaS environment is a terrible idea. I mean, great idea for locking in customers I suppose.
Wrap your GH Actions in a debugger breakpoint and connect into the live broken GH Action to inspect the machine, re-run commands etc..
At it's heart - realtime debugging for GitHub actions.
The pipelines though, are all configured using ansible. Applying the same principle I did to Jenkins, using it strictly for execution automation, logs, permission and the likes, and leaving the heavy lifting to a more trusty tool, I think I can avoid both major vendor lock-in and idiosyncrasies and nonsenses from the language.
It boggles my mind how unrefined the GitHub actions flow is. It doesn't offer a good experience by a long shot.
We have to add an action to kill duplicate runs on triggers.
The caching is marginal, at least on Windows runners because it takes forever to expand the tar ball. Not even sure from their docs / issues on the cache action if they did finally move it to use GNU tar.
Having some way to get an interactive shell on fail would be a big step up for debugging issues. Otherwise we are back to print debugging in the actions or uploading artifacts at each step so we can inspect them.
My choices here are either to use a different CI solution (but I quite like Actions, and they are better integrated in the Github UI than alternatives), or to use custom self-hosted autoscaling runners (but there doesn't appear to be a good solution available at the moment, and I definitely don't want to managed infra for CI.)
I work at Trunk.io and we have a tool called CI Debugger. It helps you understand your workflows. Instead of getting lost in logfiles and commits, you can pinpoint where things are bogging down or going haywire. It can give you a clearer picture of what's happening under the hood. Like having a heat map for the CI/CD processes.
This is pretty much possible in all other CIs.
For example, I wrote a set of workflows[1] to automatically apply a label after 2 'review approvals'.
First, a 'dummy' workflow triggering off PRs (in that context) that uploads an artifact: the PR number
Second, the 'real' workflow that runs in the context of the actual repository, set to be `on: workflow_run: workflows: - Final Review Labeler` - this pulls in the artifact, runs a GraphQL query, and applies the label if applicable.
[1]: https://github.com/goonstation/goonstation/blob/master/.gith...
Yes, I can wrap it with a simple docker action. It would be cool if that were automatic.
You can fake it by passing a stringified JSON object, but that’s bizarre.
it's apparently a fixed problem in upstream openssl beta that isnt bundled yet, but i think a custom runner would be better too (but not an option at my org)
still, some days it gets in a rut and quickly fails for 10+ runs all billed at full rate. the entire eng group is severely annoyed by it because we cross test all oses on each commit so its becoming an expensive problem. basically 3x the billing on windows ci
I found Nektos act pretty usable for local testing when I was doing a lot of GA work.
I don’t understand why people settle for less.
I'm so sick of having to duplicate pieces of my actions constantly
(So, in a sense, there's no real technology "behind" it. It's just Markdown with a little HTML templating, with Jekyll as the SSG.)
I'll see if I can clean it up into a public repository, but no guarantees :-)
> In this particular case, it took me 4 separate commits (and 4 failed releases) to debug the various small errors I made: not using ${{ ... }}5 where I needed to, forgetting a needs: relationship, &c
It really feels like a throwback to the punch card era.
I too don't enjoy writing CI scripts (despite knowing Linux administration and shell scripting quite well), it always takes an inordinate amount of time, but it also saves much more over the long term.
Could things be better? Of course; that's how software is--and this should resonate with most folks on this site. But just because some product isn't infallible doesn't mean we can't love it too.
Wow, computers doing what they're supposed to do. Pretty impressive ...
The UX on GH actions is crap, other CI/CD solutions give you way more control and tooling. GH really needs to step up their game on this one.
The thing for us is that we’re already paying for GitHub and Actions are included. I know, for a fact, GH is/was working with stakeholders using their software to get feedback on what could be better. So they are trying!
I’m not here being a fanboy by any means; I’m just saying it does what I need it to do and I’m an active user who is pleased with the service provided.
YMMV.
But that's exactly what a lot of commenters complain about.
They canned it.
https://github.com/microsoft/azure-pipelines-agent/pull/2687...