Act: Run your GitHub Actions locally
github.com
github.com
But just one word of warning: when you run `act` with no arguments, what does it do? Displays usage? Nope -- it runs all workflows defined in the repo, all at once in parallel!
This seems like a crazy default to me. I've never wanted to do anything remotely like that, so it just seems both dangerous and inconvenient all at once.
Nice otherwise though...
Use --help for help info.
shutdown /?
We are moving back to a makefile based approach that is called by the GitHub workflows. We can handle different levels of parallelism: the make kind or the indexed by worker number when running in actions. That way we can test things locally, we can still have 32 workers on GitHub to run the full suite fast enough.
I also like that we are less bound to GitHub now because it has been notorious unreliable for us this past year and we may move more easily to something else.
You drop it in your workflow and get an SSH shell into the worker, figure things out iteratively, then push when it's working.
So far I’ve not found any limitations or issues using Ubuntu runners on my OSX dev machine. A couple examples from my workflows: - building docker images - provisioning VMs with the Digital Ocean cli / http api - hardening VMs with ansible - creating/managing k3s clusters with k3sup - deploying apps with helm
I like your suggested approach of using tmate to access the worker mid-way through a run. This should make it faster to develop/debug the steps that make up the workflow. Though this doesn’t address the cycle time of push-to-GitHub/queue-workflow/watch-for-result.
I’m actually going to try combining the two techniques - use tmate to develop inside a local act runner.
if: ${{ !env.ACT }}
That said, despite its limitations, I've been using both act and tmate in combination for a couple of years. Gets the job done.
(1) Who am I kidding, of course I am a cynic.
they are all at least semi proprietary.
Concourse, Circle CI and gitlab do that.
And running your jobs in Docker is what I recommend that people at my work do, and I admin several GitHub Enterprise Server instances and GitHub Enterprise Cloud as well.
This is the one thing that Drone got very right - every Drone job runs in a container. It is built in to the drone tooling to be able to run those jobs locally on your development machine as well, and requiring containers is why that works.
If you run your CI and/or CD steps from within a container, you can run that container anywhere, and writing a small script to read your CI/CD yaml (or whatever you use), and wrap your favorite container command line tool into a working local CI/CD system should be pretty trivial.
Using containers also makes moving to another CI/CD system which can use containers as trivial as it can currently be.
[1]: https://dagger.io/
[2]: https://earthly.dev/
At home, for some hobby projects, I've been using earthly. It's just amazing. I can fully run the jobs locally and they are _blazing_ fast due to the buildkit caching. The CI now only just executes the earthly stuff and is super trivial (very little vendor lock in, I personally use woodpecker-ci, but it would only take 5 minutes to convert to use GH actions).
I am not a fan of the syntax. But it's so familiar from Dockerfiles and so easy to get started I can't really complain about it. Easy to make changes, even after months not touching it. Unless I update dependencies or somehow invalidate most of the cache a normal pipeline takes <10s to run (compile, test, create and push image to a registry).
This workflow is such a game-changer. It also allows, fairly easy, to do very complicated flows [1].
I've tried to get started with dagger but I don't use the currently supported SDK's and the cue-lang setup was overwhelming. I think I like the idea of a more sane syntax from dagger, but Earthly's approachability [2] just rings true.
[1]: https://github.com/phoenixframework/phoenix/blob/master/Eart...
[2]: https://earthly.dev/blog/platform-values/#approachability
Do you really need that GH Action for pulling Docker images / installing $language_compiler / creating cloud resources ? A `docker` / `curl` / `sudo apt-get install` invocation in a Makefile / script needs to be written once and is the same in CI as on your dev machines. The Action is GH-specific, requires you to look it up, requires you to learn the DSL for invoking it, requires you to keep it up-to-date, requires you to worry about it getting hacked and stealing your secrets, ...
A Makefile already supports dependencies between targets. A shell script is a simple DSL for executing processes, piping the output of one to another, and detecting and acting on failure of subcommands. How much YAML spaghetti do you need to write to do those same things in a workflow file?
Now my makefiles in addition to the usual "make" and "make test" also support "make prerequisites" to prepare a build environment by installing everything necessary and "make ci" to run everything that CI should check. With actual implementation being scripts placed under "scripts/ci".
The scripts do provide some goodies when they are run by GitHub Actions – like folding the build logs or installing dependencies differently – but these targets also work on the developer machine.
If it’s unbearable due to circumstances out of your control, there’s nothing wrong with adding some actions/cache steps to .github/workflows – this goes around the build: fetch previous cache before, update the cache after if needed.
The build is still reproducible outside of GitHub Actions, but a pinch of magic salt makes it go faster sometimes without being an essential part of the build pipeline married to GitHub.
If you need to install a whole host of mostly static dependencies, GitHub Actions support running steps in arbitrary Docker container. Prepare an image beforehand, it can be cached too, now you have a predictable environment. (The only downside is that it doesn’t work on macOS and Windows.)
Actually I use a similar workflow for some of the other projects that I have to work with, keeping everything CI agnostic. I incrementally add various types of dependencies to the container images where the application will be built.
For example:
1. common base image (e.g. Debian, Ubuntu, or something like Alpine, or maybe RPM based ones)
2. previous + common tools (optional, if you want to include curl or other tools in the container for debugging or testing stuff quickly)
3. previous + project runtime (depending on tech stack, for example OpenJDK for Java projects)
4. previous + development tools (depending on the tech stack, typically for pulling in dependencies, like Maven, or npm, or whatever)
5. previous + project dependencies (if the project is large and the dependencies change rarely, you can install them once here and the changing 5% or so later)
6. previous + project build (including things like running tests, typically multi-stage with the build and tests, and built app handled separately)
Though the benefits of this aren't quite as staggering, if you use a self-hosted package repository like Sonatype Nexus, which can cache any dependencies that you've used previously and make everything faster on the network I/O side. This only doesn't hold true when actually installing the packages takes up the majority of the time (e.g. compiling native code), in which case the above is still very useful.
So, an example of how the stages might look, is as follows:
Builder: Ubuntu + tools (optional) + OpenJDK + Maven + project dependencies + project build (and run tests)
Runner: Ubuntu + tools (optional) + OpenJDK + built project from last image (using COPY with --from, typically .jar file or app directory)
For my own personal stuff I also use a slightly simplified version of this, about which I wrote on my blog here, the drawbacks included: https://blog.kronis.dev/articles/using-ubuntu-as-the-base-fo...
Heck make can make all this much better by just prepending each output line with some colored prefix and timestamp. But make hasn't changed in 30 years and likely won't change.
People are proud that it "solves" things since 1976. Yes if your requirements never changed since 1976. I'm not holding by breath that it will deliver basic usability-enhancing features that one can reasonably expect nowadays.
I also suggest Bazel as a consideration alongside Make. With Bazel, you get two advantages over Make:
1. It is easier to ensure that what GitHub Actions runs and builds is the same as what you have locally, since Bazel can fetch toolchains as part of the build process
2. Using a Bazel remote cache, you do not have to repeat work if the build fails halfway and you need to make some changes before running it again.
I think the method you describe is still absolutely how it should be, but this types of interactions are why there’s gravity in the other direction.
Apart from triggers and environment set up none of those things have to be unique.
I often push complex CI logic in YAML into code where it is more easily debugged and I dont have to scratch my head to figure out how to use conditionals. Sending slack messages should always be in code IMHO.
A working example of this is at: https://github.com/nickjj/docker-flask-example/blob/912388f3...
Those ./run ci:XXX commands are in: https://github.com/nickjj/docker-flask-example/blob/912388f3...
I like it because if CI ever happens to be down I can still run that shell script locally.
Yes, because proper use of the tool cache (and other caches) significantly speed up GitHub Actions builds.
Look, I'm all for putting logic in Makefiles instead of in YAML. But not at the expense of slower (and therefore more expensive) builds!
If you mean "I want to run one build with the foo feature and one build with the bar feature. Actions lets me run those in parallel", then that is the "strategy" part of the workflow, not the "tasks" part. My comment was about the latter. ("and make your GH Actions run `make` in a script task.")
If you mean "I want to run two steps of a job in parallel and then run the rest of the job after they're complete", then shell is a much simpler DSL for that. Running things in parallel is literally a single `&` character.
>restart failed portions without retrying the whole CI, run certain sections with conditions like which branch you are on (different tasks when commit to master vs feature branch)
So can a shell script.
Yes now try waiting for all parallel tasks, and error out if at least one errors. And try separating their output so that they don't get interleaved into a big mess. And try by default hiding the outputs of commands that succeeded, only showing those that failed, except when the user explicitly asks for it.
Your "simple" shell script now suddenly isn't so simple anymore.
all: variant-1 variant-2
@echo done with all
variant-%:
@echo starting $@
@sleep 1
@echo done $@
brew install make
gmake -j2 --output-sync=target
`wait`
>and error out if at least one errors.
`wait` propagates the exit status of the waited task. `set -e` triggers the script to exit on any command failure.
>And try separating their output so that they don't get interleaved into a big mess.
`| while read -r line; do echo "task X: $line"; done`
Learn the tools instead of being so confident that it can't be done or that it's complicated.
(These can’t project their tasks over multiple GH Actions runners, eg for multiple OSes. For that you will need to use the YAML. Good compilers will already do work in parallel and max out however many cores they are given, so multi machine is the main use case. Unfortunate.)
The CI runners aren’t multi core VMs I believe so you can’t just use standard shell utilities, you have to indicate to the CI system you want to run multiple tasks.
* GNU make is sometimes unavailable
* syntax is an acquired taste
* not everything fits in a rule body
Imho, these are far outweighed by the flexibility, portability, and "least surprise" convention embodied in a Makefile.
I really don't understand why github don't release a local runner for github actions. Everyone I know which has worked with cicd wants the same, some way to run and debug the pipelines/workflows locally.
There's a way to download a much larger "base image" which in theory would make it work, but if I remember rightly it was something like 60gb of containers which I was never patient enough to download. It was always quicker to just push up to github and iterate that way unfortunately.
run the container on your dev machine or your dreaded paid service. same every time no matter what.
FWIW, while the above sort of recommends kubernetes-everywhere, I'm happy to make a bet on a service like AWS Fargate because I _don't_ think I need to iterate on container orchestration much (as an application developer). Something like DynamoDB, by contrast, seems quite treacherous to build atop, given how closely an application's code is likely to be tied to its primary database.
The first time I used GH Actions I thought there was a strong vendor lock-in element to it which I wasn't hugely comfortable with. I'm a big fan of using GitLab CI these days which seems to be a good trade off between various considerations.
This does make me wonder if you could create a sort of "local first CI" where CI is just an extension of local checks. Therefore the CI is just a check that tests that pass locally also pass on a clean machine. Obviously we don't want to run CI locally if it'd take an hour, but on the flip side, if CI takes an hour on a (typically) beefy local dev machine, it'll probably take 2 hours on a remote machine.
It’s designed to work “locally” pre-commit, giving you super fast tests, but then when you push you can also use it in your CI/CD pipeline.
We achieve the speed with massive concurrency and pre-built environments, it was built as a drop in replacement for your local test runner.
Will you evaluate this tool to see if it affords a nicer workflow for you?
I did not like configuring the GH Actions YAML files at all, but in the end it works quite nicely. The ability to do MSVC Windows (x86_64-pc-windows-msvc) and MacOS builds (for "free") is kinda nice.
So in many cases, I use a makefile with one of the github actions to run a makefile. It has worked okay in the few cases I've tried it.
For example, linting can be run in seconds with an action. If you instead clone and install dependencies before linting, you’re waiting much longer to realize you have more whitespace than you should have
Edit: also, why does it run things as root instead of with a use set up the same way as actions?
Seems like the documentation is a little bit hard to find.
Somebody wants to build a serverless- type ci system?
I think the big problem would be the marketplace, you need to implement your own or make it compatible with the GitHub one
From the top of my head:
1. Parallelization. 2. Capture of build artifacts, which can also be useful for logs of non-linear complex tests such as dependencies of e2e tests. 3. Secrets for release or artifact uploads to third party repos (e.g. docker repos) 4. Caching.
There may be more. Once you sprinkle these things left and right in your CI config, it becomes hard to move to another, even if the bulk of the actual tests you run are just "make test"
- write the main job in a Python/PowerShell/Bash script that runs locally,
- write a workflow that sets up the environment (AWS login), installs dependencies (using GitHub's facilities for that) and calls the script (often passing values using GitHub Secrets),
- push the workflow with a "push" trigger on a feature branch (for fast testing),
- push 30 fixup commits to fix all the small syntax errors, logic mistakes, and GHA idiosyncrasies I hadn't thought of,
- remove the push trigger,
- do an interactive rebase to squash all the fixups,
- force-push and merge.
It works pretty well and can be easily used locally or in other CI/CD systems.
> - write the main job in a Python/PowerShell/Bash script that runs locally,
> - write a workflow that sets up the environment (AWS login), installs dependencies (using GitHub's facilities for that) and calls the script (often passing values using GitHub Secrets),
> - push the workflow with a "push" trigger on a feature branch (for fast testing),
> - push 30 fixup commits to fix all the small syntax errors, logic mistakes, and GHA idiosyncrasies I hadn't thought of,
> - remove the push trigger,
> - do an interactive rebase to squash all the fixups,
> - force-push and merge.
> It works pretty well and can be easily used locally or in other CI/CD systems.
It saddens me that such madness is considered as "works pretty well"
I don't have a lot of experience with GitHub Actions, but is there really no better way to do this?
You could write the workflow correctly the first time. Good luck. It's not one language, you're writing PowerShell scripts inside a bunch of YAML files that reference each other with relative paths and need to call installers that were never written for unattended installs. The only way I can make it work is through trial and error.
GHA really feels like it's trying to cobble up a CI/CD pipeline from a million different pieces that were designed for entirely different purposes. It works, if you spend enough weeks on it, but it will never be pleasant to configure. I can understand why they didn't even try to make it reproducible on the users' workstations, it can break if basically any part of your system's configuration, filesystem, or environment deviates from the runners.
That's obviously not entirely GitHub's fault and the service is very practical once it's set up. My advice is to depend on GHA for their useful features (caching, Secrets, parallel jobs) and do everything else in your own scripts or in Docker.
Unironically yes. It's awful, it's ugly, but it works with no extra setup, effort, or cost (other than feeling dirty for a few minutes).
git add […] && EDITOR=true git commit --amend && git push -f […]
But alternatively, if you find yourself doing this a lot and it's really the code of the step that you're debugging (vs. debugging how the workflow interacts with Actions itself) I try to keep my jobs' steps pretty simple:
- run: ci/thing.sh
-C <commit>, --reuse-message=<commit>
Take an existing commit object, and reuse the log message and the authorship information (including the timestamp) when creating the commit.
git commit -C HEAD --amend amend = commit —-amend —-no-edit
> git amend -ei also use other incredibly informative messages like:
- oh oops
- fix for real this time
- will this actually work?
- this probably won't fix it
- a
git commit . -m fixAny use of rules, cache, includes, or ... you know, real life ... gitlab-ci constructs makes `gitlab-runner exec my-job` do absolutely nothing helpful. The circleci binary did as advertised on the tin and has been my favorite "debug ci builds locally" experience
You probably could use server-side git hooks just fine as an alternative if you self host the repo though, but I would assume if you are using github, or another hosting service, their tools are probably best/easiest/most convenient for their own platform.
Any chance for rebasing it to upsteam?
With Gitlab CI you can run pipeline locally if you install gitlab-runner. Sadly, you can run only one job at a time, not the whole pipeline
Now, the asterisk to that claim is that I have had good success running gitlab locally, with local runners, such that I can do whatever crazy thing I want to the CI variables, to the repo, to the branches, and can run the runners in "debug" mode (although usually not necessary since I can "docker exec" into them) but the idea that I can just tell my colleagues to "brew install gitlab-runner && gitlab-runner exec some-job" is for sure false