Working with GitHub Actions
jeffrafter.com
jeffrafter.com
One final aspect we started investigating was how we could completely eliminate Jenkins from our stack. We have some deeply-customized jobs that run on that box (for handling bundling of build artifacts, managing metadata for final public releases, etc). What we will likely do is write a post-build agent in .NET Core, include it within the same project repository it will be responsible for working with, and then as part of the solution build it would just get taken care of automatically. Then, we simply need to invoke one more command after our initial dotnet build to trigger the package/deployment process, all from GitHub's machines.
If anyone from the GitHub team is browsing here, I was wondering if there were any plans for caching Action state between builds. That is to say, instead of downloading & installing the .NET Core 2.2 runtime every time an action executes, it could load from some cached instance (if available). This is the only downside I see right now - Jenkins can usually turn our check builds around in about 50% the time of the GitHub Actions, but for our purposes right now we honestly don't mind it. PRs don't need to merge every 60 seconds. Once we get to release builds in GitHub Actions, I think that tune might change a little bit.
If you set a specific version, it can use a cached image and is a lot faster than a complete install.
Also have a look at their other `setup-*` actions for similar installations of Node, Python, Go...
EDIT: Found it! https://twitter.com/natfriedman/status/1164210683979812869 Looks like it is landing on November 13.
But, it took me longer to skim this article than to setup my first action. This is making things way more complicated seeming than it actually is IMO. All you need is 3 small files :)
If anyone is looking for such a get-started-right-now post, I've written one with a sample build-and-upload-to-s3 workflow [0].
It is indeed simple to get started, and I'm personally quite optimistic for Actions in the future!
I've had a a first look through the documentation, and saw a couple of interesting differences:
Actions allow a job to be broken up into a series of (re-usable) steps
Can be triggered on a broader set of events
Actually you can define global environment variables with an extra step... here is how I do it. I was quite frustrated until I worked this out!
- name: Set environment variables
shell: bash
run: |
echo '::set-env name=GOPATH::${{ runner.workspace }}'
echo '::add-path::${{ runner.workspace }}/bin'
echo '::set-env name=GO111MODULE::${{ matrix.modules }}'
echo '::set-env name=GOTAGS::${{ matrix.gotags }}'
echo '::set-env name=BUILD_FLAGS::${{ matrix.build_flags }}'I haven't been able to play with Actions to confirm what I'm about to say, but if they use a fully compliant YAML parser you can use YAML anchors to do "copy-paste" without the copy paste; I think GitLab actually fully recommended such an approach before they supported `extends:` in their pipeline document
I'm on my phone or I would link to the YAML spec and GitLab examples in their docs for the gory details
[0] https://help.github.com/en/articles/events-that-trigger-work...
While they're definitely trying out some novel ideas in the CI world, it was startlingly slow. Using as much parallelism as possible on every platform; CircleCI could do this pipeline in ~5 minutes, BuildKite (what we currently use) hosted on GCP n1-standard-1 instances, could do it in about 6, and Github Actions took closer to 10. I believe the main reason why Circle was faster was that I took some extra time to set up their native cacheing systems for things like npm dependencies and build outputs between steps; we don't have this set up on BuildKite, and that might shave off a few seconds (though, I haven't looked into it, but if we're forced to egress these caches from a BuildKite DC, say in AWS, to our agents in GCP, maybe it wouldn't save any time. The tight integration of Circle is a big advantage here).
My theory is that the big slowdown is the downloading of docker images; not necessarily the beefiness of each build container.
On Circle, we avoided the use of docker and went straight native, except in the final steps where we're actually building an image of course.
On BuildKite, we're actively encouraged to use Docker to isolate builds. Given a single agent will be shared between multiple repositories, this makes total sense. But, the agents are still durable; if a tslint step starts with a pull of the node:carbon image, that image will be cached on each machine that has ever ran the build, practically indefinitely (until the machines are recycled). So the P99 off each pipeline can be higher than that 6 minute mark, but the vast majority of builds will go much quicker.
On GHA, every step needs to be in a docker image (which I like!), but there doesn't appear to be any local image caching in the build containers. If they can add this in some capacity, I'd bet the builds would be much faster. Arguably; they know at any time what images may be needed for a given repository, because the pipeline steps are defined statically in the repository, so they could feasibly keep those cached in a pool of workers that are then grabbed for that repo. Plus, I'd bet over 50% of builds start with the top 50 most popular images on docker hub (alpine, ubuntu, node, python, etc). There's no reason why every agent can't have those pre-downloaded and ready to go, beyond cost.
Excited to see the product improve; I could easily see us switching and paying for it one day.
- You can have persistent workers since the build machines run in your cluster.
- Much better Docker layer cache hit rates. We use custom Docker images and I've never seen a Docker image cache hit on our CircleCI machines.
- Better download throughput. CircleCI appears to be bandwidth limited to about 15MB per second.
- It's much more straight-forward to programmatically generate workflows instead of hard-coding them in multi-thousand line YAML file.
Some of low-hanging optimizations on CircleCI I've used to reduce build time are:
- Using a shallow git checkout. Reduced checkout time of our 500MB repo from 30 seconds to 2 seconds.
- save_cache into /dev/shm. This cut our 1GB node_modules (yea, it's huge I know) save_cache and restore_cache from 30 seconds to 10 seconds. The HUGE downside is /dev/shm is mounted as noexec so you can't run anything in node_modules/.bin without some hacks.
Buildkite has pros and cons, like any CI platform. Managing your own agents is nice in the sense that you get really fine-grained control over their performance and cost. But, that being said, it would take some engineering to create an agent pool which can match the cost of something like Circle.
If you're running an agent 24/7, on, say, an EC2 instance, but you're not actually using it 24/7, there's some serious cost overhead there where you'd see some nice savings on Circle. The great thing about BK is the flexibility; there's infinite potential to do some really smart things, like build an autoscaling agent pool. In fact, they have an AWS CloudFormation stack which does exactly that (we don't run that, but I've looked into it and its really cool: it essentially has a lambda cron which calls the BK API to determine the number of outstanding jobs, dumps that into a CloudWatch metric, then sets up an ASG which can scale-to-zero based on this metric). As far as I can tell, this is a "one-click deploy", so management should be easy.
We run our agents inside Kubernetes on GKE. We've had some minor issues with this. Docker-in-Docker is never fun. So we'll probably look to switching to that stack (and maybe set it up so, instead of the automated ASG, we just have a timeboxed ASG between 9-5 M-F, scale to 1 instance outside of those hours, as we're pretty timezone-centralized). The advantage of GCP/GKE is that we're using preemptable instances; ridiculously easy to set up on GKE and you get that 70% cost savings. The only downside is that, every once in a while, a build just fails because the underlying instance restarts. At our scale, not a huge deal; just re-run it. Maybe happens once a week.
Again, its trade-offs. You can go with more persistent workers, and get amazing docker layer image caching. Or you can go with an auto-scaling route and save a ton of costs. It shouldn't be impossible to find a balance in there which works for your company, and I like that about BuildKite.
I don't think we'd ever switch to Circle, just because I don't see a huge benefit in doing so. We may, one day, switch to GHA. We already use Github, so that integration is nice. If the performance is looking good and the cost is acceptable, then we'd do it; CI is so fungible that it really comes down to minutia like this.
colin@layerci.com
[1] - https://twitter.com/natfriedman/status/1166757981637115904
We're working on artifact caching between runs now, which I think will also help some of the speed issues that you're referring to.
We really appreciate the feedback; if you do get around to trying it again, I'd love to know more about what we could improve.
The new actions (syntax) runs on Azure and has native VMs you can use to run commands, but you can still mix and match docker images and native commands if you'd like.
https://help.github.com/en/articles/virtual-environments-for...