Arm64 on GitHub Actions
github.blog
github.blog
They're just more efficient for certain kinds of workflows - both cheaper and faster.
If you're building for arm64 targets or android, this is a no brainer because it sidesteps the emulation requirements.
We've been offering arm64 runners for a few months now (up to 32 vcpu) with WarpBuild and our users love it.
In general, there are lots of inefficiencies in CI starting from the lack of visibility, debugging ease, performance, runner config customization, persistence, etc. We're out to solve it and make the world's fastest CI cloud ecosystem on top of existing CI providers with WarpBuild.
FlyCI website: https://www.flyci.net/ Discord server: https://discord.gg/JyCjh439da
But at 37% cheaper than x64 it's quite a better deal than what the outdated x64 CPUs.
Looks like every GitHub Actions provider is on this thread. :) I also wanted to chime in with two thoughts.
First, we recently wrote about how we enabled arm64 VMs at Ubicloud. Any feedback on the technical details is more than welcome: https://news.ycombinator.com/item?id=40491377
Second, I feel runs-on's analysis is a bit unfair to us. Ubicloud's x64 / arm64 performance and queue times above are as good as any (we deprecated the AMD EPYC 7502 line). For example, RunsOn arm64 queue times are 31|42 seconds and Ubicloud queue times are 17|24 seconds.
But the analysis says the following, "Be aware that Ubicloud has low CPU speeds and high queue times, which means they won’t necessarily end up being cheaper than competitors." I fail to see this conclusion from the numbers above. What am I missing?
Also note that I specifically mentioned that "all third-parties are good on that front, expected [sic] Ubicloud (but that may change)."
Also agree that now that you removed the outdated CPUs (was still active end of May), the analysis should mention that you have OK CPUs by now. I will fix it.
For providers that are on this page, do not hesitate to write to me if you find the analysis outdated for your specific service.
[edit] page is now up to date with new analysis
Still, you updated the above statement to, "Be aware that Ubicloud has a slightly lower CPU speed and somewhat variable queue times for x64 (but improving)."
Could you clarify what you meant by variable queue times?
According to the benchmarks, Ubicloud queue times for x64 were 18|44 secs at p50|p95 over the past month. I agree that our p95 number is higher. At the same time, RunsOn's AWS numbers are 31|36 secs. So I feel that driving a conclusion based on our p95 variance number for x64 is a bit unfair to Ubicloud.
Thank you for compiling this benchmark btw. Now, we'll follow it closely.
Once this improves I’ll be sure to update it.
As expected. Just Azure's Ampere Altra that's OLD. It's Graviton 2 era. We're at 4 now (sort of). Ampere One is still "somewhere".
GH’s macOS runners particularly were complete trash.
Plus, who doesn't love having some new hardware to play with?
We are interested in doing the same setup.
Our office is a single room with six desks, so space, noise, and heat are a larger consideration for us. I use a mini PC at home for a personal server as well, but that's just because used enterprise equipment is often a great deal.
Check back in three years once you've had to deal with OS upgrades, power outages, running out of disk space, and one of them failing.
Always cheaper if you have at least some demand. For the lazy fix even just get a few mac mini and it'd work fine.
- No OS upgrades (running Ubuntu 22 LTS), but regular update
- We've had a power outage, but not during the day, so when they rebooted, they went right back to action. Same issue with a network outage in the office.
- We have hit disk space issues due to not clearing all the data we needed to. A quick SSH and an rm or two did the job, plus another 30 minutes to an hour to update a cron to remove that portion of the cache
I see what you mean for sure, and there will undoubtedly be more things that come up at some point, but with the savings, it would take quite a few engineering hours per month to lose the savings at this point.
And I won't lie, when something goes wrong, it's a nice change of pace to deal with a server instead of code once and a while. Can't actually use that for calculations, but I don't hate when it happens so far.
This being routine engineering, I can tell you right now how it will go. Let's start!
> OS upgrades
Distinction needed: update/upgrade. Entirely automatable, some may say table stakes. Decide to update in this job or another.
Upgrades are a common problem for either hosting world. Can my software work in this new environment? Hence, CI.
> power outages
Set the firmware to return to last power state. Also buy a couple UPS' and a PDU. Fixed. A host and networks between going away isn't so easy.
> Running out of disk space
How is this unique for them to solve? Hosted workers/cloud servers have finite disks too.
Clean your workspaces in your pipelines. That's all disposing of the hosted one does.
> and one of them failing.
Someone else in the thread said it best, these are CI runners man. Capacity is down slightly. Oh no. Several cheap copies were bought to address this.
The marketing was so successful that we have to debate that yes, there is value in owning things you need/use.
You just obliterate and reinstall the base system in what 10 minutes?
Even if within 3 years there needs to be OS upgrades, you can just e-cycle the old machines and buy brand new minipc's (with added benefit of increased performance) every time you need to upgrade, and you'd still be ahead!!
At smaller scales, as long as your builds aren't stupud-slow, cloud providers should be cheaper just because they can utilize their hardware more (10x?) efficiently.
I guess you had to import the secrets, too...
Also the CEO Surya is such a great guy, we had a small networking issue that lasted for about two hours and they gave us credits for all the runners affected with that, and their customer support is blazingly fast.
For those hitting that same kind of issue (and you are many), you should check out my product https://runs-on.com. 1-click install, 1-click upgrades, and from 7x to 16x cheaper runners on AWS.
I'm still using GH Actions, just not their minutes.
We don't (yet) have support for pre-pulled repos on runners per customer. Are the large repo sizes despite fetching only the refs required for the specific context [1][2] which can help speed things up significantly.
[1] https://www.atlassian.com/git/tutorials/big-repositories [2] https://stackoverflow.com/questions/48072080/how-to-manage-l...
Thanks for the WarpBuild love!
However, my last anecdata was from a few weeks ago. You can try us out and take a call for yourself.
Our focus is on CI runners and efficiency and I think that gives us an edge on the devex. Ubicloud is definitely cheaper though.
> Larger runners are only available for organizations and enterprises using the GitHub Team [$4 per user/month] or GitHub Enterprise Cloud [$21 per user/month] plans.
> We expect to begin offering Arm runners for open source projects by the end of the year
side note: I find it mildly annoying that it's no longer "ARM" but "Arm"
https://github.blog/changelog/2024-01-30-github-actions-maco...
I've been using github-act-runner on top of a local Nomad cluster to run some of my builds on risc-v, arm64 & x86-64, it's fine, but a bit of a hassle to set up
Considering how compiling a couple of these projects on my tiny arm VPS slows down everything else, this is a welcome change!
Why surprisingly? Are there any CI/CD, let alone SaaS CI/CD providers, that don't make it easy?
Just wanted to add, I loved your post on logging Nomad allocs using Loki. Was a huge help while I was setting up our clusters at $CURRENT_JOB
Interesting, I've used GitLab extensively and hosted/self-hosted runners were trivial to use/run, so I would have expected nothing less from GitHub. They probably make a decent margin on the hosted runners at the cost of significant complexity, so if they can give that away to the orgs that will have their own (virtual) hardware anyway, with their own contracts and networking and etc., why not?
> Just wanted to add, I loved your post on logging Nomad allocs using Loki. Was a huge help while I was setting up our clusters at $CURRENT_JOB
Glad you found it helpful! If you ever want to chat about it, feel free to hit me up (contacts are on my website available in my profile).
Why would that be and how would this affect these Arm64 compiles on GitHub?
If you compile a normal application for amd64, it will work on any 64 bit Intel and AMD machine, unless you start using specialized features, which wasn't the case in the above mentioned case.
The same goes for minute crediting, where splitting things up to run concurrently is actively discouraged because they individually all round up to the next minute. For example; have 3 concurrent tasks that run 1m10s? You're billed 6 minutes. I get that a run would be rounded up, but come on.
[0] https://www.digitalocean.com/community/questions/feature-req...
[1] https://ideas.digitalocean.com/core-compute-platform/p/arm-b...
edit for additional context: An ARM server on Hetzner costs roughly 1/10th that of an x86 one on DO with the same core count and memory, and in my tests out-performs it too.
We have a lot of users migrating off self-hosted setups using `actions-runner-controller` to ours because of this. Essentially, not having to deal with bin-packing is more efficient and concurrency, uptime guarantees are nice.
Daemonsets are generally intended if you have multi-pod nodes; otherwise, you can just use sidecars.