Indeed. We are already optimizing this. And spot instances are coming soon as well :-)
267 karma · joined October 17, 2012
1. Create EC2 instance for runner #1. Find out there is no existing snapshot, so an empty volume is created and attached.
2. Runner #1 runs exactly 1 job and shuts down. A snapshot is taken for the persistent volume. That's going to be used by later runners.
3. Create EC2 instance for runner #2. Create a new volume based on the last snapshot.
4. Assuming #2 is still running while a new job comes in. Create EC2 instance for runner #3. Create volume based on the same last snapshot.
5. Whenever a runner finishes, its persistent volume gets a snapshot taken. Outdated snapshots are automatically removed.
And yes we manage the AMI that the runner uses. We try out best to follow https://github.com/actions/runner-images and will automate this process very soon so it's always up-to-date.
Edit: formatting
> gShoe is an internal joke about a product [Google] shouldn't make, and is used in many internal trainings as a placeholder for any Google product
By spoofing I meant to say that an authenticated but malicious client (intentionally or not, e.g. a clueless intern) may be able to write malicious contents to the cache. For example, their build toolchain could be contaminated and the resulting build outputs are contaminated. The "action" per se and its hash is still legit, but the hash is only used as the lookup key -- their corresponding value is "spoofed."
The only safe way I can imagine to use such a remote cache is for CI to publish its build results so that they could be reused by developers. The direction from developers to developers or even to CI seems difficult to handle and has less value. But I might be missing some important insights here so my conclusion could be wrong.
But if that's the case, is the most valuable use case to just configure the CI to read from / write to the remote cache, and developers to only read from the remote cache? And given such an assumption, is it much easier to design/implememt a remote cache product?
For enterprise edition simply email to hello[AT]naivesystems.com as noted in the README on GitHub
Disclaimer: I'm the founder.
It has been battle tested with real customers in automotive, medical devices, and semiconductors. AFAIK this is the first FOSS tool that achieves commercial use standards (extensive ruleset coverage, low false positives based symbolic execution (which Coverity relies heavily on) and SMT solver, ...)
Instead of being a service, I'm also open to sell the software+hardware solution behind it, so you can have it on-prem. Do you think that's something you would consider given the constraints on supply chain security?
One of the machines in the fleet failed to sync its clock via NTP. Once a job X got scheduled to it, the runner pod failed authentication due to incorrect clock time, and then the whole ARC system started to behave incorrectly: job X was stuck without runners, until another workflow job Y was created, and then X got run but Y became stuck. There were also other wierd behaviors like this so I eventually rebuilt everything based on VMs and stopped using ARC.
Using VMs also allowed me to support the use of the official runner images [0], which is good for compatibility.
I feel more people would benefit from managed "self-hosted" runners, so I started DimeRun [1] to provide cheaper GHA runners for people who don't have the time/willingness to troubleshoot low-level infra issues.
[0]: https://github.com/actions/runner-images [1]: https://dime.run
We also use VMs. But they are persistent. So you always see your runners as Online in GitHub UI.
We achieve this by investing in virtualization technology so that idle runner VMs do not consume too many resources. Disclaimer: I used to work for Google Cloud.
We host GitHub Actions runners for you, with faster CPUs and larger storage at lower prices than the official ones. Currently running a waitlist and onboarding customers in batch to validate the idea.