Ask HN: Is “Distributed CI” Possible? | Hacker News Reader

Ask HN: Is “Distributed CI” Possible?

20 pointsavgcorrection27 comments

Is it possible to have distributed CI in the following sense:

- The “CI” is a program that checks if the code is okay (like runs a test suite)

- Anyone can run it and get the result

- A central server/node can receive the output and SHA1 (that the code was run on) and quickly check if the CI passed

- It takes some effort to cheat the procedure—it doesn't have to be bullet proof (it's meant to be used “among friends”)

- No proof-of-work or something else that burns up electricity for questionable gain

It seems that a simple plaintext script CI is not sufficient since whatever node has run the script can just change the script: `return true`. But if you can either encrypt the program or obfuscate it enough (for a decompiler) then you might be able to implement something like this:

    <do tests>
    if they pass: calculate the output of the CI based on the stable
    success-output of the tests + a secret + the SHA1

That means that the CI program that is distributed to the nodes needs to literally be an undecipharable blob program that in principle can do anything to the node computer—another reason why this can only work “among friends”.

If it's “among friends”, isn't "change the script" out of scope?

You either trust the client to do the requested work, or you don't. If you can't trust the client to use the appropriate CI script version, then you can't trust anything the client responds with.

An example of this is how hacked DVD/BluRay drives work, where they fib to the driver about what byte is in which memory address.

You can issue the work to multiple clients, and treat it like a node failure. If you have a quorum of executions then you can trust that result. However, that's similar to the disallowed proof-of-work requirement.

No matter what, the client has to trust that the server isn't out to get them.

we had something like this. Environment was flaky and expectation was you run the test a few times till it passes.... as expected most developers who can issue put requests with curl or something would only need to "run" that test once

That's not how CI works? CI is the backend verifying that the tests pass, not that the developer has run the tests.

From what I understand, requiring developers to assert they run the tests and not run them again from a clean checkout before pushing to prod is likely a compliance problem (SOX, PCI, etc).

I believe so, but I do not think it is worth the effort and complexity for a lean team. I just recently half-built something like this that used Docker build layers to run lint/test/build/etc. Each build layer was tagged by some sort of manifest hash to guarantee that, if that particular layer was retrieved from cache, the inputs (the codebase files) were identical. The idea was to use fast developer machines to do the work and basically just check the hashes in CI (or skip the separate CI environment entirely).

It turned out all right, but I dropped it for a few reasons. Overall, it was not worth the effort and complexity. First, making it fast and safe was a PITA compared to just...running a few commands in a normal CI environment. Second, for manifest-style hashes, you have to either exclude the .git folder (which means no diffing/affected commands) or try to clean it up somehow (which obviously can be done, but if you do that in a Docker layer it screws up your caching, etc.). Third, it generated an ungodly amount of local cached data. There are a few other reasons I dropped it, but they're generally related to safety/optimization/caching/git.

You should check out projects like sigstore.dev, attestation, and in general "software supply chain security"

This is already possibly and being done by some, you of course need to trust the node, because that could swap out tool on you, but there are various ways to push the attestation further down the stack, into the hardware and CPU. You'll also want to think about how your dependencies fit into the trust model as well, lots of scary stuff can happen there (i.e. things like dependency confusion)

I'm particularly fond of Dagger, as they are doing a lot of the heavy lifting on this front, and their tool makes local -> CI a joy. You can run Dagger in any larger CI system, they focus on replacing your Dockerfiles and other build scripts as one of the next-gen solutions build on BuildKit

I think nix's hydra and binary caching would accomplish this

https://nixos.wiki/wiki/Hydra

https://nixos.wiki/wiki/Binary_Cache

Yhis can probably be achieved with distribited builds on NixOS with a mster that had all other machines configured as build servers.

In this setup the nix daemon on the master can delegate build tasks to other build machines, let's call them agents. These agents have a feature set, arch set and a speed factor to steer task distribution. The feature sets define if, e.g., a machine can compile builds in a sandbox VM. The arch set is for cross compilation support.

You could schedule code checks as nix checks, e.g., linting and validarion could be done by exporting the output as a derivation.

Not practically that I know of. The definition of "not bulletproof" is very vague; why can't the friends just agree not to "return true"?

There are some theoretic foundations in this line of work: https://en.wikipedia.org/wiki/Verifiable_computing

Sure it could be done but it's going to be slower and more unreliable than a centralized solution, and you'll need run it redundantly so you can be sure no one is malicious.

There's also problem of incentive. Why would anyone want to run your builds? So you can run theirs? In that case why you won't run yours and their theirs and save the overhead?

This is like perl's CPANTS system: The modules you upload to CPAN had their unit tests run on a variety of systems and configurations, operated by volunteers, which would report back to the module author any failures and trace output.

In retrospect, it's remarkable that it's been running as long as it has with little major incident.

Sounds like yet another attempt to shoehorn your chosen solution (blockchain?) into a problem better solved without it.

Compare to this actual working solution: I run gitlab runners on spot instances (or at home for personal projects) The runners check in with the server, get a job, process it and send the results. A CI job can have multiple parts. A server can be handling multiple client requests simultaneously.

CI is a resource intensive process so you want beefy machines but they shouldn’t run when not needed (at night if you’re in one time zone, when unloaded if you’re distributed). Enter autoscailing groups and spot instances. 4x the speed for 10% of the cost of always on and “on demand” priced instances. Or for personal projects, speedy when needed idle when not.

Compare this solution to what you’re working on. You’ll see you’re doing a lot of extra work for no additional value.

Do you know about mainframer? [1] It's a tool made for remote builds via ssh/scp. I am using it a lot for my AI training projects because I can continue to work on the code locally without my CPU and GPU going up in smoke.

[1] https://github.com/buildfoundation/mainframer

Sure, make sure your build is reproducible then use something like act (open source github actions runner) to build the executable, submit the hash when you're done.

Basically what you're describing is reproducible builds. The crypto stuff won't work because homomorphic encryption doesn't work, and neither do trusted compute platforms.

Is this just reinventing the concept of Jenkins distributed executors?

I don't know? I googled for the term and didn't find anything relevant.

But am glad if it exists.

Well, in air/space/military applications (and some others), exists very special part, extremely reliable systems, which have doubled or tripled hardware, and they use algorithms very similar to block codes/ecc (or tests in software), to constantly verify that hardware work right, and if detected failure, could automatically disable currently enabled block and switch to spare.

But for you case big problem, that you need to first write 100% specifications and 100% tests, and only then your CI could make automatic checks, and this is not typical for open source unfortunately.

How about this?

1. You update a copy of a repo which defines all inputs in a repeatable way (all inputs are either part of the repo or their hashes are verified during build). Nix does this.

2. You upload the build outputs to a publicly available cache server, like Cachix.

3. You send the changes to someone else for inclusion in their repo.

4. They verify that your code doesn't do anything that looks like sandbox breaking, and that at least all the original tests are still in place. Or they have their own way to keep the run sandboxed and to test coverage.

5. They run the code (pointing at the cache server), which should reuse the cached results and finish almost immediately.

i dev on main in teams we build in environments that are very prod like we have tests that verify code that has been deployed - no matter the location on startup we build test and deploy from local machines (and test because startup)_- this is very reliable or else our tests woudl regularly fail because of this we push to main after our deploy has worked

ofc this only works on smallish teams that push / pull very regularly and have good communication around who is deploying when.

it used to be a shout over a desk partition.

we go faster than you and break far far less than you

What problem are you solving?

Running CI is typically done internally with the stakeholders of the code all aligned on a company's path, aka shared buy-in towards a successful outcome, so why would anyone want to cheat their CI test runs? If its a public or open source project, anyone can download and run tests and look at results.

Jenkins X? https://jenkins-x.io

If your "script" is Turing-complete and the remote nodes can't be trusted, there is no solution due to the halting problem. You can't verify correctness of an arbitrary function output w/o evaluating it step by step.

Also any obfuscation can be removed and any compiled code can be decompiled.

homomorphic encryption and trusted compute platforms not withstanding (neither actually works).

Maybe zero knowledge proofs?

I wasn't getting at the Halting Problem or program analysis...

You could try Bazel with a remote cache https://bazel.build/remote/caching

But it requires some work to move all your build an test targets to bazel.

If it's "among friends," then there is no reason to talk about trust.

If it's for work, look into cryptographic signing (probably PKI) and as long as the signing keys are only issued to trusted agents, you can treat any result (plus Git hash) with a valid signature as a valid result.

Which hedge fund put you up to this?

CI = Continuous Integration

Reply on news.ycombinator.com