OSS Rebuild: open-source, rebuilt to last
security.googleblog.com
security.googleblog.com
We built ReprOS to solve this problem: https://codeberg.org/stagex/repros
"Git push" to it and it will do a build in a throw-away VM then have the host sign the artifact results and push signatures to the same or a different repo.
It will no doubt mature a fair bit more through this process
Curious, what is this about? Care to share the context?
You just need a read-only system partition, like macOS or NixOS or Silverblue.
Silverblue and NixOS are not fully reproducible, and rely on blind maintainer trust to merge anything they want, then shifts to a centralized build/signing trust model for all packages (single points of failure)
Debian has inflated reproducibility stats as it allows bootstrapping packages from binary blobs exposing it to trusting trust attacks, and overall relies on a distributed trust model where different maintainers maintain and sign different packages (many points of failure)
Stagex, which is what ReprOS is built with, relies on a decentralized trust model with 100% full-source-bootstrapped, deterministic, multi-party-signed commits, with multi-party-signed builds, (no single computer or human is trusted).
The former options are highly complex desktop focused operating systems that are fast and loose with supply chain security in order to maximize contributions and package variety.
Stagex by contrast is -not- a desktop distribution and cranks supply chain security all the way up focusing primarily on those few packages needed for mission critical build, administrative, and server use cases.
ReprOS by extension you can build from hex0 all the way up to compilers and a final .iso bit for bit identical every time, so admins can form confidence they are running unmodified upstream source code without any chance of tampering by a compromised maintainer.
Let us know if you can think of any ways we could communicate how we differ better!
But I'd also say another way to do it is to build / cross compile on two totally different machines, say Linux and OS X, or Linux and FreeBSD, or even a modern Debian and some Linux VM from 2005
If the results are exactly the same, then I think it can be trusted
I guess that's like Diverse Double-Compiling, but extended to the whole machine:
Given all builds start from hex0 you can build from an Arch Linux host and I can build from a Debian host and we are still guaranteed identical results every time.
Every altered package is reproduced bit-for-bit identically by multiple different maintainers and signed every release.
Our entire distro and release process is built around addressing trusting trust attacks which is why we must be 100% deterministic or we cannot ship a release.
I am honestly not sure how much effort it would be; I was thinking about doing it / verifying it for https://oils.pub/ , which has a large build process
But it may save time to use ReprOS. I'll look at it more - it sounds interesting!
I reverse-engineered it a tiny bit, looks like you can get a list of all builds so far like this:
gsutil ls -r 'gs://google-rebuild-attestations/**'
> I'm very excited about this project
Thank you!
> but it could really do with a web UI of some sort!
Couldn't agree more! The terminal UI exists (`./tools/ctl tui`) but is oriented towards developers on the project or to those who wish to run their own instance. Making the system and data more accessible to general users is a big priority.
It's using that fixed list from the Gist though, I haven't set it up to update and reflect new listed projects.
With guix I can bit for bit reproduce with my client machine the upstream binaries.
This seems flawed to assume that google's servers are uncompromised, its vastly better to have distributed ability to reproduce.
https://guix.gnu.org/manual/en/html_node/Invoking-guix-chall...
I find that for many npm packages, I don't know how builds were actually published to the registry and for some projects that I rebuilt myself in docker, I got vastly different sizes of distribution artifacts.
Also, it seems like this is targeting pypi, npm, and crates at first - what about packages in linux distro repositories (debian, etc.)?
There's probably a lot of people that see GenAI as the solution to Not Invented Here: just have it rewrite your third party dependencies! What could go wrong. There will also be some irony of this situation with third party dependencies being more audited/reviewed than the internal code they get integrated into.
At the same time rolling your own implementation with GenAI will be quick and easy. Outsiders are checking these, so no CVEs for these. Just sit back and relax.
> Does this require builds are done via CI
Nope! One use for OSS Rebuild would be providing maintainers that have idiosyncratic release processes with an option for providing strong build integrity assurances to their downstream users. This wouldn't force them into any particular workflow, just require their process be reproducible in a container.
> for some projects that I rebuilt myself in docker, I got vastly different sizes of distribution artifacts.
Absolutely. OSS Rebuild can serve to identify cases where there may be discrepancies (e.g. accidentally included test or development files) and publicize that information so end-users can confidently understand, reproduce, and customize their dependencies.
> what about packages in linux distro repositories (debian, etc.)
OSS Rebuild actually does have experimental support for Debian rebuilds, not to mention work towards JVM and Ruby support, although no attestations have been published yet. There is also no practical impediment to supporting additional ecosystems. The existing support is more reflective of the size of the current team rather than the scope of the project.
- https://reproducible.archlinux.org/ (since 2020)
- https://reproduce.debian.net/ (since 2024)
There's arch-repro-status and debian-repro-status respectively to show the status of the packages you have installed, but since it's not yet possible to make an opensource computer out of reproducible-only software, there isn't really any tooling to enforces this through policy.
[1]: "it means effectively a decision was made for NixOS to be a hobby distro not suitable for any targeted applications or individuals. It really sucks, because I love everything else about nix design. Instead I am forced to bootstrap high security applications using arch and debian toolchains which are worse than nix in every way but supply chain integrity given that all authors directly sign package sources with their personal well verified keys."
Nixpkgs is more valuable than Nix at this point, but also quite vulnerable. In practice it has worked out reasonably well so far, I don't know of anyone who got "owned" because of Nix.
Not anymore with the introduction of dynamic derivations (experimental)
It's hard to know what exactly your security concerns are here, but if you look at the current ecosystem of using containers and package registries, Nix is pretty clearly a solid contender, security-wise.
NixOS knowingly discards vital supply chain integrity controls to minimize developer friction and maximize package contributions. It is a highly complex Wikipedia style distribution optimizing for maximum package variety which is absolutely fine and great for hobby use cases, but use in security critical applications is absolutely irresponsible.
Guix goes some big steps further in supply chain integrity but still ultimately trusts individual maintainers.
See this chart to understand how NixOS compares in terms of threat model https://codeberg.org/stagex/stagex#comparison
Also, it seems like you're using unstable if you're having that many breakages, sooo kinda asking for it?
It does not even try to be a workstation distro so we can get away with a small number of packages, focusing on building software with high accountability.
Thankfully OCI build tooling is mature enough now that we can build using standards and do not need a custom build framework and custom languages like nix/guix does anymore.
E.g.:
> could've contributed SLSA attestations support to nix
That sounds like a great idea! However one important consideration is that while an artifact on nixpkgs may aim to replicate the function of the upstream package, it must adhere to and interoperate with the rest of the distribution. Ultimately, its 'ecosystem' is nix. Work that goes into writing and maintaining the nix build does not generally filter back upstream to impact the build integrity of, say, its associated PyPI package. So if users continue to consume from PyPI, improving nix won't serve them.
This is not to say that the long-term source of truth for packaging will remain the language registries. Just that today's reality demands we meet users where they are.
> Would love to see Google contribute to nix in this space :)
Same :)
Both nix and guix are exciting projects with a lot of enviable security properties. Many here can attest that using them feels like, and perhaps is, the future. I see OSS Rebuild as serving more immediate needs.
By rebuilding packages from the registries people already use, we can bring some of those security properties to users without them needing to change the way they get their software.
Until then they are still behind Debian and Arch Linux, which do in fact implement this with rebuilderd and debrebuild/archlinux-repro.
I.e. how would the xz backdoor be identified? Does the system have logic like "the build should not us binary bits already in the repo"? Or it's even more specific , like "all build files must come from a single directory? If it's more generic, how does it work?
Rebuilt to Last? It is a Google project, so I give it two years.
2. It will likely be well architected.
3. It is open source, so others can fork it when Google abandons it. https://github.com/google/oss-rebuild
This means two migrations in two years.
They already do, and always have. It doesn't make any sense to most of them to fund their OSS dependencies at all, because they're available for free. They should do more than what makes sense for them, and they should have to pay professional consequences if they don't.
Programmers should have enough unity to bring pressure against companies that make a lot of money from software they don't pay for. Or rather, should have had, because LLMs have changed anything.
without burden on upstream maintainers
Then I see
This is not an officially supported Google product
on https://github.com/google/oss-rebuild
And then I also see
oss-rebuild uses a public Cloud KMS key to validate attestation signatures. Anonymous authentication is not supported so an ADC credential must be present.
$ gcloud init
$ gcloud auth application-default login
Mainly because Google has horrible customer support.
It would be more interesting if they came up with something hosted on third party infrastructure. Last I heard, Google Cloud is run by Oracle executives
---
e.g. in particular the Unisuper incident led me to believe that a lot of operational stuff is being outsourced, and is of poor quality
UniSuper members go a week with no account access after Google Cloud misconfig
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
Google accidentally deleted a $125 billion pension fund's account
https://qz.com/google-cloud-pension-fund-unisuper-1851472990
I would not say this is unrelated, because operations in the underlying cloud can be a weak link in security
Although I'd certainly be interested in an argument otherwise
This fit in somewhere here?
OSS Rebuild should give that Nebraskan the peace of mind to continue their everyday heroism without being pulled away to set up security configs or debug release CI. The rest of the blocks on top can contribute the support to assure themselves and the community that those critical builds are trustworthy.