There might have been a time where it worked, but people started to use plugins for all kinds of reasons, quite good ones in most cases. Findbugs, code coverage, source code generation,...
Today, a maven project without plugins is rare. Maven brought us 95%, but there is a long tail left to cover.
For the purpose of this conversation we mostly just care about the use case of someone grabbing the code and wanting to use it in their own project. For this use case, dev tools like findbugs and code coverage can be ignored, so it would suffice to have a version of the build system with plugins completely disabled.
Code generation is the thornier one, and we can at least be more principled about it than "run some arbitrary code", and at least it should be trivial to say "this codegen process gets absolutely no I/O access whatsoever; you're a dumb text pipeline". But at the end of the day, we have to Just Say No to things like this. Even if it makes the codebase grodier to check in generated code, if I can't inspect and audit the source code, that's a problem, and arbitrary build-time codegen prevents that. Some trade-offs are worth making.
I'm not a fan of generated code. It tends to cause misery, being in a no mans land between code and not-code. But it is usefull sometimes, e.g rust generating an API from the opengl XML specs.
Sandboxing seems the least worst option, but it will still be uninspected half code that one day ends up in production.
The code was only provided in a roundabout way that was deliberately done to evade manual inspection, so that's not a failure of checking in generated code, that's a failure of actually building a binary from the artifacts that we expect it to be built from. Suffice to say, cutting out the Turing-complete crap from our build systems is only one of many things that we need to fix.
I'm not sure how well isolated the containers are (probably not very – I think GitHub gives access to the Docker socket) and you'd have to make sure they don't share secret tokens etc., but at least it might make things simpler to audit, and isolation could be improved in the future.
There is a sizable movement in the Rust ecosystem to move all build-time scripts and procedural macros to (be combined to) WASM. This allows you to write turing-complete performant code to cover all use-cases people can reasonably think of, while also allowing trivially easy sandboxing.
It's not perfect, for example some build scripts download content from the internet, which can be abused for information extraction. And code generation scripts could generate different code depending on where it thinks it's running. But it's a lot better than the unsandboxed code execution that's present in most current build systems, without introducing the constraints of a pure config file.
I’d argue that Rust’s biggest advantage here is that build scripts and procedural macros are written in Rust themselves, making them easier to read and thus harder to hide things in than autotools’ m4-to-shell gunk.
But that means it’s important to build those things from source! Part of the movement you cite consists of watt, which ships proc macros as precompiled WebAssembly binaries mainly to save build time. But if watt were more widely adopted, it would actually make an attacker’s life much easier since they could backdoor the precompiled binary. Historically I’ve been sympathetic to dtolnay’s various attempts to use precompiled binaries this way (not just watt but also the serde thing that caused a hullabaloo); I hate slow builds. But after the xz backdoor I think this is untenable.
If everything is done carefully enough with reproducible builds, I think using a binary whose hash can be checked shouldn't be a great extension of trust.
You could have multiple independent autobuilders verifying that particular source does indeed generate a binary with the claimed hash.
But determined actors with access are always going to try to thwart these schemes, so verification and testing is going to need to step up.
if you trust that they're giving you the correct hash, but not the correct binary, then you're not thinking clearly.
To put it another way: if you don't have reproducible builds and you're trying to replicate an upstream artifact then it's very hard to tell what the cause is. It might just be some ephemeral state, a race, or some machine aspect that caused a few fairly meaningless differences. But as soon as you have a reproducible build, then failure to reproduce instantly marks upstream as being suspect.
It's also useful when you're trying to do things like tweak a build - you can ensure you're importing upstream correctly by checking the hash with what you're making, and then even if your downstream version isn't a binary dependent on upstream (e.g. optimized differently, or with some extra plugins somewhere), you can be sure that changes between what you're building and upstream are intentional and not spurious.
It's clearly not a silver bullet, sure. But it's not entirely useless either, and it probably could help as part of a larger ecosystem shift; especially if conventional tooling created and published these by default such that bad actors trying to hide build processes actually stick out more glaringly.
Debian's response is looking to be mandating the source must come from the git repository, not a tar ball. And it will be done using a script. And the script must produce the same output every time, ie be reproducible. Currently Debian's idea of "reproducible" means it reflects the source Debian distributes. This change means it will reproducibly reflect the upstream sources. That doesn't mean it will be the same - just that it's derived in a reproducible way.
As for trusting the person who gave you the hash: that's not what it hinges on. It hinges on the majority installing a binary from a distro. That binary can be produced reproducibly, by an automated build system. Even now, Debian's binaries aren't created just once by Debian. There are Debian builders all around the planet creating the binary, and verifying it matches what's being distributed.
Thus the odds of the binary you are running not being derived from the source are vanishingly low.
I see what you’re saying, but I don’t buy that reproducible builds actually solve anything, especially long term. As this whole xz thing has shown us, lots of things fly under the radar if the circumstances are right. This kind of thing will absolutely happen again. In the future it may not even require someone usurping an existing library, it could be a useful library created entirely by the hacker for the express purpose of infiltration a decade later.
Reproducible builds are a placebo. You must still assume there are no bad actors anywhere in the supply chain, or that none are capable of hiding anything that can be reproducibly built, and we can no longer afford to make that assumption.
A reproducible vulnerability is still a vulnerability.
A window that can be smashed in is still a vulnerability, so there is no value in people locking their front doors.
You're right. They don't solve anything, if you restrict "anything" to mean stops exploits entering the system. What reproducible builds do is increase visibility.
They do that by ensuring everybody gets the same build, and by ensuring everybody has the source used to do the build, and by providing a reliable link in the audit trail that leads to when the vulnerability was created and who did it. They make it possible to automatically verify all that happened without anybody having to lift a finger, and it's done in a way that can't be subverted as Jia Tan did to the xz utils source.
Ensuring everyone has the same build means you can't just target a few people, everyone is going to get your code. That in turn means there will be many eyes like Andres looking at your work. Once one of them notice something, one of them is likely to trace it back to the identity that added it. Thus all these things combine to have one overall effect - they ensure the problem and it's cause are visible to many eyes. Those eyes can cooperate and combine their efforts to track down the issue, because they are all are absolutely guaranteed they are looking at the same thing.
If you don't think increasing visibility is a powerful technique for addressing problems in software then you have a lot to learn about software engineering. There are software engineers out their who have devoted their entire professional lives to increasing visibility. Without them creating things that merely report and preserve information like telemetry and logging systems software, would be nowhere as reliable as it is now. Nonetheless, if the point you are making is when we shine a little sunlight into a dark corner, the sunlight itself does nothing to whatever we find there, you're right. Fixing whatever the sunlight / log / telemetry revealed is a separate problem.
You aren't correct in thinking visibility doesn't reduce these sorts of attacks. By making the attack visible to more people, it increases the chance it will be notice on any given day, thus reducing it's lifetime and making it less useful. Worse, if the vulnerability was maliciously added the best outcome for the identity is what happened here - the identity was burned, all its contributions where also burned and so a few years of work went down the drain. At worst someone is going to end up in jail, or dead if they live in authoritarian state. The effect is not too different from a adding surveillance cameras in self-checkouts. Their mere presence makes honesty look like a much better policy.
If you trust the people giving you the hash of the binary, just get the binary from them, too.
Reproducible builds and hashes thereof aid in transparency and detecting when that transparency breaks down. Of course, it doesn't mean hackers can't hide malicious code in plain sight, but at least it makes it slightly harder to hide between the cracks as happened here.
The days of 3rd party libraries simply being trusted because they're open source are slowly coming to an end. the problem is not unreproducible builds, the problem is that we implicitly trust source code found online.
Projects relying on 100s of 3rd party libraries are a major problem, and no one seems to care. They like it when their Rust build lists all the libraries that they didn't have to write as it compiles them. They like it when they can include an _extremely_ small Node library which itself relies on 4 other _extremely_ small Node libraries, which each rely on another 4 _extremely_ small Node libraries, until you have a node_modules directory with 30,000 packages in it.
We don't know how to write software any more. We know how to sew bits of things together into a loincloth, and we get mad when the loincloth makes us itch. Of course it does, you didn't LOOK at anything you are relying on. You didn't stop to think, even for a moment, that maybe writing the thing instead of trusting someone else with it was even an option.
As we continue doing this, relying on 3rd parties to write the heavy lifting code, we lose the skill to write such code ourselves. We are transferring the skills that we need into the hands of people that we can't trust, when viewed from a high level.
We need to get back to small applications, which we know because we wrote them in their entirety, and that someone else can trust because the code for the thing is 2500 lines long and has extremely few, if any, dependencies.
We need to get away from software which imports 30,000 (or even 100) third party libraries with implicit trust because it's open source.
"All bugs are shallow with enough eyes" requires that people use their eyes.
Or in other words, the point is not for everyone to verify the build produces the expected binary since that would indeed make the published binaries pointless. Instead, most people trust the published hash because it can be independently verified and anyone can call out the publisher of the binary if it doesn't match.
I’m not sure I follow. The bar for passing review and audits is “people can understand it”, at least to the point of preventing smuggling in a custom public keys and other shenanigans – as opposed to “all valid programs in the language”. While I agree Turing-completeness opens up undeniable complexity, it’s the best we got. And there are huge variations in obscurity across languages and tooling.
It's worse than that, these small C files are required to fail unless the target build environment meets some specific requirements.
Configure scrips tend to be a very slow part of projects, so a new tool that would support parallel compile/run of these checks would also have a considerable speedup. In addition to being more reviewable, verifiable and less error prone.
I've been porting and compiling Unix programs since 5 years before Linux. There was a lot of work needed to port between Unix variants, or even worse, non-Unix variants. So I can see the problem that the autotools solve.
But I never spent much time learning the details on how the autotools work. When './configure' has some problem, it is a very steep uphill battle to get any traction on the issue.
So I'd be more in the camp of looking for a different solution than autotools.
So a first pass should probably be to eliminate as much checks as possible. But there will almost always be a need for a couple of build time checks/switches - should be supported by some kind of tooling.
https://cmake.org/cmake/help/latest/manual/cmake-modules.7.h...
So it seems considerably better than configure scripts. Probably also better than my idea :p I don't know how easy it would be for an adversary to do something "underhanded" though, or how easy that would be to spot such.
> 99.9% of all projects have the capability to be built with trivial declarative rulesets.
Only if you forbid bootstrapping, which all projects ultimately rely on at some point in their supply chain.
We could write an entire PhD thesis on the number of dire technical failings that would need to be addressed to stop this attack, so while this alone wouldn't have stopped it, it would have required the actor to come up with another vector of code injection which would have been easier to find.
> Only if you forbid bootstrapping
Codebases that bootstrap are the 0.1%. Those can be built via `bash build.sh` rather than deceptively hiding a Turing-complete environment behind a declarative one. Even if you need to have these in your trusted computing base somewhere, we can focus auditing resources there, especially once we've reduced the amount of auditing that we need to do on the other 99.9% of codebases now that we've systematically limited the build-time shenanigans that they can get up to.
At worst, you're actually creating more holes. The reason autoconf/automake exist and M4 scripts are innocuous in the first place is because the build system uses an underpowered language and developers have to turn to code generation to get around it.
If you kneecap the build system's language you're not solving problems. You're creating them.
> it would have required the actor to come up with another vector of code injection which would have been easier to find.
If make was standardized and could programmatically determine the environment its run under and write full programs then the attack vector wouldn't exist in the first place.
> Codebases that bootstrap are the 0.1%.
We have different experiences, because ime it's close to 100% especially when you include transitive dependencies. When you care about supply chain security you care about being able to bootstrap from sources for your code and all your dependencies, and it's almost guaranteed that one of your dependencies needs to be bootstrapped.
I would like to see this on the language level, not just on the OS level.
Say I have a directory with a virtualenv and some code that needs some packages from PyPI. I would very much like to sandbox anything that runs in this virtualenv to just disk access inside that directory, and network access only to specifically whitelisted URLs. As a user I should only need to add "sandbox = True" to the pyproject.toml file, and optionally "network_whitelist = [...]".
From my cursory looking around, I believe Cloudflare sandbox utils, which are convenience wrappers around systemd Seccomp, might be the best starting point.
Edit: or just use Firejail, interesting...
You mention sandboxing on the language level, but I don't think it is the way. Apparently sandboxing within Python itself is a particularly nasty rabbit hole that is ultimately unfruitful because of Python's introspection capabilities. You will find many dire warnings on that path.
Sounds like it would be impossible but maybe my thinking is just enclosed and not free.