Hacker News Reader

Comment by kijin | Hacker News Reader

What makes maintainers of major distros still rely on questionable tarballs to build packages? It's not as if these essential programs don't have a public, authoritative git repository.

Is it because of inertia, because we've been using tarballs since before VCS was a thing? Is it to reduce the burden of package maintainership, by letting upstream do most of the transpiling and autotools plumbing work? Is it because some assets and test data are only included in the tarballs? Why are they not committed, either to the same repo or (if upstream wishes to keep the main repo small) some other repo?

People have been calling for reproducible builds for years now, but reproducible builds don't mean anything if they are not built from authoritative sources with immutable commit IDs.

View on HN

The tarball, with its signed hash, is the authoritative source with immutable ID.

Not every project uses git. Not every project uses a public VCS. Git history can be changed with a simple command and is unreliable.

The point of a tarball is that it is the distribution medium of a source release. The developer generates the generated code (where required), runs all tests (eg. `make distcheck`), then when satisfied it's good, signs it and releases it. If you can't trust the developer's signed tarball you can't trust their git repo either, since it's the developer you can't trust.

A downstream, such as a Linux distro, uses the release tarball because it's the single, verifiable, published source of truth. They should be able to do reproduceable builds from the tarball (not being able to do so should be filed upstream as a bug), and for licensing reasons (eg. GPL) need to keep the tarball and make it available to anyone to whom the software is distributed.

The parent referred to "questionable" (i.e., unsigned) tarballs.

Your answer, while excellent, is answering a question that wasn't asked.

No distribution I know of uses "unsigned" tarballs. Most of them hardcode the hash of the expected tarball anyway.

Hashes and cryptographic signatures are not the same thing.

> Git history can be changed with a simple command and is unreliable.

Not if other people have cloned your repo. That's the whole point of having a distributed VCS. Your repo is no longer the single source of truth because other repos remember what truth looked like 3 days, 3 months, 3 years ago. You renounce the right to dictate truth as you wish, and gain trust in exchange.

I don't understand why this idea comes up again and again: "You trust the developer, you might as well trust the code they distribute." That makes no sense. Why do you trust them in the first place? Do you know them? Have you met them in person, had a few beers with them? Or is your trust merely based on their past performance as an open-source developer? If it's the latter, whether they have been willing to work in a way that facilitates independent inspection and verification of their output should be a very important factor in whether you should trust them at all. Hardware vendors who dump opaque binary blobs at the kernel's doorstep tend not to receive much respect, after all.

> "I trust the developer, so I trust the code they distribute."

I said "If you can't trust the developer's signed tarball you can't trust their git repo". That's absolutely not the same thing. In fact it's sort of the opposite.

Distributed cloned repos are not a single source of truth. If they differ, which one is the actual source of truth? Was the post-release changed history propagated?

A signed tarball means the signing key you use to verify the tarball has been verified by a web of trust. The web of trust requires the physical confirmation of identity of the private key holder. Yes, it's possible to subvert the web of trust by playing the long game, complete with falsified (or even real) national ID documents, active participation in social networks, and other life activities. This, though, is true of all espionage and no amount of technical measures or randos ranting on social media on the internet will prevent it.

> Distributed cloned repos are not a single source of truth.

That's the point. No person should be trusted to be a single source of truth. If there are multiple sources of truth and they don't agree, it's a red flag. A highly useful flag, in fact. The situation must be resolved before truth can be established again, and all that commotion in the bazaar will make it more likely that a maintainer gone rogue will be caught. A signed tarball by the same rogue maintainer, on the other hand, will probably pass unnoticed until it's too late.

At the end of the day, it's neither people nor any specific chunk of code that I'd like to trust. It's the system -- a system that incentivizes accountable behavior, and blocks releases if there are any red flags, so that you don't really need to trust any single person.

> A signed tarball means the signing key you use to verify the tarball has been verified by a web of trust. The web of trust requires the physical confirmation of identity of the private key holder.

Hasn't the web of trust collapsed due to an unfixable hole in keyservers?

PGP keys in practice have almost always amounted to checking a hardcoded list of acceptable keys that is periodically updated, not unlike a list of trusted CAs.

(In any case, the attack we're talking about here--a rogue maintainer creating a backdoor in their project--won't be caught by any amount of advanced signing infrastructure. Somebody was elevated to a position of trust and any reason not to elevate them wasn't discovered until after the damage was done.)

> If you can't trust the developer's signed tarball you can't trust their git repo either

The public git repo has a lot more eyes on it. You don't necessarily have to trust the developer if 1) you trust at least one of the people who reviewed the code, and 2) you have a way to verify that the code you're running is the same as the code that person reviewed

Trusted builds based on the public git repo rather than an unrelated tarball solves part 2 of that problem.

There are still nowadays projects without a proper source control.

For example, xpdfreader, source of xpdf, is distributed just as source tarballs. The maintainer just publishes them every time there is a new version.

https://www.xpdfreader.com/download.html

https://www.xpdfreader.com/old-versions.html

https://packages.debian.org/search?keywords=xpdf

There is nothing in Debian or FOSS in general that mandates having source control.

(I remember that was a point of contention with WebKit/KHTML... in the olden days of 2000-something, Apple forked KHTML to make WebKit, and in order to comply with GPL, they just published source tarballs, which were basically impossible to merge back into KHTML.)

edit:

ahh I see that the xpdf that is in debian is actually different than this one; someone forked it some time ago and it is in in git here (but no github/other forge, just .git)

https://offog.org/git/xpopple.git/

well ok, maybe I was wrong. but still, a .git folder on someone webpage is not that much more reliable than a tarball on someone's webpage

No need to look that far, ncurses is also tarball/patch-based.

Interesting.

Well, xpdf I knew :) but yeah xpdf can be nuked, ncurses cannot.

Certainly there's nothing in FOSS that mandates source control, but source control is one of the most important technologies discovered for developing software, and I'd be hard pressed to imagine a justification for relying on software that didn't use it.

It is likely that the `xpdf` project is developed under source control, but they don’t make the repository public. SQLite makes their repository public, but they do not accept contributions.

I'm showing my age here, but making version control public is a relatively recent thing; I can't think of a project before OpenBSD that did it, and it wasn't until SourceForge that the idea of projects just having source repos out there and visible became a popular idea, and that was like '99 or 2000. Most projects would just release tarballs and have a CVS repo that was limited to "approved" developers. And that was well after all the big distro build systems developed.

A list of only-archive-published projects will be interesting. 7-Zip is another. Regarding that .git web directory, is there an existent tool/script that does something similar to `git clone` for it?

yes there is, it's called git clone.

Well, unexpected. Git pretty cool.

A popular project with release-only distribution is Lua. I don't know whether they have proper source control, but even if they do, AFAIK it isn't publicly available.

There's a lot of inertia here, but also the tarball is or was the one point of commonality where you could say "this is the standard thing every upstream provides". Today almost everything has a git repo, but it still isn't 100% universal, and in the past even less so -- some projects had git, some svn, some cvs, some mercurial, some didn't use source control at all(!), but everybody provided their releases as source tarballs. So when you're designing a distro packaging workflow 20 years ago, "treat the source tarball as the authoritative upstream output" made sense, because it was the authoritative upstream release output.

Why are tarballs any less authoritative than a Git repository? Both come from the same authors. If you can't trust the tarball released by some author why would you trust what the repository contains?

as i understand it, the tooling would "see" some of the questionable configs that were in the tarballs.

Because they're packaged into tarballs they cant be checked. or or to the point they use the tarballs because its much faster...

What tooling?

Git is just a filesystem with file versions, anything you can check in a tarball can also be checked in Git. The only thing Git adds is a change history for the contents in the filesystem (a history that can be forged too - Git even has a bunch of commands for that too).

If you cannot trust the authors' tarballs how can you trust their Git repository?

> Why are tarballs any less authoritative than a Git repository? Both come from the same authors.

Traceability. There's a difference in how easily someone can insert nefarious code without it going through processes that might reveal or stop it.

Compare: "Why is a paper sack of pills any different than a factory-sealed anti-tamper bottle? You're buying them from the same grocery store..."

> Traceability. There's a difference in how easily someone can insert nefarious code without it going through processes that might reveal or stop it.

But both come from the same authors. If you do not trust the tarball because it may have nefarious how would you trust Git to not have nefarious code? The authors have complete control over what goes in Git and what processes might be there.

> Compare...

The grocery store comparison doesn't fit here because grocery stores do not make the things they sell. The equivalent to a grocery store would be the download site (BTW lets not continue with metaphors because IMO instead of making things more clear you are introducing TWO parallel channels to argue about).

> But both come from the same authors.

The Git repository comes with a history containing each and every commit, along with checksums and sometimes cryptographic signatures for it all. The tarball is just a bunch of random files.

If somebody wants to rewrite the Git history, that will get instantly noticed by anybody that had an older clone of it. If somebody adds a hack to it, it will be at the top of the history and easy to notice as well.

If somebody adds something to a tarball, there is nothing to tell you that something changed, you'll have to audit the whole thing from top to bottom (or compare it to the Git repository it came from).

Also the author isn't in control here, the tarball can get changed after it was uploaded to the server and there would be nothing in the tarball itself to tell you.

People are using PGP signatures and checksums to secure tarballs some more, but that's always an extra step and doesn't happen automatically. And even that still doesn't give you the history.

> The tarball is just a bunch of random files.

But that is EXACTLY WHAT GIT IS!, the only difference is that you also get the previous versions too. That doesn't help you at all when the authors themselves are not trustworthy. It MAY help after finding a compromise to figure out when it happened, but it wont help anyone prevent it.

What you write is about someone trying to compromising the Git server itself. This is NOT what i refer to (and even that sort of compromise would be sidestepped with signed tarballs).

What i refer to is more fundamental: the author of the project controls what is put in the Git repository itself, just like the tarballs.

The Git repository is NOT any more trustworthy than the tarballs: if you do not trust the latter then you should not trust the former either and if you do trust the former then there is no reason to not trust the latter.

> the author of the project controls what is put in the Git repository itself

The author doesn't control the Git repository. The Git repository is a shared data structure, not just bits on a server. Neither the author nor anybody else can just change the Git repository arbitrarily. They can only change it in ways allowed by Git or the alarm bells will be going off. A tarball has no such restrictions.

I do not refer to the files or bits or whatever that make the git repository in your `.git` directory. Please take a step back and read again what i wrote, it has nothing to do with what you responded with here.

> But [a bunch of random files] is EXACTLY WHAT GIT IS!

Well, yeah, but only in the same limp way that "a bunch of random bytes" describes both (A) some text file versus (B) an e-mail with valid DKIM headers with a PGP signature.

It's possible that both could arrive containing your CEO's name and an urgent demand to wire all company funds to some random city in Myanmar... but one approach is fundamentally more secure.

> the only difference is that you also get the previous versions too.

No, that's not the only difference nor the most-important.

The important difference is that the files represented by a particular git-revision all exist with cryptographically-hashed relationships on their content, and that structure prevents/deters multiple kinds of secret meddling or impersonation.

> The important difference is that the files represented by a particular git-revision all exist with cryptographically-hashed relationships on their content, and that structure prevents/deters multiple kinds of secret meddling or impersonation.

...which a developer with proper write access to the repository (i.e. the same developer that'd make the tarballs) wont need to do since they can decide what goes in there in the first place, thus making this "important difference" not at all important when it comes to the case i argue in this entire thread.

This backdoor resided in git for a year without anyone noticing.

It wasn't until it was activated via the tarball that someone found out. One could easily argue that signed tarballs are more vetted than git commits.

Every commit in git repository is effectively signed by the commit hash. You can't sneak in any unreviewed changes.

But you can sneak in reviewed (by you) changes, if you have control over the repository.

Again: both the Git and the tarballs are released by the same developers.

You never have full control over a repository that can be cloned by others. Sure, you can rewrite the history, but anyone who has been pulling from your repo will notice the conflict.

Whether you can trust the developer is beside the point. The question is how easily we can tell if someone is (still) trustworthy. Source control brings a measure of accountability into the game, making it more difficult for unusual activity to go undetected.

It is not beside the point, it is the ENTIRE point. If you do not trust the developer then Git wont help you, the developer can put their malicious code in the repository itself.

Git wont help you prevent anything, it is just a bunch of random files (as someone mentioned above) with the only difference being that there is a change history attached. But that wont prevent anyone with proper legitimate access (i.e. not trying to compromise it) to the repository from adding malicious code. All it may help with is figuring out, after the fact (of finding the compromise) when it was added.

As i wrote above, the Git repository is not any more trustworthy than the tarballs: if you do not trust the latter then you should not trust the former either and if you do trust the former then there is no reason to not trust the latter.

> Git wont help you prevent anything

Of course. Nothing does.

> All it may help with is figuring out, after the fact (of finding the compromise) when it was added.

Exactly.

And that's what I'm trying to say.

Ease of detection discourages bad behavior, just as regular police patrols help reduce crime in a neighborhood. With a public git repo, every commit increases the chance of detection long before a tarball is ready for release.

I never said that I trust the git repo. I trust the eyeballs more, and the pitchforks too.

Ease of detection is not a thing here since nothing is detected. The only thing you can have with Git is figuring out when a change was made after it has been detected - by different means.

And again, when someone can submit to a repository as developer and also make tarball releases, it makes no sense to differentiate between the two if you are worrying about malicious code.

Your original comment was why "maintainers of major distros still rely on questionable tarballs to build packages" instead of Git. And IMO the answer is really simple: because if they trust a project's developers to not put malicious code in their project, it makes no difference if the code came from Git or an explicitly provided release tarball. If such trust wasn't there, chances are the project wouldn't be part of a distribution in the first place.

> but anyone who has been pulling from your repo will notice the conflict.

Not will but can notice a conflict. Repos edit history all the time for a variety of reasons. Well written but nefarious changes to history won't immediately be recognized as nefarious.

Besides once there's multiple copies of a repo available with different histories, who is to say which is the canonical repo for a project?

Yes, and that didn't save us from the xz backdoor did it?

The git repository has been scanned by many eyes. People, other than the authors, do read the source code on GitHub or clone/fork the repo and try to build / modify when they want to contribute. By contrast, almost no one reads through the tarball contents. They just assume that it is essential the same as the git repo.

That doesn't make the Git repository inherently more authoritative or trustworthy, not to mention that if you do not trust the tarballs you shouldn't trust the Git repository either - or the developers who control what goes in both.

The issue here isn't about trusting Git or tarballs, is about trusting developers. If you do not trust the developers then getting the code via Git or tarball wouldn't make anything more trustworthy and if you DO trust the developers then there is no reason to trust the Git more than the tarballs since they are both provided by the same people who you already trust to not insert malicious code in the project.

That's the conventional thinking, but it has been proven false by this xz incidence. The maintainer of xz did not inject any malicious code into the git repo, but only in the tarball, exactly because the latter is subject to far fewer eyes and he took far less risks polluting only the tarball.

The project was already compromised, the Git repository isn't any more trustworthy than the tarball.

And if there is any relevant lesson from the xz case isn't to trust Git more than tarballs but - as someone else mentioned already - tarballs should be fully reproducible from Git.

> The project was already compromised, the Git repository isn't any more trustworthy than the tarball.

You are talking about this from hindsight. For other projects, we do not know yet if anything similar is happening. So for them the Git repo is definitely more trustworthy.

> tarballs should be fully reproducible from Git.

That's exactly the same as trusting Git repo more than the tarball.

Because it's often easier, and because they're not "questionable tarballs".

People seem to have forgotten that all of this has been working well without all that many incidents for 35 years. And these archives are published by the same people you trust to run code on your system. It's not like you can't hide this kind of stuff in the autotools soup committed to git. I doubt this would have been caught it – the "sneak it in the .tar.gz only" was just a bonus defence against detection.

I'm not saying using git isn't better, but we also need to retain some perspective on all of this.

This ibus thing for example seems entirely innocent.

The core of the issue is "who do we trust to write code for us?" Maintainers protect us from randos submitting malicious code, but who maintains the maintainers?

It's absolutely because of inertia. Cutting a tarball is a very public indicator that the software is ready to be used. A git tag or branch doesn't have the clarity of "use this" as a release tarball does.

That said, this is absolutely going to be changing now. We obviously can't keep relying on tarballs anymore. We'll find a new normal that will work for a very long time until some other critical issue arises and the cycle repeats.

We can use tarballs - they are useful as signed artifacts, but only if we verify that they are reproducible.

If you're going to download both the tarball and the got repo and verify the reproducibility of the tarball, then why bother with the tarball at all? You already have the got repo.

A git repo can vanish overnight. Git is often used to snag source, but tarballs are still crafted after even then:

https://news.ycombinator.com/item?id=39903813

Git repos only vanish if they have no clones. For the purpose of accountability, the "official" repo is not more privileged than any other clone that retains the same history, and can be verifiably recreated from any clone if it is ever lost or tampered with. (Assuming SHA-1 isn't too broken, that is.)

For archival purposes, nothing prevents people from creating a tarball that contains the .git directory as well, which would preserve not only the current state of the project but its entire history.

Git repos only vanish if they have no clones.

And? You're speaking probabilities, not certainties. It's not relevant in terms of archiving. You don't guess, you don't hope, you simply 100% ensure that 10 days, 19 years, or a century from now you can build the same thing.

I agree that adding "extra stuff" to a tarball isn't a bad idea, and in fact, many already do!

If you can archive the tarball, you can archive the git repos. If for some reason you can't, you can cut your own tarballs from the git repo and then you don't have to worry about them because you made them yourself.

Archiving an entire git repo is serious overkill, and untenable realistically. Debian has 30k packages, if not more, and some of them are the linux kernel.

My responses in these threads has been to the "why not build from git" logic championed here.

You want to build from a reliable, repeatable source. And as I mentioned, that can be git clone -> tarball, sure.

Reply on news.ycombinator.com