Comment by segfaultbuserr | Hacker News Reader

Comment by segfaultbuserr | Hacker News Reader

Parent Full thread

Historically, it was done for end-user convenience. Traditionally, each project has two separate sources, one is the actual development repository, strictly for use by developers. The next is the source tarball for end-user installation, pre-generated by developers by preprocessing the source repo - such as autotools script generation, gettext translation file generation, etc. The idea was that the end users were running on many flavor of incompatible Unix systems, so installing the full development tools such as autotools or CVS/SVN could be inconvenient. To avoid those troubles, developers pre-generate tarballs with necessary installation scripts such as ./configure for user-friendliness - so source tarballs are installers in a sense, at the middle way between actual source and pre-complied binary. On the other hand, because these scripts are machine-generated and are extremely sensitive to small chances to Makefiles, they should not pollute the canonical source tree, so they're not included in the source repository. Occasionally, under the strict Cathedral style of software development, the canonical source tree may even be private under exclusive access by the core team, end users only have access to release tarballs.

Nowadays, the repo-tarball split is largely unnecessary, but the practice remains. The xz incident became a wake-up call that the Reproducible Build movement focused on binary reproducibility, but have so far ignored tarball reproducibility. Hopefully, this problem will be addressed by the community in the future.

> Nowadays, the repo-tarball split is largely unnecessary

I think locking the source distribution mechanism to only git would be detrimental in the grand scheme. A source tarball is universal, independent of preferred tooling, easy to hash, sign and verify, and archive. Even git has loopholes in that a tag does not necessarily have to be a part of the master branch, and not everyone uses (or wants to use) github, or a similar online interface for git...

It's also worth considering that source generation tools are often keen to changing their API frequently, while their output is more weaponized against the passage of time. Autotools is significantly better about that these days, but many other tools aren't...

I think most packagers that can pull from Git can also pull from Mercurial, SVN or whatever else.

Pulling from Git doesn't really require using GitHub other than connecting to their IP. You don't need an account or any tool other than Git to clone a public repo.

It seems to me that pulling directly from the primary source of truth is a good idea these days where possible.

The Git protocol is also substantially more heavyweight than just downloading a tarball.

'Dumb' HTTP involves requesting `info/refs` which lists all the references (branches and tags) available. The client has to find the tag you want, then request the corresponding commit object. That lists the hash of the tree object, which has to be requested, and recursively the tree object references other trees and finally the file blobs.

Any object request could fail because the object isn't stored 'loose' on disk, but is instead in a pack. So the client first downloads `objects/info/http-alternates` to see if it's in a different location, then if that doesn't list anything, it asks for the list of packs `objects/info/packs`. Then it asks for the index of each pack to determine which pack contains the object, then finally downloads that pack.

Because that's very slow, Git also supports 'Smart' HTTP. That moves all the lookups server-side, with a back-and-forth between client and server about what the client has and what it wants, ultimately building a custom pack containing what you asked for. This obviously can't be cached.

In contrast a tarball is one request for one file on disk which is fully cacheable, both by the server hosting the file and any proxies on the path.

This is why things like `bower` and `DefinitelyTyped` got deprecated. Some of them were even hosting the registry itself as a Git repository on GitHub.

GitHub/GitLab can generate tarballs from specific git commits, and those are cacheable and don’t change (when GitHub deployed some changes to the format, many places noticed and GitHub had to undo the changes).

That's just a shortcut to the conventional tarball distribution method.

"The xz incident became a wake-up call that the Reproducible Build movement focused on binary reproducibility, but have so far ignored tarball reproducibility."

The reproducibility must stretch over all the outputs.

A related idea is the Hermetic Build", where the sameness of all the input is ensured, even including the build tools.

In this case, inputs, or more like the whole chain toward "real inputs".

A project can be reproducible when built from the release tarball, without being able to reproduce the release tarball from a git tag.

> Nowadays, the repo-tarball split is largely unnecessary, but the practice remains.

I keep seeing this argument, but I don't understand why CI runners can't generate the configure script rather than offloading that onto the end user requiring them to have autotools/automake/autoconf/m4/etc

If you've ever tried to fight with that on an decade old enterprise system, those tarballs with configure scripts in them are useful. They get generated with new tooling on modern distros, but run on ancient systems. Getting the right versions installed can be an absolute pain in the ass.

Reply on news.ycombinator.com