Are tarballs obsolete? | Hacker News Reader

Are tarballs obsolete?

esr.ibiblio.org

66 pointsacheron82 comments

No! Things like package managers and git, however pervasive, are agents and not the testable, serialized representation of data.

Why wouldn't you make the same file set available as a single download via plain, old HTTP(S)?

What's so hard about that? Why place something on a shelf, behind an agent that demands a learning curve, when it could also be available to a browser?

You CAN'T hash a tree of files quickly by hand. You need a program to help test the integrity of a tree with some recursive problem solving. Take tarballs away from people, and you're putting things on a shelf, out of reach, taking away a level of safety and reliability from a certain cross-section of your audience.

Not only that, but EVERY platform has a web browser, even if it's curl or wget. With a single file, available by HTTP, any platform can get to it, so if your machine dies, and you have to call a friend for help, and they don't use the same OS, they won't have to install a special tool, or hope an intervening third-party web service provides coverage.

If we're simply talking file formats, and possibly exchanging the tarball for zips, ISOs or what-have-you, as alternative serial representations of the same data, that's not even a discussion.

If you're worried about practical utility, load, bandwidth and/or availability (or even, perhaps, culling the dull, dumb, toxic and obnoxious from the herd, terrible as that sounds) then changing the agent permitted to access the resource might be one solution, but not necessarily the right solution.

> Why place something on a shelf, behind an agent that demands a learning curve, when it could also be available to a browser?

wget <filename>

git clone <repo url>

What is the difference in learning curve ?

> With a single file, available by HTTP, any platform can get to it, so if your machine dies, and you have to call a friend for help, and they don't use the same OS, they won't have to install a special tool, or hope an intervening third-party web service provides coverage.

People have been making an argument for static compiling basic tools for a long time now ( I support it ). I think git can be considered a pretty basic tool now given its popularity.

The authenticity check with GPG or SHA sums requires a gentler learning curve for a 'flat' tarball.

> git can be considered a pretty basic tool

so what about CVS, svn, Mercurial and fossil? Git is hip right now, but including it in an initrd seems excessive. Furthermore git downloads the entire repo, all the history/commits, this isn't always what you want or can handle or want to pay for.

> Furthermore git downloads the entire repo, all the history/commits, this isn't always what you want or can handle or want to pay for.

That's the default behavior, but with `git clone --depth 1` you can apparently download only the latest version of all files: https://news.ycombinator.com/item?id=10449157

Googling around, most pages confirm it. Found this one also informative: http://stackoverflow.com/q/6941889/1201863

> so what about CVS, svn, Mercurial and fossil

Why, these are pretty basic too. Especially if what you want is just to get the code. Having a bit of experience with any VCS will let you skim the relevant manpage and use another VCS in minutes tops.

> including it in an initrd seems excessive.

True...

> Furthermore git downloads the entire repo

Not necessarily? I think you can download just the source files, without any git metadata, turning git into SVN essentially. I'm not 100% sure though, but I'd be really surprised was it not the case.

---

I started my VCS adventure with RCS, then (quickly) moved to CVS, then (even more quickly) to SVN. As soon as DVCS started to appear I switched to bazaar (bzr), then git. These are the VCS I used for my own projects; I probably used a couple more VCSes in a situation where I needed latest code for some project which used something like Mercurial, or Darcs, etc.

In general, I think it's fair to say that Version Control, as a whole, is a basic tool.

>Not necessarily? I think you can download just the source files, without any git metadata, turning git into SVN essentially. I'm not 100% sure though, but I'd be really surprised was it not the case.

Partially possible. You're always fetching diffs, not files, with git. You can get close to what you said with `git clone --depth 1`, which tells git to only get history for the latest revision, but you still get some git metadata about available branches. Depending on the version of git you're using, you may also have to pass --single-branch in order to only fetch one branch's revision.

wget <filename>

more like

    wget <filename>

    <some hard to remember command with like 4 flags to unpack,
     plus hope it ends up in a subdirectory and does not put all files in this dir>
     <filename>

until you've done it enough times to remember it. Learning curve indeed.

You can generally replace

    <some hard to remember command with like 4 flags to unpack,
     plus hope it ends up in a subdirectory and does not put all files in this dir>

with

    tar -xf <filename>

    -x means extract

    -f means force

and my favorite to put in there is:

    -v for verbose

This shows you all of the files being processed, sometimes it can be really helpful to see. Try:

    tar -xvf <filename>

Actually, -f makes tar read from a file instead of stdin.

indeed http://explainshell.com/explain?cmd=tar+-xvf

If you want a stable version, you can't necessarily just git clone. (Maybe if the default branch is kept stable, but I wouldn't trust that by default.) You also have to find the right branch/tag and do a checkout.

> What is the difference in learning curve ?

Everyone knows how to use an http client and everyone has an http client. Most people do not know how to use git and most people do not have a git client. It's more of a vertical line than a curve.

    'git' is not recognized as an internal or external command, 
    operable program or batch file.

The distributed form of a project (especially one based on GNU autotools) is often not merely a tarball of the upstream repository. Often the author has pre-run some stages of the build that she feels would impose unnecessary or esoteric dependencies. Typically this involves pre-generating Makefile.in from Makefile.am and configure from configure.ac. Another common one is for the maintainer to pre-generate the documentation. The user of the distributed tarball is fully capable of re-running these steps, but they have become optional.

This is a fact often forgotten when projects switch to github and just start using the automated release tarballs github will make from the repo.

Yes! Thank you for explaining this. I don't think a lot of people realize what makes up a good release tarball.

Exactly.

Many web tool chains involve pulling from external repositories such as npm, bower or even github itself. If one of these are down you cannot deploy your application.

Even better many packaging systems such as npm or setuptools allow you and library maintainers to specify a flexible version numbers for your dependencies. If during the course of your build, test and deploy chain one of these dependencies changes your application could break through no fault of your own. You cannot rely on maintainers to not release breaking changes in minor versions, it happens all the time, intentionally or not.

At least for npm there is a solution for that problem: https://docs.npmjs.com/cli/shrinkwrap

This becomes more complex when distributions want to patch configure.ac or Makefile.am -- they need to patch the included Makefile.in files as well.

Trying to balance these two forces is the reasoning behind the AM_MAINTAINER_MODE flag.

These days, I prefer to simply use git archive for a tarball and make users run autogen.sh themselves.

See https://blogs.gnome.org/desrt/2011/09/08/am_maintainer_mode-... and the associated comments.

Yes, incluinding the maintainer-mode option can be important, which is why this line should be included in configure.ac most of the time when using automake:

    AM_MAINTAINER_MODE([enable])

This includes the --disable-maintainer-mode configure option, while leaving maintainer mode on by default which is the traditional behavior you get when you don't include that macro. This shouldn't change any behavior for most projects.

You can change the default to off with the same macro, if that is important; the key point is to include the macro so it can be changed by the automated packaging scripts used in some distributions.

I recommend Autotools Mythbuster[1] for a nice overview of how to use modern versions of autotools, including maintainer mode[2].

[1] https://autotools.io/

[2] https://autotools.io/automake/maintainer.html

To the newcomer, autoconf looks really scary. Are there any good tutorials and are there clear advantages of using autoconf over CMake, waf, scons, ninja or other hipster-language-build-systems?

I'm going to start looking at CMake to build HTML and javascript based projects soon.

I consider myself rather adept with Makefiles for Gnu make, and I also find autoconf scary (I'll be checking out the "autotools mythbuster" linked above though).

I'd guess that autotools and cmake have a lot of complexity to do C/C++/libs related stuff that is inapplicable to html/js projects, and I'd suggest almost any of the others: waf, scons, plain make ...

ninja, as you've probably heard, is designed to be the backend to some other frontend which generates all the explicit dependencies and commands. That could be cmake, it could be a custom script of yours. Then ninja identifies changed files and runs an "incremental" build as fast (parallel and minimal) as possible.

Just btw, I wrote a from-first-principles sort of tutorial on Gnu make: http://www.ploxiln.net/make.html

I found Klemens' _21st_Century_C_ has a decent intro to basic autoconf setup.

CMake has been a huge improvement over autoconf and over visual studio projects/solutions for me. I have a repeatable and multiplatform build system with CMake. Autoconf seems like the ugly and old bastard brother of CMake in comparison, and that's praising autoconf.

However I don't know if CMake is really the best fit to build html projects. In fact, I don't know what should be built about them... my composer scripts barely copy some JavaScript and CSS files to the public/assets folder, nothing fancy like what CMake has to do.

To add to this, this is the purpose of the "maintainer" targets generated by automake. To fully clean an automake project so everything is rebuilt, you want "make maintainer-clean".

Running the more common "clean" or "distclean" targets leaves not only the generated Makefile.in files, but also generated source like the generated .c/.h files from a lex/yacc parser. This is important behavior; while C compilers are available everywhere, having bison and flex (or ANTLR, or any other less-common code generation tool) installed isn't as common.

Another issue with GitHub's automated release tarballs is that they do not include submodules. Though, I've also found that submodules are often overlooked when cloning.

Unrelated, but this reminds me of 10 years ago and the horrors of running

    configure && make;

And watching it implode most of the time. Running a configure script to generate a makefile to build software makes me glad for binary distributions.

I have been very happy with how will this has worked over the last few years. These days it's a rare surprise when I download software and it doesn't compile easily. Of course I much prefer to use an up-to-date binary from the package manager, if one is available.

That's largely because you still have the package manager to resort to in order to take care of most dependencies. Prior to that, you'd still have to install (a lot of) dependencies by hand. Not all of them used autotools, not all of them used standard install locations by default... it was pretty atrocious.

In fact, I remember that one thing which kept me on Debian for a long time was that it had so many available packages. Back then Debian was even slower than today and many of those packages were a little old, but it didn't really matter to someone who was just discovering things.

It's easy to forget about them, but package maintainers are incredibly important in today's Linux (and *nix-inspired in general) world. There's a lot of hard work behind the fact that you can just apt-get whatever new program you found out about.

What I find hardest (as a manual installer) is those packages which were designed with a package manager in mind and end up depending on many third party packages rather than incorporating the needed functionality.

As an example, program X might need to compute a statistic like the skewness of a distribution. A stats package might implement the skewness calculation, so the author of X adds the dependency and calls the right function.

However, building the stats package might require a Fortran compiler and some matrix libraries, since the stats package does much more than compute skewness.

Another approach would be for the author of X to incorporate the few tens of lines needed for the skewness as part of X. This is of course more work for the author X, so I can well understand the desire to simply declare it as a dependency.

To provide a little more information: the standard Makefile target for generating tarballs is `dist` (which all GNU projects should implement). For those curious, go take a look at some of your favorite projects that implement this target and see what the resulting tarball may result in, versus the contents of the repository.

https://www.gnu.org/prep/standards/standards.html#Standard-T...

I got bitten by this a couple of days ago - I wrongly assumed that tags on GitHub were identical to the tarballs offered on the project's website. Not so much, as it turns out: https://github.com/Tarsnap/spiped/issues/26

My comment from yesterdays post on this topic:

If you're willing to hand-wave away certain things as being outside of the "center of the open-source software-release ritual", then you can get a 'yes'.

But in addition to source-distributions mentioned in the essay, which are "too tiny a minority", two other issues are:

1) If you define "git" as "pervasive" then you can declare that git is the solution. In my field, some of the tools are still only accessible via tarball/zip. Two examples are the IUPAC InChI distribution, from http://www.iupac.org/home/publications/e-resources/inchi/dow... , and VMD from http://www.ks.uiuc.edu/Development/Download/download.cgi?Pac... . PyMol, a more popular free software visualization package, is also not developed using git.

2) If you define "open-source software-release" to exclude fee-based free software distribution (perhaps it is also "too tiny a minority"?), then it preferentially excludes certain flavors of 'open-source software'. For example, I write free software, and distribute it for money. A tarball is an easy way to identify the delivery of the contractually obligated product. I can also send it via email, vs. setting up a private git server and getting accounts set up for my customers.

How much free software development is part of the hidden world of non-publicly accessible development? I don't think anyone really knows.

So if your free software baseline assumes "pervasive git", a public development repo, and no cost to access the code, then sure, the answer is a "yes". Until then, it's a "no."

And my examples show that baseline is only a subset of free software development, with no clear idea of how large it is.

Another place where tarball/zip is significantly more viable than Git: the browser (and Chromebooks as such). I know there are things like Tim Caswell's JSGit, but they're woefully unfinished and assembly-required compared to an interface like JSZip.

I am strongly in favour of tarballs for releases. Git tags can be moved, for example, so that's an extra thing to be careful about, if one is worried about reproducibility.

http://yakking.branchable.com/posts/releases/

That's my write-up on the topic of making releases, from about a month ago.

Thanks for posting your blog post link. Having only published static websites, I've been curious about the release process and your notes provide some insight, especially the parts about signatures.

Tarballs versus DVCS appears to be a false dilemma. They're independent mechanisms. The more proper question to ask here is "Are tarballs obsolete for distributing source code?".

That's what the article is asking. Perhaps a misleading title.

Correct me if I'm wrong but doesn't a git clone involve downloading the project's entire git history as well as $CURRENT_VERSION? That seems a little excessive.

You can use `--depth=1` to only download one revision, and `--branch` to specify an alternate branch or tag.

As far as I know though you can't grab a specific revision this way (I'd love to be shown otherwise) - which is an issue when setting up dockerfiles since you want to bake the sha hash which works into the file.

    git clone --depth=1

will get you just $CURRENT_VERSION

By default, yes. But you can use the --depth option to limit it to a clone of the most recent state.

While I disagree with the authors sentiment as a whole; this issue could be solved by using a shallow clone.

Ironically git actually supports TAR for this exact thing: git archive exports tar files.

You can do a low depth to minimize that if you want (--depth=1 or something like that if I remember correctly). You could just rm -rf the root .git directory after but that is a download bloat issue.

That said this should work:

    git clone --depth=1 --branch=master git://myrepo && rm -rf myrepo/.git`

but.. yeah. Not really great.

And what if you find a bug, and want to regress to a previous version?

What is so cumbersome about downloading a project's history?

Well, some projects have been around for a while. "All the history" can be a LOT of history.

Actually, even for the Linux kernel, it's only a few hundred megabyte.

"Only" a few hundred megabyte is "really a lot" in many situations.

My bare Linux repository comes to 1.2GB. A git gc run got it down to 1.1GB.

Oh, true. I just checked. I think I mixed up two numbers.

The article suggests that this is not a big deal anymore now that computers have more storage and networks have more bandwidth.

There are still a lot of networks that don't have enough bandwidth to make git trivial. Most home networks, it's going to turn a 5-second download into 60 seconds or more. Some networks, it's going to turn a 1-minute download into a half an hour.

Sure, but on the other hand, downloading a tarball for every minor release means you're redownloading a lot of code. Depending on how much churn there is, you may save bandwidth in the long run by keeping a clone and updating as you go.

Absolutely not; there are a myriad of reasons to prefer tarballs over Git:

- Offline use - Permission restrictions (Internal git networks) - True-persistent versioning (Can always delete and update a tag), no external tools needed to download - Can restrict extra garbage that may not be need to be shipped - Etc.

> Offline use

Tarballs are just as offline as git repositories...

> Permission restrictions (Internal git networks)

What?

> True-persistent versioning (Can always delete and update a tag)

Why is that of concern regarding "tarballs vs git"? If anything it's a win for git.

> Can restrict extra garbage that may not be need to be shipped

Only valid point you presented, and it's better discussed elsewhere in this thread.

>Tarballs are just as offline as git repositories...

Tarballs are easier to shoenet into an offline machine. To do this with git repositories, you would either need to deal with transfering a directory structure or, more likely, pack the respository into some sort of tarball.

`git bundle create` will do what you want.

> Why is that of concern regarding "tarballs vs git"? If anything it's a win for git.

Git tags (and branches) are mutable. Tarballs technically are, but don't move around as much. Unless someone's being actively malicious, you can be confident that no-one's going to put new code in an old-versioned tarball.

The exact same can be said of git. You're reaching. Git has builtin verification of absolutely everything it holds (tags included) since it uses sha1 for objects and commit messages, and they include the previous state... so a point in a git tree in that way is a hell of a lot more immutable than a tarball.

My colleagues complain of people moving tags and sometimes branches around, or fucking things up with a bad merge. I haven't heard the same criticisms of tarballs.

I'm not a tarball warrior, I'm just going off what I hear people saying.

You haven't heard the same criticisms of tarballs because tarballs aren't used as a version control system. These issues are specific to (fairly bad) VCS usage, not packaging usage.

Yeah, except you can change a tag and it is still verified.

The problem tarballs solve is bundling files together. That use-case will probably be with us for awhile even if it has fallen out of favor for source code.

The article is discussing whether it is worth to provide tarballs for software releases, not tarballs in general. The title might be misleading.

I like tarballs. Speaking as a non-programmer who compiles things like Vim and Scrypt, it is nice to be able to download a release without having to install a VCS first.

Tarballs are also easier to attach to an email or to share with other computers :)

The article seems to mean "tarballs are obsolete for source distribution", not obsolete on the whole.

> Here’s an advantage of the clone/pull distribution system; every clone is implicitly validated by its SHA1 hash chain.

Given that TLS does not consider SHA1 secure[1] anymore… I'm not sure that's an assumption I'd be making.

[1] in the sense that it's being rapidly deprecated; even if it doesn't trigger validation failures today, the fact that it will tomorrow is telling.

SHA1 is still good as a checksum against random errors.

SHA-1 is clearly imperfect, but not that big a problem here. The writing is on the wall for SHA-1's collision resistance. A break in collision resistance would allow a malicious software maintainer to distribute different tarballs to people of interest than to the internet at large, but would not allow a third party to replace a honestly-generated tarball with some other tarball ("second pre-image").

Since you almost certainly need to trust the maintainer not to insert backdoors anyway (a well-hidden backdoor will not be detected by the normal packaging process), trusting the maintainer not to maliciously distribute different software to a small number of people of interest isn't that unreasonable.

Tarballs can have empty directory

I love tarballs. They got a whole lot more enjoyable with all the paralell compression utilities out there. My favourite is for a file format that has fallen out of grace (bzip2) [1]. It is fast enough to make IO the bottleneck on my HDD (although not on my ssd).

1:http://compression.ca/pbzip2/

tarballs which are also usually compressed (gz, bz2, 7z, xz) may have the benefit of being more efficient to download over bandwidth constraint or size constrained links. Also being 1 file, it's only one request and it works over zmodem or modern derivatives.

Seen how any filesystem will slow down when dealing with many individual files, i'd say no.

Consider using core OS tools (wget/fetch, tar, etc.) versus the dependency tree required to run git on your system. Do you need curl? Do you need perl? Do you need a XML parser and a regex runtime engine?

Step 1: gems, npm, pip, etc. Step 2: "containers". Step 3: this.

No. Source: Veritas NetBackup.

docker brought back the frenzy of delivering software in a tarball just like slackware in the past. Long live patrick volkerdig and tar -zxvf ... -C / !

Pro Tip: You don't need the dash. That's how old-school tar is!

Even though Docker's disk and transport format are technically tar, I think that this is still a wrong statement.

tarballs are ubiquitous already. Docker actively hides that it uses tarballs (it abstracts away all interactions with them, to the point that it really is an implementation detail excluding docker export/import).

Reply on news.ycombinator.com