Road to OCIv2 Images: What's Wrong with Tar?

I think articles like this are important... we need a good way of collecting knoweldge (and especially frustrations) like this in a compact format. Everyone who thinks, "I'll use tar," in response to some problem involving archives, backups, package distribution, container distribution, etc. will at some point run head-first into one of the things that makes tar weird. Without good collections of complaints, somebody comes in to solve the problem and ends up only solving the corner of it that they personally care about.

Something I personally find frustrating is that I'd like to be able to create a tar archive without recording UIDs, usernames, or timestamps. The command-line tools aren't really built for that, and yet this seems pretty reasonable, and libraries which create tar files do this by default. (This is covered under the reproducible builds section.)

The next thing that you're likely to want is some kind of random-access archive, for which you'll use either zip (which sticks the directory at the end) or squashfs, maybe.

Finally, the last thing that I often want is just a very binary key-value format which I can distribute and randomly access with reasonable efficiency.

Looking at things from the opposite perspective, I'm personally frustrated by the cross-platform personal backup options. Currently, I'm using Duplicity or some variant, but I find it a bit of and as a result my backup schedule is rather sporadic.

It may sound esoteric but this stuff is actually Super Damn Important, for at least two big reasons:

1. Security. Right now it is hard to trust a container image. Determining provenance and/or reproducibility are hampered by the image format and especially hampered by the charming little quirks of tar.

2. Performance. There are lots of optimisations that would be available if we didn't need to stream out an entire layer before we could do anything use with it. The tar family of formats is, as their name suggests, all about creating linear files intended to be saved to a tape. On a tape random access is bonkers. For a container, random access is a frequent operation. On top of that there's the whole hassle of shipping almost-but-not-quite-identical layers over and over and over. There must be petabytes of wasted bandwidth worldwide by now.

I'm curious to hear what solution they are proposing for this, especially since there are at least two existing proposals in that field, in the form of OStree and casync: https://github.com/ostreedev/ostree https://github.com/systemd/casync/

Using Zip archives on Unix doesn't feel right, but it can solve a bunch of these problems straight away, while remaining highly portable.

A better solution might be to just emulate a block device with a sparse file and mount it somewhere with a loopback device.

As someone who was not programming when many of the customizations introduced in the article were introduced: this seems like a cautionary tale that goes against the wisdom about "competing standards": https://xkcd.com/927/

The desire to unify formats' competing back-compatibility needs created something that was (because of the standards conflicts/schisms/reunifications) extremely sub-par for most use cases, but (because of the time spent baking common interfaces) just usable enough that it remained the primary basis for storage formats for, if the author is to believed, far longer than it should have.

I wonder how many other tools that are venerated because of their age and ubiquity are similarly decrepit and broken when you peel back the curtain?

Coincidentally, I recently submitted an article by Thomas E. Dickey on the subject of tar and portability.

* https://news.ycombinator.com/item?id=18969290

The hoo-ha over tar and cpio in the 1980s when the subject of standardization came up, and the whole raison d'être of pax, is largely forgotten nowadays. But the problems that were brought to light back then live on.

The thing to bear in mind is that we already have attempts to improve upon this. The problematic formats in the 1980s were themselves products of the second-system effect, adding on various things to the original tape archive format. We are now, in the second decade of the 21st century, well into umpteenth-system effect.

This brings up a point where this article is very wrong. The pax utility did not appear in 2001. pax was in the POSIX draft back in the early 1990s, and had become widespread enough to be in reference books by 1991. The article conflates the PAX extensions with the pax utility. This, and the incorrect dates, are errors promulgated by Wikipedia, which I challenged back in 2018 at https://en.wikipedia.org/wiki/Talk:Pax_(Unix)#Incomplete_Inf... and which was probably the source of these errors in the article. Always double-check what Wikipedia says on computing topics with a proper reference.

Before being tempted to reinvent the wheel here, I recommend a look at all of the times during the so-called "archiver wars" where this wheel already was reinvented, and learning from them. Of particular note, given this article, is Rahul Dhesi's ZOO file format from 1986. It allowed for multiple generations of a given file, and archive headers marked deleted files with a flag (allowing them to be undeleted), which could be used for "whiteouts". It suffers far less from an extension mess because support for filesystem features from Unix, MS-DOS, and VMS (at least as they were in 1986) was provided in the base data strutures; including 8.3 and long filenames.

But really the basic error here is in using an off-line archive format for an on-line live filesystem mechanism. It's the wrong data structure for the job. Whereas there are right data structures, and have been for years. The history of filesystem formats development includes several cases of addressing the very things mentioned in the article, from deduplication (c.f. ZFS) through generations for deleted files (ODS for Files-11 in VMS, where ZOO got the idea from) to reproducible directory scan orders (c.f. the side-effects of using B-trees in HPFS).

Reminds me a bit of “PSD is not my favorite file format”[0]

0: https://stackoverflow.com/questions/5355708/psd-file-format#...