Hacker News Reader

Comment by boramalper | Hacker News Reader

I'm not sure if tar is a good format for backups, due to its extremely sequential nature (e.g. you can't get a list of the files in an archive without scanning the whole archive...)

View on HN

What overhead does that add in practice? Like a second or so to scan a few GB of tarball? Does that matter?

It often is, especially if the tarball is compressed (especially if you use anything but Zstandard) or on a network store rather than fast local ssd. Typically a rule of thumb is a hard drive with a spindle may five around 100mb/sec throughput, so that is likely the bigger bottleneck, followed by cpu if using slower compression algorithms.

Zstandard at a high compression level gives a good tradeoff of decompression speed vs underlying physical throughput, be it network or spindle.

Even so, yes, the sequential nature of tar is not great for this reason.

Squashfs files, which can be used on Linux and Mac (using fuse for OS X), are a very good alternative for archIves, though without some of the features described in the linked article.

you need to download the entire thing to scan it.

if it already resides locally on a seekable media , there's litle drawback

I guess it can be cumbersome as well if there are many small* files (since you need to iterate over all of them) until you find the one you are looking for.

Think of it like a linked list.

*: As opposed to a few large files.

This issue occurred to me as well (especially for encrypted tarballs). My workaround was to create a listing of the files that accompanies the tarball, such as: "backup.tar" and "backup.list".

Reply on news.ycombinator.com