Hacker News Reader

Often new things are met with an excess of skepticism but I agree here.

I'd take this more seriously if the format was documented at all, but so far it appears to be "this implementation relies on sqlite and zstd therefore it's better", without even a specification of the sql schema, let alone anything else.

The github repo contains precompiled binaries of zstd and sqlite. The sqlite builds appear to have thread support disabled so not only will it be single writer it'll be single reader too.

The schema is missing strictly typed tables, and the implementation appears to lack explicit collation handling for names and content.

The described benchmark appears to involve files with an average size of 16KB. I suspect it was executed on Windows on top of NTFS with an AV package running, which is a pathological case for single threaded use of the POSIXy IO APIs that undoubtedly most of the selected implementations use.

It's slightly odd that it appears to perform better when SQLite is being built with thread safety disabled (https://github.com/PackOrganization/Pack/blob/main/Libraries...) and yet the implementation is inserting in a thread group: https://github.com/PackOrganization/Pack/blob/main/Source/Dr.... I suspect the answer here is that because the implementation is using a thread group to read files and compress chunks, it's amortizing the slow cost of file opens in this benchmark using threading, but is heavily constrained by the sqlite locking - and the compression ratio will take a substantial hit in some cases as a result of the limited range of each compression operation. I suspect that zstd(1) with -T0 would outperform this for speed and compression ratio, and it's already installed on a lot of systems - even Windows 11 gained native support for .zst files recently.

The premise that we could do with something more portable than TAR and with less baggage is somewhat reasonable - we probably could do with a simple, safe format. There are a lot more key considerations to making such a format good, such as many you outline, such as choices around seeking, syncing, incremental updates, compression efficiency, parallelism, etc. There is no single set of trade-offs to cover all cases but it would be possible to make a file format that can be shared among them, while constraining the design somewhat for safety and ease of portability.

Hello, and thank you for the notes. Unfortunately, your points seem to be mostly wrong, so let me clarify them a little. Do not worry; many people misunderstand SQLite and its abilities.

- Single implementation: Sure. Working with SQLite convinced me that nobody cared to reimplement it, as it worked so well that nobody wanted or needed to rework it. I may write an unpacker just to prove that it is not hard at all to read SQLite format. The complicated part is the SQL engine (and many other features that are not used in Pack), and for Pack, you can live without it.

- SQLite does not require a disk. It has a memory option. Pack can have piping and properly will. I did not implement it because, well, it is too new, and I do what I feel is needed first. You can subscribe to the newsletter on the site (https://pack.ac/notes) or follow GitHub.

- Of course you can read SQLite without reading the whole file. It is a database, not a tar file.

- SQLite is highly optimized to read the lowest amount of data, and it has layers of smart caching. There is a reason it is used on almost any device that has a computer on it, even smartwatches.

- Of course the archive is safe from changes in unpacking. It will be opened in read-only mode, guarded by the OS and file system, and Pack also uses code isolation, which prevents calling write on any file.

- There are a lot of tools that help repair a damaged SQLite file. Pack too is guarded with transactions. The file will not get corrupted unless the disk goes corrupt; the mentioned tools come handy then. And in today's world of SSDs, the risk is shrinking rapidly.

- On unpack, Pack reads, decompresses, checks, and writes in a multithreaded. So yes, parallel reading is possible and done in Pack.

- I suggest trying Pack for yourself. It gives you the feeling you need to have to be sure.

I am very curious how you are going to get SQLite working with piping, especially on extract... It's pretty common to do stuff like "curl ... | tar xvf -", so that you can start extraction the moment first kilobyte of data arrives. This really saves a lot of time, as disk + network work in parallel.

A less common tar's feature is packing on compress -- stuff like "ssh remote tar cvf - ... > local-file.tar", which skips temporary file on remote machine, and also saves lots of time in transfer.

But for both of those, sqlite's "memory" won't help you there - memory or not, you still need to have the entire file to read it. So if you just store file contents in the sql database, then you have to fetch everything up to the latest byte before you can get any data out.

Maybe you can have index in sqlite, and append data as-is... but where would you put that index?

if you put it in front (like squashfs), you need to produce entire metadata before writing first data byte.. and that should include compressed sizes too (assuming you want to support random extraction), which means you cannot stream file out until you finish compressing all the data. And also sometimes you will not be able to add files to the archive without rewriting the whole archive (if the index grows and you didn't leave enough padding). This might be OK, but definitely should be mentioned.

If you put it at the end (like zip), you will be able to stream file out during compression, but fast decompression would be impossible. Also, you'll forego any sqlite transitional guarantees - since the database will be created in-memory, and only written at the very end once all the files are written.

So frankly, I don't see how you can win on a streaming front, unless you really have a custom format and "sqlite3" is just a small part of it.

(Another problem is there is not even a short spec - how is sqlite3 used, what is your schema, and so on. And I am sorry, but I am not going to read the source code just to figure this stuff out).

> SQlite requires real on-disk file

You can run SQLite with an In Memory database, I use it quite a lot for unit tests.

https://www.sqlite.org/inmemorydb.html

I think that point is about it not supporting streaming compression where the output of the packer is immediately fed into something like a pipe or a TCP connection.

You can do this with both tar and ZIP. If all you have is SQLite, you need to fully create the local database (be it in a file or in memory) before you can transmit it somewhere else to be stored or unpacked.

Yes, but I think they meant something like this:

pack archive | ssh host2 pack extract

Which is a weird gripe - in this use case, `tar` makes most sense to use. It's not like this format claims to be a full replacement for tar or anything close to that.

That's exactly what they do - they explicitly call out this format as replacement to tar and zip at least:

> Most popular solutions like Zip, gzip, tar, RAR, or 7-Zip are near or more than three decades old. While holding value for such a long time (in the computer world) is testimony to their greatness, the work is far from done.

> Pack tries to continue this path as the next step and proposes the next universal choice in the field.

Oh, for some reason I thought they promote it as archival container for situations with a lot of files.

Yes, I am. Pack can hold millions of files with no problem. One field in which it shines is that, aside from being fast at processing large amounts of data, it can process many small files much faster than similar tools or even many popular file systems.

About piping, it can be done, and it is on my list. I will finish features based on their popularity and making sense.

As Pack has random access support, you can choose a file in a big pack, and it can stream it out to the output. It is already able to unpack partially to your file system (using --include="file path in pack"); streaming/piping it would not be a problem.

> No parallel reads from multiple threads

Sqlite supports parallel reads from multiple threads.

It even supports parallel reads and writes from multiple threads.

What it doesn't really support is parallel reads and writes from multiple processes.

Not when threading is disabled, as it is in this project.

It is not disabled, as you think. It is secured by the internal code in Pack. Almost all parts of Pack are multithreaded.

it’s not safe to disable SQLite’s thread safety as you do here: https://github.com/PackOrganization/Pack/blob/main/Libraries... and then do your own locking. You attempt to pass the flag at open time to enable serialized mode, however quoting the SQLite docs for the build flag you set:

  Note that when SQLite is compiled with SQLITE_THREADSAFE=0, the code to make SQLite threadsafe is omitted from the build. When this occurs, it is impossible to change the threading mode at start-time or run-time.

SQLite’s APIs are often hazardous in these ways, it really should be erroring rather than silently ignoring the fullmutex flag, but alas.

Sure, but single threading isn't an inherent part of sqlite as OP implies.

> SQlite3 automatically means:

...

> - No parallel reads from multiple threads

Ad 2: SQLite has an in-memory DB option.

Then you cannot transfer through a pipe something that's too big to fit in memory.

Pack may not be it, but it would be nice if Tar would go the way of the dodo. It has all the flaws that you mentioned (and more!).

It does not in fact have most of the mentioned flaws. It's pipeable, immutable, continuous, trivial to repair, safe for append-only writing.

It literally has none of the issues mentioned though. Not that it doesn’t have limitations but those listed aren’t them.