Fun with File Formats
blogs.loc.gov
blogs.loc.gov
The `shared-mime-info` database from freedesktop-dot-org is probably more worthy of contribution than these government-backed databases, at least in terms of number-of-end-users. New type definitions in their database will improve the entire Linux/BSD ecosystem (both desktop and server!) because it's consumed not only by fd.o's own `update-mime-database` utility but by many language-specific type-identification libraries too https://gitlab.freedesktop.org/xdg/shared-mime-info/-/blob/m...
…including (shameless plug) the new Ractor-based Ruby type library I've been working on in the wake of the `mimemagic` drama earlier this year: https://github.com/okeeblow/DistorteD/tree/NEW%E2%80%85SENSA...
However their data is mostly focused on text/document types for obvious reasons :)
I took a serious look at trying to contribute to the Archive Team's "just fix the file format problem" effort a while ago. I ran into the problem that they require contributions be public domain/CC0. This prohibits material derived from the vast corpus of FOSS (even the most permissively licensed stuff—e.g. MIT, BSD, etc.) Considering how rare public domain/CC0 is, essentially everything would need to be done from scratch.
Pragmatically (but not legally speaking, INAL) you could probably get by with summarizing and linking (especially to web archive).
"Licensing" is hard. The "Open Specifications Promise" [1], which covers a bunch of Microsoft-designed file formats, is merely a covenant not to sue.
"Provenance" is tricky. For example, much of the knowledge of the Apple iWork formats were derived by reverse-engineering the source programs and extracting protobuf definitions. Many open source projects have freely copied from each other, making detailed analysis tricky [2].
[1] https://en.wikipedia.org/wiki/Microsoft_Open_Specification_P...
Boy do I hate these kinds of after-the-fact, speculative "explanations"—anachronistic rationalizations that come out of a mental exercise to justify what is by retroactively (and non-authoritatively) trying to invent a plausible why.
And if you know the personality and values of the progenitor of the whole thing, then you know that this better-safe-than-sorry deference to hair-splittingly nuanced IP hurdles (that in practice don't really exist) is not the MO.
Besides that, requiring public domain/CC0 doesn't solve either of those problems.
I wonder if the Google LLC v. Oracle America, Inc. ruling helps this in any way. Defining a specification (either with public docs, RE or empirically) of closed format for the sake of reading the data or long-term archival strikes me as emphatically free-use. But I could also easily seeing it ending up patent-troll bait.
Extraction of content is a separate fun ;-)
Second thing, for example it didn’t work with compound objects. Especially for things like word embedded in excel, etc. Similar for XML-based types, zip-based, …
Plus rules aren’t very flexible - I’d used lisp-like language to describe detection patterns - that made it quite flexible in describing complex detection logic.
The information contained in this document has been released into the
public domain and is not considered to be confidential or proprietary
although still the copyright and property of Lotus Development Corporation.
It would be great if all of the new format were at least documented as this one.
What I'm looking for is basically something that ticks all the boxes of LOC's 7 Sustainability Factors. Is able to store arbitrary binary data, along with metadata, in a single file. Easy to pack/unpack, particularly in Python. I'm pretty well covered for image/audio/video formats, but none of those lend themselves to n-dim arrays.
Ogg, matroska, and HDF5 are all container formats which can ostensibly do this, possibly even well. Ogg and matroska are supposed to be able to support arbitrary data steams and metadata, but actually finding tooling to do the operations I want to do has yet to bear fruit. HDF5 is insanely complex, and I have some concerns about performance and data corruption risk.
Tar is tempting but it does some really goofy things, like ending an archive with 1kB of zeros.
What actually looks most promising as an existing "format" is the RIFF family of chunked encoding. Dead simple, python stdlib even has a reader. Only real downside is 2*32 ~= 4 gigabyte size limit, though this can be overcome by using a "stream of chunks" model.
Alternatively, a stream based on Msgpack, with some slight constraints, is very appealing.
Sqlite3, also very tempting.
If I did write such a "format", it would definitely be built on the primitives of either RIFF, Msgpack, or Sqlite-as-application-file, so it'd be less of a "file format" and more of "existing format with some opinions about schema".
Protip for anyone also thinking about making their own file format: make sure the API/format supports some sort of arbitrary metadata model! Apache Feather and Parquet are pretty recent, but don't really support writing metadata, you kind of have to hack around it. IMHO, this is a huge oversight, especially considering the internal metadata is literally just JSON-encoded fields.
Edit: actually just reread the follow-up to "moving away from hdf5" and it mentions ASDF format. This looks really promising. YAML metadata header with binary blocks. Normally it's not a good idea to start with UTF8 and later go binary, but if you know when to stop reading text, it's not the worst. Also, it's a name collision with the asdf version manager (and also anything else named after rolling the left hand home keys)
https://cyrille.rossant.net/should-you-use-hdf5/
https://towardsdatascience.com/saving-metadata-with-datafram...
https://en.m.wikipedia.org/wiki/Resource_Interchange_File_Fo...
https://msgpack.org/index.html
There's a couple of squicky things, for one, it looks like the extension model may allow RCE behaviors by deserializing arbitrary classes. It looks like it uses a URI/registration based system so by default, it is limited in what types it can try to load, but it's plugin-based, so a library you import could theoretically expose you. But that seems relatively low risk.
But overall this format is really cool. Supports linking across multiple ASDF files, which I can see getting messy, but this is an incredibly powerful technique and they do it in a sane way.
- Has readers/writers/APIs in almost every language under the sun
- It's data tree is a file tree, leaving it very simple from an abstraction standpoint
- No inherent arbitrary metadata mechanism but plenty of ideas (in the wild, even) for storing arbitrary metadata in JSON or XML or YML "files" side-by-side data files
- Used in many examples in the wild (DOCX, XSLX, ODT, JAR, etc)
With low compression levels, you can actually tune things to get faster write times than raw bytes, depending on CPU speed vs IO speed. So that's cool.