Hacker News Reader

Comment by Lammy | Hacker News Reader

In addition to this resource and UK's equivalent (PRONOM/DROID, also mentioned in the linked post), I've found ArchiveTeam's wiki to be very useful for obscure file format details: http://fileformats.archiveteam.org/

The `shared-mime-info` database from freedesktop-dot-org is probably more worthy of contribution than these government-backed databases, at least in terms of number-of-end-users. New type definitions in their database will improve the entire Linux/BSD ecosystem (both desktop and server!) because it's consumed not only by fd.o's own `update-mime-database` utility but by many language-specific type-identification libraries too https://gitlab.freedesktop.org/xdg/shared-mime-info/-/blob/m...

…including (shameless plug) the new Ractor-based Ruby type library I've been working on in the wake of the `mimemagic` drama earlier this year: https://github.com/okeeblow/DistorteD/tree/NEW%E2%80%85SENSA...

View on HN

> I've found ArchiveTeam's wiki to be very useful for obscure file format details: http://fileformats.archiveteam.org/

I took a serious look at trying to contribute to the Archive Team's "just fix the file format problem" effort a while ago. I ran into the problem that they require contributions be public domain/CC0. This prohibits material derived from the vast corpus of FOSS (even the most permissively licensed stuff—e.g. MIT, BSD, etc.) Considering how rare public domain/CC0 is, essentially everything would need to be done from scratch.

You should reach out to them and let them know. I don't know what the alternative would be - are there better CC licenses out there? Dual licensing CC+MIT/BSD?

Pragmatically (but not legally speaking, INAL) you could probably get by with summarizing and linking (especially to web archive).

There are two problems that reasonably explain the decision of only accepting public domain info: licensing and provenance.

"Licensing" is hard. The "Open Specifications Promise" [1], which covers a bunch of Microsoft-designed file formats, is merely a covenant not to sue.

"Provenance" is tricky. For example, much of the knowledge of the Apple iWork formats were derived by reverse-engineering the source programs and extracting protobuf definitions. Many open source projects have freely copied from each other, making detailed analysis tricky [2].

[1] https://en.wikipedia.org/wiki/Microsoft_Open_Specification_P...

[2] https://github.com/jazzband/tablib/issues/114

> There are two problems that reasonably explain the decision

Boy do I hate these kinds of after-the-fact, speculative "explanations"—anachronistic rationalizations that come out of a mental exercise to justify what is by retroactively (and non-authoritatively) trying to invent a plausible why.

And if you know the personality and values of the progenitor of the whole thing, then you know that this better-safe-than-sorry deference to hair-splittingly nuanced IP hurdles (that in practice don't really exist) is not the MO.

Besides that, requiring public domain/CC0 doesn't solve either of those problems.

> "Provenance" is tricky. For example, much of the knowledge of the Apple iWork formats were derived by reverse-engineering the source programs and extracting protobuf definitions. Many open source projects have freely copied from each other, making detailed analysis tricky

I wonder if the Google LLC v. Oracle America, Inc. ruling helps this in any way. Defining a specification (either with public docs, RE or empirically) of closed format for the sake of reading the data or long-term archival strikes me as emphatically free-use. But I could also easily seeing it ending up patent-troll bait.

There is also Apache Tika (https://tika.apache.org/) - file format detection & content extraction library

Tika also stores its type definitions in the freedesktop format internally: https://gitbox.apache.org/repos/asf?p=tika.git;a=blob;f=tika...

However their data is mostly focused on text/document types for obvious reasons :)

Also the magic number database for guessing the format of a file:

https://www.darwinsys.com/file/

magic database has its own problems based on the language itself

What kind of problems?

Primarily because it’s very hard to express complex conditions

Reply on news.ycombinator.com