Hacker News Reader

Comment by mike_d | Hacker News Reader

From what I can gather they are trying to compose a database of information on what application generates/can read a specific file, primarily by file extension.

This already exists in libmagic (https://github.com/file/file) and can be used on any BSD/Linux system by typing `file [filename]`

View on HN

The goal is a bit more expansive than libmagic, it is a wiki of file format resources, including extensions, software, format specifications, related formats and more.

Yeah, it includes mechanical specifications of physical media so, say, alien archaeologists can get data off a VHS.

People are reading the title thinking it's about filesystems. It's about cataloging all human information storage methods, from phonographs to zip drives to word perfect files.

libmagic is quite primitive and can't reliably handle the complex data types, like, MS Office documents with embedded objects (especially the versions before 2007), files with common container types (jar/war files are detected just as zip), etc. - the signature language is quite primitive from my point of view. Plus it's quite slow...

I've spent a long time (> 10 years) developing code for detection of file types (without relying onto the file extension), and extraction of the data... And everything was need to be done very fast & reliably. The last project was the content detection for the filtering web proxy, where every millisecond counts. And for signatures, there was a lisp-like language that allowed to describe very complex detection rules, although it's sometime was still necessary to go to the C++, for things like, OLE2 parsing, XML parsing, or listing files in Zip files, etc.

From the open source alternatives, it makes sense to look to the Apache Tika that has both content detection & extraction capabilities, although it's written in Java...

And - in case of need - on Windows there is Trid:

https://mark0.net/soft-trid-e.html

which has also a couple online options and a database of signatures/magic numbers:

https://mark0.net/soft-trid-deflist.html

There is a Linux version too! But sadly, there is not a library which would be incredibly useful for analysing, for example, blobs in a database, or categorising random files uploaded by a user of a website.

libmagic focuses on identifying file type using magic numbers and other properties. The wiki has links to resources for programmatically extracting image size or color palette, for example.

libmagic can also extract metadata for images.

Reply on news.ycombinator.com