libmagic is quite primitive and can't reliably handle the complex data types, like, MS Office documents with embedded objects (especially the versions before 2007), files with common container types (jar/war files are detected just as zip), etc. - the signature language is quite primitive from my point of view. Plus it's quite slow...
I've spent a long time (> 10 years) developing code for detection of file types (without relying onto the file extension), and extraction of the data... And everything was need to be done very fast & reliably. The last project was the content detection for the filtering web proxy, where every millisecond counts. And for signatures, there was a lisp-like language that allowed to describe very complex detection rules, although it's sometime was still necessary to go to the C++, for things like, OLE2 parsing, XML parsing, or listing files in Zip files, etc.
From the open source alternatives, it makes sense to look to the Apache Tika that has both content detection & extraction capabilities, although it's written in Java...