You might think so, but 'file' has its share of buffer overflows, integer overflows, attacker-inducible infinite loops, etc. It does some more extensive parsing for some kinds of files, and some of those end up with edge cases. Here's one from two days ago, a buffer overrun in parsing ELF files:
https://access.redhat.com/security/cve/CVE-2014-3710. Others:
https://security-tracker.debian.org/tracker/source-package/f...