If security is really important to you though, you should move image processing to separate isolated servers anyway, and verify magic bytes as described by this site.

I was checking magic numbers before sending files to GraphicsMagick but I'm going to go through my code again and make sure that this is always happening.

The Graphicsmagick code base includes 3 of the 5 coders that are mentioned (URL, MVG and MSL). 2 of those use LibXML and url.c suspiciously uses LibXML's nanoftp and nanohttp.

  Under Unix, all text (non-numeric) substitutions should be
  surrounded with double quotes for the purpose of security, and
  because any double quotes occuring within the substituted text will
  be escaped using a backslash.

  Commands (excluding file names) containing one or more of the
  special characters ";&|><" (requiring that multiple processes be
  executed) are executed via the Unix shell with text substitutions
  carefully excaped to avoid possible compromise.  Otherwise, commands
  are executed directly without use of the Unix shell.