Except an attacker can strip a referer header: if you fail open like that, you leave yourself open to attack.

See http://blog.kotowicz.net/2011/10/stripping-referrer-for-fun-... for examples