It would, but it would create other problems also. If I were to link to a raw.github.com URL on HN, it would be disallowed because the Referer header would be "incorrect".
Not to mention the fact that referrers are an optional part of the http spec, so requiring them is bad form to begin with.