Does your solution assumes referrer will not be manipulated on the client side?
Of course, anyone can code their own browser to lie about headers. It doesn't make much sense to specifically open yourself to vulnerabilities though.
There's downside, though - you can't inspect JSONs by simply opening them in a new tab.