Set the header and call it a day, and at this point browsers should default to same-origin. The only outlier is IE 11, with 0.5% usage, and it is so grossly out of date it's pretty reasonable to just dump.
I remember in the very early Internet trying to raise an alarm that a lot of people didn't realize the privacy implications of referral headers (run a website and you could find all sorts of crazy niche discussion forums, see the resources people were referencing, etc). I certainly am not claiming prescience, but it was amazing how little anyone cared. Mind you, I also once promoted the notion that browsers should only ever send strongly, expensively domain-specific hashed passwords from password elements, and that too was pooh poohed. Several billion pwned passwords later...