I believe you only lose the referer header when switching between http and https.