You're right, of course.
Let me expand my argument a little.
We have two security domains, associated with the first and second server. The first server has sensitive URLs, the second receives these in the referrer header.
In making the first connection, the client and first server get to make some security policy decisions: the crypto used in securing the connection, the availability of countermeasures against known TLS protocol vulnerabilities, client and server authentication methods, etc. These parameters are complex, and you end up with a connection with some security level given some attacker model. If you think about it really hard, you can come up with a quantitive estimate of what security you might get from the connection in terms of attacker work -- anywhere from zero (trivially broken) to 256-bit security (very good).
Assume our first connection has a 128-bit security level and has countermeasures for TLS vulnerabilities. This is good going. Now we serve over this connection a https link, which the user clicks.
Unfortunately the client and/or second server is poorly configured and we only manage (for the sake of argument) a 40-bit security level and no TLS vulnerability countermeasures. Now we've reduced portions of the first connection to the security level offerred by the second connection. This is really very surprising.