I think there's different components to pitch memory, including both relative and absolute pitch. (I suspect there's more than that, too - some people seem to just hear a chord as a single unit while others pick out the individual notes in it, some people have a very good ear for timbre.)

I don't have perfect pitch, but I actually seem to have anchored on A=443Hz. When my violin teacher insisted on 440Hz it would cause me discomfort the same way you describe, just feeling wrong. And I've found that when I try to tune an instrument from memory, I'm consistently sharp. (Assuming I'm not off by a whole-tone, because I don't have perfect pitch.)