I still however feel drawn to certain frequencies even without reference or other instrument playing and those frequencies I am drawn to are actual notes. I know 440 Hz is somewhat arbitrary sure, but 99% of the music we hear is somewhat referenced to it.
To me there is certainly a difference between playing a song with an fundamental A 440 Hz or an fundamental A# 466.16 Hz. Not even just when playing them next to each other, but also when you play them days apart. It happened more than once to me that some instrument had been detuned a semitone up or down and I noticed because the resulting music felt different. All of that of course playing solo with no reference to compare to.
Let's not forget, that our brain is not neutral towards frequencies. We can hear whether something is made of metal or wood, whether it is massive or hollow, whether it is controlled or uncontrolled. Deeper usually means bigger regardless of overtone structure. Higher can for example also mean faster (e.g. wind). Parents can hear in a slightly higher pitched noise their kid makes whether they have hurt themselves even if the noise is the same they make every day. Of people speak higher they are excited etc. A lot of that is about relative potch and overtone structures, but even if you take that away, some part that is absolute remains.
If you have a semitone of a difference in some physical vibrating object that can be a noticable difference in mass.
I am not saying transcendental statements like "key X is the most melancholic" are true, what I say is that even to those without perfect pitch hearing a semitone difference might feel different without reference (especially since our sound systems might be able to reproduce one bass fundamental better/punchier/clearer than another).