No one believes abstracta have a physical location -- they lack physical properties. The claim "2 + 2 = 4" is true -- and clearly not true invirute of anything anyone thinks... if we kill that person (/people), it is no less true.
Indeed, if numbers don't exist (for example), do we suppose that we can't communicate issues of quantity with other species (, & possible alien life, etc. etc.) ? (If we can, what shared things are we talking about when we quantify?)
It seems deeply implausible to say that our use of number is circumstantially psychological -- any description of reality is going to be indispensably quantitative --- quantity is what we are talkng about. We are not talking about ourselves.