For a number like 2 to exist, the implication is that there are two things in the universe that can be exactly equal.
Even if there are two or more of something (not a settled question in physics), the idea that you can add two such groups of two and have it be equivalent to some other group of four of that thing is only an abstraction in our heads.
In reality, if you have two apples in two pockets and I have four in mine, all we can say is that there are eight regions of the universe we like to call “apples”. 2+2=4 isn’t true in the same way that Sherlock Holmes’s address is because the former depends on a shared illusion/set of abstractions.