My point is that physical reality cannot be made of something that doesn't exist physically, such as mathematical objects, abstract concepts or more generally ideas of any kind. Do you not agree that this must be true?
Proving the universe isn't made out of "mathematical objects" in particular is equivalent to the difficulty of proving it's not a "simulation". This is one of the red lines that tells you you've gone too far; you can't prove that. You can't even non-circularly define such a thing in this context anyhow, let alone prove anything.
If not, then whose to say whether the mathematical object is 'real' or just a perfect description? Is the difference meaningful?