Philosophical problems regarding the fundamental nature of reality aside, this short clip is relevant to your question:
> https://www.youtube.com/watch?v=tCUK2zRTcOc
Translated transcript:
Physics is a "Real Science". It deals with reality. Math is a structural science. It deals with the structure of thinking. These structures do not have to exist. They can exist, but they don't have to. That's a fundamental difference. The translation of mathematical concepts to reality is highly critical, I would say. You cannot just translate it directly, because this leads to such strange questions like "what would happen if we take the law of gravitation by old Newton and let r^2 go to zero?". Well, you can't! Because Heisenberg is standing down there.