I believe we are in agreement on this. Intuitively, and under the assumption that the physical world exists (although I have no clue why or how it exists), I feel that mathematics is formed inside of the physical world. The experience of being able to practice mathematics probably emerges from brain structures which have evolved in the physical world.
Still, there are some problems that I run into when taking these thoughts further. How, for instance can one apply deductive reasoning or apply Occam's razor in a context where these are not available?
I am also intrigued by your earlier remark that "math is just a model that is surprisingly applicable to the real world." (emphasis mine). This brings to mind "The Unreasonable Effectiveness of Mathematics in the Natural Sciences". Perhaps there is an easy way out for believers of anti-realism.
Would it be an interesting hypothesis to say that the real (physical) world that we observe is limited exactly by the way that our sensors and brains take shape in it? I'd like to think of this as the antithesis to "in the beginning there was nothing" -- I'd rather think that outside our physical world "there is all and everything"; we just seem to be able to reflect only on part of it. The unreasonable effectiveness of mathematics hints at a correlation between how brains work and what physical laws there are. Perhaps Emmy Noether's ideas on symmetry may lead to some clues here as well.
In this way it would not be surprising at all that mathematics is applicable to the real world, as it is so almost by definition. This is obviously not the same interpretation as Max Tegmark's, but it does hint at some kind of interplay between a mathematical world and a physical world.
Unfortunately, I can only make this theory work for myself intuitively. I have no grasp on what it means that the physical world is part of something bigger. In a way, it seems to be moving the goalposts, similar to how some people believe we are somewhere in a nested series of simulations. And I feel quite uncomfortable in using logic, concepts, abstractions and what have you, which are part of the human brain context, and possibly not of the context that I magically believe our physical world to reside in.