If it's all physics, where does the math to describe it come from?

For Tegmark, that's enough to argue reality is every possible universe of mathematics.

We should distinguish the field of physics, which is a human study, from the world, which it attempts to map or model on a fundamental level. We can say biology and chemistry are made up of the fundamental stuff of the world which physics seeks to understand, but that's different from saying chemistry and biology are just the science of physics. We have different domains for a reason. How that cashes out in the world itself are metaphysical questions of ontology, emergence, reductionism, mereology and Platonism/universals. Also the status of causality and laws of nature.

But then again, philosophy is also a domain of human inquiry. The world is just whatever it is, however we think it best to describe. Problem is that our different domains of descriptions and questions don't always fit easily with one another. So to say it's all just the domain of physics is to mistake one map for the territory.