Some guy by the name of Hartry Field claims to have successfully generated a logical axiomitization of Newtonian mechanics, without using math, only qualitative concepts like "betweenness."[1] Math can extend this model and is consistent with it, but it's not necessary in principle. Of course, this does not mean that you don't need math in practice, in the same way that continuations are the only control structure you need in principle but in practice you use loops and function calls and switches or whatever. What it does mean, from a philosophical view, is that math isn't necessary the quintessential 'language of the universe' that we thought it was.
As far as advanced mathematical physics goes, our problem is that we stop being able to easily understand what the math represents when you go deep enough. A first derivative is understandable as velocity, the second as acceleration, and multiplying that by mass to get force has some intuitive notion. If you look in the middle of a bunch of scribblings, it's possible to put the equation into words and it will make sense and have a physical interpretation. Special relativity and the beginnings of quantum mechanics came about because we had intuitive concepts, and were able to fit math around them. In modern physics, the math has come first, and the intuition later or not at all. Quantum phenomena still have competing interpretations (probability fields, many-worlds), which may not even be mutually exclusive. Once we figure out what in the hell is going on, the math may prove to be unnecessary, but right now the math is all we have to go on, and we sort of have to treat it like a literal truth in order to get anything done.
[1] http://en.wikipedia.org/wiki/Philosophy_of_mathematics#Ficti...