It's not just that. It's also that a lot of physics can readily be explained and understood without mathematics. It's just that with mathematics, it's easier to demonstrate the consequences. It's important to remember that Einstein got at relativity by understanding the right assumption to be made.
Just to be clear on this: I fully agree that in advanced physics, you sometimes you have to blindly 'do the math' and can determine the physical interpretation of the result only afterwards. However, that doesn't mean that there is no physics without mathematics, it doesn't mean that there isn't a lot of physics that can be understood without mathematics and it certainly doesn't mean nature is mathematical in essence. Hell, it isn't even physical in essence, unless you can reduce our minds to physics.
The notion that 'nature is mathematical in essence' is not logically seperated from assumptions about the object/subject and mind/matter dualities (and about whether they are dualities in the first place) that I doubt most of the phycisists espousing these notions are willing to support, because they are rather incompatible with other parts of their worldview.