This is something that gets repeated again and again but I don't find this puzzling at all. Mathematics is an extremely general tool for modeling all kinds of things, so if something follows some sort of rules, you most certainly can use mathematics to describe, model, and analyze it. While I am not aware of any specific results or hypotheses, I would still guess that mathematics - or even some small subset like set theory - is in some sense universal for modeling things in a way similar to universal models of computation like Turing machines or lambda calculus.
I can not even imagine a universe with laws of physics that could not be tackled with mathematics. Even if the universe was pure chaos, everything happening totally randomly without any recognizable rules, we still could try throwing statistics at it, at least ignoring the fact that such an universe would probably not be a place for life to flourish. Finally mathematics does not exist in a vacuum, we invented large parts of it specifically to deal with real world phenomenons. And then of course »some« more that does not describe anything existing in the physical world.
Somewhere deep down below there are certainly many puzzling issues, from why there is a universe at all and why it is the way it is to human consciousness and why and how we are able to make sense of the universe, but that mathematics can describe the universe does not seem to be one of those, at least to me.