The thing is, the universe and mathematics are totally separate entities. The universe did not start off "as math", nor is it defined by it.

We, as clever little beings found similarities and analogies between the universe and mathematics and leverage those to make conjecture regarding the form and nature of the universe. However, having a "complete" knowledge of either mathematics or the physical universe does not confer complete knowledge of the other.