Yes, we've all seen the xkcd[1] but you've misunderstood it. Physics applies mathematics but mathematics cannot derive physics in the way that a complete physics (and a lot of compute) could derive chemistry and biochemistry.
Math isn't attempting to describe a physical universe. It provides the substrate upon which such a description can be expressed and validated - found to be consistent with itself - but many valid descriptions do not describe our universe. Physics is the empirical search for the correct mathematical description of our universe.