The important distinction between math and metaphysics is that the substantive content of the field of mathematics does not turn on whether mathematical realism is true. Mathematics tells us something about the world by telling us what can be true about the world. We know that there are prime numbers because we know that no such world exists where prime numbers are finite. But this is just saying that a system with certain axioms have certain relationships, or necessarily exclude certain relationships (e.g. whatever axioms of arithmetic you take).
On the other hand, there is no substantive content in metaphysics (at least as far as ontology goes) if the statements do not pick out real entities.