Mathematicians do care about how much "black magic" they're invoking, and like to use simple constructions where possible (the field of reverse mathematics makes the central object of study). For example, Wiles' initial proof of Fermat's last theorem used quite exotic machinery called "inaccessible cardinals", which lie outside of ZFC. Subsequent work showed they weren't needed.
Another good example of mathematicians caring which 'house of cards' their results are built on is the search for an "elementary" proof of the prime number theorem (i.e. showing it doesn't rely on complex analysis).
Edit: here's a great related discussion on MathOverflow, bringing in analogies from CS: https://mathoverflow.net/questions/90820/set-theories-withou...