Related: do people have good recommendations of "casual" math books for more advanced topics? I.e. ones that an educated reader with a background in math can read through in one pass (unlike most textbooks) but nonetheless have real math in them? (And has fun exercises!)

I can start with some. Feynman's Lectures on Computing. Scott Aaronson's Quantum Computing Since Democritus (though it assumes some background knowledge of quantum computing). I think Colin Adam's "The Knot Book" (on knot theory in topology) as well.