For whatever reason people are often tempted to victimize themselves and assume that everyone has it better than them. Which is an incredibly slippery and dangerous slope(looking at the news over the last few years). And with that in mind, "Factfulness" by Hans Rosling is a __MUST__ imo. Once you wrap your head around the facts, a next good choice is "The Black Swan" and "Antifragile" by Nassim Nicholas Taleb, despite some (to a certain extent justified) bashing on tech people. Note to self: need to pick up "Skin in the Game".

Completely with you about Spivak, as far as calculus goes.

Physics: recently picked up Walter Lewin's "For the Love of Physics" and it's a masterpiece. Didn't get the chance to finish it because of the pandemic and it got locked in the office but it appears he's managed to cram in an entire university course in one book.

Biology and anatomy - "The Selfish Gene" by Richard Dawkins is brilliant entry point for people with limited knowledge on the subject.

Chemistry - no idea, that's the one subject which I hated with a passion since I was a child. Very paradoxical, given that physics was arguably my favorite subject ¯\_(ツ)_/¯...

History - Yuval Noah Harari's books, though somewhat anecdotal as far as history is concerned. I'd say there are way too many to list here and there is way too much to read about all major events in history to fit in just a few books.