There are philosophy of math books like Joel David Hampkins. But given the breath of your question you might be better served by just a general philosophy text. Like something covering an introduction to metaphysics or even better 'philosophical logic'.

A more rigorous approach would be to study mathematics as mathematics (and not through traditional philosophy). In that case you would have to go down the path of mathematical logic, proof theory and model theory. Then after that fall into a pure theoretical CS / metamathematical rabbit hole.

A good intro to formal logic is "forallx". The key to truth in logic being the concept of tautology.

There is also the cognitive linguistics approach of focusing on the genesis of these abstractions from underlying metaphors. There is a famous book by George Lakoff on this subject.

More generally my favorite wider contextualization of the subject of reason, meaning and formalism (though it might seem distant and antiquated) is C S Peirce and his triadic approach to meaning and philosophy.

I have a very simple article on communication, reason and philosophical logic that gets across this triadic perspective to seeing these things: https://medium.com/the-sphinx/what-is-meant-by-a-reason-part...