The book that "clicked" for me is: Cryptography Engineering: Design Principles and Practical Applications, by Ferguson/Schneier/Kohno (2015) [1]
In plain language it walks through what I wanted to know, in a modern and paranoid perspective, as a readable narrative, from the point of view that we want to design each of the basic crypto primitives ourselves.
[1] https://onlinelibrary.wiley.com/doi/book/10.1002/97811187223...