> The particular thing that comes to mind repeatedly when reading this is the fact that more or less all of mathematics can be derived starting either from set theory or from logic theory.

I don't know if this is the actual foundation of mathematics though - we're seeing more advances in category theory, the Russell-Whitehead project of reducing mathematics to pure logic is generally considered a failure, and set theory's bogged down in issues of axioms in the wake of Cohen's proof of the indecidability of the continuum hypothesis. It's probably better to see these foundational projects as providing windows into the mathematical universe instead of being the actual substance of mathematics.

After all, we do mathematics without pure logic or sets all the time. Axioms are chosen for their elegance and ability to describe conceived mathematical concepts, not the other way around.