Defense can win any single encounter, it cannot win every single encounter.
Perfect defense does not exist. Especially once we get down to people. Even in your scenario, those "good libraries with lemmas" are now the point of attack. Find a weakness in that and you find a weakness in thousands of systems.
And that does happen all the time. Commonly used libraries are found to have privilege escalation vulnerabilities. They get patched, then something else, somewhere else, is discovered. And sometimes the patches themselves open up other vulnerabilities.
As for social engineering being more difficult to pull off, that's not true at all. We're all looking to streamline the process, we're tired, we're sleepy, we're bored, we just don't care, we hate our boss, we just want to watch the world burn. Those are all ways we fail even the most airtight security measures.
Tell me the provably correct patch for willingly handing over my credentials to a malicious party.