All the criticism of cryptographic agility that I have seen has involved an attacker negotiating a downgrade to a broken protocol. But if the protocol is not yet broken, then being agile isn't a concern, and if/when the protocol does become broken, then you can remove support for the broken protocol, which is what you'd be forced to do anyway, so a flexible approach just seems like a more gradual way to achieve that future transition.

Being distrustful of untested post-quantum algorithms is fair. But you can always double-encrypt a message with both classical and post-quantum algorithms, which requires an attacker to break both of them in order to decipher it. This guards against both short-term algorithmic weaknesses in our nascent post-quantum algorithms, as well as long-term threats to classical algorithms from hypothetical quantum computers.

Key sizes are larger, yes, but not show-stoppingly large; on par with secure RSA key sizes, which is tolerable. Kilobytes, not megabytes.