Formal methods seems to be perfect for verifying protocols, specially finding design flaws, here an example of an ongoing effort to verify TLS

https://www.mitls.org/

https://github.com/project-everest/mitls-fstar