Why not use certificates as your primary authentication for SSH? Facebook has a great blog post on implementing this at scale: https://engineering.fb.com/security/scalable-and-secure-acce...