It has all the pitfalls of putting your front door key under your doormat. Except you're doing it in a world where everyone can materialize a unique key out of thin air, and you can always instantly tell your door which of those keys should or shouldn't be allowed to open it.
sshpass is a curse, even calling it a crutch would be improper flattery. It's a dangerous cheat that accomplishes nothing except impeding people from otherwise spending the 20 minutes it takes to figure out SSH keys.