Why? You still have a password. You now have a password you don't know, that's entered on your behalf by querying some aspect of your phone, but it's still a password in a database that can be compromised if the database is compromised.

I now need to pretend to be your phone, instead of pretending to be you, but that's not such a leap, security wise.

If some library becomes common and used by multiple sites, now you're even worse off, because now, you have the same "password" on multiple sites, and you are relying on each site salting this "password" for your security. Since you now no longer have the option of using different passwords on different sites, your security is now completely out of your hands and you are at the mercy of each person using this "mobile authentication library" to properly salt before saving it into their database.