When a third-party loses your password, you change it and life goes on (you are not reusing passwords, are you?). When a third-party loses your fingerprint, what do you do? Cut off your fingers? Grow a new hand?
Yes. The fingerprint information is supposed to be irreversibly hashed. But so are passwords. And yet we still see them being stored unencrypted. No. I'm very careful with my fingerprints.
Also, the data on my phone must be some of the most valuable data I have. When I lose my phone and you gain access, you immediately own all my online identities. My phone has SSH keys, it has the password to my Email account stored in its keychain (very handy for all these "I have forgotten my password" features on all the sites), it has my 2 factor authentication tokens - I can't even begin to imagine how bad it would be if somebody had access to my phone.
Yours is probably as valuable to you.