On the flip side, anyone who obtains the physical device likely has access to the key (your fingerprints are all over it)
Of course all this assumes a particular attacker model and for a lazy attacker that doesn't know how to use the fingerprints to fool the sensor, everything is fine.
Another downside is being possibly forced to unlock my phone at airports, since as far as I'm aware, passwords are protected by the fifth amendment whereas fingerprints are not. In this case it might be smarter to turn off one's phone entirely when crossing borders into countries where one can be compelled in this way. This doesn't even go into all the other kinds of pressure they might apply to make me unlock my phone anyway though, so there's not much protection here aside from leaving my good phone at home.