You raise very good points, but it's not like there aren't ways to break into a house without signs of forced entry.
To my mind, privacy is often the most important difference. There are a lot more private things on my phone than in my house, honestly -- a phone these days is often almost an extension of the mind. Which is why we should hold it to a higher standard.
I personally have no problems with TouchID/FaceID, I think the secure enclave is great, but I still think there's a lot of room for improvement, better sandboxing of data, better tripwires for when data is accessed/transferred, etc.