Sounds like Latch is just garbage. Our August locks consist of me talking to the watch (or phone, if that's your thing): "hey dingus, unlock the front door". I can do that after a run when I'm still 50 feet away, or whenever is convenient for me to do so. Of course, leaving the house is just the opposite, me shouting at HomePod on the way out the door: "hey dingus, lock the front door". And before I'm out the front gate, I hear the chime confirming that the door locked.
Latch sounds like they just turned your phone into a really shitty physical key without the convenience of an actual twist-it-in-the-hole key physical key.