This seems like a major bug with crypto. I don't have physical possession of my bed when I get up from it and go to work, but I've yet to come home and discover a grifter, or conman sleeping in it.
Likewise, if I lose my car keys, society still manages to acknowledge that I am still the owner of my car.