The way I understand IPFS, the address is based on the sha-256 hash of the content. If no one wants to host your object, or even if the object disappears from the network, you can put it back. Because it is based on a cryptographic hash of the content, no one can reuse the address for anything else, and putting it back won't change the address. This is unlike HTTP where you can lose your domain name, provider or change the referenced content.
So as long as a single person has your object it will not disappear, and as long as a single person is willing to host it, it will stay available, that person can be the owner of the NFT.