Property rights would inherit. So one of their relatives or heirs. If they had no one to inherit the restriction it would go to the state - but the state would have gotten the land unrestricted in that case anyway.