For the specific example you provide you could also use a prompt-based mask to prevent it from editing the person.