When you afford SD img2img influence with a high strength parameter, so it can change the colors of a photo, you also are letting it have enough influence to redesign the room at random and not just changing the colors
Dreambooth trained on the ground truth photos might help a bit