I've had a lot of luck using Invoke.AI's solution. It allows for a lot more control over what the AI model is allowed to modify and lets you tailor each prompt for each individual edit.
Are you running on your own hardware? I've found the latest commits to be less stable and it's hard to know what to run.