With some CPU offloading I'm able to run SDXL at 1.5it/s on an RTX2070S w/ 8GB VRAM.

When used with ControlNet it still runs but more layers offloaded at 1.4s/it