Why bother with the safety checker if the model is running locally? I wonder how much faster it would be if the safety checks were skipped.
edit: I tried it out by copying this pipeline file locally and then disabling the safety checker. https://raw.githubusercontent.com/huggingface/diffusers/main...
On my M1 macbook, did a test of 10 images, including the one-off loading time. With checker: 10.51s, without safety checker: 9.48s. So not that big of a hit.
Doing a search for "nsfw" in all subdirectories seems to turn up all the files you need to edit.