I suspect that the M2 air is thrashing the disk pretty aggressively. Diffusion models rerun the same model once per step, so for a generation with 50 steps, you copy the entire model in and out of memory 50 times. That's going to kill performance.
I believe the model is copied into ram once upon calling StableDiffusionPipeline, unless the mac implementation partially loads the model due to only having 8G of ram.
It's only copied to VRAM once when implemented correctly.
M1 is a unified memory system and doesn't have VRAM.