That's probably due to swapping due to the 8GB of RAM. People who have run Stable Diffusion on M2 airs with 16 GB of RAM seem to get performance that is in line with their GPU core count.