Do you use it on vercel, maybe?

I had the same experience of slow refreshes on the NextJS project I worked on (running locally) and the other seasoned NextJS developers didn't think it was unusual.