Docker is a lot slower on OSX. If you switch to ARM images that helps (especially stability).

A bigger impact is using the virtiofs (beta) file system mapping/sync. The existing one is horrendously slow to the point of being unusable.