Containers on anything other than Linux run in a Virtual Machine. Depending on the platform, you may or may not have hardware acceleration. It's either using the software emulator (slow) or hardware acceleration. On the new M architecture, I believe Docker now supports Rosetta2 emulation via Apple's Hypervisor Framework so it is near native performance.

As far as Vagrant being faster than Docker, it's possible the qemu backend it uses was hardware accelerated and for whatever reason the Docker VM wasn't? Or perhaps you were using Docker Desktop which does take up a lot of compute capacity away? Try using the docker cli without docker desktop and see if it works better for you.