You can't run x86 Linux quickly on macOS. What you can do is to run ARM Linux quickly on macOS and run x86 programs inside of that Linux with Rosetta quickly.
Am I missing something? Is this quicker just because you are emulating only the program and not the OS?
Yes. If you run x86 Linux in a VM on an M1 you are emulating/translating every single instruction; with a native OS you are only translating the application.
It is quicker because Rosetta is a proprietary Apple technology which apparently works superb, but only supports userspace code. Emulating x86 with qemu does not work as fast, unfortunately. I'm not qualified enough to judge whether it's qemu shortcoming or it's just not possible to emulate entire VM as efficiently, but the practical result for today is that you want to use Rosetta.