I agree, I tried going the route of having a separate server to do my dev on, and it's a pain if anything with your network changes, and when you're not at home and connected over a VPN, then you're in a worse position than you would be compared to just spinning up a VPS.
I run Docker on my M1 Macbook Pro and have had no major issues running amd64 images using the M1 specific builds for Docker.
How slow are the amd64 images compared to equivalent arm64 images?