How much slower?
IIRC the Docker implementation for OS X runs a tiny, carefully-configured Linux VM transparently in the background. This isn't that dissimilar from the Windows Subsystem for Linux from a user standpoint.
I've done plenty of Docker work on OS X; the only real weirdness is that you're running Linux in the containers and OS X outside. Eventually, though, before switching completely to a native Linux desktop, I moved deeper into the containers and would barely ever see the OS X command prompt any more.