As others have mentioned, use Docker Compose instead of vagrant. That way you don't have the overhead of learning two different systems (compose and the Docker command line are very similar).

If it takes more than 20m to get a new dev machine setup from a fresh OS, you probably haven't pushed enough of your environment into images.

Don't be afraid to ignore the rules about what and how much goes into a container as those are for production images (eg: customer facing). It's prefectly fine for this usecase to treat it more as a mini-vm than the 'process on steroids' you use for production.