FWIW I do the same thing but with docker.
Exposing the docker daemon on the network and setting DOCKER_HOST I’m able to use the remote machine as if it was local.
It’s hugely beneficial, I’ve considered making mini buildfarms that load balance this connection in a deterministic way.