The three big differences are:
1. Docker doesn't deal with running processes (like postgres or redis), only the filesystem state
2. Docker doesn't have enough isolation, so you'd probably need to run it within qemu or firecracker for compliance in bigger teams
3. Docker-in-docker is still pretty painful, if you need to do anything nonstandard like change the size of /dev/shm, access /dev/kvm, or load kernel drivers, it'll take custom configuration.
Often times there are long-living processes which rarely change but take a long time to warm up. The Bazel [1] agent for C++ projects, the buildkit [2] state for docker, or the running Postgres or Redis server for a cloud native app for example.
It's why running "docker build" twice on your laptop is so fast, but running "docker build" in CI seems glacially slow.
> why is docker-in-docker a requirement, and how is that easier than qemu in qemu or qemu in docker or whatever?
The example given was running "docker-compose build", so you'd need either docker-in-firecracker (this post), docker-in-docker, or docker-in-qemu. You'd almost never run docker-compose build on bare metal in practice, because you'd immediately need to send the images you built somewhere else in order to use them.
[1] https://bazel.build/ [2] https://docs.docker.com/develop/develop-images/build_enhance...
I'm not experienced enough with Bazel to comment on that.
Docker in practice usually doesn't, but in theory it can and should.
A checkpoint/restore demo, of live-migrating running docker containers between servers, including running process state, was shown at Dockercon 2015. I can't find a video right now, but it was demoed then.
The "docker checkpoint" command has existed since docker 1.13.0 (released 2017), which allows checkpointing and restoring the running state of containers, including running processes, with some limitations. https://docs.docker.com/engine/reference/commandline/checkpo...
The reality is just unfortunately that these features are experimental, have been for the last 7+ years, and have taken far longer to mature than would be expected.
Interesting that you're folks now use firecracker. I assume it now fills in adequately for the previously homegrown tech at layerci [0]?
In terms of doing this in a CI env like actions where you may have different types of machines serving you, it may be problematic as the machine specs need to pretty closely match.