While they're definitely trying out some novel ideas in the CI world, it was startlingly slow. Using as much parallelism as possible on every platform; CircleCI could do this pipeline in ~5 minutes, BuildKite (what we currently use) hosted on GCP n1-standard-1 instances, could do it in about 6, and Github Actions took closer to 10. I believe the main reason why Circle was faster was that I took some extra time to set up their native cacheing systems for things like npm dependencies and build outputs between steps; we don't have this set up on BuildKite, and that might shave off a few seconds (though, I haven't looked into it, but if we're forced to egress these caches from a BuildKite DC, say in AWS, to our agents in GCP, maybe it wouldn't save any time. The tight integration of Circle is a big advantage here).
My theory is that the big slowdown is the downloading of docker images; not necessarily the beefiness of each build container.
On Circle, we avoided the use of docker and went straight native, except in the final steps where we're actually building an image of course.
On BuildKite, we're actively encouraged to use Docker to isolate builds. Given a single agent will be shared between multiple repositories, this makes total sense. But, the agents are still durable; if a tslint step starts with a pull of the node:carbon image, that image will be cached on each machine that has ever ran the build, practically indefinitely (until the machines are recycled). So the P99 off each pipeline can be higher than that 6 minute mark, but the vast majority of builds will go much quicker.
On GHA, every step needs to be in a docker image (which I like!), but there doesn't appear to be any local image caching in the build containers. If they can add this in some capacity, I'd bet the builds would be much faster. Arguably; they know at any time what images may be needed for a given repository, because the pipeline steps are defined statically in the repository, so they could feasibly keep those cached in a pool of workers that are then grabbed for that repo. Plus, I'd bet over 50% of builds start with the top 50 most popular images on docker hub (alpine, ubuntu, node, python, etc). There's no reason why every agent can't have those pre-downloaded and ready to go, beyond cost.
Excited to see the product improve; I could easily see us switching and paying for it one day.