I don't expect many teams to volunteer to suffer this sort of slowdown and complexity in the near term.
Since it's all BuildKit, you can swap `docker buildx build` for `depot build` and it works exactly the same - I made a depot/build-push-action to drop in place of the docker/build-push-action in GitHub Actions.
It also has a persistent SSD cache disk for each builder, that was my other pain with GitHub Actions, time saving and loading layer cache was negating the speedups from cache hits - with a persistent disk, there's no saving or loading.
Anyways, combo of having a local cache and running on real ARM machines gives like an order of magnitude speedup to builds compared to the QEMU emulation.
Still a new project, not yet officially launched, and hosted services aren't for everyone, but exactly as you said, the status quo is amazingly painful.
You can add hosted arm GitHub runners or register arm hosts for docker and see down to earth build times.