Nix would let you do just this already with potential binary cache and reproducibility.

I’m surprised it’s not more widely used for embedded. Maybe due to larger images? That’s a solvable problem.

Maybe best of all it’s fast. Yocto is painfully slow.