Updating causing a rebuild isn't really a Nix or even a build tool problem though, is it? cache.nixos.org can't provide everything for every architecture all the time.
With multiple types of machines involved, you want some sort of binary cache setup, ideally automated with CI. There's niks3 [0], celler [1] (a more active fork of attic [2]), hydra [3].
0: https://github.com/Mic92/niks3
1: https://github.com/celler-cache/celler