Obviously a solvable problem to split build and test but perhaps the time savings aren't worth the complexity.
https://src.fedoraproject.org/rpms/rpy/pull-request/4#reques...
Your question made me look up Arm's history in Fedora and came up on this 2012 LWN thread[1]. There's some discussion against cross-compilation already back then.
I personally found cross compiling Rust easy, as long as you don't have C dependencies. If you have C dependencies it becomes way harder.
This suggests that spending time to upstream cross compilation fixes would be worth it for everyone, and probably even in the C world, 20% of the packages need 80% of the effort.