The good news is that Clang can cross-compile fairly easily. Much better than gcc.
The bad news is that there are a surprising number of missing libraries on Ubuntu/ARM64. For example, Eigen3. And although the code is fairly compatible, there's some extra cognitive load in learning to debug on ARM. For example, calling a virtual function on a deleted object crashes differently.
I'm willing to put up with it for ARM's advantages in battery-powered applications, but I wouldn't just to save a few bucks on cloud servers.