Extremely aggressive optimizations are really special though, and they tend to require rewrites when new CPU extensions release (and workarounds to work on older hardware). If you rely on super low level ultra-aggressive micro optimizations your code is going to have a relatively short shelf life, different ISA or not.

The vast majority of the code written for any given computer or smartphone doesn't have this level of sophistication and optimization though. I'd wager that for most code just changing the build target will indeed mostly just work.

It won't be painless but modern code tends to be so high level and abstracted (especially on smartphones) than the underlying ISA matters a lot less than in the past.