I write software that ports to numerous processors. It's a non-issue. Only issues I've ever seen are casts that break the C spec, and those are ugly anyway. Casts to unaligned boundaries can cause SIGBUS on ARM. Otherwise multiarch is only hard if you are doing SIMD.
If you are working in a higher level language it's a total non-issue. You will not notice.