Othar than word size (eg. i386 vs. x86_64), endianness (eg. PPC vs. x86), alignment (eg. ARM vs. x86) there's plenty of other low-level issues like parameter passing conventions, call frame handling, exception handling, presence or absence of optional instructions or registers, and a number of other subtle gotchas that could affect a port. Instruction probing is one of the ones we run into where everything builds find and fails at runtime.

If you're working at a high enough level you're probably fine. Like the difference between being an auto mechanic and using the app to hail a ride.

In my packaging work I encounter hard coded assumptions about pointer width, endianness and presence of atomic operations.

All Debian packages are built natively. There is no cross build going on here.

Assembler optimizations without fallback to pure C for other architectures for example.