But yeah, if you did weird things like mucking with hardware registers directly or used undocumented fields of a struct, then yeah, you could have a problem.
One common problem was when applications assumed all memory was reachable by the custom chips; the later addition of fast ram (which was ram that wasn't slowed down by custom chips accessing it, thus starving the CPU of memory cycles) could break those. Another example is using the upper 8 bits of pointers, which was safe on 68000 but not on 68020 and up which had a full 32-bit address bus. Making assumptions about OS structures was also a favorite of many hackers, and something that invariably broke on later OS releases. However, if you stuck to the API you were fine, and many applications run without problems on all OS releases from 1.2 to 3.1.
Too bad AGA was too little, too late. If it was released a few years sooner, say with the A3000, it might've been different. ECS barely added anything over OCS, other than extra chip RAM. There were a few modes hardly anyone used.