GCC transition from 3 to 4 didn't break the ABI. The last required ABI break was 3.0 to 3.1.
The ABI between g++ transitions is also platform and architecture dependent. Mac was PowerPC back then. Sun used Sparc. Not everything is always x86, even on Linux.
I still would characterize it a fairly stable ABI, not something that "can always break", especially compared with other vendors that break the library ABI every release.
[1] A new libstdc++ C++11 ABI was added in GCC 5.x but it is optional even in C++11 mode.