The ABIs are quite tightly coupled though. There's e.g. a dependency to glibc, so either you make a compromise and use the oldest common glibc or ship packages for various glibc versions. So unless everything is built from source, this approach has its limits.
edit: of course it's possible to use static linking but even then there's a residue of calls that are dynamically linked at runtime. (That's why there's flatpak, snap, ... which bundle dependencies redundantly)