You might be right that it was not due the a.out format alone.
Now I remember that at the same time there was some huge migration from libc5 to glibc!
Maybe it was composition of libc5 + a.out + egcs that gave us speed(no swappng) back then.
I might do some benchmark on "PCEM" emulator (only proper emulator for old hardware as it retains original speed of cpu, memory and hard drives) in my spare time.