Unfortunately, it seems that, at least for gmp, the shared objects balloon in comparison to all other architectures. It is about three times bigger (6000 instead of 2000kB): https://packages.debian.org/sid/libgmp-dev. I am hopeful that this may improve with extensions, though I know little about the details.