The obvious one is a smaller .so may have fewer pages of interest. If each function you call takes half a page, and you call 1/4th of the functions, chances are good you need to load more than 1/4the pages, because sometimes you'll call both functions in a page you load and sometimes you won't. In other words: if the .so is smaller, you get better packing.
The other explanation is prefetching. Lets say all functions take a full page, so packing into a page doesn't matter. When you call a function, which demand pages, chances are the kernel is also going to load adjacent pages; if all the functions you call are adjacent, prefetching saves you significant i/o time. Otoh, if your functions are sparse in the backing file, you will have to wait for i/o; which may be (marginally) delayed because of prefetching on earlier loads.