It used to matter in the past, but it seems like these days the effect is not very significant...
Last year we ran a test which involved artificially bloating one of our IPAs from 160mb to 260mb* and compared the app performance metrics* before and after the change, we saw no meaningful difference between the two versions.
By using the app thinning mechanism, you can create a different IPA sizes for different devices.
* CPI, CPS etc.