+1000 I'll tend to look at the speed coming off the server instead. At least I know that I can control 100%. If the user is on a dial up modem, not my problem ;)
Compress data with gzip is probably the big one for modem users, but also batching scripts and CSS together, reducing image sizes, re-arranging code to reduce blocking during rendering.
These are all things under your control, and with current mobile browsers, these are things you really must care about
You can do something to make like better for those on mobile and dialup with server settings, image compression, and the like. You can't change their connection speed, but you can focus on their experience.