> Is there a way I can test this more scientifically?

How about the following?:

1. Choose a dozen or so public files on hosts known to have high bandwidth (Google or Microsoft perhaps). Maybe choose a couple of public legal torrents too.

2. Write a script to download all the chosen files (either sequentially or concurrently), and emit timing information.

3. Run that script a couple of times a day at random times. Try to cover weekdays and weekends, days and nights.

4. Repeat 3 but running a speed test first each time. See if there is any statistically significant difference.