Ads aren't the first thing people look at though. If the rest of the page loads and content is visible then i won't leave because that ad I just had to see didn't become visible.
For example, if you are involved in a 3rd party bidding where you bid on some other network's impression, you'll have a deadline to respond with a bid. Let's say that deadline is 500ms (it's not). 100ms increase in median would lose _significant_ amount of impressions in that case.