Look at page speed vs. bounce rate in Google Analytics for any well-trafficked site. People are frequently casually clicking around, and the faster you have content up on the screen the more casual users will engage.

By analogy, to get into a different mindset, think about channel-surfing on the TV. If other channels show a picture in 0.2s, and as you flip around there's a channel that takes 0.8s to show a picture, are you more or less likely to surf past the slow channel?