Amazon found every 100ms slower the site loaded, they lost 1% in revenue: http://www.gduchamp.com/media/StanfordDataMining.2006-11-28....
Walmart.com found a very large change in conversion rate based on page load times: ( http://www.slideshare.net/devonauerswald/walmart-pagespeedsl... slide 37)
My own customers have seen 4x different conversion rate, when grouped by page load time. (i.e. same slide as walmart page 37 above, where the peak is 4x higher than the lowest performing group).
Your business probably won't fail because it's slow, but it will certainly make less money because it's slow.