I do actually, more for the effect on credibility than for anything else. Just as users have "itchy back buttons," I think they also have "meh, this is the last straw, I'm going to try out the competition" moments.
Can you cite a case where this has actually transpired? Obviously in the extreme example where the site is never available I can see it having an effect. But spotty downtime I've seen actually have the opposite effect - users become more loyal. On one forum I frequent the "500 errors" are sometimes looked forward to since they usually signify a big news event is happening. Reddit and twitter are other sites that don't seem to have been negatively affected by spotty downtimes. I do think that regularly (excruciatingly) slow performance can have an impact though - one of the reasons I left digg with V3.
I don't think companies are going to publicize cases of losing customers due to downtime. But just to turn it around a bit, think of someone coming to a site for the first page. What's the user experience for them if the site is down?
i recently quit quickbooks online because the site went down for over 24 hours within the first week of my signing up
I tend to think no. For a small site with few users, no one will care. Once a site has become large enough, downtime with press coverage seems to create a virtuous cycle of social proof, e.g. Twitter, Facebook, etc.