Yes, speed matters. This is just anecdotal evidence, so feel free to dismiss it:
We recently switched hosting companies and our site load speed went from an average of 10 seconds to the new server speed of just under 1 second.
Our monthly revenue instantly doubled. People stayed on our site longer, they learned more about our products, and thus were more willing to shell over their hard earned cash.