It's gone both ways. In the late 90's, I was a co-founder of Ruby Lane and we had an agreement with eBay to spider their site and they paid us for traffic. When they decided to end the contract, we removed them from our search engine.

But later, eBay successfully sued Bidder's Edge to prevent them from spidering eBay's site:

https://en.wikipedia.org/wiki/EBay_v._Bidder%27s_Edge

Courts in later cases have disagreed with this ruling.

The bigger issue IMO is that it can get pretty hard to spider a site if they are actively trying to prevent you from doing it. It turns into a not-so-fun game of cat and mouse.