> But if you're searching enough to eat up Google's bandwidth, they're paying for that data and they're under no obligation to keep serving you as a client
Do you not see the irony?
Do you not see the irony?
Search engines don't allow people to scrape them (resorting to blocking after scrapers ignore robots.txt) because they don't get anything similarly valuable in return.
(Disclosure: I work for Google, though not on search.)
Search engines crawling millions of sites each with---on average---a few MB of data distributes cost globally.
Extracting terabytes of index data from a single search engine's repository consolidates the cost on the back of that repository's bandwidth provision.
These are not symmetrical cost structures.