Monopolies, lobbying and protectionism got in the way of keeping the web truly machine readable. There's tremendous value in restoring some of it.
Even just considering the parts of Google that it takes to bring you the N blue links part of the Google SERP, the web scraper is probably the least interesting and significant piece of technology in the stack. It's beyond reductive to say that Google is in essence a large web scraper, or a web scraper of any kind. It is like saying that a person is, in essence, the world's largest mouth.
I think it's incredibly difficult to build a profitable business in this space. The number of customers who a) need to scrape the web b) aren't sophisticated enough to do it themselves and c) are big enough to make $$$ from are small. The important bit is always processing the web pages for whatever content is salient to the given customer. Which means that you need to deliver the web pages to them. So effectively your business is providing nothing more than URL lists and potentially some additional metadata compared to what the customer would get if they fetched the pages themselves. There are definitely some other complexities you could resolve, but it's hard to imagine that those benefits would be enough to build a business on.
Exactly and that ship has long since sailed. The good ship Web 3.0 (semantic web) launched in ‘99 and was a ghost ship until recently when it was boarded by crypto pirates now flying the web 3.0 flag.
> There's tremendous value in restoring some of it.
To this comment and OP, my startup is using web scraping to pre-populate machine-readable data for a DNS-based protocol called NUM [0]. So as others have said, whilst web scraping itself may be difficult to build into a viable business, it can be a key component of a viable business. Email in my profile if you want to discuss.
0. https://num.uk/blog/we-crawled-5m-uk-websites-and-published-...
The Semantic Web would have given much better search among other benefits.
Just scraping upon scraping.
:-)