Hacker News Reader

Have you got any proof of Google scraping from residential proxies users don't know about, rather than from their clearly labelled AS? Otherwise you're mixing entirely different things into one claim.

View on HN

That's the whole point. Websites that try to block scraping attempts will let google scrape without any hurdle because of google's ads and search network. This gives google some advantage over new players because as a new name brand you are hardly going to convince a website to allow scraping even if your product may actually be more advantageous to the website (for example assume you made a search engine that doesn't suck like google, and aggregates links instead of copying content from your website).

Proxies in comparison can allow new players to have some playing chance. That said I doubt any legitimate & ethical business would use proxies.

I don't think parent post is claiming that Google is using other people's networks to scrape the web only that they have a strong incentive to keep other players from doing that.

No, there are other scrapers that Google doesn't block or interact with. You can even run scraping from GCP. This has nothing to do with "only Google is allowed to scrape". They even host apps which exist for scraping data, like https://play.google.com/store/apps/details?id=com.sociallead...

Reply on news.ycombinator.com