Hacker News Reader

Comment by nurettin | Hacker News Reader

And google marks your ip as a bot. I did this three years ago in one of my DO proxies. Google still asks me to confirm a captcha when I do a search.

View on HN

My company is behind a giant honking NAT and we get blocked all the time. Unusual volume for our "residential" IP block I suppose. Google seems to do some averaging across CIDR blocks to detect likely bots. Any chance somebody is abusing your proxy IP? Is your proxy part of an IP allocation thats underutilized?

An easy way to get around it is to login to a Google account. Google allows you to use even the most trashed and abused TOR exit nodes as long as you're logged in.

Indeed. If you're looking for programmatic web search, I'd suggest you go the Bing API route.

I commented earlier on this post about it being impossible for a bot to do search on Google. Turns out I turned to the same resource you're recommending. I'd also recommend sending a python bot to your favorite news sites once a day for updates instead. Same deal, Beautiful Soup. On the other hand, Google has Custom Search, which is $100 a year for 20k queries. I use that as well.

If you want news, use RSS. Almost every news site is using it.

Google CSE seems like the most fitting solution, which appears to do pretty much the same thing as the article, but without all the hackery nonsense:

Step 1 - Setup a CSE to search entire web: https://support.google.com/customsearch/answer/2631040?hl=en

Step 2 - Use the CSE API: https://developers.google.com/custom-search/json-api/v1/over...

It looks like Google is shutting down the paid version of Custom Search Engine: "On April 1st, 2017, Google will discontinue the sales of Google Site Search, the paid version of Custom Search Engine. All new purchases and renewals must take place before this date. This product will be completely shut down by April 1st, 2018." Source: https://cse.google.com/cse/

This depends on the rate at which you are making requests. If the rate is low, then Google doesn't bother. I have successfully scraped Google by waiting longer between requests.

Reply on news.ycombinator.com