Google Search NoAPI | Hacker News Reader

Google Search NoAPI

blog.databigbang.com

10 pointswslh7 comments

I was an early user of the SOAP search API - definitely disappointing when that was shut down.

For several years, I have been using the Bing search API for very low volume stuff - works really well.

Google has told me (and other users of the now deprecated AJAX Search API) to switch to their paid Custom Search API.

The new API is missing some features that were in the old API, which is unfortunate. And, also unfortunate, the most accurate Google results (Local, Web, Image, etc.) come from scraping in ways like this article describes. Hopefully, Google will address these issues in its Custom Search API. If they don't many people will still continue to scrape.

But the Google Custom Search API is a different thing. It's a search based on a set of sites. What's important is to have an API to search for the "whole" indexed Internet, and Google is the most advanced search engine in town. If you play with more complex queries you quickly note the difference between Google and Bing, Yahoo or Blekko.

Yes, which is what we thought too. But, you can setup a "custom search engine" with some default site (we used www.google.com), then change the settings to that it uses the entire web, and remove the site that you originally put in.

The Bing and Yahoo APIs are definitely a lot better, IMO. But, as long as Google wants me to use their Custom Search API, then that's what I'll use.

If you go to Google and look at the html source code you’ll be astonished to see pure Javascript obfuscated code.

If you turn off javascript on your browser, I'm pretty sure you get straight html. That used to be the case anyway. Most scripts don't signal javascript-support.

The CSE API replaces it and provides a JSON based API: http://code.google.com/apis/customsearch/v1/overview.html

you can easily scrape google without having to use something as bulky as htmlunit.

When I teach people how to scrape, exercise #1 is scraping Google with Ruby Mechanize and Nokogiri. xPath is your friend

Reply on news.ycombinator.com