Scroogle down | Hacker News Reader

Scroogle down

scroogle.org

32 pointsleftblank24 comments

Why would Google care about this? The ad uses Google's results and strips ads and cookie data, meaning they are taking from Google and giving nothing in return. Further, they use an unsupported method to scrape Google.

I can see the social value of anonymizing searches but Google shouldn't bear any blame for product changes that break derivative, unsupported third party products.

As a product manager, it's a nightmare when you're afraid to change (or retire an older version of) a successful product because some third party was using it in a unsupported way.

Alt: https://duckduckgo.com/ (https://duckduckgo.com/privacy.html)

Also see: https://us2.ixquick.com/ (https://us2.ixquick.com/eng/protect-privacy.html)

I'm amazed that they have been able to run this service for 5 years - the fact that Google hasn't stopped them in that time says more about Google that it does about Scroogle.

That was exactly what I wrote below :)

I figure google doesn't mind scroogle at all, maybe is even amused by it (but that's reading the tealeaves). Someone inside google has had to push some buttons to permit scroogle to continue the scraping for as long as they did, normally they would have been auto-blocked in an eye blink.

I think Google wanted to shut them down because Scroogle offer an SSL search service, so all searches are encrypted. Too much privacy, Google would say.

The number of searches through scroogle per day compared to google directly is so low that I don't think that was a factor here.

How do you know the number of searches that went through Scroogle was so low it wasn't a factor? It may well have been a factor. Or it might not have been. We simply don't know.

http://siteanalytics.compete.com/google.com+scroogle.com/

I meant that Google don't want SSL searches, at all. Even a small service is still too much for them.

Why not use something like http://www.google.com/xhtml?q=google%20mobile%20search ?

I misunderstood you for a moment. I thought you meant to use that instead of scroogle.

Now that I know what you mean, yes, that looks like an ideal solution. The Scroogle people are moaning for no reason. It's not Google's job to provide them with the layout they prefer. How hard is it to scrape Google anyway? It's not Flash-based or anything. Jeez.

Scraping google for an extended period is in fact incredibly hard, but not because of the lay out.

I'm fairly sure that someone on googles management has had to give scroogle a free pass on the scraping because at any serious usage level they'd normally have been blocked long ago. I'm guessing they have weighed the advantages of having scroogle out there versus the disadvantages of a bunch of bad press. They might even think it is good to have sites that are openly critical of google out there.

Scroogle should count their blessings that it has lasted as long as it did, now they should get their act together and fix the bugs in their scraper.

Does Scroogle really generate enough traffic for Google to even notice, or bother taking any action? Does anyone use it on a regular basis?

About 6K uniques on a daily basis. If they sent that to google without some trickery they'd be banned within a day.

any more like this?

I would think that the ajax API could also be twisted to their needs. It requires an API key that Google puts some limits on, limits that they might want to apply to Scroogle in any case. http://code.google.com/apis/ajaxsearch/

The ajax api returns completely different results than the normal search and in a different order, so it's not worth using if you want to "actual" google search results.

Actually, their docs claim that the limits aren't really related to the API key. The API key is just so they can contact you if they start blocking you:

  http://code.google.com/apis/ajaxsearch/key.html

That may just be disinformation though. For example, if you can spread your usage over multiple IPs, you might be allowed more searches than if you told them your API key for all your requests on multiple IP addresses.

It seems like the Slashdot and Hacker News comments have missed perhaps a key point. If any of you know of other search engines that meet these two criteria, please let me know:

1. Queries are passed as x-www-form-urlencoded via HTTP POST not as plain text strings via HTTP GET e.g. www.example.com/?q=heres+my+query+for+you+to+look+at vs www.example.com/cgi-bin/do.cgi

2. Queries of numerous diverse users at numerous diverse IP's are aggregated through a small set of IP's (Scroogle's), and Scroogle's servers are not scattered across the globe.

While this may not offer "privacy", it certainly makes it more difficult for a server to filter and/or redirect based on query string and approximated geolocation. Scroogle is obviously not catering to the common marketing platform. Otherwise they would be more interested to know what people are searching for, and where they are are searching from.

Scroogle's design makes it difficult for Google to serve up "personalised" and/or "localised" ads and results. (Even if they did not go through the trouble to scrub the ads and dump all cookies this would still be true.)

Some users may appreciate such relatively "raw" results.

Correction: There is no ad scrubbing involved. The results pages Scroogle scrapes (output=ie) have no ads to begin with. And, for whatever it's worth, they trigger less cookies from Google (only the PREF cookie is used, when I checked).

To reiterate what I said before, the difference is Scroogle allows the user query to be submitted via POST and over SSL.

The POST method effectively limits what the server can do with the data and what it can trigger automatically in the browser, without user input. Perhaps this is good for the user (more control), and bad for the marketer (less control). As such, it could be a controversial topic.

RFC 2616 Sec. 15.1.3 discussed the use of POST in the interests of privacy.

It would be nice if the output=ie option worked with Google Scholar/Patents and if Scroogle then supported Scholar searches.

It's really too bad. I kind of wish that Google would offer a stripped down, encrypted search page that promises not to store anything.

Kind of like how browsers have a "privacy mode", Google could also offer a "private searching mode".

It should be no surprise that Google's API's will never let users use the HTTP POST method. Those HTTP GET query strings are "valuable data"; Google is a business, not a charity. Hence Scroogle is doing something unique (AFAIK) by letting users use HTTP POST.

Another HN discussion about the Scroogle outage here: http://news.ycombinator.com/item?id=1337099

Reply on news.ycombinator.com