Google2Csv is a simple Google scraper that saves the results on a CSV file | Hacker News Reader

Google2Csv is a simple Google scraper that saves the results on a CSV file

github.com

68 pointspanos_sa32 comments

If you're looking to do any serious search engine scraping, you'll find yourself needing to use proxies.

For my thesis which required millions of datapoints, I used this tool: http://www.scrapebox.com/

This seems like what I was lookingf. What kind of proxy were you using?

Nice, until the CSV part really. I mean you have a nice DataFrame there but then it gets serialized using probably the worst format out there for totally variable data like what you scrape from a site.

Unless the file uses the actual ASCII record separator you'll end up with a CSV file which can only be read by a handful of software, after telling it explicitly what the separator and quoting rules are. And even then it's hit or miss. And likely it does not use the RS because even though the chances it's unambiguous greatly increase and the RS was actually meant for that, software doesn't typically use it because when CSV was invented it's existence was unknown or ignored beacue it's not human readable (I guess, don't really know actually) and so the sad story began.

As you can see: I'm not a fan of CSV :) Just today I again had to waste time because at one point in the development of this otherwise fine piece of software I'm working on - even though I knew I'd regret it - I allowed it to export CSV files. Customer moved software to another machine, forgot that they once told the CSV exporter part to use the system settings, and now has CSV files with a comma separator (you know, the C in CSV). Oh the irony, that's not what they wanted.

While I couldnt agree more with your disliking of CSV, I have yet to find another more generic human-readable table file. Of all the pandas dumping methods there is no more widely available format. I find the times a CSV is utterly broken and I have to manually write the parsing logic for it quite rare actually, but then again it varies per field. What format would you rather prefer?

JSONL (jsonlines.org)

http://jsonlines.org

pandas can directly write to parquet

parquet has a very narrow use case. it's great for internal operations but interoperability is limited unless both sides use one of the same handful of tools. csv is supported by literally every tool ever designed to work with data.

Anything unambiguous really, whatever works for the job. So unless the job is looking at very limited amount of data and drawing conclusions from it just using your brain, that means you need some kind of visalisation and processing tool. Like pandas or excel since we're talking tabular data. I.e. actually the whole premise of 'let's look at data in a plain text editor' is sort of wrong, imo. It's too limited and hence not the right tool for the job most of the times. You can't do anything but the simplest things with the data anyway in a plain text editor, so from the moment your data is more than a couple of rows in a table you're better off with another tool anyway.

thanks for the feedback, I just update it, you can now have csv, excel or JSONL file

CSV is totally fine, and I don't understand why people have such an issue with the format. It is very simple, here is the entire specification: the file should be encoded in UTF-8, and fields are separated by the , character. Fields may be quoted with ". If a field contains any of [",\r\n] then it must be quoted. The quote process is: replace all " characters in the field with "" and then surround the field with ". I would expect a competent programmer to be able to implement this specification as an interview question!

The existence of software that produces braindead CSVs is beside the point. Don't use that software. Just do the above and everything will work.

The answer to your first question is in your last paragraph. CSV is an abysmal format for data exchange because the standards are weak and implementations are extremely varied.

If you're writing an (en|de)coder for a specific or limited target, then sure it's fine. Otherwise I would steer clear of it.

here is the entire specification: the file should be encoded in UTF-8, and fields are separated by the , character

This exactly is part of the problem: how many people do you think would come up with this specific specification? You do realize in real life CSV gets used for anything which is plain text and separated and perhaps even quoted? Also 'should be' UTF-8? By what authority?

The existence of software that produces braindead CSVs is beside the point.

No, it is not. Especially not because there's so much software. And because it's not braindead, it's just doing something else. Which is bascially fine since there's no standard.

Just do the above and everything will work.

Yes, in your personal bubble. Beyond that it's totally insufficient.

Lots of people like to use backslash escaping, and you are so used to it that you used it in your spec even though it leaves the ambiguity of whether \n and \r are literal or the LF and CR characters. I feel like it's perfectly expressing how it "contains the seeds of its own destruction". How much data expresses a double quote as "\"" instead of """"?

It would be easy to use CSV if you never did any data interchange with it.

> Scraping google search results is illegal.

IANAL, can someone please elaborate? This sounds wrong in several ways, one of which being that Google results are almost exclusively scraped from somewhere else already.

It is decidedly not illegal to scrape web sites. This was recently established by a court [1].

[1] https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn

Tangentially related, this would be a fun target for the anti-trust lawsuits. Force google to open its search results as an API service. Allow them to charge for it, sure. But make it an open API.

(Isn’t it weird they don’t offer this already? Makes you think.)

Further to the practical/technical issue of being blocked, there is a legal issue: this way of coding it (= not using an API key) violates Google's Terms and Conditions.

Are Google's terms legally binding if you are just searching? I don't recall ever agreeing to them in that case.

> is the ToS legally binding if you are just searching

Based on the LinkedIn case my guess is no, because it could be considered "generally public". But that ruling is highly suspect and likely to go to the supreme court, imo.

The onus is on them to prove that I agreed to some ToS.

If it's publicly accessible, it's pretty much a public space, thus Google ToS is BS. It's the equivalent of walking by the Google office and taking a photo of the building, IMO. This wouldn't hold up in court.

Additionally in a court decision from last year, accessing public data is legal. https://parsers.me/us-court-fully-legalized-website-scraping...

That's pretty rich coming from a company that scrapes websites for a living.

I made a wonderful little Google scraper in Clojure once. I was surprised to see I got IP blocked after only 20 or so searches.

A rite of passage.

This is a great way of getting your ip permanently blacklisted and swamped by captchas.

It's not going to be permanently blocked, but the low rate limits are annoying, so for any seriosu use a set of proxies is must.

I don't know about being blocked. I have a vps where I used a similar naive script and it got blacklisted by google. I sometimes tunnel through it. It has been getting captchas from google the past 7 years. It is pretty permanent.

This is a cool project. I've used Scraperr[0] for years, but it's always great to have alternatives.

[0] http://scraperr.com/

Neat!

Have you run into issues with getting blocked by Google / issued captchas?

Did something similar using requests, there's a rated limit around 80+ request (not sure how long it lasted though) no matter how I delay each request.

My work around is to run in on the cloud with multiple IPs, so I just switch between. Mine's a one time thing so this is workable.

If you're querying for news, Reddit might be a wise choice, they have quite a bit of free APIs afaik.

Reply on news.ycombinator.com