Fast scraping in Python with Asyncio | Hacker News Reader

Fast scraping in Python with Asyncio

compiletoi.net

96 pointswatermel0n21 comments

Code samples on your blog are really hard to read. (the syntax highlighting colors make it impossible)

I brought down the white background to #aaa and it made the code much easier to read.

It's very interesting to see the Python libraries starting to experiment with the power of the new 'yield from' statement.

it becomes more interesting when you start read streaming server request and then pipe into streaming client request.

`yield from` and asyncio makes everything much more simple.

You can also use grequests (originally a part of requests). It has a grequests.map() function, that I use exactly for this scenario.

I used that at some point, but I tend to prefer the explicit yield of asyncio to the implicit context change of greenlets. (This blog post is a good explanation of that: https://glyph.twistedmatrix.com/2014/02/unyielding.html)

Here are a number of asyncio examples, including Guido's web crawler - https://code.google.com/p/tulip/source/browse/examples/

Looks ok. I agree with author many people roll their own.

Depending what you want to scrape, this might work better for you.

https://www.mashape.com/stremor/stremor-content-extractor-fa...

It will scrape articles even if they are in multiple divs. It even works on magazine layouts like TheVerge.com that doesn't work in Readability (py, api, or js).

This looks very similar to a Python library called Blackmamba I have been using on Linux for async requests.

If you need support down to Python 2.6 and similar functionality it's worth checking out:

http://rootfoo.org/blackmamba

https://github.com/rootfoo/blackmamba

I'm not sure if it's just me, but after years of writing Python code with various async I/O libs and then used Go for a while, it felt so much better when the language/runtime handles async I/O for me and I do not have to think about event loops at all.

Python used to be the clean and simple language for me, but all those ”hacks” to get async right just makes the final code ugly. The cleanness of Go wins me over instantly.

Even though the two languages are not directly comparable, the foundations of Go seems to me much better and solid. It reminds me of Rob Pike's essay “Less is exponentially more” http://commandcenter.blogspot.ca/2012/06/less-is-exponential...

Try gevent or eventlet. Real pythonistas use these, instead verbose java-esque coroutines.

To paraphrase JWZ, "Python is big because your needs are big".

I don't see anything "hacky" to get async right. It's a simple library. It's just not first class (then again, no reason it should be, isn't it not first class in Rust too?).

Even better use gevent or eventlet. Then don't have to worry about @asyncio.coroutine, yielding, vs yielding from and do so on.

Like the article of @madjar in the comment above explains. There is a reason for definitely not using gevent.

I read that article and don't agree.

Those methods (gevent and eventlet) are the default and typical way of building concurrent python services.

That article conflates threads, shared state and explicit IO awareness and dispatching. Those can be different in each case.

Python doesn't have isolated heaps like Erlang but it is still possible to have lightweight threads with a queue and not share by convention data between lightweight threads.

Having or not having @coroutines or yields or deferreds or inlineDeferreds doesn't make a difference in that case.

In general, a set of jumping callback and errback functions are a lot worse concurrency mechanism than threads.

A callback chain is a poor man's threads, it is just spread non-locally across the whole code without a way to properly synchronize.

Threads done right, should make it easy to organize data locally. This is quite the opposite of what that article claims. A set of spread around callback functions, do the opposite.

Whether yields or inlineDeferreds are used are almost orthogonal to the above. All they do is help you localize data (which callbacks don't) but now you have to explicitly care where and when you do IO. You might not even care in real life but they force you to. That is stupid.

Oh and library fragmentation. There are now 3+concurrency frameworks for Python. They all look different. And because of this bubbling of IO primitives up to the surface of the API, make it almost impossible to share libraries between them.

Clearly this would be faster than using the requests library to scrape pages in serial, but would it also be faster than using the requests library in several threads at once? Besides the simplified syntax, what is the advantage of asyncio over a threaded scraper?

I would say the performance are equivalent, with maybe some overhead if there is some heavy thread spawning. I'll run some benchmarks at some point.

One of the biggest advantages is the fact this is not multi-threading, so context switches can only happen at precise points (yield from). This makes it easier to reason about the code.

I'd really like it if articles on scraping included robots.txt support.

The robotparser library works fairly well. When I used it, it choked on a particular not-to-spec robots.txt file, but other than that it was fine.

Copypasta of dead comment:

nashequilibrium 13 hours ago | link [dead]

rp = robotparser.RobotFileParser()

rp.set_url("https://news.ycombinator.com/news/robots.txt")

rp.read()

# Reads the robots.txt

rp.can_fetch("*", 'https://news.ycombinator.com/news')

>>>> True

I tend to use scrapy when doing python scraping but its a massive setup, this will be good for smaller projects.

s/scrapper/scraper/

Reply on news.ycombinator.com