Hacker News Reader

Cool site! Curious to know what do you use for scraping, without getting blocked?

Thanks a lot! We use scrapy[1]. Our crawler's behavior is very polite so most of them won't block us. Easyjet for example is very strict against crawlers and will block most of them(and us).

[1] https://scrapy.org/

> Easyjet for example is very strict against crawlers and will block most of them(and us).

It seems that easyJet has outsourced its anti-scraping protection to the company called "Distil Networks"[1].

I wonder how could they block screen scrapers using Chrome on Android with mobile IP addresses.

[1] https://www.youtube.com/watch?v=43wh3beHSBo

Reply on news.ycombinator.com