Hacker News Reader

Comment by toolslive | Hacker News Reader

fetching html and then parsing it navigating the parsed result (or with regexp) is what used to work 20 years ago. These days, with all these reactive javascript frameworks you better skip to item number 5: headless browsing. Also mind that Facebook, Instagram, ... will have anti-scraping measures in place. It's a race ;)

View on HN

I've found that a lot of the time that's not needed. Often you'll find the data as a JSON blob in the page and can just read it directly from there. Or find that there's an API endpoint that the javascript reads.

I find this method works best. Skip looking at the page and instead watch all the network requests as the page loads.

ZAP HUD Proxy is the best option here...

Load the page on it and it shows you all the request being made along with the payload as it happens.

Find the one you need, copy the data, endpoint and HTTP verb and recreate it in your language of choice :D

It's not all bad, many modern sites just expose a JSON API that can be used. It really depends on how protective and large the company behind it is.

This. Even relatively simple websites are much harder to parse today. I did a minor side project for a customer scraping some info and anti-scraping measures were in full force. It feels like an all out war.

Such as? I've never encounter anything I wasn't able to overcome.

Usually starts from simple to difficult. User agent stuff, IP address detection, aggressive rate limiting, captcha checking, browser fingerprinting, etc.

It can be overcome and, admittedly, I am new to this so for me that means way more time spent trying to make it work. The odd one that I got stuck on for a while was a presented list where individual record held pertinent details, but the list had, seemingly randomly, items that looked like records on the surface, but were not ( so those had to be identified and ignored ). Small things like that.

Still, I would love to learn more about your approach if you would be willing to share.

please click on all the traffic lights you see below.

Once recaptcha is in the mix it'll get tricky pretty quickly. Everything else is easy to overcome most of the time.

Recaptcha comes to mind.

That said, there are quite a few services which battle these systems for you nowadays (such as scraperapi - not affiliated, not a user). They are not always successful, but they have an advantage of maaany residential proxies (no doubt totally ethically obtained /s, but that's another story).

Datadome, Incapsula

Reply on news.ycombinator.com