Hacker News Reader

Comment by tyingq | Hacker News Reader

PyPpeteer might be worth a look as well. Basically a port of the JS puppeteer project that drives headless Chrome via the Devtools API.

As mentioned elsewhere, using anything other than headless isn't useful beyond a fairly narrow scope these days.

https://github.com/pyppeteer/pyppeteer

View on HN

I think you'd be surprised by the amount of website you can scrape without an headless browser.

Even Google SERP can be scraped with a simple HTTP client.

That's what I meant by narrow...a known set of sites and data you want to extract.

I imagine, for example, building on the SERP example might hit a wall if you added logged in vs not logged SERPS, iterating over carousel data, reading advertisement data etc.

From what I can observe, 2/3 websites can be scraped without using a headless browser.

How do you deal with JavaScript then?

There's an official Python library for Playwright as well: https://github.com/microsoft/playwright-python

I am often contacted by people who ask me to scrape a dynamic/JS rendered websites. You might be surprised to know that many of such dynamic websites are actually depending on some API end-point which is being accessed via some AJAX like functionality which you can access directly and get the required data. I often faced the situation where data was not fetched via some external source was already available either in data-field or some JSON like structure hence no need to use Selenium with headless browser.

Sure. This one happens not be Selenium.

Reply on news.ycombinator.com