Hacker News Reader

maltzon Mastering Web Scraping in Python: Crawling from Scratch

You should probably take a look at this.

https://www.zenrows.com/blog/stealth-web-scraping-in-python-...

maltzon Mastering Web Scraping in Python: Crawling from Scratch

YouTube has "var ytInitialData" & "var ytInitialPlayerResponse" params hardcoded in HTML. No need to run JS!

maltzon Mastering Web Scraping in Python: Crawling from Scratch

Yep, seems so. https://www.zenrows.com/pricing

maltzon Mastering Web Scraping in Python: Crawling from Scratch

If pages are constructed client-side, the content you are looking for is either hardcoded as JSON in the HTML or loaded via XHR request. Scrape that.

maltzon Mastering Web Scraping in Python: Crawling from Scratch

Playwright. It can be easily used with JS, Python, Go, Java, etc.

maltzon Mastering Web Scraping in Python: Crawling from Scratch

It's part of a series of blog posts that talks explicitly about crawling. There are indeed other links that do better explaining advanced extraction techniques.

Extraction => https://www.zenrows.com/blog/mastering-web-scraping-in-pytho...

Avoid blocking => https://www.zenrows.com/blog/stealth-web-scraping-in-python-...