How to build a DIY web scraper in any language

levelup.gitconnected.com

49 pointskld875 comments

I've been told I should've shared the friend link to remove the paywall (my bad), here it is: https://levelup.gitconnected.com/how-to-build-a-diy-web-scra...

Make crawlers not scrapers.

Make open APIs, not scrapers.

The problem is that most companies/developers still aren't "Open API first"... So, people must build scrapers.

Depends on your use case. If you just need specific data from a known location, build a scraper.

If you’re looking to index as much data as possible, build a crawler.

The complexity of a crawler is significantly more than a scraper, so it’s not always the right choice.