Ask HN: 37% of people are using Python for web scraping. What do they scrap?

6 pointsggregoire2 comments

Just saw this stat in this survey: https://www.jetbrains.com/lp/devecosystem-2019/python

I'm not familiar with web scraping. Wondering what are the most common applications of it?

I can think of a few like public users data collection, prices comparison, news aggregation… Are there some other obvious ones I am missing?

To crawl literally anything. For my last project I got more than 10 million linkedin profiles to build a social network for entrepreneurs.

So data can give you power to build anything

Most people are probably using scrapers and crawlers to collect data on sites that either don't have an open API or the API is difficult to use.

Examples:

What can you then do with the data?

Monitor competition or your own brand

Sentiment Analysis

Gather sales leads

Machine learning

Generate content (blog posts, building a custom job board) Find cheap flights