Ask HN: Is API scraping more common now than page scraping?

4 pointsthinkloop2 comments

Given that many/most sites are SPA-style now that make structured http requests to retrieve their data (after initial page load), does modern scraping target underlying api's now, or is it still centered around parsing markup? If the latter, why?

View on HN

There isn’t a general rule. It depends on the website architecture. If there is an API you can use it is better for you and less tolling on the website (given you know what you are doing), otherwise scraping it is.

doesn't have any sense to make manual crawling and parsing, having tools like crawlera or proxycrawl makes it super easy to get data in API style way. So why complicate?

Reply on news.ycombinator.com