Charles Proxy for Web Scraping

My flow was always:

1) Look at the site (nowadays using your browser console)

2) Evaluate whether the "interesting" data is server-side or client side. Seems nowadays a client side request is more normal, typically resulting in structured data.

3) "copy as curl"

4) Code

Admit I have never used Charles or heard of it. The mobile app debugging looks useful though I have used a MITM proxy in the past for the same advantages.

Generally the process is simply that, figuring out the requests of a site. More effort may be needed to circumvent rate limiting, but to me that's an almost entirely separate issue.

For the Charles license, I got many years of service from, I think, $40. If the product fits you, I suggest it.