Hacker News Reader

Usually JS based websites operate with some internal API under the hood.

Inspect those, and then directly hit those with something like Python or Go.

I prefer Python (requests, lxml, BeautifulSoup).

Mimic the headers that your browser is using.

Have you tried this already and run into issues?

I've done this sort of thing and it does work well. The caveat is you usually need a cookie/token after logging in, so you still need to fake that part as well as manage the session state.

It does work well, though some sites certainly detect the behavior after a while.

Right..

I'm in an antiquated space (logistics and wholesale) so most websites aren't hard to scrape and even mimic the auth.

The other avenue I've had success with is sniffing android app traffic to reverse engineer the API calls.

Reply on news.ycombinator.com