This is something I find a lot of web scraping tools miss. Are there any you'd recommend that specifically deal with things like async JavaScript content loading, or loading content based on what you click on a page (e.g., in Single Page Apps)?
Usually the response is in JSON and you can ignore the original page. You might have to auth/grab session cookies first, but thats still easier than working with the HTML.
Imagine an async-loaded list, that continues loading more content as it comes in, until it displays all of the content available to the backend.
When would you know such a list is finished loading?
This sounds insane, but it's pretty easy and common for an ambitious UXer to key in on, and is something I've seen in production pages.
(In the event you are a UXer, please include some sort of status update! Even an overlaid spinner that disappears solves the problem.)