Offtopic:

What are some good frameworks for webscraping and PDF document processing -- some public and some behind login, some requiring multiple clicks before the sites display relevant data.

We need to ingest a wide variety of data sources for one solution. Very few of those sources supply data as API / json.