Selenium, although I'm using a wrapper library that uses it. I only query each company every few days or so which probably helps to not get banned IP-wise but also rotate them. But many of the company job links are through external sources too (lever, greenhouse, etc.) which don't seem to mind

The company data was gathered online for a long time until I found https://www.thecompaniesapi.com/ (which now is the source for much of that data)