At Profitmind, we're building massive-scale ecommerce datasets to use in AI training/inference, and we need a junior engineer to help develop the scraping infrastructure for product data. You'll be reverse-engineering undocumented APIs, handling anti-bot systems, and dealing with edge cases like pagination limits, rate limiting, and sites that change their protection schemes without warning. It's fun work! The technical side involves analyzing a site's network requests, deobfuscating and reading obfuscated javascript, and implementing simple HTTP request scraping to full browser automation. You'll also work on the infrastructure layer: state management for resumable scrapes, deduplicating products, data integrity, and monitoring systems to detect when sites change.
The work you'll be doing is in the hot path of our company, so the systems you will build need to be performant and maintainable. You should have solid Python skills and experience scraping ecommerce websites and APIs. You should also like the slightly-obsessive investigative nature of the work.
If you worked in scraping or botting in the past, please hit me up!
Reach out directly - gray at netail.ai