Ask HN: How would you architect a scraper system?

I’m looking into how large scale scraping systems work. I’m building https://cardog.io was interested in the “best practices” for strict formatted scraping systems. I’m thinking of using kafka handle the scraped data intake and transformation. Does anyone have any experience building something like this?