Hacker News Reader

Comment by chadwebscraper | Hacker News Reader

I appreciate the response! I also agree - happy to add some clarity to this stuff.

Bot protection - this is handled in a few ways, the basic form bypasses most bot protections and that’s what you can use on the site today. For tougher sites, it solves the bot protections (think datadome, Akamai, incapsula).

The consistency part is ongoing, but it’s possible to check the diffs and content extractions and notice if something has changed and “reindex” the site.

100k URLs is a lot! It could support that, but the initial indexing would be heavy. It’s fairly resource efficient (no browsers). For scale, it’s doing about 40k/scrapes a day right now.

Appreciate the comments, happy to dive deeper into the implantation and I agree with everything you’ve said. Still iterating and trying to improve it.

View on HN

Re-indexing seem sub-optimal. I can't think of a use case where people care if the design changes. Even some content changes are not going to be interesting. Someone corrected a typo, updated punctuation, that kind of thing... such things are just noise if you are trying to react to content changes.

Your system needs to know not only what changed, but whether or not it matters. Splitting meaningful content from irrelevant noise is exceedingly important. If you know that, you do not need to re-index because you can diff only the meaningful content.

As far as the 100K URLs, each URL has between 200 and 1000 sub-pages beneath the top-level page. They all need to be periodically scanned for updates, while capturing that distinction of noise vs. meaningful change. I've actually got code that does the needed work - it is scaling it up to that level that I didn't want to take on.

I'm not sure what you mean by no browsers. My existing scraper uses headless browsers, in order to capture JavaScript-driven content and navigate through a SPA without having to re-load at every URL change. If you are not using even a headless browser, how are you getting dynamic content?

Let me clarify, it just reindexes if the structured data changes, so it ignores layout changes. So it diffs the extractions.

Would be curious to try it out on your sites if you want to shoot me a few over - I can share my email.

It does use a browser to find dynamic content but does not afterwards.

I tell ya what - I'm currently working on finalizing a contract with someone for related work, but if that falls apart for some reason, I might pick this back up independently, and if so, I might be interested in chatting more. I'd recommend adding your email to your profile, and I can reach out if appropriate.

Just to set proper expectations, though, my current contract negotiations are already fully agreed, we're just waiting on the legal team to finalize things, so the odds of me seeking a new partner are not high.

Just did - great chatting with you. Best of luck & looking forward to (potentially) hearing back!

Reply on news.ycombinator.com