HNHacker News
TopNewBestAskShowJobs

veggies_4us

2 karma · joined February 13, 2018

submissionscomments
veggies_4us··on Ask HN: Best tool for technical web crawler data (entire internet)
I have tried publicwww but over 80% of the results were either duplicates domains, dead links, or didn't contain my search term.

I'm glad I only wasted $49 and didn't opt for the annual license.

veggies_4us··on Ask HN: Best tool for technical web crawler data (entire internet)
They gave us the option of either running a report on their entire index, OR if we had a list of domains to search explicitly (which we didn't have).

Our use case was different, though, we wanted to find websites using a javascript library, and then run a javascript function on all of those sites to extract account information and specific html patterns.

But yes, their larger custom reports index did the trick, would recommend.

veggies_4us··on Ask HN: Best tool for technical web crawler data (entire internet)
they don't advertise it well, but nerdydata's custom reports had some excellent results... way more than their regular search website shows.

They were also able to customize our search and extract specific data from the page. pretty advanced stuff.