Yahoo open sources Anthelion web crawler for parsing structured data
github.com
github.com
- Athelion is a plugin for Apache Nutch, which is the web crawler part and has been open source for a long time.
- As far as I can tell, Athelion parses structured data (microformats, microdata, RDFa...) but that's not the most interesting bit. The online classifier of pages and scoring of new links discovered looks like the real important piece.
- Actually, the other two parts of the plugin are modifications of existing Nutch plugins.
All in all, I can't wait to have some time to see it working.
I have no idea if this is what was used here but I know this is an example where I'm sure they didn't out source the search work to Google.
I have no idea if this is still the case.
They did things like Yahoo webpage hyperlink connectivity graph, and tons of other research.