HNHacker News
TopNewBestAskShowJobs

pkandathil

93 karma · joined February 8, 2011

submissionscomments
pkandathil··on Overview of Text Extraction Algorithms
Can you provide more detail on the comma rule.

I do text extraction for lynk.ly. First we clean the HTML for the page. Then remove a bunch of tags we consider not useful like script or embed.

Then we looked at all the opening tags and checked if they have specific text in them like "hide", "display:none". If it does, then we skip it the tag.

Finally we get text from a tag if its above a specific threshold.

This seems to have worked best for us.

To see it in action, check it out at lynk.ly

pkandathil··on Kinect for Windows SDK to Arrive Spring 2011
pretty cool cant wait
pkandathil··on Breaking the web with hash-bangs
Yeah I found the jquery solution works best. Th href is for the crawler and for your website you override the onclick.

good one @hybrid11

← PreviousPage 2 of 2