If you're hitting HN's access thresholds, you might consider incrementally parsing data from http://hnstream.com
That may help with analyzing comment content at least. Not so much with news rankings though.
That may help with analyzing comment content at least. Not so much with news rankings though.
You're right. I have limited frequency of crawling due to HN's access thresholds.
hnstream will help me. I will try it.