Hacker News Reader

Comment by pg | Hacker News Reader

You already have the data I'd use: the text of the comments.

If anyone wants to try to train a filter to detect this sort of comment, I'd be very interested to see the result.

I'll consider that a challenge...

To any that have experience getting comments data from HN -- what's the fastest, most polite way to do this? And am I correct in remembering that there's some aggressive rate-limiting for crawling the site?

Don't crawl the site, please. The place to get data is the HNSearch API.

There's a database(quite old) of HN posts and comments here:

http://www.btscene.eu/details/2240774/Hacker+News+Database+o...

Any plans to try to get a dataset for supervised ml? Perhaps collect the top 4 comments from all front page threads and post a survey on HN asking HNers to rate those comments for skepticism/dismissiveness?

I might bet that upvotes to middlebrow dismissal would be highly correlated with downvotes to the article itself, if we had downvotes to articles. In fact, I believe that these comments rise to the top because people are downvoting the article vicariously, by upvoting a rebuttal, however vacuous. In order to confirm this hypothesis, though, you'd have to collect data by implementing a downvote button on articles -- though it would not necessarily have to do anything in article-ranking terms ;)

(That might introduce a confounding factor, though--namely that by alleviating people's urge to downvote the article by giving them a [nonfunctional] button to do just that, people might stop upvoting the dismissive comments. Hmm....)

>You already have the data I'd use: the text of the comments.

Wouldn't that require real AI though? I thought for a minute that NLP (Natural Language Processing, not the other meaning(s) of the acronym) might help, but then thought that it may not work for cases where the comment is quoting another comment. Note: I'm not at all an expert in any of those fields, just interested.

Sounds like a job for Sentiment Analysis [1]. Modern systems are pretty good at discerning negative from positive comments.

You could probably find a way to mark negative and positive comments. Whether the resulting algorithm would be fine-grained enough to semi-reliably mark 'middlebrow dismissal,' I really don't know. Actually, as somebody who has worked on that stuff in the past, I don't think it would be very easy.

[1] http://en.wikipedia.org/wiki/Sentiment_analysis

Hadn't heard of it; thanks for the link.

Thanks for the link.

Agreed I don't think it would be an easy task, but I wonder how would perform a "bag of word" approach.

Harder part as I see it would be to categorize the comments on middlebrow dismissal / Not dismissal. It seems like we would be spending more time preparing the data than in the algorithm itself.

Welcome to statistics/data science/machine learning.

The hardest part is always getting the data into usable form. Its not as much fun as fitting models, but its definitely the majority of any role where people pay you to do this kind of stuff.

There's a lot of good research on forums (pm me if you want a bibliography i collected for a previous role), and short texts have become a bigger deal post Twitter. I completely agree with pg on the somewhat annoying nature of comments such as the GP.

Hey... Thanks for the offer! How can I contact you? I didn't know we can pm users over here.

I did have worked with ML before but mostly with images which are (IMHO) way easier to put in a format depending on the problem.

My email is obsfuscated in my profile. Should be pretty clear, conditional on your humanity.

Bing Liu is one of the researchers working on this. I've discussed Amazon review fraud detection with him. http://www.cs.uic.edu/~liub/

It might be easier to investigate submissions first and filter at this level if a trend is discovered. That is, a certain type of submission might attract a certain attitude of comment.

Reply on news.ycombinator.com