Stop Words as Social Signals (a result that goes against IR common knowledge) | Hacker News Reader

Stop Words as Social Signals (a result that goes against IR common knowledge)

cpdiehl.org

85 pointshamilton11 comments

One of the two linked papers (AnnualReview.pdf) discusses formal settings, including those with disparities in power:

'Brown & Levinson’s (1987) polite-ness theory takes into account an individual’s efforts to preserve the “face(s)” of others with whom one communicates. For example, they propose impersonalizing the speaker and hearer by avoiding the pronouns I and you, using past tense to create distance and time, diminishing the force of speech by using hedge words such as perhaps, using slang to convey ingroup membership, and using inclusive forms (we and let’s) to include speaker and hearer.'

Amazing that [to, to, on, my, to, on, be, on, to, to ... etc] can reveal so much. One of the linked articles ( http://homepage.psy.utexas.edu/homepage/faculty/pennebaker/r... ) describes how the relative frequency of the use of "I" in communication is a powerful indicator of status (less "I" => higher status).

Stop word distributions are also sufficient for identifying gender in most English corpora with about 80% accuracy -- and you don't need much text, either.

[Edit to add linky: http://u.cs.biu.ac.il/~koppel/papers/male-female-text-final.... ]

I wonder would James Chartrand have been found out ... http://news.ycombinator.com/item?id=994284 [Why James Chartrand Wears Women’s Underpants]

I put her four posts written right before the "reveal" post through the first gender guesser I found on Google, and they all came out female. That proves essentially nothing, granted, but this has been studied fairly extensively. If you want to do twenty minutes of work with automation you can grab everything she's ever posted on CopyBlogger and run it through your classifier of choice. Then, heck, run the guys on CopyBlogger against it and see if you peg them right, too.

Natural language processing remains my first love.

He seems to be claiming that in the phrase I don't feel comfortable signing up for a $7,000 extra flight without talking to you guys , it's the word to that signals the manager/subordinate relationship. It seems to me that ... without talking ... you is the best clue.

In English, subjunctive case usually requires a to, and people asking permission use the subjunctive a lot. Is that the signal he's finding? Is that the best NLP can do?

I was always uncomfortable with stopwords. Ignoring some data smells like overfitting, or perhaps a naïve algorithm (as in, it has no concept of structure – which is often the case, for computational reasons).

That said, the last time I dabbled in IR, my algorithm choked on stopwords. I should revisit it.

Good Lord that is counterintuitive!

See also SpotSigs ("...stopwords may however be very good indicators of the actual interesting parts of a web page...")

http://infoblog.stanford.edu/2008/08/spotsigs-are-stopwords-...

Outside the old post-2001 TIA/NSA efforts, what can this be used for?

Maybe for determining social hierarchies for mind share in product recommendations or advertising?

Yeah, I can't think of one noncreepy way to use this.

I think the following phrase is unnncessary "a result that goes against IR common knowledge"

IR aims to retrieve information and not sentiments from the text. This kind of work is related to sentiment analysis, where use of stops words as signal is common. Thus the whole argument that some holy criteria in IR has been shown to false is clearly incorrect. Even original authors don't make such assertions.

Reply on news.ycombinator.com