Hacker News Reader

Comment by minimaxir | Hacker News Reader

The trick behind NLP that's surprisingly under discussed is that you must use apples-to-apples comparisons between datasets. Using simple tweets as a proxy for style against a professionally-edited NYT op-ed is silly, but this analysis (along with the previous analyses) is transparent about that fact.

View on HN

The OpEd reads more like a speech. I'd assume there are some publicly available speeches (or videos with subtitles) that can be analysed instead.

Some articles mention that anonymous sources specifically take note of the speech patterns of other people so they can use those to throw off the trail. I'd be very surprised if the author didn't at least try to do that, or agree to have the NYT rewrite each sentence while preserving the meaning.

It's an interesting exercise and can and might be able to rule out some authors, but it seems like someone harboring such opinions and wishing to express them to some extent (and has with others -- in the resistance) will be discovered in short order.

Outing this person might be an even bigger story than the OpEd itself. But I put my money on old fashion journalism rather than text analysis.

Presumably the candidates for consideration have speech writers and staff who would either write a given speech outright or influence it heavily. So, that would likely rule out any speeches related to their position.

The speechwriters and staff are in some cases valid guesses in their own right, but don't have a unique corpus.

In the podcast interview (“The Daily”, also from the nyt), the contact inside the NYT says pretty clearly that they didn’t rewrite anything, to avoid detection or otherwise. I believe the quote was something like “if they write something that’s used to identify themselves, it is what it is”.

I’d like to see a comparison done against Bob Woodward’s work. As you say, 140 char micro logs are a horrible training set for analyzing a long form essay.

You think the anonymous administration official is Bob Woodward?

Do you mean "opaque"? "Transparent" usually means that someone is forthcoming, but I don't see your sentiment reflected in the article.

Also it assumes that writers don’t change their style. In this case all a fake writer has to do is drop a word like “lodestar” in the document and everyone thinks Mike Pence wrote it.

If I were the author I would take my own essay and then run it through algorithms like this and massage my word choices to match other people.

This is obvious. So irritating that someone went to the trouble of doing the analysis on an inappropriate data set.

Not really, they are just showing off their library in a novel way. The author doesn’t even pretend that it’s going to produce an accurate result.

I don't see that transparency at all, nor in the previous analyses linked to in the article. In fact, the articles seem to explicitly be on the side of their methods being definitively able to identify "the author behind New York Time’s op-ed" [1]; this article is prefaced by the title and question "who wrote the anti-Trump New York Times op-ed?", leaving out any criticism to its methodology, making me doubt any other claim Robinson would unreservedly make.

[1] https://github.com/mkearney/resist_oped

The caveats section of the one you linked to is in the .Rmd, not the .md: https://github.com/mkearney/resist_oped/blob/master/resist-o...

Reply on news.ycombinator.com