Who Wrote the Anti-Trump New York Times Op-Ed? Using Tidytext | Hacker News Reader

Who Wrote the Anti-Trump New York Times Op-Ed? Using Tidytext

varianceexplained.org

114 pointsehudla72 comments

The trick behind NLP that's surprisingly under discussed is that you must use apples-to-apples comparisons between datasets. Using simple tweets as a proxy for style against a professionally-edited NYT op-ed is silly, but this analysis (along with the previous analyses) is transparent about that fact.

The OpEd reads more like a speech. I'd assume there are some publicly available speeches (or videos with subtitles) that can be analysed instead.

Some articles mention that anonymous sources specifically take note of the speech patterns of other people so they can use those to throw off the trail. I'd be very surprised if the author didn't at least try to do that, or agree to have the NYT rewrite each sentence while preserving the meaning.

It's an interesting exercise and can and might be able to rule out some authors, but it seems like someone harboring such opinions and wishing to express them to some extent (and has with others -- in the resistance) will be discovered in short order.

Outing this person might be an even bigger story than the OpEd itself. But I put my money on old fashion journalism rather than text analysis.

Presumably the candidates for consideration have speech writers and staff who would either write a given speech outright or influence it heavily. So, that would likely rule out any speeches related to their position.

The speechwriters and staff are in some cases valid guesses in their own right, but don't have a unique corpus.

In the podcast interview (“The Daily”, also from the nyt), the contact inside the NYT says pretty clearly that they didn’t rewrite anything, to avoid detection or otherwise. I believe the quote was something like “if they write something that’s used to identify themselves, it is what it is”.

I’d like to see a comparison done against Bob Woodward’s work. As you say, 140 char micro logs are a horrible training set for analyzing a long form essay.

You think the anonymous administration official is Bob Woodward?

Do you mean "opaque"? "Transparent" usually means that someone is forthcoming, but I don't see your sentiment reflected in the article.

Also it assumes that writers don’t change their style. In this case all a fake writer has to do is drop a word like “lodestar” in the document and everyone thinks Mike Pence wrote it.

If I were the author I would take my own essay and then run it through algorithms like this and massage my word choices to match other people.

This is obvious. So irritating that someone went to the trouble of doing the analysis on an inappropriate data set.

Not really, they are just showing off their library in a novel way. The author doesn’t even pretend that it’s going to produce an accurate result.

I don't see that transparency at all, nor in the previous analyses linked to in the article. In fact, the articles seem to explicitly be on the side of their methods being definitively able to identify "the author behind New York Time’s op-ed" [1]; this article is prefaced by the title and question "who wrote the anti-Trump New York Times op-ed?", leaving out any criticism to its methodology, making me doubt any other claim Robinson would unreservedly make.

[1] https://github.com/mkearney/resist_oped

The caveats section of the one you linked to is in the .Rmd, not the .md: https://github.com/mkearney/resist_oped/blob/master/resist-o...

Yeah so a linguist already looked at this

https://www.thedailybeast.com/this-linguist-helped-catch-the...

> “What’s been done so far is not scientific and simply silly,” he said.

People. Don't get too drunk on your own kool-aid. Do I need to write an op ed that there are adults in the room :P ?

This is a very quick dismissal, when the article itself mentions this! It's entire purpose is to show off their neat library, not to give a substantive prediction on who the author is.

The author’s purpose is to demonstrate how to do this in R, and is upfront on how this is purely for academic exercise since he is comparing an op-ed to tweets. Much of the article is dedicated to analyzing how the analysis failed (it ends up recommending that Trump wrote it largely because of the presence of Russia in both that twitter account and the op-Ed)

I still recommend giving the article a good scan.

It's worth noting that Sec. Mike Pompeo used to be the director of the CIA and would probably know someone who could tell him about NLP and how to avoid it. The difference between identifying Shakespeare and identifying present-day politicians is that the politicians alive today have a chance of knowing about technology used today!

https://en.wikipedia.org/wiki/Mike_Pompeo

Not sure if you've ever worked at a large company but that isn't how it works. CEOs don't know most of the people at the lower levels other than at company functions e.g. roadshow. They spend 99% of the time with their direct reports and the board.

And even if Pompeo did know a Data Scientist who could tell him about NLP why would he ask them how to avoid being detected when writing a scathing op-ed piece ? You don't think that person would potentially leak ?

I'm not sure, perhaps if he had been presented with briefings containing evidence derived from these techniques he might be aware of their existence.

At any rate, I would expect him to be familiar with the idea of analyzing a text to find a leaker. It just seems like he in particular would be more careful.

But who knows? Speculating about this stuff is fun. You have some great points, remember how incompetent Manafort was?

Right, a useful tool would be one that allowed an author to modify verbiage to mimic another author such that analysis like this would identify the wrong person.

Is it also not somewhat equally likely someone who wanted to remain unknown to mask their own lexicon so as to make it harder to detect who the real author(s) were?

If someone were to do something like this, would it not be in their interest to assume someone else's lexicon or, at least, coopt the lexicon of others in general to throw of the lexicographic 'scent'?

In other words we can't only make the presumption that this is the author(s)' natural language.

At the end of the article the author discusses this, quoting this tweet: https://twitter.com/dmimno/status/1037490608015925248

"Now might be a good time to remind everyone that “distinctive phrases” and rare words (high TF-IDF) are not as good for stylometry as subtle differences like “and” vs “the” ratios. If you can easily notice it, someone can easily spoof it."

Then again stylometry would be the thing you'd expect to change most between a tweet and an article published with style checkers and editors.

Is "'and' vs 'the'" ratios an actual thing? I tried searching for it, but as you can imagine "and vs the" does not return any useful results on google or ddg.

here's a shout-out for stylometry:

https://books.google.com/books/about/To_Couple_is_the_Custom...

(obviously, biassed since I'm related)

fails Occam's razor

This is another abuse of pre-packaged logic which I see regularly. Occam's razor dictates that when there are competing hypotheses - the one that makes the least number of assumptions tends to be true.

This doesn't mean "automatically assume the simplest explanation is gospel truth, and if someone suggests otherwise, simply say the magic words 'Occams Razor' to them".

Maybe this occurred. Maybe it didn't. But dismissing it with what's essentially a prepackaged argument adds very little to the conversation.

If I were to leak info, I certainly would try to hide my way of writing. Not sure how to do it effectively. Possibly run it through Google translate and back a few times. Then patch it up to make it readable?

It seems easier, although maybe riskier, to have a trusted friend (or even your attorney) ghostwrite it for you. That would also avoid reintroducing your idiosyncrasies during the "patch it up" step.

If it were me, I'd make an outline and have my spouse write it (assuming I had a good relationship with my spouse).

Another obvious choice would be someone at the New York Times. Even if we could figure out who then all we'd have discovered is dubious proof on someone who might know the identity of the source, which isn't terribly useful.

Occam's razor does not necessarily apply in cases where there is an intelligent agent actively trying to deceive you, by means including but not limited to taking advantage of your application of Occam's razor.

One would assume that the author, who decided to publish a time-insensitive piece anonymously, was smart enough to employ language to intentionally misdirect analytical methods.

One can assume, but I don't know if you watched the Twitter/Facebook congressional hearings last week but it doesn't seem like a high level of technical competency is required to work in government. Text attribution probably seems like magic to most folks without a stats background.

I think the editors likely worked the content over as well to mix multiple styles into it, which would make analysis like the OP difficult or impossible. If you know people will analyze content using common tools, you can too and keep mixing until its too fuzzy to be id'd.

They say they didn't, but that doesn't mean the author didn't do it themselves.

I have a suggestion, which others may have thought of: can you randomize the style using programs?

1. You write an essay.

2. [Word-level randomization] An algorithm looks at it, as a first cut, replaces uncommon words with common equivalents.

3. [Sentence-level randomization] Changes each sentence to a randomly selected grammatically correct equivalent.

4. [Paragraph-level randomization] Adds spurious repetitions, replace some paragraphs with automated summarized versions.

i.e. destroy the rhetorical effect, but get the message across so as to defeat NLP-based identification. If the content of the message is important, and the final version is okay to the author, this might be safer for anonymous disclosures.

That might defeat NLP-based analysis, but the use of obfuscation and the methods employed still tells you something about potential authors.

If someone in an organization writes an essay disparaging that organization, but does so in an unnatural style that seems intended to defeat author identification, then you know that they're aware of stylometric analysis, and have created or downloaded a tool designed to interfere with assigning attribution. Depending on the organization, that might shrink the candidate pool more drastically than clustering based on sentence length, favorite words, or other linguistic quirks.

Perhaps it would be enough to just Google-translate the text to some other language and back?

Yes, translating different paragraphs into different languages and back might work.

I wonder if there are any ethical issues in outing someone using these techniques.

It's a bit like a 0-day if the proof is convincing enough. But publishing many dubious proofs that all point to different people isn't really much of an ethical issue.

No more so than a 'news'-paper filling its pages with non-news.

Op-ed is cheap, bulky and quick - but it's still not news.

Analyzing writing patterns to determine the author has been a tool around for decades and predates statistical techniques like TF-IDF.

How is that relevant? Are old things automatically ethical?

Well, they're common knowledge.

If I refuse to do something millions of others can easily do, it won't change anything.

This situation reminds me a little of the Unabomber, who was finally caught entirely because somebody (his brother) simply recognized his writing style.

Ted Kaczynskis writings were probably influenced a great deal by his mental illness, which would make him an outlier. In contrast, those in bureaucracies will likely tend towards milquetoast language over time, making recognition more difficult.

1) Unabomber was caught based on content more than style (his brother recognized the theme and called it in).

2) Unabomber sent a very long screed to NYT, not 140 characters.

3) Unabomber had a very unique theme, not an anti-Trump screed which is quite common. Which, based on that alone there is really no telling who did this because of volume of interested parties. ("I'm a senior administration official"). Ya whatever. Guess it provides something for bored people to get frothed up about, but it may be about as real as Cinderella's Castle at a Disney World. Except not nearly as well done.

I think if I planned to write a piece like this, I'd give an outline to a close confidant, such as my spouse, and have them write it. Assuming said confidant doesn't have a ton of public writing samples, this would frustrate any textual analysis.

Or write an outline, have someone at the NYT write it, then send it back to you and edit it for clarity and accuracy.

Someone someone at NYT could just writing it without the outline.

Either way.

I did a end-of-semester project on stylometry on my masters, and no-where in our literature review did we see anyone comparing TF-IDF's scores for authorship attribution.

Other than some guy on twitter saying TF-IDF, on what grounds is this classifier chosen?

Other mention that "This is about showing off the library". In that case, the authors should chose to show case their tools for a problem they are ment to solve.

"Check out my hammer, here's how to use it: First, you get some screws..."

I really wish some news outlet was circulating the headline “text analsysis shows Trump to be most likely writer of anonymous memo!”

The op-ed was a plant, personally approved by Trump. This has literally been his standard MO for decades: plant fake stories and weaponize the press for his own use. And it works. He effectively ended the entire Manafort/Cohen news cycle overnight.

Think about the plausibility of this story for even a moment. There's a list of less than 10 people that could have possibly written the op-ed, and what it amounts to legally is a written admission of high treason, a felony punishable by death. Do you seriously think any reasonable person would ever do that and assume they would get away with it?

The entire administration has been one massive psy-ops campaign against the American public. Don't listen to a single word they say. Just watch their actions, and react with your own judgement accordingly.

>and what it amounts to legally is a written admission of high treason, a felony punishable by death.

Here is the legal definition of treason in the US (at the federal level).

"Treason against the United States, shall consist only in levying War against them, or in adhering to their Enemies, giving them Aid and Comfort. No Person shall be convicted of Treason unless on the Testimony of two Witnesses to the same overt Act, or on Confession in open Court."

Nothing in that letter comes close to fitting that definition.

> He effectively ended the entire Manafort/Cohen news cycle overnight.

The Manafort Cohen news cycle was already ended by the Woodward book preview. All this letter did was make the accusations in the book look even more plausible.

I haven't heard anything at all from my peers about the Woodward book. I have heard about the essay. So if it was a plant, it was geared towards the book and not recent trials.

This goes a bit beyond his past attempts to manipulate the press with fake personas. The op-ed claims that staffers genuinely considered his removal via the 25th amendment due to his being mentally unfit. That's a damning admission, and one that effectively substantiates a great deal of the insider criticism and evidence of dysfunction that's been published. That's not a conversation any president would be willing to further.

It'd be like an 18th-century king trying to sniff out would-be usurpers by announcing to his court that he's illegitimate. It might throw enough blood in the water to get someone to bite, but even after lopping off a few heads, the statement--true or false--is going to haunt you. Some statements can't be taken back.

Not to mention that it doesn't really fit with his personality; he has incredible difficulty accepting even meaningless criticism, or letting it go for that matter. Approving an op-ed to go out that implicitly suggests that he's mentally unfit is not something he'd go along with for any reason or perceived benefit.

> Do you seriously think any reasonable person would ever do that

As the other reply points out, this really isn't treason. The President is not the country.

> and assume they would get away with it?

I don't think they even thought that far. They probably wrote this in an emotional state after watching (or even attending) John McCain's funeral. McCain's referenced with respect near the end of the article.

This seems like a short-sighted attempt to announce heroism as an inspiration to others. It certainly hasn't had that effect, but I try to keep Hanlon's razor in mind:

"Never attribute to malice that which is adequately explained by stupidity."

> This has literally been his standard MO for decades: plant fake stories and weaponize the press for his own use.

Trump's history of planting fake stories involves positive stories about his sexual prowess and popularity - "an anonymous source says Trump is great with the ladies" sort of stuff.

Planting highly negative stories about himself would be a massive shift in MO.

Its a good theory .. it completely supports the deep state argumment and allows for Trump an excuse to root out moles.

I can even see nyt editor falling for it to sell newspapers.

Everyone wins, right.

Amusingly, the analysis concluded that Trump was actually behind it.

The whole thing is pretty stupid

For the downvoters, you have to appreciate how stupid the whole is. Nothing was revealed we didn't already know. Republicans widely panned the entire thing.

Trump has already talked about taking on the establishment, and that's who this is.

He now has an excuse to fire if anyone doesn't do exactly what he asks.

Trump could already fire people. This changes nothing.

Unless "gutless anonymous" is a fool, they got someone else (e.g. NYT staffers) to write the actual text. Better to look at individuals who had recent physical contact with NYT senior level employees since they would certainly avoid telephone calls and email. Fairly likely the manifesto was written on a Remington or Underwood.

Jim Dao, who edits the op-ed page at the NYT, says that they did not do anything to alter the writing style:

> He said there was no special effort to disguise the person's writing style, for example by rewriting the piece in some fashion.

https://money.cnn.com/2018/09/05/media/new-york-times-resist...

And everyone tells the truth.

If you don't trust Jim Dao about this, why would you trust him when he says the author was a White House official in the first place?

Step 1: write the Op-Ed Step 2: using google translate, translate the article from English to a different language, let's say German. Step 3: still using google translate, translate the German version into a different language, let's say French Step 4: translate from French back to English Step 5: have someone else re-edit it back into English

If you're that paranoid, why would you trust Google?

Or, just use the same analysis that these bloggers used and then change your text to reshape the data. Anyone who is dealing with governments would know to watch out for exactly this, it's the first thing that people would watch out for.

Op-Eds aren't typically edited or re-written by the staff. They are meant to be the voice of the submitter, for better or worse.

Better yet,analyze emotions and truthfulness using ML against video statements made by officials who deny they're the author.

Reply on news.ycombinator.com