“Should this even be released?” Deep learning tool that may be used for doxxing | Hacker News Reader

“Should this even be released?” Deep learning tool that may be used for doxxing

github.com

99 pointsZuider61 comments

There are papers on how to do this, and many approaches work. Traditional statistical methods[1], support vector machines,[2][3] (software available at [4]) and a random forest algorithm [5] have been shown to work, more or less. This isn't a new idea. All this new code does is let us compare how deep learning does on the problem.

[1] https://www.aclweb.org/anthology/E/E99/E99-1021.pdf [2] http://ceur-ws.org/Vol-1391/126-CR.pdf [3] https://www.aaai.org/ocs/index.php/FLAIRS/FLAIRS13/paper/vie... [4] http://www.cs.waikato.ac.nz/ml/weka/ [5] http://ntv.ifmo.ru/en/article/15185/kompyuternaya_kriminalis...

Yeah, it should be released.

I mean sure, there are various 'immoral' uses for it (like doxxing), but there are also many good ones. Such as:

1. Working out who wrote a bunch of anonymous reviews on Amazon or other such sites, which could be used to stop fake reviews. You actually mention this usage in your article.

2. Being able to identify troublemakers in a community (such as a forum or a social networking site). I'm sure a lot of administrators would love to know if that suspicious looking new guy is the alias of a banned troll from a few weeks back (posting through a proxy server).

3. Literary analysis, like working out who wrote many anonymous works of fiction. Or as someone said below, determining which parts of Shakespeare plays were actually written by Shakespeare.

4. Crime solving. If it works anywhere near as well as you say, it could theoretically help unmask the Zodiac Killer, or perhaps even Jack the Ripper (if any of those letters were real).

All the uses above would be a net positive for humanity, and would be great possible uses for a deep learning tool like this.

Don't let the worries about its usage by 'bad' people overshadow the good you can do by releasing it.

Well, if it works, it probably only works for a little while. This is because people who don't want to be identified can run it too, and they can change their prose until they can't be identified.

Whats worse, after a while, and enough trolling in different styles, this tool might ban more and more legitimate attempts to communicate.

>"2. Being able to identify troublemakers in a community (such as a forum or a social networking site). I'm sure a lot of administrators would love to know if that suspicious looking new guy is the alias of a banned troll from a few weeks back (posting through a proxy server)."

Orwellian.

It would certainly run the risk of false positives.

This is a good point though. Even a 5% chance of it misidentifying someone as a troublemaker would be a problem for an online community site, and it's likely the actual chance is a fair bit higher than that (because it's likely not been tested on a particularly large scale in this context).

So, what do you when a troll persistently and constantly attacks your community website?

As in, flames everyone to a crisp, posts as much porn as possible, tries to incite a civil war between a few members that might not be on good terms with each other or the staff and registers hundreds of accounts, some of which stay semi dormant until they strike?

Because that can happen very easily online, especially if you get the ire of someone with a lot of free time and very few morals. Or if your site ends up at war with a troll site/gets raided by 4chan.

Do you avoid the hassle now, or wait until the situation blows up and half the site is now in the middle of it?

Ban behavior, not people. You can convert non-productive people to productive people, most of them only want to be noticed or accepted and past a certain point there is only so much you can do to block anyone anyway. Stylometric analysis would just be another simple hurdle to cross for a persistent person.

The idea that this tool would be useful for community management is terrible.

You can convert non-productive people to productive people, most of them only want to be noticed or accepted

I'm a moderator of multiple online spaces.

A few months ago, in one of them, a user got too heated and started flinging insults at someone else. As was standard policy for the place where it was occurring, I issued the user a ban of a few days (enforced cooling-off) and pointed to our guidelines on how to behave.

This user then proceeded, over a period of months, to continually harass me, send me increasingly graphic threats, and try to track me down in real life.

Pray tell, how exactly would you go about "converting" such a person to be productive? I come to you since you are apparently quite the expert on it, or else you wouldn't be giving out advice to just "convert" people.

Do we really need machine learning deanonymisation tools to identify trolls so unsubtle and obvious?

I voted to release the tool, but you've set up this false scenario with the intention of knocking it down easily and discrediting the opposing argument.

In my day, we'd use moderators: put the forum/list in a mode where every new post goes through a human review first. After a while, things calm down, and you revert to the anything-goes default.

I don't know how well that would work for high-traffic forums of today, but it can scale pretty easily by employing many moderators.

http://www.netlingo.com/word/moderated-mailing-list.php

ISTM that any site that is capable of controlling spam, is also capable of dealing with trolls. Sure spam is more repetitive than trolls, so it's easier to notice automatically, but there is also much more spam, so chances are that at least some will get through. If you have moderators authorized to zap spam, let them zap trollage as well.

That's especially troubling because the easiest and cheapest thing for any commercial service to do is to ban forever and without appeal any user that presents a hint of trouble to the algorithm. If these services have near monopolies, or share data with all of the other services like CLUE does for insurance... Its like a digital death sentence.

Ask anyone who's ever lost an ebay or a google account to an algorithmic burp and was essentially banned for life without appeal or even human oversight.

What about Russian trolls that have infected the internet, trying to manipulate public opinion or just collect data on regular citizens? Or Correct The Record? Or just about any other mass propaganda campaign? This would be very helpful

Well not necessarily.

1 and 2 could both be a bet negative for humanity just as easily as they could be a positive.

Yeah, I just realised a few negative uses for 1 right now. Like those cases where a business threatens to sue anyone who leaves negative feedback, and hence such a tool could be used to unmask anonymous reviewers giving a honest evaluation of their products/services.

Or maybe an odd case where it turns out the author of a book or creator of a product finds out someone they know in real life left the negative review and physically attacks them or something.

Essentially killing anonymity on the web is not worth any of these. Facebook alone provides a huge data trove of people's real names linked to posts that this machine could parse. Anyone with any form of professional presence on the web would possibly be vulnerable to any comments long enough to be matched on other sites.

Unfortunately if the tech exists, it will be released, but I don't think the positives outweigh the negatives here. The chilling effects in terms of comments alone would be bad.

I'd guess if it can be used to identify you, it can be used to adjust your new posts until they don't appear to be written by you: for a practical example, a Chrome addon that detects when you are making social media posts and automatically flags identifying information like a spellcheck does. So the thing can be used to counter itself.

Aren't these more or less the arguments used to justify anti-cryptography legislation? And when articles about banning encryption come up on here, people say pretty much the same thing: "Don't let the worries about its usage by 'bad' people overshadow the good".

In fact I believe this tool is even more worrisome, because there are a very large number of non-tech savvy people who express their dissonant opinions simply under the mask imparted by internet anonymity. I imagine most everyone here on HN have at least once made an anonymous account to post a comment somewhere that they would rather not have tied to their identity. This is an avenue that is necessary for the preservation of free speech. Remember that things like treating black people as equals, giving women the right to vote, gay rights, etc were and in some ways still are taboo subjects that bring the wrath and ire of the power du jure.

All that said, I still don't see any reason why such a tool shouldn't be released. Why? Even though I believe the tool to be harmful, it's better to know that it exists, know its capabilities, and most importantly know how it works. It'll end up in the hands of the wrong people anyway, so it's better to at least get it into the hands of the right people who can possible do something to combat it.

" Even though I believe the tool to be harmful, it's better to know that it exists, know its capabilities, and most importantly know how it works"

This is a really important point. Even if you don't like the possible uses of the tool, it's either "release it now and make it possible to defend against" or "don't release it, and hope the likes of the NSA don't develop their own version".

If the tool is as accurate as the author implies, I think it would be great to release it for literary analysis. For example, it might help to determine which fragments of Shakespeare plays are written by Shakespeare and which, if any, were remembered incorrectly by the actors who reconstructed them after his death.

Yeah but conversely, or, rather additionally it could be used to unmask anonymous authors who for political reasons wish to remain anonymous, like whistleblowers, dissidents, etc.

There already exists a counter tool to help against this kind of privacy invasion

https://github.com/psal/anonymouth

I was just going to ask if such a counter tool existed already. Wouldn't be too hard to put a web UI to this and make it easily accessible. Thanks for sharing.

I just tried to get anonymouth working, but unfortunately even after fixing the invalid code issues/errors preventing compilation, it crashes after you fill out about 6 screens worth questions about where various types of text content is located:

    >>>>>>>>>>>>>>>>>>>>>>>   LOGGING STACK TRACE   <<<<<<<<<<<<<<<<<<<<<<<<<
    java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
    java.util.ArrayList.rangeCheck(Unknown Source)
    java.util.ArrayList.get(Unknown Source)
    edu.drexel.psal.anonymouth.utils.FunctionWords.run(FunctionWords.java:60)
    edu.drexel.psal.anonymouth.engine.DocumentProcessor.processDocuments(DocumentProcessor.java:140)
    edu.drexel.psal.anonymouth.engine.DocumentProcessor.access$000(DocumentProcessor.java:39)
    edu.drexel.psal.anonymouth.engine.DocumentProcessor$1.doInBackground(DocumentProcessor.java:70)
    edu.drexel.psal.anonymouth.engine.DocumentProcessor$1.doInBackground(DocumentProcessor.java:67)
    javax.swing.SwingWorker$1.call(Unknown Source)
    java.util.concurrent.FutureTask.run(Unknown Source)
    javax.swing.SwingWorker.run(Unknown Source)
    java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    java.lang.Thread.run(Unknown Source)
    
    ( ErrorHandler ) - Fatal error encountered, will exit now...

It's too bad the flow isn't more along the lines of "Give me some docs from one author, and the other document you want to test. Okay! Here is your result!"

From this trace it looks like the file jsan_resources/koppel_function_words.txt is missing/empty/invalid.

Thanks for the info, that file appears to be a clean and intact list of english words, except it is under the path:

    anonymouth/src/edu/drexel/psal/resources/koppel_function_words.txt

And down the rabbithole we go; after copying the file to src/jsan_resources we now get a new error:

    >>>>>>>>>>>>>>>>>>>>>>>   LOGGING STACK TRACE   <<<<<<<<<<<<<<<<<<<<<<<<<
    java.lang.NullPointerException
    edu.drexel.psal.anonymouth.engine.InstanceConstructor.getAttributes(InstanceConstructor.java:182)
    edu.drexel.psal.anonymouth.engine.InstanceConstructor.runInstanceBuilder(InstanceConstructor.java:121)
    ...

    >>>>>>>>>>>>>>>>>>>>>>>   LOGGING STACK TRACE   <<<<<<<<<<<<<<<<<<<<<<<<<
    java.lang.NullPointerException
    edu.drexel.psal.anonymouth.utils.TaggedDocument.makeAndTagSentences(TaggedDocument.java:380)
    edu.drexel.psal.anonymouth.engine.DocumentProcessor.processDocuments(DocumentProcessor.java:184)
    ...

I'm curious how you can know the tool is 95% accurate, if it's being tested on real world data, such as from reddit etc.?

I can only assume it was tested on a synthetic dataset perhaps.

Also I'm wondering how many unique users are present in the dataset, along with the volume of content for each user.

You can easily turn any dataset with labeled authors into a de-anonymization dataset: split each author's writings in half and give them different IDs. Now you know the true answer for every pairwise combination.

Yeah, that's a good point. So with that approach you could even get results of accuracy from data from a single social media source at least I guess.

The problem with that approach is that it is training the classifier to solve a different problem than what is claimed. It assumes that there is no difference between how people write when they are identifiable and when they are not (since it would only train on identifiable samples that are anonymized after the fact). Further, it would require solving the problem of identifying authors between different media--which would be a huge achievement on its own.

A great test set for anyone trying to do this: look at the Scott Adams sock puppet controversy on Metafilter [1] and see if you can train something on his public writing to match the "PlannedChaos" commenter's posts and Adams' own tweets. It is probably the closest you could get to a "pure" training set in the sense that presumably Adams didn't think he'd get caught. (And if he did, and therefore did alter his stylometrics, then it's even a better challenge.)

[1] http://www.adweek.com/galleycat/scott-adams-caught-defending...

If there's a 5% false-positive rate, then if I give it an unidentified text and the posts from 1,000,000 identified Redittors, it's going to give me 50,000 possible authors? That doesn't seem either useful or troublesome...

That in and of itself would not be more than marginally useful or troublesome, but usually it won't be used by itself. Usually it will be combined with other evidence that you already have via Bayesian reasoning or intuition.

For example, suppose someone is revealing on Reddit details about some business dealing of yours that should have only been known by people who are under NDAs. If you intersect the set of 50000 Redditors returned by the deanonymizing tool with the set of people under your NDA, and that intersection is not empty, then the leaker is probably one of the ones in the intersection.

Satoshi can probably be revealed using these techniques. It's a double-edge sword.

If it works, prove it by unmasking Satoshi Nakamoto.

This is exactly the type of thing I'm afraid of when I read the link. Being doxxed due to something I did is one thing, but being doxxed due to something I didn't do is something else altogether.

Can you imagine how much it would suck if you woke up the next morning and the entire internet is convinced that you're Satoshi Nakamoto or a pedophile due to a false positive from this program? There is no due process and no chance of appeal; your social life is simply over at that point. All because of a 5% chance of a false positive.

That's already what happens with false rape/pedophile accusations due to America's love of "trial by media". One false rape accusation and your photo is all over the local media. Your life is ruined.

I've addressed this in part in my own reply to the parent. But the larger one is that with 1) pervasive information and 2) very cheap analysis or assertion, you're hugely increasing the potential for this type of abuse.

The limit is in attention paid -- the public has a limited capacity to absorb information, and there are a few hundred, perhaps a thousand or so "top celebrities" at any one time.

And some of those can attain a highly significant level of immunity to criticism. Ronald Reagan's presidency was the most scandal-prone in recent memory, and yet his moniker was "the Teflon president". William Jefferson Clinton took far more flack for far less, and Barak Obama takes the hit for complete fabrications. Meantime, a major party presidential candidate advocates overt violence to protesters and various other views ... and is only embraced all the more strongly by his supporters.

The dynamics of this are odd.

That's among the true terrors of a global information domain society. It's not that you can prove the goods on anyone, it's that you can plausibly assert them. And proving a negative is very, very difficult.

That said, I'm not sure the genie can be rebottled. It's an area of privacy in which law rather than technology must be applied. Including, say, exceptionally fierce penalties for misuse.

It's already been tried, apparently without success.

My point exactly.

I wonder whether the code actually exists, or if the whole thing is just intended to provoke discussion.

This is valuable code and should be released, but the product's naming and one-line about already betray it's intended purpose (as the author envisions), providing an additional vector of criticism.

In cases like this, I feel erring on the side of being less explicit tends to help. Leave just enough out to let everyone read between the lines, and put the pieces together.

Don't say it's a "Machine learning algorithm to connect anonymous accounts to real names", say it's a 'speech pattern analyzer', or say it 'allows comparison of speech patterns for likelihood of same author'.

Good point, but personally I think it's a breath of fresh air that not only has the author elected to forego the doublespeak on describing this tool, but they also sparked a discussion on the ethics of using it.

It should be released. Its the equivalent of stopping medical research because it could be used for biological weapons.

The people who need this for evil purposes will develop it whether it's released in open source or not.

This should be released, if only so that we can learn how to evade it. Perhaps it could be broken by only writing anonymously in a different language from that of your public life, for example.

Doxxing does makes me feel bad. Fixing the problems that led to for example throwaway Ask HN posts is in the long term a better solution, though may be easier said than than done. (Yes, I mean doing the Ask HN posts non-anonymously instead.)

It would be interesting to create a tool with the opposite functionality which would take writing in one's own idiosyncratic style as input, and output writing in a more generic style. Better still if it improved composition and presentation.

I wonder if this could work when confronted with "hivemind" twitter accounts shared with lots of different people. I don't see how it could get enough signal from the noise to allow someone to unmask any identities.

This entire post and the debate surrounding it, is frankly stupid. What does 95% accuracy even means?? Consider face recognition, even when there is good gold standard for matching faces (human judgement, since human are good at recognizing faces), determining accuracy of Face recognition algorithms is still challenging (E.g. Megaface challenge). When it comes to a piece of text written by an author its even more difficult. There are several practical problems too, such as how do you distinguish Quotes and copy-pasted paragraphs from rest of the text.

This sounds like a beginner who created a dataset, with a flawed metric. And is now going around claiming 95% accuracy, using "Deep" learning. And equally clueless commenters are hyping it up.

Why stop at claiming 95%, hell even I can create a "dataset" and a "deep learning" algorithm and get 99.9%.

I am not discounting that there are legitimate stylometric analysis methods, which have been peer reviewed. But please lets not hype "Deep learning for doxxing". This just sullies the real progress being made in deep learning.

Agree. Rather than release the code, release the testing data set and/or test methodology used to make a claim of 95% accuracy; no one is harmed and it should give a good sense of the claim. I wouldn't be shocked to find that the score was achieved by overtraining on a very small and narrow data set (small number of identities), and that the model isn't generalizable.

That said, identification may be a more tractable problem if you have a limited population, additional metadata for features, and normalized writing samples (comparing anonymous reviews to identified reviews, or within community posts, as opposed to trying to compare a set of anonymous tweets to an identifiable dissertation).

> This entire post and the debate surrounding it, is frankly stupid. What does 95% accuracy even means??

Generally in supervised machine learning a claim of X% accuracy means that when tested on a large dataset for which the correct result is known and that was not part of the training dataset or validation dataset, it classified X% of that dataset correctly.

Typically you gather a big dataset of labeled data and then split it randomly into training, validation, and test sets. A 50/25/25 split is common. If the learning approach you are using does not need a validation set, then 70/30 training/test is common.

How reliable such an accuracy estimate is depends on how well your dataset matches the characteristics of the datasets people will be using your trained system on. His 95% accuracy report is probably reasonably reliable when his software is used on anonymous posts on the forums where he gathered his datasets. It would probably be less reliable looking at anonymous posts on, say, a bagpipe maker's forum.

Huh... of course I know definition of accuracy, and how its calculated.

However even in supervised learning, accuracy is only used in very limited cases such as multi class classification. For a whole bunch of problems including the one being discussed its a poor and in some cases a biased metric. E.g. consider a heavily unbalanced problem 99% positive labels. By predicting all instances with majority label its possible to get 99% accuracy. There are several better metrics, False Accept rates, Precision Recall curves etc.

Without knowing how the dataset was collected, did the "username" leaked into the dataset, etc. its impossible to evaluate such outlandish claims.

The whole moral and ethical debate is non-sequitur, and harms legitimate deep learning research.

I understand this concern. It's like the old 20 Newsgroups data set for testing classification, where supposedly you're distinguishing the topics of conversation between comp.graphics, sci.electronics, talk.politics.misc, and so on...

...but what the most effective classifiers do is memorize the names and signature blocks of people who posted in each newsgroup.

People tend to think and write in the same familiar memes and patterns - I doubt very much it could discern individuals based on text samples.

that this exists means that other similar tools exist which we don't know about (because any good idea is simultaneously discovered by multiple unconnected people around the world), so release it

This will lead to political dissidents being outted.

so is this how Roko's Basilisk will find you?

Yes

Yes, at least for demo purposes.

Reply on news.ycombinator.com