U.S. military to analyze 350B social media posts to understand popular movements
bloomberg.com
bloomberg.com
Many interesting things can be done with this kind of data (and it's what originally got me interested in data science https://www.springer.com/us/book/9783319140100). That being said, the specific solicitation (https://www.neco.navy.mil/synopsis/detail.aspx?id=537809) doesn't seem particularly exciting or novel. Just an overgrown grad school project to keep the students busy. With some Python experience and patience to collect the data, pretty much anyone can do something similar.
You can find the slides for one of my old (PyData 2015) tutorials on social media mining here: https://www.slideshare.net/bgoncalves/mining-georeferenced-d...
Yeah, the Overton window has now moved, fascism is now acceptable. /s
Some may ask "why is the military involved" the answer relates to the fact that today's military have realized that foreign actors will weaponize technology (e.g. internet, social media) to serve their goals. To ensure that the goals of your nation are protected, the military has moved more and more towards the cyberspace.
Here are a few benefits that can come from such endeavor:
* Relief efforts - How do people react in social media before, at the time and after a disaster happens? How can social media be used to connect to the ones in need? How can X entity use social media to advise groups in a risky area (e.g. landslide) to diminish damage and risk of life?
* Trolls - Are there cues that can be used to determine when someone is a troll? Can that information be used to help society become more resilient in the future?
* Training - Can this research help dozens/hundreds of personnel to gain valuable know-hows and capabilities in the cyber arena? Will that knowledge translate in a more capable entity?
Another question to ask is whether X entity follows your ideological view and whether there is enough oversight. If the answer is no to either/or, then as part of a democracy, you should raise your concern with representatives & the people around you.
> Our research aims to provide enhanced understanding of fundamental social dynamics, to model the evolution of linguistic communities, and emerging modes of collective expression, over time and across countries ...
> As a central requirement for this research, we seek to acquire a large-scale global historical archive of social media data, providing the full text of all public social media posts, across all countries and languages covered by the social media platform. We aim to use this research to advance knowledge through scientific publications. This data will also be used for pedagogical purposes in the classroom, giving students new opportunities for thesis research and the development of “big data” analytic skills.
How can the Department of Defense do a better job at interfacing with the general public to better explain intentions, motives etc... with the work that we do?
There are many positive and negative things that could result from something like this, and it isn't the fact that the Navy is doing it that scares people. It's the fact that the US government is doing it.
That would be akin to asking [insert company] if it will possibly ever use data collected from one thing on a different project at some later date. That is a complete unknown, so it wouldn't even be appropriate to put into a proposal, as they have specific formats.
NPS is a graduate study program, with a research component and long history of academic publishing. Whether that data could possibly, someday be used beyond that is completely out of scope for this solicitation. Further, if other groups in the military wanted similar data, they would likely not go to NPS for it, for a variety of reasons.
I think at the end of the day though, if someone has an assumption that everything the DoD does or messages to the world is somehow subterfuge, there's no amount of discussion that would reverse that feeling.
Similar undertakings have no doubt already been made in private inside other organizations.
I'm just explaining why people feel uneasy about this kind of thing. It's because of who's boss.
I'm also a realist and understand that this kind of analysis is inevitable and will eventually be continuous. I'm not arguing that it shouldn't be done.
I've just been assuming this sort of work has been going on since then.
The idea of the military arm of the U.S. government predicting the next Arab Spring reminds me of that classic mind-bender of a quote, which I think is often misunderstood as a defense of lying to the public, when it is about something deeper:
> The aide said that guys like me were "in what we call the reality-based community," which he defined as people who "believe that solutions emerge from your judicious study of discernible reality." I nodded and murmured something about enlightenment principles and empiricism. He cut me off. "That's not the way the world really works anymore," he continued. "We're an empire now, and when we act, we create our own reality. And while you're studying that reality -- judiciously, as you will -- we'll act again, creating other new realities, which you can study too, and that's how things will sort out. We're history's actors . . . and you, all of you, will be left to just study what we do."
https://www.nytimes.com/2004/10/17/magazine/faith-certainty-...
From Wikipedia:
> The free energy principle is that systems—those that are defined by their enclosure in a Markov blanket—try to minimize the difference between their model of the world and their sense and associated perception. This difference can be described as "surprise" and is minimized by continuous correction of the world model of the system. As such, the principle is based on the Bayesian idea of the brain as an “inference engine.” [Karl] Friston added a second way to minimization: action. By actively changing the world into the expected state, systems can also minimize the free energy of the system. Friston assumes this to be the principle of all biological reaction.
But before then it used to be common knowledge that you always post your real life details on the internet, at the very least your name and email, often your phone and office address too
Monitoring pacifist groups based on publicly available information may respect rights technically but it is not a good look. It indicates either major bad intentions to subvert democracy or sets the stage for any who plan to abuse it while being peripheral to their nominal core functionality.
To give a crude analogy it may be technically legal to note the comings and goings of my neighbor and every aspect of their home security but it would be very understandable if they think I plan to burgle them.
Another possibility is that the military like a large company has many arms and legs and highly possible that it is something that is already done, but by some seperate part of that collective that the rest are not aware of. I've seen a company the size of 100 have two people do the same project for nearly 6 months before they realised about the other and the duplication of work. More people, more separate divisions/locations, more chance duplication on some levels happens.
For me, I suspect the later, but equally mindful that as a tactic - doing something and then after you have done it, say your going to do it to gauge reactions - is a tactic that would not be out of place in any military intelligence collective.
Or maybe they have learned a few things by reading Sun Tzu.
Perhaps not as individuals, with enough personal meta data your identity can be reverse engineered.
That aside, anyone you believes there is no hidden agenda here is naive. Any understanding will have a second benefit for doing the opposite (i.e., preventing movements).
That depends on who wins the war, and thus writes the history books. From the point of view of the British, the American Revolution was an act of terrorism and treason.
Attempts by the military to understand (and, one would assume, prevent) popular uprisings would nip a second American Revolution in the bud, for better or worse.
The linked article refers to an attempt to entrench the status quo, and not everyone is pleased about that.
The US military is interested in mapping these activities and developing countermeasures.
Information/disinformation operations have been a part of warfare since... ...forever it seems. It is - certainly - nothing new. The medium has merely changed and that's what important to the staging of the front[s], as it were.
You should pick up a copy of Eric Frank Russell's WASP[0], if you haven't read it before. It's a very good read in this area.
Sometimes, countries protect themselves using revolution in other countries. For example Lenin wanted to make the revolution in Germany, as they believed Russia was not prepared for revolution. Lenin was captured in Germany and released with the condition of not trying it again on Germany(or he would be shooted and killed) and sent to Russia, where he started the Soviet revolution.
For example, United States wanted to control the Panama channel, so they supported an "independence" movement against Colombia, so a new "free" country was created, that by "coincidence" gave the US the total control of it for a long time. Even today, the US is in control of it.
Before that, when the US felt the Spanish Empire was weak, they supported the "Cuba(and Philippines) libre" movement. People in Cuba and Philippines were very happy for the help until they realized the helpers army did not go away after the "independence". In Philippines the US exterminated over a million people, forcing everybody to speak English.
After Panama, the US continued supporting coups in other America's countries and around the world.
One of the most important movements the US supported was the popular movement in Persia, now Iran, because they wanted the British Empire to leave(so their own companies could enter and control the oil).
In the end, this movement converted Iran from secular country to a radical Islam one, and the Americans were ousted.
For the US it is routine creating havoc around the world, as the hegemonic power in the world that they are. All empires before had done that too.
They would love to be able to support popular movements in China or Russia as it is a threat for US hegemonic power in the future. Getting united China to disintegrate in several smaller countries would be a great victory.
But Chinese people already play the game.
If that message comes off abrasive it's only because I am genuinely confused, but would like to know where you're coming from. There's always a risk of us living in our little bubbles and so I make every effort to try to see things through the eyes of others, especially when I simply cannot otherwise empathize with their views on any level.
Bigger question is why in the world this would be made public. I think it's a 100% safe bet to say that the military/intelligence has been engaged in this research for many years already. I see no positive benefit to making this information public aside from preempting 'leaks' / creating preemptive PR.
[1] - https://en.wikipedia.org/wiki/COINTELPRO
[2] - https://en.wikipedia.org/wiki/FBI%E2%80%93King_suicide_lette...
There's probably a cultural component to what side of this one comes down to, how much privacy you can expect (and be granted by law) in public spaces is different between countries.
I don't have to worry about someone seeing a crude joke made 10 years ago and starting a massive deplatformization campaign against me in the name of social justice.
When I make a post, it's there forever.
I know your analogy seems to make sense, but these two phenomenon are incompatible. You cannot translate this digital phenomenon to that physical phenomenon.
I agree, they aren't the same in that regard. The expectation is similar, that's what I was trying to say. When people make a crude joke or call somebody an asshole on Twitter, they don't expect that to be read by everybody.
Somebody compared a public post to standing on a soap box. If you're standing on a soap box, you expect what you say to be heard publicly. I believe that's primarily because you usually don't stand on a soap box, so being that exposed really stands out. If everybody stood on a soap box all the time because the floor is lava and soap boxes are heat resistant, that would quickly change and people wouldn't associate a soap box with "public". The early days of social media certainly had that, it was new and everybody was acutely aware of it. That changes with exposure, now it's no longer an outlet to let the public know about your opinion, it's a chat where you talk one on one or in small groups (for most users, very few have large followings and get a lot of visibility) to friends and strangers alike. Having some tweet go viral is the equivalent of some reporter overhearing your private conversation and mentioning what a vile person you are in his column.
If we're keeping up with analogies, that's like saying someone doesn't expect what they say on a soap box to be heard by anyone. That's a false expectation, however.
We have to strike a balance between ideology and pragmatism. The best compromise is likely preventing the government from having access to data we consider private, while understanding anything on the other side of the line is fair game and accessible to friends, family, politicians, corporations, and historians alike.
The only way we can achieve this is by limiting what can be done with this data aka fixing the US government. We cannot, for example, have public or inferred health data used to prevent people from attaining health insurance.
That's my point. If "soap box speaking" was as common as tweeting or posting on Facebook or speaking to a friend in a pub, people wouldn't expect it to be different. When social media changed from "technology pioneers use it carefully" to "everybody uses it and rarely thinks twice before hitting send", the expectation of the average user shifts.
> while understanding anything on the other side of the line is fair game
I do agree: Fair game for any person. As soon as technology enters the game (scrapers, databases etc), it changes. Similarly, you can sit in a park and catch fragments of what the people walking by talk about, that's all right. If you're using a directional microphone to listen in on people on the other side of the park, that's not. A user doesn't expect to the target of a microphone or a webscraper/data mining operation.
If people don't expect this, it's because they don't understand it.
Let's work on educating people instead of warping reality around the ignorant.
In that case, you're allowed to remotely listen in on everyone as long as they're in a public space.
> If people don't expect this, it's because they don't understand it.
Absolutely, and because everything is done so that they either don't understand it, or quickly forget it. Change Twitter's send button to "I want everyone, including my enemies, the government, journalists and my mother to read this" ... and you'll pretty much kill Twitter.
> Let's work on educating people instead of warping reality around the ignorant.
The ignorant make up the majority. We can't educate everybody about machine learning to the required degree where they can make an informed decision about it. The same goes for financial instruments or law. Instead, we just outlaw the most egregious transgressions ("no, it said on page 277 of the contract that we get all the money he'll ever make in return for this bicycle") and base what's allowed on what a reasonable person would expect.
You're willfully continuing to ignore the soap box analogy.
I'm also sure the government will shamelessly make the argument that because you're telling something to a friend in a "public place" it means it's not a private conversation, so it's fair game for mass-collection and real-time analysis. It may have already made this argument in some court cases, but I don't recall exactly.
What the court does is muddy the distinction between a theoretical reasonable expectation of privacy and the actual expectation of privacy resulting from how things currently are.
As in, I expect conversations in my car to be private, but if all cars had microphones with cell connections my government would argue that I can't expect privacy because clearly my environment already has privacy-compromising features. [0]
It's cyclical and deceitful and until we stop playing ball and change the language, the farce will continue.
[0] https://www.forbes.com/sites/thomasbrewster/2017/01/15/polic...
On the other hand if the government turns on all connections and starts to bulk process all of them to find "interesting" discussions, and then complies dossiers based on these which are then used to determine probabilities of behaviours... and issue sanctions, constraints and awards based on these.. Well. My bet is that almost everyone would object.
Somewhere between these two extremes, absolute surveillance and extremely limited state power lies the correct position. My own view is that it should be much closer to the limited position than state omniscience, the core question is do you want freedom, or do you want safety. In the modern world you (I, anyone) can't have an absolute of both. Absolute freedom would result in the dominance of lunatics with assault rifles (cf. anywhere in the modern world where state power collapses). Absolute safety means that your neighbours and family will regularly face state sanction - even if by some chance you don't.
The fly in the ointment is that I fear that state omniscience is stable in that a society run in that way could continue indefinitely. On the other hand the extreme of liberty (given assault rifles) is clearly not sustainable (in that school, hospital, roads, power, water, cannot be delivered). So we may see a world in which a core territory (China) is run by big brother and a periphery (to steal William Gibson's term) is allowed to seethe and boil in anarchy.
To avoid this we need to develop civil society and generally learn to get along with each other, and we need to punctuate this with action to lock up lunatics and psychos who are taking advantage of us. We also need to find a way to neutralise a core that is seeking to actively destabilise the periphery. Interestingly I think that HN is one of the very few places where this kind of action can be developed.
In your vision of a perfect world. Personally, I’m fine with this. They aren’t using call logs or other private data. Information on social media is public or semi-public. Fair game, and I’m curious about the results.
(Agree that a more productive public debate is needed. But that shouldn’t start by presuming values.)
These “should we do X” questions tend to be useless. If it can be done, it will be done, absent multilateral action. Being able to predict social upheaval for humanitarian and geopolitical positioning purposes is obviously useful.
The government using such models to suppress activism in the U.S. is bad. But knowing how such phenomenon work is also a precursor to preventing its abuse.
One problem with the current state of the privacy debate is this sort of hyperbole.
No, I’m not. I delineate between information people intentionally cast publicly—even if they don’t understand the consequences of doing so—and information that occurs in public. My location, tracked by my phone and router to various apps’ servers, occurs in public. My tweets, on the other hand, are intentionally broadcast. There is a lot of grey area between these poles, but intent broadly inspires my thinking.
Extending “I’m fine with the U.S. military analysing social-media posts” to “okay with someone following you 24/7/365 every moment that you step outside your house” is hyperbolic. Framed as a question, on the other hand, it invites discussion.
As such, my point stands.
> Information on social media is public or semi-public. Fair game, and I’m curious about the results."
Perfect world achieved!
Then there was also the case of a Pentagon contractor leaving their AWS buckets open exposing billions of scraped social media posts. [1]
As much as people can reach for benign reasons for "research" like this, the reality is that it's not solely done to "understand" movements but rather to better influence them [2].
[0] https://news.vice.com/en_us/article/zm5k4a/the-troubling-lin...
[1] https://www.upguard.com/breaches/cloud-leak-centcom
[2] https://www.theguardian.com/technology/2011/mar/17/us-spy-op...