The Surveillance Engine: How the NSA Built Its Own Secret Google
firstlook.org
firstlook.org
[1] https://www.techdirt.com/articles/20140203/11143926078/paral...
I thought it was amazing that the government spent so much time discussing the call records being logged.. when they are doing so much worse. Maybe that's how they keep people focusing on what the government wants to talk about? (aka look over here, nevermind that thing over there...)
This find is way worse than call detail records..
That doesn't mean the call records issue wasn't important. After all, they pick their torture and drone assassination targets based on that alone, in many cases. But considering the Internet is such a big part of everyone's lives these days, I think NSA getting a firehose from the big carriers and cable providers is a bigger deal, yet I've heard very little talk about denying NSA access to the undersea cables.
I agree doubly so however - the phone records are near the very bottom of the list when it comes to the severity of what Snowden's(and possibly other's) revelations have brought to light.
When folks tell us crazy things, like the government is tracking every place you go and your opinions through your cell phone and social networks, we're supposed to say something like "That's extraordinary. With extraordinary claims, we require extraordinary proof" Then, if they persist, we're supposed to say something like "Such a program would require far too many people to keep a secret. We couldn't even keep the atom bomb a secret. The government is terrible at keeping secrets. Such a claim is just too far-fetched."
These are the traditional things taught to people who are supposed to be clear-headed and rational. It's the way we engage crackpots without taking them too seriously.
These responses seem to have failed us miserably in the current circumstances. As it turns out, yes, that's what they were doing, and yes, it was extraordinary and required lots of people to keep incredible secrets. But it still happened.
These things keep happening in the realm of automated surveillance, both by the government and corporations (and worse, when corps do it and the govt scoops it all up later) that would have been considered completely whacked just ten years ago. The stuff of paranoid fantasies.
Our tools of rational inquiry have failed us.
I have not heard a single conspiracy theorist suggest any of the things disclosed by Snowden, and I have yet to see evidence of RFID chips and the government monitoring us through our microwaves, or that Person of Interest is based on a true story.
Extraordinary claims still require extraordinary evidence, and that evidence usually comes from someone involved, not from a random blog/site on the internet, or a youtube channel.
The one thing everyone misses with this, which is my main concern, is less that the entire NSA/Insert Alphabet Soup here can track me, but that any one individual within these organisations who decides they don't like you for any reason can majorly screw you over. It is not the whole that concerns me, it is the parts.
If these evidence-attenders and conspiracy theorists were maligned as being on the lunatic fringe up until recently and were fully exonerated, what else could they have been right about? This question has the government running scared, scrambling to reassure the public that the now-redeemed former crackpots are still off the rocker, and have nothing else that is relevant to say about things which the public doesn't yet believe.
The public failed these conspiracy theorists because they accepted the establishment friendly line that they were kooky-- the public accepted the false associations and intentional counfoundment made between people with legitimate but not mainstream ideas backed with evidence, and various cultish, indefensible fantasy theories like aliens controlling the government, and the moon landing being faked. Thankfully, the public received a gut shot when Snowden went public with the truth. Maybe next time people will listen a bit closer to voices that the government would love to suppress.
https://firstlook.org/theintercept/document/2014/08/25/shari...
Netscape. With an AIM launcher in the status bar. Accessing TOP SECRET//COMINT//NOFORN resources.
Would it have been more secure if they used IE 6?
Now we have proof. So that's that.
Also, given the strong "disincentivization" for whistleblowing that is going on at intelligence agencies, I am not at all surprised that it took a few years before somebody went public with it.
Hell, it used to take decades before this kind of stuff became public. Just this year we were made aware that the West German spy agencies opened _all_ cross-border inter-German mail. You know, the kind of thing that we in West Germany were told only the East Germans would do. That was kept secret for more than 20 years after it stopped being done, even though thousands of people were involved.
I personally keep to a simple rule. If something is technologically feasible[0], economically viable and someone has an incentive for it, it will happen, period. This rule of thumb successfully filters out weather control crackpottery while correctly identifying surveillance capabilities (and no, it's not hindsight bias; the only thing I was surprised about Snowden revelations was that it was underwhelming).
[0] - and keep in mind that pretty much anything is.
The technological ones. I was expecting (and I still believe) that they're tapping much better into the Internet that it is revealed, and that there are many more hardware backdoors out there.
> Countries now also have the ultimate argument for the balkanization of the Internet.
I agree. This is very bad. I didn't say I was underwhelmed with outcome (though I am with the reaction of public and of Internet companies), just with the revealed capabilities of the NSA.
I would already know exactly which employee to target based on your dubious interests gleaned from your internet searches.
You'd have to have pretty big balls to fuck with a three letter agency like that.
Makes sense, though. The people who are the best at math are going to be the people who are the best at grokking and expanding upon the existing understanding of the math involved in cryptography.
But for contractor, there are job boards for people with clearances.[2] Although you get a pay boost of at least 10-15k, I'm sure the bureaucracy is awful.
I also wouldn't assume that all the top people go to biz. Some very talented people work in gov and academia. Not everyone is guided primarily by money & scores.
[1] http://www.math.uh.edu/~tomforde/Web/Jobs.html [2] http://www.clearedconnections.com/
Also you have to be an American citizen with parents who live in America. At least I know this is the case for civilians who want high security clearances.
Maybe that is what you are experiencing?
[1] https://firstlook.org/theintercept/2014/08/20/u-s-military-b...
[2] https://thebulletwire.com/u-s-military-bans-glenn-greenwalds...
The only thing I can think of, is internal issues at political level (who said what lies to whom to achieve more funding about his work, etc.) which even that won't take long before the interested party eventually finds out.
Once the military decides that they get to ignore laws which are silly, they will decide they can ignore laws which are unenforceable in practice, and then the Republic will fall.
You either have a culture of civilian control of the military, and a nation of laws, or you don't. The Church Committee is what resulted from the last time the Intelligence Community decided that laws are silly; is that what you want to go back to? Before you answer, keep in mind that the NSA was actually following the laws this time, so the situation as you see it post-Snowden could easily have been far worse.
There is a memo from a few years ago that prohibited accessing Wikileaks from personally-owned computers, but the order was rescinded.
"Google was also contracted to provide the search technology and servers which makes up “Intellipedia”, a massive, incredibly secure database through which over 100,000 United States spies and intelligence professionals from 16 government agencies share information. And the NSA has purchased numerous servers from Google, utilizing their search technology to analyze and organize massive amounts of secure data all over the world."
At this point I think it safe to assume its the exact same code Google used in 2010, just a bit dumber, since Googles search engine only learns by being used, and since there are far fewer using it, it learns slower.
[1]http://questioneverything2012.wordpress.com/2012/03/14/googl...
So it is news, to those of us who haven't read this blog that makes an unsourced claim.
Besides, from the Greenwald docs, we've seen that Intelipedia is simply a giant Wikimedia instance, nothing fancy.
PRISM is bad enough in US hands, but can you imagine if Oceania/Arstotzka got their hands on the data? (irrelevant countries as to not have to argue that)
Anyway, however damaging it was, my point is that some slides with a bunch of code names, "data-sharing partners", hints at legal justifications and the bureaucracy set up to carry this out, and maybe some high-level descriptions of how the systems work, is a lot different than releasing the actual code that does it, private keys to the servers involved, etc.
It's not like graph theory and data mining are elite subjects only NSA knows, so some high-level descriptions aren't going to help the bad guys much. Handing over complete source code would.
If the NSA could justify the time and expense to make a search tool, there must be a lot of searching needs... which aren't legal for the NSA to send to the domestic-oriented agencies.
Also, more explicitly-stated details about how powerful "metadata" can be. Obvious to anybody who understands INNER JOIN, but this is still new for most people.
//Of course, I'm sure all this ignoring of warrants and the 4th Amendment, to quote a DOJ lawyer in the recent EFF mess regarding Jewel v NSA, is all for the "national security of this company". (emphasis mine)
The problem is (as it's always been) that they aren't just criminal suspects and they aren't all foreign. But that was the problem whether or not the NSA built a "Google."
It's even more important for us to know what kinds of data about us domestic law enforcement has access to, than it is to know what foreign intelligence has access to.
The power of metadata is implicit. No one at NSA is arguing that metadata isn't powerful, that's the whole reason they're collecting it.
What the NSA is arguing is against the meme that they're trodding roughshod over the Constitution and illegally scooping up data (as they have done in the past). On the contrary, this time they've been very careful to try to remain within the law, and one of those arguments is by pointing out that, according to the law, some types of information are less protected than others.
Greenwald can educate all he wants, but the problem isn't that Americans are stupid or whatever you all seem to think. Americans on the large have been assuming the NSA is doing far worse than what Snowden accused them of, because Americans (on the large) expect that the NSA (or some other branch of the government) is actually able to intercept and listen in on "those jihadis' phone calls and emails".
You're talking about a generation of Americans who grew up listening to PBS stories hounding the Federal government agencies of the Bush and Clinton Administrations for "not connecting the dots", and for having a "lack of imagination", and for "treating terrorism like a law enforcement" problem and for being hamstrung by a bunch of civil liberties protections.
This is by and large the same generation that decided to vote against privacy when they bought into to having their digital lives collectively managed by Google, Apple, Microsoft and Facebook. Explain the low-level bits and bytes to them all you want, but they've already voted on the big picture they want to see.
I'm surprised such a thing took so long to be revealed. If you've got as much data as the NSA has, wouldn't you want a Google like search engine to be able to search through it? It makes so much sense which is why I am surprised some people are surprised about this.
If Snowden, as one person, accomplished what he did with (the majority of Americans would say) good intentions, imagine what a team of people, who were just as smart if not smarter, could do if they didn't have morals guiding their actions. Imagine how well a team could cover their tracks, dot all the I's, and cross all the T's, compared to just one guy.
This is all common sense, but it bears repeating.
Imagine the other end of the scale - where in fact every detail about everyones lives is wide open and available for everyone and anyone to access. Willingly. Freely. A new order of celebrity: total telepathy.
Do you think we'd be dealing with terrorism, then? Would there be the idealist, killing souls, for a little private time?
It's possible the program has been reigned in since these slides were authored but that seems unlikely.
seems like a lot until you consider how many indexed pages Google has: http://i.imgur.com/EqIJAoL.jpg
why not throw in grains of sand or atoms in the universe?
More on topic: Whether you can compare the index sizes or not mostly depends on their signal to noise ration. Given the number of Google search request per month and assuming and average of 5 viewed results, you can put a lower bound of 99% of pages that are never seen by anyone. As the search results should greatly overlap the actual number of relevant sites is probably orders of magnitude less than 1%.
Of course they built a search engine. Wouldn't you? Don't you have similar at your workplace? We use them all the time. Think about web interfaces built on top of ElasticSearch, for example. Is that not a 'search engine'?
It's the opening it up to LEO. Taking homeland security-justified warrantless searches and turning those over to criminal investigations is far beyond unconstitutional.
Except, that it's not.
It's all about how the government comes into possession of information that makes it the fruit (or not) of the "poisonous tree".
Once the government legally learns of some bit of information, there's no requirement by itself that they have to "pretend not to know" something is going on. That's why a third party voluntarily divulging information to the government doesn't cause that evidence to get thrown out of court: There may be a cause for civil or even criminal action against that third party, but that doesn't invalidate the evidence.
Now, Congress has passed statute laws limiting information sharing between intelligence agencies and law enforcement precisely because of the threat of having this all-seeing eye subvert democratic government, but the limits were never complete exclusions, even before 9/11. And the reason statute laws were needed is because there was no barrier to this activity from the judicial side alone.
Parallel construction came in not because evidence was illegally gathered, but because giving the (fully legal!) chain of custody for evidence derived from intelligence sources would have quite naturally have "burned" that source or method, so the NSA would require law enforcement agencies using that data to use alternate (again though, legal) means to make a case in court, in order to protect their intelligence source. It's the old apocryphal dilemma about whether to let Coventry be bombed to protect ULTRA, applied to the post-9/11 world.
The Reality Distortion Field appears, and people believe, because they want to believe.
It depresses me, the lack of intelligent discourse.
Most tech people I meet actually believe that the NSA records and stores all telephone calls. It's depressingly stupid, but I have given up arguing, logic and sense are not welcome when the NSA is the topic.
Unfortunately, with the NSA, independently verifiable proof of anything, especially doing the right thing, is impossible to attain.
Proof:
* NSA Collects ‘Word for Word’ Every Domestic Communication, Says Former Analyst http://www.pbs.org/newshour/bb/government_programs-july-dec1...
* NSA collecting phone records of millions of Verizon customers daily http://www.theguardian.com/world/2013/jun/06/nsa-phone-recor...
* NSA surveillance program reaches ‘into the past’ to retrieve, replay phone calls http://www.washingtonpost.com/world/national-security/nsa-su...
* NSA spying flap extends to contents of U.S. phone calls http://www.cnet.com/news/nsa-spying-flap-extends-to-contents...
Also, none of your links in any way prove that the NSA stores all phone calls. None of them. 1 is hearsay from someone, another talks about metadata, and another says that the NSA can record any phone call (yes we know that). Nowhere is there any proof that all calls are being recorded.
This is what I am talking about. It doesn't seem possible to have a conversation about the NSA on Hacker News that is based on logic, reasoning or sense.
The original claim was "the NSA records all phone calls".
This is saying, "the NSA records all phone calls, in a certain, non-domestic country, for a 30-day window", which is nowhere near the same.
You would think that simply accusing the NSA of what it's actually doing what be good enough, but that's only rarely the case here at HN.