Drug Agents Use Vast Phone Trove Eclipsing N.S.A.’s | Hacker News Reader

Drug Agents Use Vast Phone Trove Eclipsing N.S.A.’s

nytimes.com

335 pointsdanso105 comments

Take a look at what happened when the drug dealers were doing the exact same thing:

In a police raid, Cali cocaine cartel leader José Santacruz Londono was found to have assembled a database that contained both the office and residential telephone numbers of U.S. diplomats and agents based in Colombia, along with the entire call log for the phone company in Cali, which was leaked by employees of the utility.

A $1.5 million IBM AS400 mainframe was loaded with custom-written data-mining software. It cross-referenced the Cali phone exchange's traffic with the phone numbers of American personnel and Colombian intelligence and law enforcement officials. The computer was essentially conducting a perpetual internal mole-hunt of the cartel's organizational chart. Santacruz could see if any of his lieutenants were spilling the beans.

They were. A top Colombian narcotics security adviser says the system fingered at least a dozen informants -- and that they were swiftly assassinated by the cartel.

Ref: http://www.mail-archive.com/eristocracy@merrymeet.com/msg000...

Longer story here: http://cocaine.org/cokecrime/index.html

That story can't possibly be true -- he would have only had access to metadata, and I've been told many times that metadata surveillance is completely harmless.

The existence of the data would be enough to infer a relationship that shouldn't exist at all.

woosh

I think you were looking for this... http://i.imgur.com/YwgbT16.gif

From the document:

Protecting the Hemisphere program is a formidable challenge. We have taken the following steps to try and keep the program under the radar...

The fact that our government creates and then goes to such great lengths to hide programs that it knows the public (and most likely the courts) will vehemently object to shows the level of contempt that government officials and employees have for the very people they were hired to serve. I don't know that it can be fixed, but it is a very serious problem.

Maybe they don't want drug pushers and users to know they have this either?

Security through obscurity?

…actually works outside of theory. We're not talking cryptosystems here, we're talking about intelligence and counterintelligence.

You don't call a spy's disguise security through obscurity.

Note that I'm not condoning mass government surveilance, only objecting to its criticism on the grounds of "security through obsecurity".

Even real-world cryptography can benefit from security through obscurity. It's all a time game- how fast can my attackers attack my security, how much time do I need to buy. If security through obscurity buys you time, why not?

"Buys you time" for what? For fixing it? Why not do it right in the first place?

The danger in this line of thought is that security breaches only have to happen once for real damage to ensue. For software companies, when it happens, I always expect to see a clear explanation for why it did happen and in case of stupid architectural problems, I tend to avoid that company in the future.

Of course there's a difference between software and real world undercover operations performed by government agencies. Placing an agent undercover is both a gamble and a race against the clock. The government knows the risks involved, the agent knows the risks involved, human casualties can happen but that's part of the contract so to speak. Nobody willingly enters such operations without knowing the risks involved.

But if you're gambling with customer's data, be prepared to explain that to the angry customers when the shit hits the fan.

As far as I know, we've never yet created a cryptographic algorithm that's withstood more than 20 years of scrutiny. We could assume this trend continues into the future, and that any crypto algorithm we do create will be broken within 20 years. This means that encrypting a piece of data isn't some magical eternal protection--it just seals it in a time-capsule that'll "degrade" after 20 years or less. (Sometimes far less.)

But, so far, this property has also been pretty much irrelevant: almost all the things we want to do by passing along a secret are time-sensitive, and breaking the secret 20 years after the fact doesn't really buy you anything. Being able to impersonate the SSL key of Microsoft.com-as-it-was-in-1993 doesn't let you do anything to Microsoft.com-as-it-is-today.

This policy scales down, of course: in military comsec terms, you only need the encryption on operational details to last until the day after the operation is carried out. After that, your "secret" has become "plain" (something quite obviously blew up, etc.) and so the enemy breaking the encryption on the orders won't tell them anything they didn't already realize by hearing the explosion.

This is why the military keeps multiple different kinds of ciphers for different levels of secrecy, by the way: they assume that the more things they use a particular crypto algorithm for--the more signals the enemy gets to intercept that use that algorithm--the more enemy sigint folks will be put to the task of breaking that algorithm. So "top secret" encryption isn't meant to withstand any more scrutiny than "secret" encryption; it's just generally a bunch of orthogonal crypto primitives to the ones in the merely-secret crypto, and only used rarely, for the kinds of orders that need to stay secret long after execution (e.g. covert ops on allies.) Thus, enemy nations will have comparatively little reason to have analyzed and broken it--and breaking the secret-level ciphers won't help them, because of the orthogonal implementations.

You're right of course, however we need to make a distinction here.

First of all there's the issue of how strong is an encryption algorithm. For example RSA is based on the problem of factoring large numbers, a problem that's generally considered to be hard as we know of no efficient algorithm for solving it. But we haven't proved that factoring large numbers will remain a hard to solve problem in the future. The NSA could very well have custom hardware for efficiently factoring 1024-bit primes by now and the upcoming quantum computing is a real threat. If they haven't done it by now, 1024-bit keys will become breakable in the future, however 2048-bit keys are another issue entirely and 4096-bit keys will probably stay unbreakable.

But, even if breakthroughs in solving the integer factorization problem will be made in the future, as long as P != NP then perfect encryption is possible. In fact, we already know of encryption schemes that are provably unbreakable even with unlimited hardware at disposal, the problem being that they are also hard to implement, so we ended up with making tradeoffs.

Second, it's far easier to attack a particular implementation, to bypass the encryption algorithm entirely, e.g. attacks against the key generation system, side-channels, the protocol of the software system we are talking about, etc ... because software always has bugs, as in zero-day exploits that one could make use of.

For this reason - if indeed the military is using different encryption algorithms for different security levels, algorithms that aren't used in the wild, then to me that's a pretty bad idea, as far more often than not it's the implementation that's broken, not the algorithm. And in case of inside leaks, the implementation is always easier to get a hold of, compared to the key.

Military comsec basically has to assume the implementation will be immediately made available to enemy sigint, since they build crypto implementations into things like secure phones which can just be stolen off a dead soldier and pulled apart.

The implementations can be upgraded in the field when a flaw is found, as with any firmware (and frequently an implementation will be cycled out for a different one even if it is thought to be unbroken, just to put any time that's been put into breaking it to waste.) But enemy governments are precisely the people with enough resources, and reason, to want to break entire algorithms.

The thing is, it is a hard problem--so they only bother to break algorithms where they know they'll get big rewards for doing so ("top secret" doesn't usually mean more valuable to enemies, after all; usually it just means "fewer people should know this ever happened.") A standard secure phone will have all the Suite A and Suite B ciphers[1] built into it, but since so many more transmissions will be using Suite B ciphers, there'll be comparatively less strategic advantage in cracking the currently-used Suite A cipher before it's cycled out for the next one. So Suite B ciphers sometimes do get cracked during their "useful shelf-life" and have to be immediately switched, while Suite A ciphers are usually left alone.

---

[1] http://en.wikipedia.org/wiki/NSA_Suite_A_Cryptography, http://en.wikipedia.org/wiki/NSA_Suite_B_Cryptography

Diffie-Hellman and RSA are older than 20 years, aren't they?

Also DES has never been `broken'---only brute-forced.

Seemed to work for Skype. Took quite a while to reverse engineer the protocol, and I'm not sure there are any proper cleanroom implementations. That's a solid business win for them.

It did not work for Skype[1]. This conversation has been about security through obscurity. You're describing their competitive advantage because competitors couldn't build external interfaces to the protocol; that's not security a security win, it's a business win, as you said.

[1] http://en.wikipedia.org/wiki/Skype_security#Flaws_and_potent...

Please point out security flaws in Skype's voice protocol. That list is a list of problems and flaws with Skype's software (which is, in general, shit). It doesn't seem to document any crypto failures. The largest security failing listed there is that it pulls ads over an unencrypted connection.

For all we know, the core Skype protocol may be perfectly implemented. The Wikipedia link states there's no peer-review.

So because we can't study it we assume it's perfect? That's the essence of the fallacy of security through obscurity.

As a corollary, just because you can't do a quick Google search for 0days in iOS or Windows doesn't mean they exist. In fact they do, and they're bought and sold on black markets or are kept secret by governments and the like.

You don't assume something is secure because you can't readily access documented flaws. You assume something is secure when it has undergone rigorous peer review, which, as you stated, does not exist.

Your argument seems to be that you can't simply find a laundry list of Skype flaws floating around. This is true. But it says positively nothing about the security or lack of security regarding Skype's protocol.

You said flat-out it did NOT work. I'm saying that it's not determinable, and so far, no published security holes in Skype exist. In fact, no real good details exist, despite plenty of people trying. Skype's probably the most popular IM/Voice/Video protocol in the world.

I agree that Skype's protocol may be terrible. But you cannot state that obscurity didn't help. "No one" is even able to connect to Skype, let alone break it, at this point.

Is this an argument in favor of the practice? Because, honestly, drugs should be legal to sell and to purchase. If all drugs should be legal and if the trade in drugs should be legal, then how can any program like this designed solely to enforce drug laws be justified? In that event, who cares what drug "pushers" and users know?

Well, honestly, drugs should not be legal to sell and to purchase.

What world are you living in? Do you think these guys deal weed? Have you seen what (crack) cocaine does to people?

These cocaine pushers are in the business of destroying people, and they certainly should be apprehended.

It is less expensive and less destructive to treat addicts than it is to conduct a Drug War.

"Well, honestly, drugs should not be legal to sell and to purchase."

Care to cite any reasons here? See, when it comes to cocaine, Congress has not revisited the debate since people said these sort of things:

http://query.nytimes.com/gst/abstract.html?res=F70F1EFA3A5A1...

http://www.druglibrary.org/schaffer/history/negro_cocaine_fi...

Yeah, you read that correctly. Black guys who use cocaine become more accurate with a gun and will attack white women! Also, Jews are selling it. Cocaine also makes it nearly impossible to kill a black man using a standard issue handgun, so let's upgrade the caliber.

"Have you seen what (crack) cocaine does to people?"

Yes: I have seen the children of the wealthy doing cocaine at college parties, then going on to get high-paying jobs on Wall St. Truly ruinous, truly!

How about we drop the boogeyman and drop the anecdotes and start citing some sources? You claim that cocaine destroys people; let's see the proof.

Might be beating dead horse, but people want to drink alcohol snort coke or shoot up heroin. Put as many laws as you want, that is not a way to solve problem, never will.

You can't forbid people being stupid by law. Also, abuse of narcotics is a side-effect for bigger problems, ones that is easier to ignore. Will we get there when we are able to speak about it, I wonder, I wonder my friend.

Many drugs are horrible and ruin lives. But they ruin fewer lives than do the laws making the sale and purchase of drugs illegal. If every penny spent on enforcing drug laws and incarcerating drug sellers and buyers were spent on education, treatment, and poverty programs, drug use would fall in the United States by orders of magnitude.

One of the more pernicious effects of making the sale and purchase of drugs illegal is that people who sell and purchase drugs are sent to prison. Most people sent to prison for these offenses are themselves addicts either trying to feed their habit, or to pay for it. These are the people that the drug laws are intended to protect, no? But as a result of going to prison they:

* are frequently denied effective treatment for their illness, and therefore continue as addicts inside and outside of prison;

* are effectively exiled from the legitimate economy, becoming a burden on society both before and after incarceration, and increasing the likelihood of continued drug use;

* cannot take care of their families, increasing the burden that we all carry that we would not otherwise carry;

* are mixed with a violent element, whose influence will certainly cause many to commit non-drug crimes that they otherwise would not have.

The prisons turn users into addicts and addicts into career criminals.

Some people seem to believe that we need these harsh laws in order to catch, as you put it, "cocaine pushers" who are "in the business of destroying people". In fact, narcotraffickers are entirely the creation of US "Drug War" policies. Decades of supply-side enforcement has made these people so wealthy that they can purchase entire governments. The government of Mexico, the world's 14th largest economy, is populated in some percentage at almost every level by agents of the cartels. It is "Drug War" policies that create the profits to pay for this corruption.

Demand-side policies along with controlled and regulated distribution of drugs in the US will decimate the narcotraffickers. Without customers willing to pay ridiculous margins for their product, they will simply go out of business. Although some former narcos will attempt to leach off of society through kidnapping and other crimes, a lack of local community support, money and political cover will mean that aggressive policing will finally be able to stomp them out in time.

Also consider the case of Oxycontin, a legal narcotic manufactured by Perdue Pharma, "a privately held pharmaceutical company founded by physicians and now located in Stamford, Connecticut". Oxycontin is among the most destructive drugs today. Oxycontin is primarily acquired by addicts either by getting doctors to legally prescribe it, or by purchasing excess pills from individuals to whom it was legally prescribed.

"Drug War" policies are entirely ineffective a combating this kind of practice; as a result, prescription drug abuse is growing at a faster rate than nearly all other forms of drug abuse. However, it is almost certain that addiction-treatment regimes instituted to reduce the use of currently-illegal drugs will also be effective at reducing the illicit use of currently-legal drugs.

And if for some reason you think that education and treatment programs can't have a significant (if not massive) effect on addiction rates, I would ask you to take a look at the following two data points:

http://articles.latimes.com/2013/jul/12/science/la-sci-sn-ci...

http://www.cato.org/publications/white-paper/drug-decriminal...

    If every penny spent on enforcing drug laws and incarcerating drug sellers and buyers were spent on education, treatment, and poverty programs, drug use would fall in the United States by orders of magnitude.

I have no doubt that's true. But even if the war on drugs was stopped, the money wouldn't go to helping the poor. The U.S. only has right wing parties, so the money would just end up paying off debt or lowering taxes.

    One of the more pernicious effects of making the sale and purchase of drugs illegal is that people who sell and purchase drugs are sent to prison. 
    ...
    The prisons turn users into addicts and addicts into career criminals.

(hard) Drugs are illegal where I live too (The Netherlands). But we don't have such harsh laws at all. Possession of drugs means a fine and perhaps a few days in a cell sobering up.

It is not the war on drugs that is destroying the U.S's lower class, it's the ridiculous prison sentences. No country in Europe has as many drug related problems as the U.S., and in no country in Europe drugs are legal (except maybe portugal?)

You are absolutely right in everything you say. But legalizing crack cocaine does not sound like an optimal solution to me, sure perhaps you'll succeed in overthrowing the drug cartels. But then you have the issue of regulating or even organizing the legal (private) distribution of drugs. I don't see that ending well at all.

The prisons and the U.S.'s messed up legal system are the problem they are what should be fixed.

> It is not the war on drugs that is destroying the U.S's lower class, it's the ridiculous prison sentences.

I don't see these two things as separable.

> But then you have the issue of regulating or even organizing the legal (private) distribution of drugs. I don't see that ending well at all.

Perhaps you are right, but it is my opinion that it is the very illegality of drug abuse that makes it so difficult to treat, especially for the harder drugs. If you read the Portugal study you find that addicts that are not threatened with jail time or other criminal sanction are more likely to seek and pursue treatment. And Portugal has a small fraction of the funds for treatment available as the US would have if it repurposed it's "Drug War" budgets for treatment, so you could imagine that such policies would be more effective here.

One key additional concept from Greenwald's report is that individuals who choose to avoid hard drugs are not dissuaded by their illegality just as they are not dissuaded by the harm that these drugs cause. Furthermore, individuals who would otherwise choose not to do drugs don't change their minds because the criminal sanction disappears. Choosing to do hard drugs involves a decision-making process that requires a person to disregard signifiant harm to their person. This insensitivity towards risk on the part of the addict means that criminal sanction simply does not have the deterrent effect one would expect; inversely, non-drug users are deterred primarily by the ill effects to their health and wellbeing, so drug's illegality barely enters into the equation.

>> It is not the war on drugs that is destroying the U.S's lower class, it's the ridiculous prison sentences.

>I don't see these two things as separable.

That's the entire problem of the US in a nutshell I guess :)

"then you have the issue of regulating or even organizing the legal (private) distribution of drugs. I don't see that ending well at all."

We have been very successful at regulating tobacco and alcohol. Yes, you can find moonshine if you look really hard, but that's just the point -- almost nobody wants moonshine, people prefer regulated liquor. Sure there is black market, unregulated tobacco and teenagers manage to buy it, but the vast majority of people who smoke buy their tobacco legally.

For that matter we have also been overwhelmingly success at regulating pharmaceutical drugs, to the point where a black market exists for them as replacements for illegal drugs. There is a reason recreational opiate users want pills: the regulations on purity, dosage, etc. Even methamphetamine is available by prescription (for narcolepsy, obesity, and ADHD treatment), and the pharmaceutical stuff is a lot safer, because of regulations.

In reality we know how to regulate drugs, including extremely dangerous drugs like alcohol and tobacco, and even "hard" drugs like methamphetamine. Maintaining a regulatory system is not the problem here. The real problem is that the war on drugs is profitable. One of the most ironic facts of lobbying in today's world is that "The Partnership for a Drug-Free America" receives money from alcohol, tobacco companies, and pharmaceutical companies. There is also the matter of politicians having figured out that they can always portray themselves as "tough on crime" by pushing for drug arrests. Police officers unions are fighting for their members jobs by lobbying for maintaining or even expanding the effort. The executive branch has also figured out that the war on drugs is a great excuse for expanding executive power -- even to the point of the attorney general's office having gained the authority to declare drugs to be illegal (and then prosecute people for possessing those drugs).

Legalization and regulation are the answers our society really needs. We need to disband the DEA, repeal the controlled substances act, pass a constitutional amendment that forbids all such prohibitions, and set up a regulatory framework. It is not likely to happen, for the reasons outlined above and because we have had so many decades of propaganda that people have trouble with the idea of alcohol being drug or of methamphetamine having medicinal use.

Out of curiosity, what is it about crack cocaine that has you so terrified? There are far more dangerous drugs out there...

    Out of curiosity, what is it about crack cocaine that has you so terrified? There are far more dangerous drugs out there...

Sorry for the late reply. I disagree with you, and I was a bit tired so I stopped discussing :P I'll answer your question:

The combination of accessibility, addictiveness and health effects. I live opposite to an addiction treatment center and see crack addicts every day. It's addictive like tobacco is, has stronger mental health effects than alcohol and because of its low cost is more accessible than any other hard drug.

I know alcohol is very dangerous too, but 99% of alcohol users manage their addiction in a way they can still manage their lives adequately. With crack and other hard drug addictions you will find the odds reversed.

Have you seen what alcohol does to people? It kills over 80,000 people a year in the U.S. http://www.cdc.gov/alcohol/fact-sheets/alcohol-use.htm

You think you're protecting people with that attitude but you're actually supporting a system that kills even more.

Sanctimonious nanny-staters like you are the ones that let the madness continue. You should be ashamed of your ignorance and complicity.

I think we need to take a step back and get to the part where it makes sense to use these tactics against drug users.

The point of my comment wasn't that we shouldn't use technology to enforce our laws (though our drug laws are ridiculous and counterproductive). It was that our local, state, and federal governments operate as a collection of thousands of fiefdoms that run for the most part without supervision from any other agency. Each of those fiefdoms can implement programs like this in secret, and no one can challenge them because no one knows about them. Every so often one of these programs is identified and disbanded after years of litigation, but hundreds more will soon appear in its place.

The simplicity of implementing such programs combined with the near impossibility of identifying and eliminating them makes this an insurmountable problem. Our government seems to have adopted the mantra of the criminals they claim to dislike so much: If you don't get caught, it's not illegal.

Fix it? Sure, the US Founding Fathers had all been there, done that, gotten the T-shirt, and showed us how:

(1) Have freedom of the press so that the press can put out the information needed for an informed citizenry.

(2) Have freedom of speech and assembly so that the citizens can learn, discuss, and decide.

(3) Have the Bill of Rights so that the government can't just run roughshod over the citizens.

(4) Have three branches of government so that there can be checks and balances and keep, say, the Executive Branch from becoming dictatorial, self-serving, self-perpetuating, and tyrannical.

(5) Have the President and the Congress elected by the citizens, directly or nearly so and, thus, have them elected out'a there when the citizens don't like them.

Then, presto, to solve the problems, citizens just need to become informed and, then, behind some curtains pull some levers. Done. All the powerful, wealthy, connected, crooked dirt bags in government float belly down down the Potomac River.

Net, we just need to become informed and pull the levers.

And, there's another way: Bring law suits, e.g., citing the Fourth Amendment.

So,

Q. Go to war in Syria?

A. Not without money voted by Congress.

Q. Implement ObamaCare?

A. Not if the citizens and/or courts don't like it.

Q. NSA spying on US citizens?

A. Not if citizens vote to stop that stuff and/or court cases get brought and are successful.

Q. The spying on US citizens as in the OP?

A. Not if citizens don't like it and/or there are suitable court cases.

Q. US Customs taking mobile electronic devices of US citizens and extracting and keeping all the data?

A. Not if, right, the usual, citizens and courts stop it.

Etc.

The US government is flawed on a deep constitutional level. The incentive structures and oversight mechanisms are broken. The constitution is pretty much a joke at this point. The founders knew it would become outdated quickly and recommended rewriting it every 25 years or so.

> The founders knew it would become outdated quickly and recommended rewriting it every 25 years or so.

Interesting! Where was this mentioned -- in their personal correspondence, or a more public document?

It's a concept rather than a specific quote: http://en.wikipedia.org/wiki/Living_Constitution

Here's the well-known rationale from Thomas Jefferson: http://press-pubs.uchicago.edu/founders/documents/v1ch2s23.h...

(1) Have freedom of the press so that the press can put out the information needed for an informed citizenry.

At the time the bill of rights was written, the term "press" did not refer to any sort of news organization. It meant a literal press - the idea was that publication by anyone with the means to publish was not to be restricted.

Nowadays everyone has the modern equivalent of a press so it is unfortunate that the meaning of "press" has been narrowed to a poorly-defined set of semi-official news distributing organizations.

Good! Now with blogs we can return to the original meaning!

And if none of that was enough, have enough guns to force a regime change.

I continually don't understand this argument. Any group of armed and motivated people intending on taking down the government would be stopped, one way or another, before they can assemble a well-organised militia. The presence of guns might make you safer in a number of narrow scenarios but not against anything as large as a state-wide police force, let alone the US Army.

You see to forget that the height of military technology of the day was the Kentucky or Pennsylvania long rifle.

The militia was intended to be the primary form of military organization.

Fundamentally, an armed citizenry was to be the final check against the abuse of power by the state.

The Iraqis managed to kick us out with rifles and improvised mines. Not even a single artillery piece, armored vehicle, aircraft, nor any technology less than 50 years old, against $2 Trillion worth of US anti-insurgency doctrine.

"Hired to serve", but it brings to mind the following quote from Noam Chomsky, regarding who is being served:

"Governments should not have this capacity. But governments will use whatever technology is available to them to combat their primary enemy -- which is their own population,"

It shows contempt for and fear of privacy advocates, who are a small minority of the people they were hired to serve.

The article didn't address the question of why AT&T has phone records going back so many years. I doubt that five year old data about a customer would be useful for marketing, considering that they have the most recent data. Nor is old data useful for billing once the time period for contesting a phone bill has elapsed. Network capacity planning could be done with anonymized or aggregated data. So it would seem that the only reason why they hang on to all this data is because the government asks them to, or because they can make money selling it to the government (or both).

Now, think about what adverse effects such old data could have on justice. For example, let's say a friend of mine from school, who I spent a lot of time talking to on the phone a decade ago, decided this year to become a drug dealer. The government could start investigating me based on this stale data, despite not having any reasonable suspicion that I was involved in any crime.

I think that what we really need is a privacy-oriented phone company built on the model of DuckDuckGo, which doesn't keep any data about customers beyond what's necessary for billing purposes. But maybe the U.S. already has laws that would make such an ethical phone company illegal.

I wonder if you could do it as an MVNO.

If you operated an MVNO (Mobile Virtual Network Operator), wouldn't your traffic still be going over AT&T's or some other carrier's network? They'd have access to the information whether you wanted them to or not.

I don't know if they keep billing data for MVNO customers. And, a $500mm/yr MVNO would have the power to negotiate contractual terms which regular individual or business customers wouldn't; not like showing up with pitchforks at exec houses in the middle of the night when violated, but with huge payments required at least.

From my brief research, they still handle all of the SS7 part of things as an MVNO, so even if they don't have 'billing' information, they definitely have the call origination/termination records/history.

The article says that AT&T has data not just on its customer but on any traffic that passes through its switches. I suspect that's much of it.

Theory #1 – Because deleting stuff at scale is harder than you think. If you've got structured data (think "relational database, much of which is pretty much guaranteed to be append-only") and you've optimised it's on-disk structure for reporting (think "highly indexed, possibly even with both the index data and the transaction data being stored in ways that take advantage of known query patterns and physical disk geometry) – removing data from it is likely to be _much_ more effort than just flipping a "deleted" flag and continuing to expand your storage pool. This is even more true if you'd also need to consider replicated copies/archives/snapshots/backups. Facebook/Google/Twitter et al store everything, not just because they think it'll make them more valuable to advertisers, but because deleting data from distributed/sharded/backedup/archived databases is more expensive than just marking it "deleted" and leaving it in place.

Theory #2 – The NSA (or it's predecessor or a related agency) have been paying them to store whatever they can since whenever it became possible. If you're prepared to ignore the privacy implications, it's obvious that some _tiny_ percentage of that data will become useful for law enforcement purposes. Unfortunately – when the "privacy implications" are considered in terms of "Is _your_ privacy more important than _my_ career?" – it's pretty clear that a very powerful cohort of law enforcement and intelligence agency decision makers say "Hell no! I just need one or two more big successes and I'll get that promotion I want! Of course it's worth monitoring every single person on the planet to give me a shot at the executive bathroom and an office with a door!" (or, less cynically but equal in consequence "Should I listen to every phone conversation in America, if it might possibly mean we can stop the next 9-11? Yeah, I think it might be…").

Because deleting stuff at scale is harder than you think.

If you've got an age-based retention policy, and you've built in the capacity to delete the data, not really. Periodic purge. I'm not saying this is trivial, but it's not hard. The system was clearly designed not to delete the data.

Your theory #2 holds much more water.

The optimist in me wants to argue that there may be OPEX vs CAPEX reasons to have not invested in "building in the capacity to delete", or possibly even tax implications of development budgets vs ongoing maintenance which might reasonably explain why you'd choose to build such a system as "append-only, archive indefinitely". The pessimist in me fears you're correct…

Re: theory #1- there are commodity solutions to the queriable data warehouse problem, e.g. Vertica or Greenplum. Looking at these can give you an idea of what might be hard. You've just described Vertica pretty accurately (column-oriented storage, materialised on disk for particular query patterns, ad-hoc deletion comparatively costly). And ageing out in Vertica is easy- you just drop a partition. I can't imagine a custom system would be built without so e provision for this.

Sure, if you were building it from scratch today, but Vertical was founded in 2005 - the article talks about data going back to 1987. Cynical-me suspects the NSA has been "requiring" them to retain all that data since before it became easy/practical to expire it properly.

27 years ago was 1986.

Are you sure that 27 years worth of data storage within the bulk of that time frame was easier and/or more cost effective than deletion?

" … let's say a friend of mine from school, who I spent a lot of time talking to on the phone a decade ago, decided this year to become a drug dealer. The government could start investigating me based on this stale data, despite not having any reasonable suspicion that I was involved in any crime."

Next time you're pulled over for a "random traffic stop" or some inexplicably-enforced minor violation (brake lights out, rolling thru a stop sign, singled out in a stream of traffic doing ~10 over on an interstate), I assume the words "parallel construction" will loom large in your mind – along with the memory of that old girlfriend's junkie ex, and your college room-mate's best pal who used to deal weed…

Money.

Because the government requires they retain it.

"The government pays AT&T to place its employees in drug-fighting units around the country. Those employees sit alongside Drug Enforcement Administration agents and local detectives and supply them with the phone data from as far back as 1987."

"Unlike the N.S.A. data, the Hemisphere data includes information on the locations of callers."

"Some four billion call records are added to the database every day"

No wonder phone number portability got pushed through.

man, that makes sense. nice call.

It's funny, we thought they went to all that trouble for our benefit. We're a bunch of rubes.

Even if you switch phone numbers, using Hemisphere they can determine your new number. They look for a new phone number calling all your old contacts.

Absolutely. But allowing/encouraging people to keep a single identifier sure makes things easier, and cuts down on your database joins.

Do you really think the DBA for a project that doesn't official exist gets to write consumer protection policy?

Can you imagine any nontechnical organization doing something as complicated as passing a law just to cut down on database joins? Seriously?

Portability makes it easier for people

This is a side effect and, as other people told it, completely not needed.

Even easier than that, people switch SIM cards but keep the same phone (this may not be very common in the US, still, it happens)

This "Protect the Program" stuff sounds like another "Parallel construction" debacle. They are not talking about hiding the program from the public, they are talking about getting the information elsewhere after you used Hemisphere.

Basically, this is a private backyard deal the DEA made with AT&T. Apparently our laws have deteriorated to the point where AT&T lawyers believe they can do this legally.

Another day, another peek behind the veil of large scale (institutional and codified) data abuse.

We don't have to put up with them poking their fingers in our lives for much longer. We can build around them. We can make them irrelevant. It'll take a more concerted effort than building social networks for cats (where you spend four years of your life posturing about "changing the world" and "being successful" then the next day you cash out for big acquihire fake-success bucks).

Stop studying $FAD_OF_THE_WEEK and go back to your math and ML fundamentals. Play around with it. Build good things. Then build bigger good things. Build good things for good people.

If you haven't already seen The House I Live In, I recommend watching the documentary online. It will change the way you think about drugs and the United States' prison system. The movie doesn't focus specifically on technology, but it explores the by-any-means-necessary approach to law enforcement that helps facilitate things such as the Hemisphere Project mentioned in the story.

Trailer: http://www.youtube.com/watch?v=a0atL1HSwi8

I recently read an article about prisoners who had cell phones from which they posted to their Facebook pages, replete with images of their drugs and snack stashes, even facilitating communications between inmates in different prisons.

It was only after investigative journalists brought the situation to the attention of regulatory agencies that the prisons clamped down on the security breaches within the prisons.

http://www.nydailynews.com/news/crime/tennessee-inmates-post...

They didn't clamp down on anything. All they did was make an example of them. I bet the other inmates 'helped'. They still have access to cell phones and Facebook, they just are more careful now about this sort of stupidity.

The government pays AT&T to place its employees in drug-fighting units around the country. Those employees sit alongside Drug Enforcement Administration agents and local detectives

If the government pays your salary and you are embedded in the DEA, you are a government employee. Saying these people work for AT&T is patently absurd.

Exactly. People get so worked up about the technical distinctions, ones that are mostly drawn by the IRS for tax purposes, that they confuse the form for the function.

It is no different than Booz Hamilton "contractors" working on site at NSA and other TLAs. If the money comes from the government and you take direction from the government, even indirectly, you are an agent of the government.

It seems pretty clear that the mobile operators are the biggest commercial organization threats to personal liberty. Their only real competition is the banks.

Cash is really easy to use and is accepted everywhere. There's no similarly elegant solution for mobile connectivity if you want to stop revealing metadata information.

Well, not really everywhere. If you tried to buy an airline ticket with cash, you'd be subject to a lot of scrutiny.

Doesn't negate your point, but I checked out of curiosity, and at least with American Airlines, it is easy to buy an airline ticket with cash. http://www.aa.com/i18n/reservations/paymentOptions/cashPayme...

"Legal tender for all debts, public and private".

> "Legal tender for all debts, public and private".

Yes, debts being the operative term here. You don't incur a debt until you receive the product. Airline tickets are paid for in advance so the customer never incurs a debt.

Given the number of network connected machines that cash passes though (ATMs, ticket machines, counting machines, ...), I'd assume that such machines would be routinely scanning the serial number of each note. Couple that with an account number, or the CCTV that is usually pointed at any cash handling machine, and I'd posit that the anonymity of cash is eroding.

Let's not forget about the big internet companies that participate in PRISM.

This story alone is reason enough to seriously start working on a program and schedule for:

- legalizing drugs and

- selling them in a controlled environment,

- taxed and

- accompanied by a heavy investment in related education in schools nationwide

...instead of throwing billions at killing our most basic rights.

The example of Portugal: http://www.spiegel.de/international/europe/evaluating-drug-d...

We already have "related education" mandated as curricula in several different grades in every school system in the country. It's not uncommonly ridiculous propaganda, with blatant urban legends dramatized for your FUD, which is widely rejected by the student body.

What we need is a non-criminalized mental health system that actually functions (addiction neural wiring being a subset of mental illnesses that can be acquired through the victim's actions), and a poverty / diminished capacity safety net that tries to redress problems in the way of marginalized people rejoining civil society.

Essentially everything else would self-resolve after legalization.

David Simon pointed out something like this when PRISM itself was first leaked. The courts and law enforcement have combined for some pretty incredible types of surveillance and investigation possibilities, and have done so for years and years.

Is there any doubt as to the fascist nature of the USA now?

Call it "corporate cronyism," or "corporate fascism," but whatever it's called, it's no land of the free anymore.

Well, unfortunately the word "fascist" isn't particularly meaningful anymore. But there certainly isn't any doubt that Fourth Amendment protections for documents in peoples' homes haven't been considered to be extended to digital communications moving through various service providers' pipes, and that various government agencies have been having a field day with surveillance data they've been able to legally justify collecting based on that lack of Constitutional protection. It seems largely like a widespread failure to create language protecting digital privacy in the same spirit as analog privacy was originally protected. Maybe that's fascism, but it seems less useful to use words without agreed upon meanings.

My bad for using sensational terminology, but the heart of the matter remains.

The privatization and collusion between government and private entities at the expense of We the People has brought the USA to a point where it's obviously become the land of the corporation and the home of the wage-slave.

Lobbyists with vast funding avail their corporate sponsors of representation which is denied to those who lack such deep-pockets.

Only those with similar vast funding are able to attain high level elected positions within our government, from whence they enact legislation which furthers the divide between the plebiscite and the proletariat, which falls right into line with Marxist "liberal-democratic" ideology, where the government becomes the tool of the Bourgeoisie at the expense of the Proles and the Plebes.

This socioeconomic stratification is quite visible in today's USA, and programs such as "Hemisphere" exemplify this situation.

I too find the exclusion of electronic communications from 4th Amendment protections to be an egregious failure of our Constitutional checks and balances, yet due to my own limited understanding of the situation I have always considered our electronic communications to be analogous to the "... papers and effects..." outlined in the 4th Amendment, but my motivations are different from those who have been appointed to interpret our legal structures.

The very notion of "Too big to fail/jail" is yet another example of holding those with vast resources to a different standard when compared to the average Joe Public who worries about how he's going to continue to feed his family as well as keep an acceptable roof over their head.

Where it leaves us is at an increasingly disadvantageous position, unless we are of the Bourgeoisie and able to purchase our way through the system on par with big banks who launder cartel drug profits with impunity; where political appointees can perjure themselves before Congress with impunity; and where every day words are redefined by our public servants as they maneuver their way through the system with similar impunity as relates to their questionable acts.

I agree with a lot of what you're saying, but I actually don't think the recent surveillance scandals are the best example of corporatism. It is very unclear that corporations (for instance AT&T, in the article) are actually colluding with the government, rather than being legally compelled to comply.

"Unlike the N.S.A. data, the Hemisphere data includes information on the locations of callers."

Unlike the publicly known N.S.A. data.

Also, as if call status matters.

It can't just be AT&T, it must be regional bells, the other major carriers, small companies, everyone.

?? There are only 2 regional bells now, and one of them is called AT&T.

The article notes that:

"Representatives from Verizon, Sprint and T-Mobile all declined to comment on Sunday in response to questions about whether their companies were aware of Hemisphere or participated in that program or similar ones."

If they were not participating in Hemisphere or similar programs, I'd expect them to be quick to say so.

If they were asked to comment on Sunday (on a weekend), then it's probably more likely that they wanted to confer with their legal departments before denying with any confidence.

This is especially in light of the recent public/media PR confusion with technology companies around PRISM.

The connection to counternarcotics is a distraction from the real story here. Legalizing drugs changes nothing. The article points this program was used in routine law enforcement, e.g., bomb threats.

I'd like to know more about the "administrative subpoena" process. How often are innocent people's phone logs dumped? Why not just install cameras in the TVs and be done with it?

If you look at slide 18, it looks like 96% of requests were for drug cases.

Yet another reason to legalize drugs.

When they say it goies back to at least 1987, I think they are referring to the Daytona database of call detail records (CDRs). That project was active in the 80's, but it goes back much farther, and it may contain most of the CDRs ever created. In it's early stages, punched cards, and later magnetic tapes, were trucked to Bell Labs and subsequently AT&T Labs, where the Daytona system was operated. In it's heyday it was the biggest database on Earth.

There may be a successor to Daytona by now, and "all the CDRs, ever" is not as impressive as it once was, but it is likely that joint telco/government projects like Daytona have been collecting and analyzing call data since there was call data to analyze.

I would not be surprised if they are still using Daytona's query language.

HN and the internet community reacted with consistent and vigorous outrage towards Facebook and Google for complying with subpoenas, and over unconfirmed rumors and suspicions that they were selling user data to 3rd parties (not just ad targeting internally). We had boycotts and alternatives to try to move off these services.

Where's the boycott of the PSTN? Why aren't we angry at AT&T for actually embedding staff in the government? Why aren't we encouraging people to switch off their services?

Probably because telcoms are already considered the worst offenders, and assumed to be evil? No change to the status quo = no uproar.

"Crucially, they said, the phone data is stored by AT&T, and not by the government as in the N.S.A. program"

The irony of this, is that it's exactly the tactic used by drug dealers. In order to avoid the risk of possessing illegal substances, they coerce drug addicts to hold their stash. The courts call this 'constructive possession'.

Key term: administrative subpoena. Judges, we don't need no stinkin' judges.

http://www.wired.com/threatlevel/2012/08/administrative-subp...

For the P. Simon inclined, rather than just pointing to the obvious slope we've slipped down since Nixon:

Whoah God only knows, God makes his plan The information's unavailable to the mortal man We're workin' our jobs, collect our pay Believe we're gliding down the highway, when in fact we're slip sliding away...

http://www.lyricsfreak.com/s/simon+and+garfunkel/slip+slidin...

I wonder if this system has a search capability or it strictly limited to phone # searches. I also wonder if tech to speech works well enough to convert all these calls to text.

Reply on news.ycombinator.com