“What Is That Box?” – When The NSA Shows Up At Your Internet Company | Hacker News Reader

“What Is That Box?” – When The NSA Shows Up At Your Internet Company

readability.com

326 pointswikiburner94 comments

We had to facilitate them to set up a duplicate port to tap in to monitor that customer’s traffic. It was a 2U (two-unit) PC that we ran a mirrored ethernet port to.

[What we ended up with was] a little box in our systems room that was capturing all the traffic to this customer. Everything they were sending and receiving.

And yet his lawyer could have written a truthful denial that they'd given the govt "direct access to the server". See how that works?

I used to work for a webhosting company, and had similar experiences. We'd get requests for Men In Nice Suits to come in, rack up a nice non-descript 3u box -- this was a few years prior to this experience, so I'm certain that the tech's improved since then. As was described, the box just sat there, eating power, under orders Not To Touch Under Any Circumstances, until the federales came back in to take their box back.

Thinking back about it, again, this seems a lot of how the feds can keep things like this from getting out. The people that know are given the gag orders, the sysadmins racking and unracking know it's better for their careers, and their not staying out of jail, not to say that they have weird boxes on their network which have mirrored ports going to them. It's there, it's suspect, but the consequences for discussing a suspect box make it difficult to really discuss things.

I am so glad that I don't work under gag orders like that. It just doesn't seem ethical to be paid by a customer to spy on them. I understand why most would say nothing, but it must be so tempting to inform the customer.

It just seems very inefficient to store stuff on a hard drive for 9 months, no? If it's important enough to snoop, shouldn't you want to get the word out sooner?

So, mount webcams in the datacenter. Point them at your racks (front and back, to show cables). This isn't a bad idea in any event, because sometimes it's good to know what Figby Tenthumbs recabled on Monday morning when he was hung-over.

Now make access to the cameras public.

"What's that new box?"

"We can't say."

"Ooohh, I see. Noted."

Watch the watchers watching.

That's a similar mechanism to a warrant canary. I like it, and haven't heard of it being used, but I doubt the effectiveness.

Though, I suppose if a company suddenly had to take down their datacenter video feed for unspecified reasons, that would be tell tale.

But could the FISA court order the datacenter operator to loop the feed? We'd never know. And that's scary.

Meh. Pay an intern to drop a copy of today's newspaper in front of the camera every morning.

You could trivially detect a loop by examining noise.

You could trivially defeat that by injecting noise.

You could less trivially defeat that by looking for "random" behavior in the datacenter, such as people walking by, vibrations caused by folks moving about, blinking patterns of the ubiquitous network activity LEDs in your rack and other racks, etc.

I think a loop is trivial to detect.

Have your own server's LEDs blink in pseudo-random sequence; have a script monitor the video feed, alerting if the sequence of LEDs doesn't match what is expected

This is the culmination of a very nice subthread. While there is certainly a huge need for a systemic fix, the subthread shows how technology can help us fight the problem on a temporary case-by-case basis. Brilliant!

Also, our names are on lists now.

[I despise the fact that I am making a joke like this, and that it's probably true. I'm writing my folks in congress again.]

Mount a watch/calendar on the wall.

Not if there was a clock in the frame with accurate date and time.

And then the day the webcam broke, customers fled in droves.

Or, the government simply says, "You have to say that the new server is a web server".

This is why I'm always skeptical of things like "warrant canaries" and the like. If the government can require you lie to your customers about whether they're being monitored, surely they can control the terms of disclosure about what's going on?

I wonder if something like making a donation to the EFF for every week that a warrant has not been served would work. Could they compel you to keep donating?

Maybe, or donate on your behalf, but they could compel you to say you're still donating, and the EFF says that they don't disclose donor info except as compelled by law - so I assume they could be compelled to keep reporting so.

But honestly, this is getting so ridiculous. How would people know that you'd stopped donating? If you had a webpage that said, "We donated this week", the government could just require you keep putting it up. You could give the EFF the ability to disclose your donation - but what, your customers are going to call the EFF every week? The EFF is going to set up a webpage that your customers are going to go to, in order to verify that your donation was received?

Then, let's say you have 500k customers. You get a FISA request or whatever for 1 of them. You stop donations, some webpage somewhere gets updated, and....what happens, exactly?

First off, the customer would have to want to check however mechanism is available, since you certainly can't notify them. So once they check, what do they know, other than someone on the service has received a government request.

Then, now that they know this, what do they do about it? Migrate off? To whom? If all it takes is one request for customer data to trigger the warrant canary, shit, all the government has to do is make one request for every company with a warrant canary. Now there are no available services.

At which point what do your users do? Do you go with an un-canaried provider? Hope that you're not the target of the warrant?

Then, look at it this way, someone, perhaps a less scrupulous provider, will simply lie about not having received a warrant, in order to get the business.

While it's a fun thought exercise, there's absolutely no practical way to do what people are suggesting without using a system that would be so vulnerable to manipulation as to be useless.

The Riseup folks had a webcam running when one of their servers was seized for running an anonymous remailer. The FBI returned the server two weeks later, without ever notifying anyone that it had been seized or returned.

https://mayfirst.org/fbi-returns-server

I like the way the larger woman in the background is looking up at the ceiling as if for cameras. Also that the male short-haired government agent is wearing a suit, as one would expect. And then looks ~directly into the camera at one point.

Maybe I just don't pay enough attention to this, but this is the first place where I've read that Google and other large companies are being paid for monitoring their customers. This is making my sleaze-o-meter spike. What are the rates like? Is it per user? Per message? Per kilobyte? It certainly couldn't be per arrest...

Sometimes it seems like the rabbit hole just keeps going deeper, but then you realize it's a damned sewer!

They don't offer anything up with that last point ("bigco's selling data"). As far as I can tell it was pulled from thin air. They're understandably upset -- a lot of us are -- but I don't think that quite justifies making unbased accusations like that (and if it has basis I'd love to hear more).

No, this is not new information. Companies do get paid, a lot, to facilitate this. This was known before the leaks, but leaks confirmed it. Now there are articles all over if you search. Example:

http://www.cbsnews.com/8301-201_162-57593273/verizon-at-t-ge...

"In its letter to Markey, AT&T estimated that it collected $24 million in government reimbursements between 2007 and 2011. Verizon, which had the highest fees but says it doesn't charge in every case, reported a similar amount, collecting between $3 million and $5 million a year during the same period."

I'm not sure how I feel about that (making the companies pay for it doesn't seem fair either), but it certainly doesn't leave us with incentives in the right places.

$24 million over a year period paid to AT&T doesn't fit my definition of 'get paid a lot.'

AT&T had profits of (approximately) $7 billion on revenue of $31 B in 2012. So these reimbursements amount to something like 0.015% of revenue, or 0.1% of profits. For a firm that size, there are much greater incentives to lobby for favorable rules on employee health benefits, spectrum allocation, or consumer relations. I mean, over that time period the CEO of AT&T earned about $100 m, so in theory he could afford to take the hit for all that fee revenue personally and still make more money in a year than most people see in a lifetime.

I'm not expressing support for the NSA's activity here, I think this sort of data vacuuming is quite dangerous. But viewed in the context of actual corporate revenues, I don't think it's realistic to say the government is bribing big business here, because the amount they're paying is trivial, and thus largely reflective of the change in the actual operating cost of the corporate data centers.

First line should say 'over a 5 year period.' oops.

I recall cricket was the cheapest. Can I just get a line item on my phone bill for NSA surveillance instead of paying it through my taxes? :)

When you receive paper from the government under FOIA, you are required to pay for the photocopies. Responding to your FOIA request isn't a mission directive, so it doesn't deserve budget money. It's just something the government has to do. You are requiring the government to use resources (toner, paper, time) so you have to pay for them.

Surveillance by government is the same way. Police and three-letter agencies are using engineer time, bandwidth, and potentially rack space of service providers complying with warrants. They compensate providers for those resources.

Similarly, if the police kick down an apartment door, they're supposed to compensate the landlord for the cost of a new door. If your municipal police department wanted to wiretap your cell phone, they would have to pay Verizon/AT&T/whatever a monthly fee just like you do.

Your statements are all valid, but it still doesn't detract from the parent's statement that [sic] something about companies being paid to spy on users is just plain wrong.

your statement is illogical.

Parent broke down why the companies ask for compensation, and it's reasonable. Why should the government get to take up resources for free?

The problem I see is, all this are asked to be kept secret. How much money the Government gives the Telcos, ISPs and websites like Google, Facebook etc. for monitoring people and how much these people charge etc. are being kept secret.

You create a ghost. Create fear of the ghost. Tell people that only you can protect them from the ghost. But you don't tell them how they plan to protect them from the ghost, nor are you willing to disclose how much you spend to protect people from this ghost.

Don't forget - when the ghost doesn't attack you claim credit for stopping the ghost.

Yes. I think, the whole dialogue on terrorism should move away from the abstract concept called 'terrorism'. Any disgruntled group which sees itself as the underdog against a very powerful entity will resort to terrorism. You cannot wipe out terrorism from the face of the earth, like you cannot wipe out car accidents. The governments the world over are asking for enormous powers, selling us the dream that there will not be one innocent life lost because of another terrorist attack. They are dumbing down the actual issues behind these problems.

People should realize that only bringing focus to the real issues and not blanket regulations and restrictions on freedom is going to have some real effect.

Why don't governments create the new laws or policies time bound and specific to particular issues. If they see Al Queda activity in US, make it public. Release information on the organizations. People behind these organizations, the people helping to fund these organizations. Create embargoes on countries and organizations funding these organizations. And do them more effective and open manner than how it is done now.

Fill the media with real issues and educate people who sympathize with terrorist organizations. Give a platform for these people to redress their grievances. Create more opportunities for leaders of supposedly 'terrorist' organizations and to have more debate and dialogue with others.

The data is handed over because a sovereign nation issued a legally binding court order. Google hands over data when required to do so by court order, otherwise it doesn't.

You can't pay Google for private information unless you have a court order, and Google is compelled to hand over data whether or not the feds can pay. The data is not being sold. Google is only being paid for resources it is already legally obligated to spend.

Indeed. I think it is somewhat analogous to the US Army blocking access to the Guardian's website because of the classification of the Snowden leaked documents hosted there.

Everything follows in a logical order when viewed from one perspective, but seen from the perspective of a normal person who cares less about internal procedures and more about general governance it is obvious the emperor has no clothes.

http://www.guardian.co.uk/world/2013/jun/28/us-army-blocks-g...

Why is it wrong?

If the government wanted to have a room in an office building for a long period of time to spy on a company in the sam building it is reasonable to charge rent & for utilities. I don't see how this is different?

(I agree in principle that warrantless spying is wrong. But if a warrant is issued I certainly don't see why a company shouldn't charge for their resource use)

On the other hand, no-one gets compensated for doing their taxes.

In Australia at least the cost of preparing your tax return is tax deductible. For that matter the cost of visiting your accountant in order to do your taxes is tax deductible.

In the US you can deduct the cost of a tax preparer or the cost of DIY tax preparation software if you itemize deductions.

Nope, but everyone has to do them. But not everyone get their door kicked in. Or you could just tax people for the money the state will be paying for people to fill in there taxes ...

Most people take the standard deduction. Would you feel better if they said that was for filling out the paperwork?

It makes sense to ask the government to cover your costs when you're doing something on a court order, and by god you aren't going to lowball it. On the other hand, it should hurt to participate in something like this, and a corporation of any significant size doesn't have a conscience to pain it (no matter what the feelings of the people animating it.) If the government reimburses companies for a generous estimate of the costs, they won't have to worry about the industry lobbying against this kind of coerced cooperation in domestic surveillance.

This is a really common arrangement when the costs involved are non-trivial. Even in civil litigation, complex e-discovery often involves cost recovery for searching through and locating records.

I had a similar experience related to a more typical criminal investigation and an individual computer. There was a warrant signed by a federal magistrate, our counsel reviewed it, and they were professional and respectful of our operational concerns.

The problem here is the secret court, secret warrant, etc. the rest is fine with me, the police should be able to investigate crime with appropriate oversight.

http://gizmodo.com/5898964/this-is-how-much-carriers-charge-...

What is the alternative. Charge zero dollars irrespective of the amount of data requested?

Cox Communications charges $2,500 to fulfill a pen register/trap-and-trace order for 60 days, and $2,000 for each additional 60-day-interval. It charges $3,500 for the first 30 days of a wiretap, and $2,500 for each additional 30 days. Thirty days worth of a customer’s call detail records costs $40.

Comcast’s pricing list, which was already leaked to the internet in 2007, indicated that it charges at least $1,000 for the first month of a wiretap, and $750 per month thereafter.

[ http://www.wired.com/threatlevel/2009/12/wiretap-prices/ ]

It would be really interesting if this revenue was Google's 2nd best performing business after Ads.

It's not always the NSA. Some of my datacenter friends told me stories about times when "a box" would appear and they were officially to not go within 6 feet of it. Of course, actually working on neighboring customer boxes meant sometimes violating that (without telling anyone), but for the most part they would stay away.

I seem to recall they were chasing down online pill vendors this way. One little box with power and two Ethernet ports can collect a whole bunch of evidence, after all. They get what they need, and then they remove it.

This was 10 years ago... or more. I can only imagine what happens now.

I think the this excerpt is a fine description of the problem with secret courts and so on:

"These programs that violate the Bill of Rights can continue because people can’t go out and say, “this is my experience, this is what happened to me, and I don’t think it is right.”"

Wait. They show you the warrant requiring your compliance. But you don't get to keep a copy of that paper?

How do you later prove that you were required by law to make the actions that you did? How do you ensure that you comply completely with the instruction if you can't compare your action to the original warrant?

That seems strange.

> How do you later prove that you were required by law to make the actions that you did?

The gag order attached to it prevents you from telling anyone but your attorney that you received the court order. Having the paper wouldn't help you prove anything when you are not allowed to acknowledge its existence.

Wait, the government can ask me to pretend that a physical object is not there?

Do they really expect engineers to start ignoring the laws of physics?

> Do they really expect engineers to start ignoring the laws of physics?

Well apparently they have been for a while already.

Exactly, I'd go into CYA mode and try my hardest to get a copy.

Sounds like a great way to create mayhem: talk to isp, fake a gov warrant. Say they can't copy it. Get them install your box. Do mayhem. If anybody comes, there is no paper trail. profit. :-(

Direct link to the story:

http://www.buzzfeed.com/justinesharrock/what-is-that-box-whe...

This is likely the reason why lots of tech firms give the NSA access to their servers. It's better than having a box installed on your network.

http://news.cnet.com/8301-13578_3-57593538-38/how-the-u.s-fo...

I'd say from technical or engineering POV it's a whole lot less hassle and less likely to break things to just mirror a switch port.

A switch port tends to carry traffic for multiple customers. Supposedly Carnivore does minimization, but I think companies like Google would rather collect data themselves to make sure it's done right.

"A number of [larger] companies are getting paid for the information. If you go establish a tap on Google’s network, they will charge X amount per month. Usually the government pays it."

This is directly contrary to what every "larger" company has repeatedly stated in response to Prism. People actually think that the companies are not only forced to keep silent, but release public statements lying?

That every major telecom/ISP charges for wiretaps and other information pulls has long been public knowledge. Reimbursing them for government requests is a line-item in our public federal budget.

http://www.forbes.com/sites/andygreenberg/2012/04/03/these-a...

Yes - we have seen language contorted into meaninglessness by by lawyers and courts. "Waterboarding is not torture." "Data recordings are not data collection." "Drones cause one civilian casualty per hundreds known dead terrorists."

I am a fervent believer in the power of government to do enormous good - but is necessary that those actions be public or they will invariably be abused. Whenever you contemplate government acting in secret, you must weigh that against the cost of that action being abused - because at some point it will be.

I'm angry and will let you imagine a link to foaas.com.

If everyone said go stuff yourself and published it on the internet, eventually they might get the message. But no one wants to be force-fed in Cuba.

Under such circumstances, why not extract as much monetary compensation as possible from the government and donate it to the EFF, ACLU, or similar?

I'm very curious to know if these little black boxes could function as a MITM. I mean, if you're already there mirroring everything that's going across...

Just guessing, but since it "just" gets all of the traffic that is passing in and out of the other server(s)/switchport (probably much like a hub), i don't think it'd be able to interfere with the monitored servers traffic.

Also, it'd kinda tip off the monitored person(s) if there suddenly was another hop on the route to their server, no?

No. Its entire job is logging. Doing MITMs could very potentially lead to information leaking that shows surveillance is going on. Pretty much any switch a datacenter's going to use has port mirroring, which allows for a passive, invisible tap of a server.

I do wonder if the NSA insists this functionality is there...

Just an unfortunate side effect. There are lots of legitimate uses for port mirroring in troubleshooting and monitoring a network, like intrusion detection and performance monitoring.

This was probably the FBI. And if the data center's switch couldn't mirror the customer port (I can't imagine any data center would use a switch that couldn't - but it is within the realm of possibility), the investigating agency would probably provide a 1U switch along with the 2U server that could handle the mirroring and then they could force the data center to connect the customer through that switch instead.

I wonder what the website was... The only one I can think of that might possibly get this treatment might be Maddox, but thats total speculation of course.

Was what I thought too. Maddox would perfectly match the sort of semi-incompetent, this-guy-seems-creepy surveillance that someone who wants to cover their ass would do. Anyway, XMission is not that small, so there's no way of knowing.

It could easily have been a community association or other similar group. A group of gardeners? Well, they could be eco-terrorists. A bunch of software enthusiasts? Why, these open-source people are borderline commies, better check them out.

This is routine activity for police authorities, unfortunately.

I think it was related to ELF/environmental activism/terrorism. I'm not sure where I saw that, though.

Dumb question, but the author kept running a TOR node at a site he knew was under NSA surveillance? That doesn't strike me as very responsible...

From how I read it, he did that after the experience, when it wasn't under surveillance anymore (well, at least not from inside the datacenter), as a sort of atonement.

Someone more knowledgeable than I should confirm or deny this, but my understanding was that TOR should be resilient to a single point attack like that. On the other hand, if they can watch packet timing on a significant fraction of intermediate nodes, there is a problem.

And, hypothetically, the FISA authorized box was only getting traffic from the one site, and not the entirety of network traffic. The room 641A attack is far more problematic.

TOR is resilient to a single point. However, if an adversary can compromise both the entrance and exit node, then they can de-anonomize the traffic.

As I understand it, TOR uses encrypted layers each of which tells the node where to send the partially unwrapped message on to.

So, if I encrypt something to, say, three layers and sent it to some TOR nodes:

1-2-3

1 knows it came from me and knows to send it to 2

2 knows it came from 1 and to send it to 3

3 knows it came from 2 and to send it to wherever

No one point on the system, IIRC, is meant to have the information necessary to compromise the entire chain. Though, if you could compromise a sizeable portion of the network, you'd be in with a significant chance of compromising any given message - which I find hard to believe that the government hasn't done.

I think the only reasonable thing to do would be to somehow encourage (without directly encouraging) people to try a variety of "test traffic" on your box, with the subtly-concealed intention of rooting the fed's box with traffic addressed to your own box (and thus not "intentional", and done by other people anyway). For a "hacking contest" or something.

Then you're charged with obstruction of justice or something.

They were mirroring the port of a single server, not capturing all traffic to the entire site. The TOR node would have been unaffected.

Thanks for speaking of your experiences with the rogue spy apparatchik which has recently reared its ugly head and I'd also like to thank you for running a Tor node.

Together we stand, for freedom. For America.

/me wonders: why a link to readability who is just framing buzzfeed.com and not a link to the original site?

And why do 245 people upvote it without noticing this link bait?

Also, the buzzfeed site has a relatively good layout. None of that overlay toolbar bullshit or popup shit. In fact, this readability link adds an overlay toolbar to the site (one of the main reasons why I use readability in the first place).

So do these companies approach the government and say "Hey, give us $1000000 per month and we'll just give you full access."?

Your numbers are inflated, but that's literally what AT&T and Verizon did. They approached the government.

This sounds like something that warrants a hyperlink, even when considering the cost of making one.

Source please.

could someone run an isp with a completely public inbox, so that they couldn't receive a FISA without it becoming public?

are companies obligated to have a private means of contacting them?

I imagine if someone tried that, the NSA/FBI would merely say, "hey, head down to our office, we want to talk to you about something."

Continuing that I wonder if you could have webcams in nearly every area of the building and make it all publicly accessible.

If you were in a one-party consent state (ideally without notification required), you could just transmit audio logs of "all incoming phone calls, for customer service and quality control purposes" to a third party, and the third party could choose which to publish. The third party would be under no obligation to keep it secret, and the caller would not know which third party to contact to get an injunction to prevent publication.

I would be tempted to quit my job on the spot if I'd receive one of those orders... (Not earning that much anyway so I could deal with it.)

What if this ISP instead cancelled the service of the business to be tapped?

The whole American society is gagged and that's the problem. If you don't like unconstitutional actions of the US Government then you are called: 1. Traitor 2. 9/11 Truther 3. Terrorist

That's where the apathy originates from.

I strongly believe that in the USA of today saying out loudly that a radical change is needed to get the country back on its Constitutional track could make one a terrorism suspect. If they can label 82-year old nun a terrorist and try her in court on this nonsense, then why not me or others who speak out loud ?

Reply on news.ycombinator.com