NSA OSS Technologies | Hacker News Reader

NSA OSS Technologies

nationalsecurityagency.github.io

427 pointsandrewke105 comments

This caught my eye:

> https://github.com/apache/incubator-pirk

> Employing homomorphic encryption techniques, PIR enables datasets to remain resident in their native locations while giving the ability to query the datasets with sensitive terms.

I can imagine a few scenarios there. One perhaps is when db admin should not find out what someone, possibly working on a classified project is querying.

Or say one compartment / project collected the data and now they want to share it with another project. Those read into the second project don't want to reveal to the first one what they are querying because it would reveal classified information.

Another scenario is a database which has results of possibly illegally intercepted communications. If the NSA can argue that the Constitutionally defined "search" doesn't occur until someone actually performs a search (as in runs an SQL query over the data). Then having PIR capability means being able to break the law but only let as few people as possible do it.

Also https://github.com/redhawksdr is pretty damn impressive. It looks like a complete parallel implementation of GNU Radio. Completed with an IDE and such. Wonder how it compares?

This is pretty common in the commercial world too, and something I've done more than once myself. The obvious use case is storing medical records.

In the UK personal medical records are often stored by systems integrators in datacentres with nebulous locations, and need to be accessed by third parties for things like underwriting life insurance policies.

To protect the data (compliance with the EU data protection act) it's encrypted in transit AND at rest. Access to data by third parties is managed through AMRAs (access medical record authorisation), which are completed by the third party, authorised by the data owner (private individual) and given to the data owner's general/dental practitioner or pharmacist, who is able to access and decrypt and appropriately share the sensitive data.

Are AMRAs kinda like "stored procedures" then?

AMRAs are like physical documents with an instruction to the information guardian, with a signature from the information owner, authorising access.

A lot of it is electronic these days, and is automated to the point that an individual authorises access by clicking a link in an email that calls an endpoint that in turn releases a token and URL to the requestor to view the appropriate records.

So like OAuth?

Partly. OAuth is authentication (who I am), which is part of it, but the real point is authorisation (what I can do).

All valid points, but I think it has more to do with this research being popular in the encryption community. One of the Coursera teachers was working on something like that.

An example of use is video games that help prevent hacking and modification.

I think that there could be some great consumer usage for having the ability to encrypt data, but still be able to search it.

So looking at the apache incubator project pirk is retired due to inactivity [1]. Is anyone interested in reviving the project? [1] http://incubator.apache.org/projects/index.html#pirk

I guess an example might be that the NSA has access to other partners information (e.g non-five eyes like Germany) and doesn't want their DB folks to be able to see that they are quering against the databases (e.g Merkels phone number)

This is exactly what PIR is for. A good intro is chapter 6 of the (awesome and ahead-of-its-time) book Malicious Cryptography. One of the authors works at the agency now, I believe.

Intriguing indeed. Could this tech also be used to query a database or service to plot a route (say for navigating with your car) without revealing the route?

Violating the sanctity of the captive portal license agreement! https://github.com/iadgov/goSecure/blob/master/scripts/wifi_...

The captive portal license agreement?

That code just bypasses a captive portal. It's basically automatically clicking "I agree".

Behold!

The captive portal license agreement!

That's nice to see the NSA is contributing to the OSS community. I just randomly picked one of the NSA GitHub repositories, analysed it with VersionEye (https://www.versioneye.com) and found already 25 security vulnerabilities. Who is the best person to contact in this case? Here is the security report: https://www.versioneye.com/user/projects/59479cd06725bd00123....

They say it clearly themselves:

> The government benefits from the open source community’s enhancements to the technology.

They're hoping that by putting this code out there, unwitting dupes will then collaborate with them to contribute to the surveillance state.

Can someone explain how some of projects can be MIT-licensed (or anything-else-licensed) as they claim? Aren't they necessarily in the public domain given that they're works of the U.S. Government?

Certain government agencies and subsidiaries are exempt from having their work considered "government work" and can thus claim copyright if they want. I'm guessing the NSA is such and agency. Also if the work was actually done by a contractor then there are other exemptions.

Thanks, but I'm not sure that's it. For example, when I look at at [1], I see an apparent contradiction with [2]. It almost seems like they don't know what they're doing, but surely that's because I'm misunderstanding what's going on?

[1] https://github.com/NationalSecurityAgency/DCP/blob/21c8d3efe...

[2] https://github.com/NationalSecurityAgency/DCP/blob/496402fa9...

Well if they are forking or contributing to other software or any type of derivative work, they probably have to retain the original license by law

That's not the case at least in the example I just gave though, right?

It could also be based upon patent rights, I know that employees get a 50% ownership of all patents, maybe that's part of it?

I've trialed Apache Nifi and it's very powerful. It's also a little unsettling as you use it thinking about how the NSA used it ...

NiFi looks pretty interesting, what's the learning curve like? Looks like a supercharged yahoo pipes.

No wonder Hortonworks wanted to integrate it.

FWIW, all your crypto is used by NSA as well.

https://github.com/ozoneplatform/owf-framework looks very interesting - NSA wrote their own BI tool?

With the selling point being that it tracks you less than other tools. In fact, avoiding tracking seems to be a prevalent line on those projects...

Yes, it's understandable, but still ironic.

OWF came up in a meeting recently -- may not be as awesome as NSA wants you to believe.

The Ozone Widget Framework is a mess, albeit a mess that's usable once you learn its quirks. The idea is basically web components, portlets, etc. rehashed with some shared service capabilities added in. Something I find interesting and rarely see mentioned: OWF was built from Apache Shindig, an Open Social spec implementation Google donated to Apache. For more information on OWF, see (PDF) http://www.dataintelligencellc.com/uploads//DI-OWF-Capabilit...

Yeah OWF is kind of OK, we tried to use it but the docs aren't straightforward.

Can you share any other info?

The Ozone Widget Framework is horrible, you have to build your application sets to function standalone as well as in a modular compatibility with other widgets. Things break frequently and for the longest time Silverlight was a requirement too (well after it was canceled). It was absolutely horrible when I used it.

reddit have a comment thread about this.

Once comment described it as a "shit show".

Apparently its terrible and they tried to get the company to rebuild it and they made it the same.

Another comment comment on how "buggy" it was.

Or that the only dev that willing to work with it are contractors that want money.

> Or that the only dev that willing to work with it are contractors that want money.

Greedy b*stards! </sarcasm>

I wonder if all the people who are really suspicious of it in here realize that this (releasing their projects as OSS) has been a thing for a while.

SELinux

Accumulo (a popular NoSQL distributed key-value store)

Apache NiFi (data processing system)

etc.

It's nice to see a push for open security and solutions rather than secretive offensive counter-security. Thanks. :)

One does not exclude the other though

One of my favorites is the Speck cipher, which has been released before:

https://en.wikipedia.org/wiki/Speck_(cipher)

I'd be very interested in more public cryptanalysis of this. It's a damn simple cipher to implement, and if it were at least as secure as say Salsa20/12 it'd be very nice for all kinds of applications.

Does the fact that many of these havent been updated in months or years mean that these are really old projects that effectively hold no value to the NSA and arent close to any of their core operations?

For every one of these projects someone spent a considerable effort do the paperwork to get it pushed out, have it signed off as sanitized, non-embarassing, etc. It is a lot of work to get that done in bureaucratic and risk-averse organisations.

If there had been a community contributing back I expect that there would have been more activity, but if it seems like noone will, would you spend your time pushing out regular updates?

Or they've received updates that make them too important to release? But likely it's just that they haven't had any major updates, and NSA internal bugfixes aren't important enough to push out, or aren't given out due to worries that that would be a national security issue somehow

There's a lot of neat things there. (This one looks interesting: https://iadgov.github.io/goSecure/)

Also interesting is splitting the repos: that the NSA and IAD have different repos, and that one seems focused on defensive tech while the other is publishing analysis tools.

I know there's a lot of people who aren't fans of the NSA (or what they do), but I think most of us can see a need for a military-grade organization to research defensive technologies for helping secure our infrastructure. I don't think many of us would be unhappy with the NSA if that's all they did. (Or phrased another way: most of us are unhappy because of how they conduct intel work or compromise defensive capability for offensive ones, eg, that whole business with ECC.)

So I think it's important to respond positively to things like the IAD github page, even if we're not fans in general.

I think you're right. It's sad to see many people are looking at these tools and performing a sort of "Allegory of the Cave" by extrapolating, then, the evils that can be done with these tools.

Something, mostly common sense, tells me that we will not find some smoking gun to a crime here in these OSS repos...if anyone wanted that, they can refer to any number of leaks.

Ultimately, I'm happy to see this stuff shared, happy to see others use it and happy to see the OSS community build on it.

There's multiple reasons why you wouldn't want to use these newly released open-source projects. First one is, like you said, the danger of a backdoor. The second one is that due to the very long list of non-ethical and illegal practices of the organisation you don't want to contribute or depend on them.

Bleach has been used in bombs around the world, should I not use it because there's a chance it's been used for (perceived) evil?

Cluster bombs have such blatant propaganda spread against them by well-meaning but naive individuals. Don't be fooled - look at all their positive uses!

That's not a like for like comparison, one is a bomb designed for destruction, one is software designed to solve a problem. The software isn't designed to kill, maim, or destroy, if it can be used as part of the process that doesn't make the software itself evil. Should we throw out linux systems since linux powers many control systems in war machines?

One thing to remember is that the NSA isn't a monolith: there are factions and differing opinions inside of the agency.

It's likely someone spent a fair amount of political capital to draw attention to the agency by emphasizing their public projects and trying to engage with the wider public. If there's a negative response to that, it only lends weight to the voices inside the agency who are against that sort of thing.

I, for one, prefer the NSA to be working on defensive technologies in collaboration with the tech community to any number of things they could be spending the resources on -- and think we badly need their expertise and help to secure domestic assets.

So I'm going to say "good job!" when they're doing things I like and save my criticism of their other behaviors for more appropriate moments.

I think collaboration is fundamentally more powerful an instrument of change than shunning is.

"military-grade" doesn't mean anything in the context of crypto. We all use "military-grade" crypto every day. If your argument is that somehow the armed forces are better at computer security than the rest of us, because "military", then I reject it wholeheartedly.

They are criminals and should be disbanded. The US intel community is full of cheats and liars, straight to the top.

http://www.hasjamesclapperbeenindictedyet.com

In my experience what military-grade crypto really means is crypto that complies to requirements stipulated by various laws and directives related to protection of state secrets. This includes widely used cryptographic primitives (AES, SHA...), various NATO, or NATO member specific primitives (often with weird interfaces, eg. DES-like checksums in keys and such), complete cryptosystems for particular usecase (often with questionable security under security models used by academic cryptographers) and various utter nonsense that only exists in order to comply with aforementioned legal framework (eg. various "solutions" for connecting two systems with different security classification without actually connecting them)

In other words, military-grade means used by military, which has no meaningful correlation to security.

Well that is a bit uninformed. Military grade (at least the US context) means algorithms and implementations analyzed and approved for use by the NSA. Today this means 'Suite B' crypto like AES, RSA, ECDH, etc. It should also mean dedicated hardware or certified implementations, physical key fill, etc. However, the words 'military grade' are frequently abused by sales to mean a badly performing variable time noise spewing implementation of AES.

In days past the 'commercial grade' crypto was often not real crypto, like voice scramblers, using 40-bit DES (when govt was using Triple-DES), XORing against a non-cryptographic PRNG keystream repeatedly, all sorts of rubbish.

That is mostly what I meant. Military-grade means approved by NSA or it's equivalent in given state. What I tried to point out is that such approval does not necessarily mean that such cryptosystem is secure for your application (eg. various tactical radio encryption systems, "military DRM"...) or even secure and meaningful at all ("data diode", various NATO TS approved quantum cryptography things...).

I wonder why https://www.iad.gov (linked at https://github.com/iadgov) is not using a TLS certificate trusted in normal browsers. I cannot visit the webpage as it uses DoD Root CA, which is not installed on my computer.

Having the US department of defense be able to forge certificates for every site world-wide, in every major browser - out of the box - might be a little too much, even with the CA system as broken as it is.

On the other hand, if you run your own CA and mostly care about your own users - using a cert signed by your own CA makes sense - to a certain extent.

I think the question was "why aren't they running this public website with a cert signed by a widely trusted CA"?

Well it is widely trusted by everyone on NIPRnet...

So TLS (X.509) only allows serving a single certificate. You have to choose to serve one trusted by people you need not to be hacked (your own CA) or a commercial one to reduce in general the likelihood of being hacked. I can see why they chose the first option.

Obviously if websites were not signed by one of the 'root trust' paths but by several, and the reputation of each of those trust paths, and the host itself, was tracked in a decentralised secure database where trust was built over time, that would be better.

Oops, I just suggested blockchain snake oil would solve something.

A danger is that if you use their tech, you might become dependent on their tools.

What, like Microsoft?

I suspect there are a lot of very incredible computer programmers at the NSA and they're probably using just regular open source non security related tools every day. It's good to see that they're contributing back to the OS community what they can.

Let's not forget: these people may be skilled, but they are working against every principle of our community.

https://www-androidauthority-com.cdn.ampproject.org/i/www.an...

And you pasted an amp link? Was that deep sarcasm?

The world is full of shades of grey, and black and white 'they are all evil!' is just pointless and dumb.

It wasn't intentionally ironic.

Where did I call anyone evil? I didn't even vilify people - I specifically focused on the work that these people are performing. It works to make the world a worse place.

How should I criticize them?

For breaking the law and working to undermine the Constitution? Is that not a big enough claim?

Maybe blame specific people (the DIR NSA Hayden, Bush, Obama). There was a strong culture at NSA of NOT spying on Americans until those clowns came along.

But spying on other people is okay, simply because where they had the audacity to be born?

Nations are fictions and depriving rights to a group that you permit to others based on nationality is entirely unjust.

I blame the specific people who built these systems to collect and process data for the military.

Would you rather that nations didn't spy on each other? I wouldn't. Effective spying helps prevent war, and lack of intel leads to wars.

If the British had had better intel they would never have invaded Afghanistan to repulse an imaginary Russian annexation. If intel had been listened to, instead of deliberately ignored/fabricated by Bush we wouldn't have invaded Iraq because of imaginary WMDs. If Kennedy hadn't actively understood the intel on Cuba, and just left it to the fears of the generals, the US would have invaded Cuba. The biggest fear of the Warsaw Pact is that the West would invade -- which was the West's fear also -- to the point that a lack of intelligence on Western force movements almost resulted in a counter-premptive invasion.

So maybe nations are fictions, but they are pretty powerful ones that most people are happy to roll with, and that makes the consequences real. These same people can agree on a set of collective norms that control who/what/where can be surveilled.

The propaganda that there are many ISIS terrorists embedded in Western nations is the key lever which will be used to transition from military/national intelligence to the surveillance state. This can't be won by railing against the NSA, but by countering (islamic and fascist) extremism, false reporting and propaganda.

And yet in the mean time we are being outflanked by traditional nation state adversaries using strategic propaganda campaigns. And they won't hesitate to spy on us. I want NSA(/GCHQ/DGSE/8200) working hard to prevent that, instead of navel gazing illegal programs about spying on environmentalists/unionists/politicans.

What about using the state infrastructure/technology to spy in the Brazilian state oil company and steal engineering secrets to hand to american competitors?

one of their links is to a security enhanced linux android.

https://source.android.com/security/selinux/

Also don't forget SELinux https://github.com/SELinuxProject/ which is a project that comes from the NSA as well

SELinux is on the list :)

I see the PR department is getting smarter...

NSA has an offense side and a defense side, and they're not that tightly coordinated.

They also have several offense sides, none of which are tightly coordinated. :)

At least they merged / are merging IAD and SID, less confusion more chaos! https://www.washingtonpost.com/world/national-security/natio...

Federal government is required to open source at least 20 percent of its code now:

https://sourcecode.cio.gov/

There are pretty big national security exemptions which the NSA could use without any question: https://sourcecode.cio.gov/Exceptions/

That makes this effort even more commendable.

The cynic in me wonders which 20% we'll get. The repo linked here is pretty impressive, so I may have to eat crow. I would have expected to get every html template and vba macro ever written, instead of the value add stuff.

My thoughts exactly. That 20% will probably provide you with a map of Russia while you're flying over the Swiss Alps. Some things are better not learned (or known) at all than learned (or known) the wrong way.

Wow, that's great. The US being a first mover in OSS again, a pity that Germany doesn't have this.

That was exactly my thought as well.

Femto (https://github.com/femto-dev/femto) looks pretty interesting to me just based on some work I've needed to do in the past that it would've come in handy for.

It looks like the last commit was over a year ago, though. Is there information I'm not seeing of whether these projects are actively maintained (or still in use at NSA?).

Im also curious, without digging into it deeply is it something comparable to Solr or Elasticsearch? If so i wonder how it stacks up

This is nice. It should all be about protecting electronic systems. To help individuals and companies build resilient systems. It protects the USA and the world economy as a whole. It should never be about spying people or even catching criminals IMHO.

If they're looking for pull requests, they won't find any salvation in the OSS world.

It's generous to have all in royalty free license.

Why should it be generous? It's been paid through taxes.

Well, it's generous for non-americans ;)

A small compensation for the invasion of their privacy...

Fun sidenote: I was the very first civilian to contribute to their GitHub project back in July 2015, when SIMP was the only project they had up on GitHub.

It was literally a one letter change in the README file, but I still have the privilege to call myself the very first civilian to contribute to the NSA's open source project: https://github.com/NationalSecurityAgency/SIMP/pull/1

I'd shake your hand

Why are people so welcoming to the filthy spies invading citizen privacy?

Army is created to do arguably worse things than spies. There are people who completely reject the idea of military - but the bulk of society tend to accept it as a unavoidable and tries to control it and make it least evil possible.

Some spying is probably unavoidable in the current world - and just like with the army we need to think how to control it and make it civilized. Getting on a high moral horse only makes the matters worse.

Side note *The Army/Navy/Marines/Af/and Coast Guard share employees with the NSA. As in, close to half the NSA employees are D.o.D Military.

I really don't understand it, to be honest I think it's unbelievable. I'd say we, as in the IT community, shouldn't touch anything coming out of the NSA with a 10 feet pole, even if we're absolutely sure it's not some kind of morally dubious attack tool.

Fuck the NSA!

YES

These assholes are hoping the OSS community contributes back to reduce our own privacy! wtf

> The government benefits from the open source community’s enhancements to the technology.

Some of the are defending against other filthy spies.

Down With All The Filthy Spies!

Basically, it's filthy spies all the way down.

Grumping about it on hackernews doesn't really do anything either... it's the same as Facebook thoughts and prayers posts and clicking like to send well wishes to victims of the latest disaster. There is no better example of the IT community feeling strenuously about something and the general public just not caring than privacy. The Cypherpunks were tolling the privacy bell nearly thirty years ago and were on the cover of Wired, which is pretty mainstream for a topic like this. Nothing happened and without a very different mindset in the public I don't think anything ever will.

This being hackernews I'd like to turn the conversation to something more constructive: the idea of whether the coder doing the work is just doing a job or should feel culpability and suffer the consequences of working for "the bad guys" has probably come up in many places. Heck someone could probably write a Master's thesis on whether "Evil Scotty" in Mirror, Mirror was actually evil. I mean he did keep the torture booths running for Evil Kirk, but he also helped the good guys get back to their parallel universe.

So, seriously - do we feel the IT guys working at places like No Such Agency are forever unclean or are they just doing a job? Clerks never settled the issue...

>Randal: There was something else going on in Jedi. I ever noticed it till today. They build another Death Star, right?

>Dante: Yeah.

>Randal: Now, the first one was completed and fully operational before the Rebel's destroyed it.

>Dante: Luke blew it up. Give credit where credit is due.

>Randal: And the second one was still being built when the blew it up.

>Dante: Compliments to Lando Calrissian.

>Randal: Something just never sat right with me that second time around. I could never put my figure on it, but something just wasn't right.

>Dante: And you figured it out?

>Randal: The first Death Star was manned by the Imperial Army. The only people onboard were stormtroppers, dignitaries, Imperials.

>Dante: Basically.

>Randal: So, when the blew it up, no problem. Evil's punished.

>Dante: And the second time around?

>Randal: The second time around, it wasn't even done being built yet. It was still under construction.

>Dante: So?

>Randal: So, construction job of that magnitude would require a helluva lot more manpower than the Imperial army had to offer. I'll bet there were independent contractors working on that thing: plumbers, aluminum siders, roofers.

>Dante: Not just Imperials, is what you're getting at?

>Randal: Exactly. In order to get it built quickly and quietly they'd hire anybody who could do the job. Do you think the average storm trooper knows how to install a toilet main? All they know is killing and white uniforms.

>Dante: All right, so they bring in independent contractors. Why are you so upset with its destruction?

>Randal: All those innocent contractors hired to do a job were killed! Casualties of a war they had nothing to do with. All right, look, you're a roofer, and some juicy government contract comes your way; you got the wife and kids and the two-story in suburbia - this is a government contract, which means all sorts of benefits. All of a sudden these left-wing militants blast you with lasers and wipe out everyone within a three-mile radius. You didn't ask for that. You have no personal politics. You're just trying to scrape out a living.

I mean there's an interesting question here because if (hypothetically) someone takes what is in their mind a moral high road and says "I will never hire an ex-NSA coder or sysadmin" then hopefully that person doesn't work for Facebook or Google, because really the difference isn't a matter of kind as much as degree, malware notwithstanding.

Some very cool stuff.... useful from both a practical and anthropological standpoint.

Reply on news.ycombinator.com