We've got thousands of pages of contracts with Booz Allen: Help us analyze | Hacker News Reader

We've got thousands of pages of contracts with Booz Allen: Help us analyze

muckrock.com

105 pointsmorisy27 comments

People get excited about things like this and say things like "this is REAL investigative journalism". Or "I wish mainstream journalists would start doing this".

Look, I read documents like this all day long. So do investigative journalists. It's just that very few people care about the stuff we find until it turns into an international spy thriller.

Just because you don't care about it on the state/local level doesn't mean we don't put in the work.

Edit: You can look at some of the great investigative journalism you're probably not paying attention to here: http://www.ire.org/blog/extra-extra/

"I wish mainstream journalists would start doing this".

Spin it however you want, fewer and fewer mainstream journalists are able to focus on this. I work with a dozen or so newsrooms a year on FOIA, CAR, and other investigative reporting training, and in most newsrooms only 1 or 2 hands go up when I ask who has filed a public records request in the past year.

MuckRock was built to change that: To make it easier to add document-requesting and analysis to the daily news cycle, to help non-journalists get interested, excited, and able to request and share documents, and to enable large scale-FOIA projects like request BAH files from dozens of agencies.

And we've found that lots of people do care about it at the state/local level, but news organizations are a) terrible about marketing their work. b) assume readers aren't interested in the nuts and bolts. c) let that work die in obscurity, content to try to win awards with it rather than to truly engage the public.

So yes, this is a flashy dump of documents, but we've helped hundreds of others do meaningful, smaller scale stories, and whatever way you cut it, I don't think there's enough groups doing that.

I also keep my own list of interesting, public-records driven stories:

http://pinboard.in/u:morisy/t:to_foi/

Encourage anyone who is interested to reverse engineer them and try those requests in their city, state, and country.

Don't get me wrong, I love Muckrock! Wasn't trying to say you guys aren't necessary. I file FOI requests for a living, and I'd love to be in touch with you guys about a number of things - best practices for turning government records into structured data, the importance of reforming records retention on the government side to reduce the burden on public records officers, fixing FOI on the local level, etc.

Feel free to get in touch with me on Twitter: twitter.com/patdennis

I'm not sure what your point is, is someone saying that you're lazy?

I was just trying to preempt the inevitable "investigative journalism is dead" comments.

Judging from the comments so far, it may not be so inevitable. However, there is still the law of averages, I guess.

This is the kind of stuff I wish Wikileaks had stuck to -- mostly or completely legal, beneficial to national security, etc. They are still challenging powerful entities and supporting change, but without becoming the story themselves.

Likewise. I really liked WikiLeaks at the beginning but it became clear after some time that there was a definite agenda behind their actions. The impression I've had for years is that they will oppose the U.S. for no better reason than that it's the U.S. Which is, if you think about it, just like opposing any maligned group for what they are instead of what they do.

In other words if we start from the assumption that the U.S. should be a "bastion of the free world" then it would make sense to tirelessly note instances where the U.S., corporations, states, etc. fall from that ideal.

But diplomatic cables describing what kind of women Berlusconi likes, while interesting, are hardly indicators of the collapse of a free society. Those types of things were released because they were embarrassing, nothing more. And while I'm certainly glad that Assange is able to stroke his ego by thumbing his nose in Uncle Sam's face, I don't actually hate my country; I want it to be better.

I have to strongly disagree. Wikileaks has released minutiae about other countries as well. It isn't only about showing up hypocrisy from the US, it is about lifting the veil of secrecy that all governments misuse whether it is hiding big stories or trivial ones.

Saying that wikileaks has an agenda to oppose the US is basically confirmation bias. Leaks about the US get news coverage in the US. Just for starters, look at the 2 million emails from Syria - clearly most of those are about trivial matters too, but most people in the US haven't even heard about the release at all.

> I have to strongly disagree. Wikileaks has released minutiae about other countries as well.

Doing bad things to other countries (again, because of what they are instead of what they've done) doesn't make me feel better about when it's done to mine.

Here's some other countries I don't hate: Russia, China, the U.K., France, South Africa and in fact the list could go on as long as I have time.

> Just for starters, look at the 2 million emails from Syria - clearly most of those are about trivial matters too, but most people in the US haven't even heard about the release at all.

There's no polite way for me to say this so I'll just spit it out; no one cares about Syrian government emails. That's why no one in the U.S has heard about the release. Gossip is as destructive for international politics as it is for small towns.

If there's big news to release then by all means go and do so, but governments should be able to maintain diaries too.

I'm having a hard time understanding your point of view, it seems to have no unifying principle. But if I take your last sentence as what you really believe then the obvious response is that once governments can be relied on to not abuse their ability to keep secrets then they will have earned the privilege to maintain their diaries in secret.

And anarcho-libertarian types will claim that a government can never be relied on to do that, which is just as well as saying again that the government could not be allowed to keep secrets.

Since I'm not an anarcho-libertarian type and you clearly are not, I don't think what they might hypothetically say matters all that much.

While I agree with you, regardless of how you feel about the US, it's undeniable that the publication of the war logs/cables essentially ramped up opposition to wikileaks and, combined with internal strife, seriously impaired the ability of the organization to continue releasing new stuff. So it's cool that other groups are now taking over that mission.

who made the "press" shift its attention on wikileaks rather than on the docs they leaked ? your favorite DC spin doctors. So please , dont blame the messenger , or you are as stupid as the government think you are. And they do think you are stupid , trust me.

Releasing 250k classified documents is a news story itself, so it was inevitable that the story would shift to wikileaks. (shifting to Julian Assange was some DC spin, though, but he sure didn't help things.)

But they didn't release the cables at first, they released an encrypted file. Then a journalist who didn't like wikileaks that had access to the file published the password in a book. Since the files were then available to those with resources to know about they password, they felt it was reasonable to release them completely so people could see if they were mentioned in the documents which were at that point for all intents and purposes public.

A good way to search through the text of the documents is DocumentCloud, which also offers some analysis tools:

https://www.documentcloud.org/search/group:muckrock

Modify the search terms as desired (i.e., "Booz")

Is there already some decent crowd sourcing text analysis software out there?

Some simplified version of Nvivo with a wiki-like user management would do wonders for something like that.

1. Button "I commit to analyze 2 / 12 /22 pages". 2. You get a very short tutorial on how to do it. 3. User is assigned relevant pages, starts tagging the documents for specific content and highlights + comments lines worth a closer look.

I would be definitely up for that. With enough traction one could just compare inter-coder reliability and establishing a high quality analyzing process (i.e. getting the same document analyzed by multiple users).

How about django-public-project? The following is from their description at http://django-public-project.readthedocs.org/en/latest/:

Django Public Project (DPP) is a specialised content management system written in Python/Django for building an information website around big projects. It is open source (BSD licence) and can be used by civic groups to monitor the progress of publicly funded projects but also from within governmental institutions itself to bring transparency in the proceeding of a project, inform the interested public about what is going on and thus enhance the acceptance of the project by the different stakeholders.

It is currently being used by the investigative comittee of the Berlin city parliament in order to investigate the massive clusterfk that is the city's new and very disfuntional airport (BER). The source code can be found at: https://github.com/holgerd77/django-public-project

Regarding participation of ordinary citizens, the docs say:

On DPP websites users can comment on nearly everything and connect their comments to various elements of the project. Comments are then shown alongside every connected element. All admin users having activated the “Receive new comment emails” in the admin user settings receive an email after someone sent a comment and can then activate the comment to be shown on the website.

http://django-public-project.readthedocs.org/en/latest/users...

If anyone wants to search through MuckRock's recent documents via the DocumentCloud interface, they're also available over here: http://www.documentcloud.org/public/search/group:muckrock

Edit: Whoops -- I see that Michael's already commented along the same lines.

To expand a bit, you'll be able to do boolean searches among the documents, scan through excerpts around your search query, look at extracted entities (people, places, company names), and plot any dates mentioned in the documents on a timeline.

Now this is investigative journalism!

http://taxanalysts.com/www/features.nsf/Articles/557540D8F53...

I think this might provide some interesting background.

What exactly is the point of this exercise? Booz Allen is a big consulting company. They have lots of contracts and SOWs, most of which are for really lame things. You are not going to find a contract that says: "provide 3 big brother spies to read everyone's mail".

Of you're trying to prove that that government spends lots of money for consultants... Well, we all know that already.

Also, unless you really know what you are talking about, you are not going to be able to spot most of the petty, obvious or really offensive contracting practices. The Feds have their own language re: procurement.

Based on the ones I've seen in other contexts, the key is to be able to aggregate. If you do, you can often find really interesting stuff from unclassified contracts around the periphery, then presumably FOIA more, but knowing how the system works helps a lot.

A while back there was a request posted to fbo.gov for an online sockpuppet management system. It's possible that some important details can be gleaned from these contracts as well.

They should be using plagiarism-detection software to strain out the boilerplate commonalities and focus on the differences (as well as a single copy of the commonalities).

Reply on news.ycombinator.com