Using wget and grep to explore inconveniently organized federal data | Hacker News Reader

Using wget and grep to explore inconveniently organized federal data

gist.github.com

55 pointsdanso20 comments

If you are not going to learn AWK, R, Python or Julia the next best thing is Open Refine.

> OpenRefine (formerly Google Refine) is a powerful tool for working with messy data: cleaning it; transforming it from one format into another; extending it with web services; and linking it to databases like Freebase. Please note that since October 2nd, 2012, Google is not actively supporting this project, which has now been rebranded to OpenRefine. Project development, documentation and promotion is now fully supported by volunteers. Find out more about the history of OpenRefine and how you can help the community.

TXR: http://nongnu.org/txr

This reminds me of Manning's case.

"That Manning was convicted of computer fraud seems to suggest that using wget on a U.S. government computer to download large numbers of files can be considered the digital equivalent of trespassing – even if it's on turf you're otherwise allowed to access."

http://www.washingtonpost.com/news/worldviews/wp/2013/07/30/...

Well, sort of. In the Manning case, the government argued that wget was unauthorized on government-owned client machines. Their argument in that case wouldn't apply to the public's use of wget to send requests to government-owned servers.

Then we need a scan of all government machines to see if there was a precedent of it being installed across many thousands of machines to suggest that this was not an actual individual breach of policy. and show that the government was remiss in its ability to enforce the policy at all.

Government information security and technology people, in my experience, are not necessarily forgiving or understanding in any way. And I mean "understanding" in both senses of that word. They neither care to understand your needs nor necessarily understand much about technology. I see so much extra work done just because the security person in charge of writing a procedure had no understanding of what the phrase "volatile memory" means. And that's just one instance from my work; I wish I could share more but it all makes me sad.

All this is a very long way of saying that some of these people may not have even understood what wget is used for, or believed it was unauthorized for use even though it is a standard tool.

ProPublica has some great material on techniques for data journalism, including scraping and transforming unfriendly data formatting:

https://www.propublica.org/nerds

The URL is nerds??? I look at the page and there was nothing on there that seems to show it as a acronym either. Just the title of the blog. Not sure I like that.

It's what their News Apps team calls themselves. https://twitter.com/propubnerds

I'm skeptical of a complaint like this from someone who voluntarily chose the handle "baldfat".

Bald and Fat is what I am. 6" 1" 230 pounds and I am bald like Cpt Picard since I was 25.

Nerd has a negative connotation as opposed to Geek. I know seems weird but getting called nerd was very negative before Geek/Nerd was cool.

Definition: 1. a stupid, irritating, ineffectual, or unattractive person. 2. an intelligent but single-minded person obsessed with a nonsocial hobby or pursuit: a computer nerd. [1] http://dictionary.reference.com/browse/nerd?r=75&src=ref&ch=...

I was looking for these articles when I posted earlier, very nice tutorials for EG that data you want that is only presented as a Flash applet:

https://www.propublica.org/nerds/item/doc-dollars-guides-col...

One interesting thing the author mentions -- interesting despite being somewhat irrelevant to the topic -- is the astonishing fact that the vast majority of people don't know how to use Ctrl+F.

I submitted it here: https://news.ycombinator.com/item?id=10321439

Its so sad there isn't a site with everything in plain sensible formats. data.gov seems like a good start. But a plain text UI (like gopher or ftp) seems better. A subfolder for each state. And a subfolder for each department. Maybe something like Linux's /proc for laws and federal data.

This feels like a data set that might already be on the amazing https://enigma.io

Enigma's secure URL isn't working. The working address is http://enigma.io

That's disappointing. I actually didn't even check that they used HTTPS, I figured it was safe to assume they do. Guess not.

Enigma is a great service and they might have a list of the Section 333 exemptions but even with the sophistication of their discovery algorithms, I'd be amazed if they had this as structured data...mostly because right now, it exists officially as a list of PDFs with some basic information, e.g. operator name and stated mission. Extracting location of operator probably wouldn't be too hard...detect if PDF follows standard letter format and then run address extraction algorithm on the first (or 5) last lines...

Obviously, someone from enigma.io could give some manual direction (such as deciding that city/state are actually important data points to extract), but it probably wouldn't be as detailed as dataset as what The Verge and Bard College have manually curated [1]

[1] https://github.com/voxmedia/data-projects/tree/master/verge-...

Anyone know of a quick trick to extract the English text from html files?

Try:

  lynx -nolist -dump

-dump dumps the formatted output of the default document or those specified on the command line to standard output.

-nolist disable the link list feature in dumps.