Hacker News Reader

Comment by chaps | Hacker News Reader

Just be VERY careful when using any of these open data sets. Datasets will undoubtedly be filtered by some unwritten criteria. It's best to use these datasets as a sanity check for some things, with follow-up FOIA requests for the serious work. And even that will be filled with similar filtering. Source: spent the last ten years working with open data and FOIA in Chicago.

Also, here's a tweet-fitting one liner I wrote that downloads all Socrata CSV datasets across all gov instances:

  mkfifo f;echo a>f&xargs -E, -a<(cat f) -I{} bash -c "wget http://api.us.socrata.com/api/catalog/v1?scroll_id={} -O-|jq -r '.results[]|.resource.id+"\","\"+.metadata.domain+"\"/api/views/\""+.resource.id+"\"/rows.csv\""'|tee <(cut -d, -f2|xargs wget -P{} --content-disposition)|tail -1"|tee f

View on HN

It's not just the filtering. It is also that many datasets are full of errors. I wrote a data analytics tool that I would test out on a bunch of open data sets like this one. One of my favorites was the Chicago crime data. The tool could quickly find (and fix) a bunch of errors. I tried contacting the people who published the crime data and even offered to show them how to use my free tool to clean up the data. They could not have cared less that their data set had large numbers of errors in it.

interesting! what sort of errors did you find and were there any sort of take aways your had other than lack of care by the data provider?

Each crime report would specify (among other things) what crime was reported and what city block it had occurred. Using my tool, it was easy to analyze specific crimes over time for specific areas. One of the problems was their database would have multiple spellings for the same location which made it much harder to do the analysis since you had to account for that.

This is a common problem, especially for "crime" datasets and very emblematic of the kind of data capturing processes for each reported "crime" (really, they're some of the worst datasets because of the extent of imperfect information and incentives to capture some "crimes" some vs others). Send it through a geocoder like geocodio to resolve most of that problem. It won't be perfect.

This strikes me as a Wikipedia Article type caveat. Sure the warning applies to certain categories of article/data, namely the ones where people are jockeying to have the authority of an encyclopaedia / data to reinforce their political stance, but on any sort of non political topic neutrality is a non issue. I fully anticipate that say, police and crime data by demographic shouldn't be trusted. Other data, like say pounds of recycled garbage collected per census tract, is probably fine.

I mean, sure? I'm very much saying this for police datasets, where for example, a "crime" database might not include homicide information.

And no, I don't think it's fair to say that the amount of pounds of recycled garbage collected per census tract would be remotely accurate. Too many private companies with private incentives. Just a taste, for example: https://www.wbez.org/stories/bga-investigation-reveals-major...

What FOIA request methodologies would you recommend to ensure you’re getting the most comprehensive data possible out of an agency or government body response?

One of the most productive things I've seen is to request for "data sufficient to show" rather than describing the data you're looking for. That way you have a bit more wiggle room in the space of interpretation. Another way is to just look up forms ("filetype:pdf site:yourmunicipality.gov" in google works wonders) that are online -- if filled out reports have text filled in by a computer, then that information was likely filled into a database, and the form you're looking at is a product of a multitude of SQL queries. In Illinois, Hites v. Waubonese compels an agency to run a SQL query if data is represented in a database, so being explicit about the existence of that law can be helpful. I'm sure similar can be said about other states. Also, be sure to explicitly request your data in an excel compatible format (some states legally require it).

Also, have a look at the Reporters Committee for the Freedom of Press's FOIA guides per-state. https://www.rcfp.org/open-government-guide/

Incredibly helpful, thanks for the reply. These tips will make their way into my requests immediately.

Anytime!

Just curious: for a municipality with limited resources and very minimal understanding of data governance, why should a city government follow in the footsteps of Boulder?

Do the missteps of Boulder make the case for other municipalities to not pursue open data in a similar way, given FOIA is sufficient?

Reply on news.ycombinator.com