Why shouldn't these records be available for free, even better yet accessible thru an API? Would the 'bad apples' still be able to operate in any area of government if there was true transparency?
1. A lot of the paperwork is all over the place. It has to be gathered, which takes time.
2. Same is true for the digital info, all over multiple databases. It takes a programmer or few to focus on pulling it (and getting all the necessary permission to do so).
3. Everything, once compiled, has to be reviewed to ensure it is on target and that any necessary info has to be redacted. The problem is that no one wants to take responsibility for a massive data dump without checking ever single word/number. This is because, while column X or form Y should only have publically available information on it, the people who processed the forms or input the data may have put information that would need to be redacted (SSNs appear in all sorts of places where they shouldn't). So the programmers will not give any sort of guarantee for the data pulls and the people pulling the physical files won't give any sort of similar guarantee for the physical files. This means the FOIA office (or what ever they are called there) has to do a line by line, word by word, number by number redaction.
4. To make this worse, many times the staff is not fully trained and has outdated equipment. FOIA has to print out files, use a black marker on both sides of the paper, and then scan it back in and print out again (and this is per policy, so the worker can be disciplined if they do anything else, even if it is more efficient). The programmers often are tasked to go through largely undocumented decade old (or older) systems.
5. Even without the political nature of the work, so even for a completely benign info request, the work is often time of low priority and for most of the staff involved it is on top of their normal day duties. Often times the staff is already underpaid and overworked, and with the info request being of low priority, it gets largely pushed aside.
It would be ideal to have it all accessible to an API, but having this done would take multiple miracles that would be better spent on obtaining world peace and ending world hunger.
In other words, the risks avoided by paying the best heuristic computer known to man -- a human -- far outweighs the cost of the resource.
> COP considered dumping the raw data online for anyone to view, but decided against it because it would involve the publication of sensitive personal details—things like mental illness, drug addiction, or cases involving minors.
My personal thoughts: I'd venture to say that all info would eventually get tossed into the "sensitive field" as a matter of a slippery slope. I also think that there's a conflict of interest in keeping notes that are "bad" over "good" (easily retrievable, shareable, and legible). It's self-serving to have physical paper notes. They can easily be "lost", "destroyed", and they only exist in a physical form: an oversight official has to come to the physical precinct house to retrieve the records.
Last time I was in a county office, the application they used was extremely dated. It looked well over 15 years old. Many governments work with really dated software with a skeleton IT crew that can barely keep even that working, much less rewrite it.
Because its easier to cover up non-digital records. Just like sometimes the boss wants to discuss things in person or over the phone rather than in storable text format.
Something no one is discussing about this supposed treasure trove of SS numbers:
1) once its out there, its out there, its like pretending your email address is secret after the spammers are already using it. And after 50 zillion examples of incompetent security, all our numbers are public to absolutely everyone except the rawest of general public (like my neighbor). Anyone I wouldn't want to have access to it, already has it.
2) as per above, they paid $22500 or whatever (hopefully prorated with the low rate or providing the docs) because someone from the city is supposed to be cleaning that data out of the file already. Otherwise that shady guy who owns the money laundering err I mean upstanding and ethical payday loan biz down the street would have already purchased a list of police reports, and done his thing with the SS numbers contained.
3) the easy and cheap part is editing out 123-45-6789 regex numbers. The hard part is in the rare situation of judicial involvement (those people rarely do their job, as per the article's claims) there might be a gag order as part of an out of court settlement or similar.
As an aside, the software used for EMR on the Android tablets is... a horrific abomination.