Hacker News Reader

Comment by stevage | Hacker News Reader

Argh. I used to work a lot in open data, including a stint within the open data department of a major city government.

This kind of behaviour does nothing to advance the cause. It just perpetuates the belief that open data and FOI are massive waste of time and resources and open legal risks for no good reason.

I'm also pretty surprised that people think email metadata from government is legitimate open data. Do you think it should be public how many times you have emailed government and gotten a response, and to which departments? I don't.

View on HN

> I'm also pretty surprised that people think email metadata from government is legitimate open data. Do you think it should be public how many times you have emailed government and gotten a response, and to which departments? I don't.

on the surface i agree with you. beneath the surface, other governmental units already possess all the metadata for your emails/phone calls, it would be a weird bit of asymmetry to say it’s okay for them to have your metadata but not for you to have their metadata.

it’s not a perfect mirror, for obvious reasons, but if OP went on to show that this metadata is powerful and push that neither side should be collecting that type of metadata, then that’s a single legitimate use i can think of for this kind of FOIA.

>it would be a weird bit of asymmetry to say it’s okay for them to have your metadata but not for you to have their metadata.

But that's not what this is. This is saying it's ok for them to give your metadata out to anyone who asks.

None of this is open data. Open data is data that's released on a discretionary basis; often times a FOIA request for the same data found within open data has a legal requirement for redaction. We see this happen all the time. What's worse is that at the executive level, eg chief data officers are often forbidden by legal teams from interacting with the media and adjacent orgs. I've literally been told by a CDO that they cannot speak with me.

My go-to phrase for open data is, "open data is a lie". Because at the end of the day, we have zero legal recourse to validate that what is provided through open datasets is complete, both in terms of available columns and available rows. Very rarely will it ever be explained why, or even if, information is missing. And the subsequent effects of that lead to a deep public misunderstanding of what's going on. And much of it is intentional out of explicit fear that the public will misinterpret the data. So we go through FOIA instead and we end up going through legal battles. There's a reason I've had to do ~10 FOIA suits.

tl;dr: open data is nice, but its lack of rigor and accountability makes it effectively useless for anything that requires depth.

Perhaps you missed the bit where I said I worked in open data for years, including inside government.

They are two very different processes, yes. Open data tends to connect you with technologists and subject matter experts who want to talk about their field. FOI tends to connect you with lawyers and governance gurus who want to find reasons not to give you what you want.

Around here the FOI processes are getting slower and less forthcoming, in defiance of the law, and ridiculous requests like these ones do not help make the case for their support.

I've spoken to many people inside government who work on the open data side. To be blunt: the signal to noise ratio isn't very good in terms of whether someone actually knows what they're talking about. So please consider my reluctance to take much of what is said as a product of being lied to for years, both intentionally and unintentionally.

Yes, FOI processes are getting slower and yes we're going to keep suing. What other choice do we have? One thing to consider is that SO MUCH of that is a product of gov agencies simply not preparing their documents in a way that makes them FOIAable. So much can be done that simply isn't -- a significant amount of which is very intentional (eg, police records) in an intentional violation of the law.

We're not trying to be dicks in all of this. We just want to know what's going on. So yours (and many others in your position)'s clear desire to think of us as adversaries rather than simply people who want to know what the fuck is going on is blinding your vision.

Get out of your head as a person in government and start thinking as a person who's at least empathetic towards the frustrations of a tired public that constantly lied to, deceived, and fed communications that are perpetually weighed down by legally reviewed PR releases. We can do better, but sure as hell not with your defeated attitude.

One thing to consider is that 'government' literally constantly advertises the opportunity for you to get credentials to their system and prepare documents themselves. They will pay you to do this.

> So yours (and many others in your position)'s clear desire to think of us as adversaries rather than simply people who want to know what the fuck is going on is blinding your vision.

Yea, I don't work in his position or any government position, but you are being adversarial. And, whatever you're trying to do, you're being a dick in at least this comment.

Maybe. But again, please try to view it as an expression of years and years of frustration. The stakes are high with this shit. When we are denied access to records, the implications are often as serious as literal deaths, sexual violence, and systemic abuse of millions of people.

I highly recommend you read this (shameless plug), which shows just a sliver of the nonsense that we have to deal with: https://southsideweekly.com/cpd-routinely-denies-foia-reques...

I am familiar with the system and many serious problems with it. Some people who want to fix things about government go work there. You are complaining that they think of you as an adversary, but it's clear that you are thinking of government employees as adversaries, who have been frustrating you for years, and at least sometimes you're clearly being a dick about it.

You're attributing way more malice than I think is warranted. Take a breath, friend. I've been largely calm and collected in this conversation, but you've (ironically) raised this to a higher adversarial point. I have no ill will towards you.

Funny enough, I tried to get a gov job recently. A chief data officer job in a niche group which I think I'd do a damn good job at. They denied me immediately because I don't have a degree, despite my qualifications. So yeah, I actually agree to a point that joining gov can be a way to 'fix things'... but, welp?

But even further, I have tried working with gov agencies on issues I've felt deep anguish towards. In late 2019 I did a significant amount of free data analysis of Chicago's parking tickets for the Chicago mayor's office to show how excessive ticketing can be. They reached out to me, they said they loved the work, but did shit-all with it. Later that year, the mayor ended up doing a 180 on the desire to have more-just ticketing policies. A month after I gave them that analysis, in a FOIA lawsuit trial, the city's counsel threw me under the bus by arguing that my end goal in that litigation is to gain access to their data so that I can modify it. Completely baseless argument that actively ignored that I put significant effort in trying to help with the highest office of the city. For free. So dude? I've fucking tried. In the end the city acted (and acts!) like a bully for the sole purpose of winning their cases. When that's the standard of behavior we receive, it's a damn wonder we even try to make any intentional effort to be kind.

Also, when I've been told over the phone and email countless times, "we are not obligated to answer questions under the law"... even for the most basic of things.. it's pretty clear where the disconnect is.

I could go on with these stories, but since it really sounds like you didn't read the article I shared -- please do, and you might better see where many of us are coming from.

In Sweden, when you contact the government by web form or mail, they warn you that every communication is by law part of the public record (metadata and contents), and you should not include anything sensitive that you do not want divulged.

Reply on news.ycombinator.com