Open data is not a panacea
mathbabe.org
mathbabe.org
If I was to point out a problem with open data, it's that it has the potential to break our social institutions by facilitating aggressive min/maxing behavior on a large scale. It's not that Goldman Sachs is going to use the data against the general public -- they're going to do that whether the data is public or private -- but you make it public and then the middle class can use it against the poor (and then the poor are going to revolt).
For example, suppose we create a good, easily accessible map of places to live and put down all the information someone in the housing market would like to know: Home prices, tax rates, quality of the schools, etc. Now home buyers can min/max those variables when making a purchasing decision. In general that's good -- when we buy a home, having that information benefits us. We can find a place that has extremely low taxes when we don't have kids in school, and choose between the places with excellent schools to find the one with the lowest taxes when we do.
But then local governments react by trying to find a niche. They let their schools go straight to hell in order to attract childless professionals through low tax rates. Or they zone all the low cost housing out of the school district so that they can maintain good schools at a low mill rate because everyone who can afford to live there is affluent. What they most certainly don't do is provide good schools or other government services to poor people, because that has a deleterious effect on the value for money that middle class people are looking for when they're shopping for a tax jurisdiction, and when they do that they lose their tax base and thereby their ability to continue doing it.
Of course, such things happened before public data, but you make the data more widely available and you make the behavior more common.
And before the libertarians get too happy about this state of affairs, recall that the de facto default defense against aggressive competition is collusion. Which we [pretend to] ban in the private sector, but between governments they just call it centralization and "federal standards" and impose it on you from the top down the second they can break a filibuster. Which is, of course, miserable and inefficient because it destroys local control and creates a systemic failure if they get it wrong.
Which isn't to say that the data availability is net negative. The incentive it provides is for governments and corporations to be efficient, and efficiency is usually (indeed almost always) beneficial. But the data lets the user of the data decide what to optimize for. So the problem comes when the thing we choose to optimize for as individuals is not the thing we would choose to optimize for as a society.
>The Goldman Sachs’s of the world will always know how to make use of “freely available to everyone” data before the average guy.
Because the Goldman Sachs's of the world have the data. They buy it from proprietary databases or hire underlings to look it up in dead tree format in the government records office behind three filing fees and six miles of red tape. If there is a buck to be made doing stuff like that, you will find the financiers making that buck.
Putting the data online may help them a little, because maybe now some things that weren't cost effective become cost effective, or the money they once paid to clerks and bureaucracies can now go to marble and champagne. But it helps other people more, because even though it doesn't make the playing field totally flat, it makes it more level than it was before.
But there is an alternate future. With open data (and increasingly cheap computing and communication devices), community organizers and the poor themselves can see exactly how they're being screwed over by the government. It will no longer be a nameless, faceless they that's inflicting pain and misery on them, but instead, specific actions, undertaken by specific individuals in specific offices. Corporations and governments won't be able to hide behind their usual practice of security by obscurity, since, with open data, it'll be possible for any citizen with a modicum of spare time and spare computing resources to run data analysis and come up with possibilites for why things are as screwed up as they are. We're already seeing this, with tech related laws and regulations. Open data makes it easy for any individual with an internet connection to track the progress of legislation. This was a major factor in galvanizing opposition to SOPA and PIPA. Making more data sets open and freely available will, in my estimation, benefit individuals far more than corporations or governments. Open data will do to closed-door decision making what full disclosure did to 0-day vulnerabilities.
You're assuming that there are actually specific fat cats smoking cigars in back rooms planning all of this out. That pretty much never happens.
What actually happens is that different tax jurisdictions have different laws and people self-select into the jurisdiction that provides them personally with the best value in government services for their tax money. That means the affluent taxpayers go to the jurisdictions that take only a modest tax bite and use it to provide only services that the affluent consume. The jurisdictions that don't do that are then forced to cut services after their tax base is eroded by so many of the high income taxpayers leaving.
The local officials in those places are not sitting in their offices trying to think up ways to screw over poor people, they're just trying to figure out how to keep the lights on. They have to do something to attract and retain more of the taxpayers who pay as much or more in taxes as they consume in government services or they fall into the death spiral of service cuts and tax increases that induce more emigration and capital flight which erodes the tax base and requires more spending cuts and tax hikes.
Open data helps leveling the ground. It may be used in pernicious ways, but it is true of any progress we've ever embraced. The wrongful use of data you depicted probably already happens. Transparency and openness usually mean these cases get more widely decried and ultimately regulated.
If you take from the middle class to give to the poor and the next city over doesn't, those middle class people vote with their feet, and then you no longer have that money to spend on anything, and your remaining constituents are upset with you when the drop in revenue necessitates a drop in spending.
Which is ostensibly what the majority in that city wanted, because they voted for officials who set things up that way. And then once that happens, the people who want things that way (i.e. the middle class) from other cities move to that city, and the people who don't (the poor) leave for some other place, which bolsters popular support in that city for things remaining the way they are.
I do think it's important to draw attention to the potential downsides, so thanks for contributing this article.
Given the upside of open data I can't imagine a scenario where closed data > open data with respect to the general public.
If you have to be a techie to get value from this data then there must be a market for easier to use data analysis tools for non-technical people, which I'm sure the market will fill. (Checkout my profile page, I'm building one right now).
making data available to everyone for free is the better way.
1. If i got the data i can make a list of strange companies and put this information out to the Internet.
2. How says that this Information is correct? you can scan my facebook profile what you will find: that i live in Australia but i dont. Yes my Ip give it away where i live but i can use a proxy. I like this and i like that but do i Really?
3. School/University Rankings are full of facts that are wrong and dont cover the reality.
Sry but if you only see data in the aspect to analyze it you could miss the beauty of the data. MP3 Files are just data but what makes them special? the noise we get out of the speaker.
So lets not think how we can abuse the data. Lets think about how with this data we can make a better world FOR EVERYONE !
I use data, and I benefit from access to data. But my use of data doesn't make someone else worse off, and others' use of data doesn't generally make me worse off.
If they have faster computers than me, that's great.
Equating research and analysis to war sounds silly, and I think it's a really unhealthy worldview.