The Socialist Origins of Big Data (2014)

Big Data is getting better at giving us lots of odd correlations -- people buy poptarts before a hurricane. Uber knows where people are going to go. This information is extremely useful.

But what we're missing here is this central fact: correlation data is only good inside a very limited set of preconditions. Once you have a WalMart or an Uber, it helps them operate better. It does not have the ability to create the next WalMart or Uber.

That means that Big Data, as it is now, will always be able to continue to optimize within a limited system, but will not be able to see outside of that system. Big Data will not be able to create the next paradigm-changing thing like Uber, because paradigm-changing things are by definition outside the scope of the data already collected.

Recently there was a a study published by some MIT students about startups. It ran a bunch of numbers and gave you advice: pick a small name, operate outside the valley, use older workers, and so on. But as somebody pointed out, you really need to have a great idea, spot-on execution, and market traction. If you have that, the rest of it doesn't matter. More to the point, if you have all of those things the MIT guys came up with, they're not going to give you the other things you need. As it turns out, the things you need for a great startup are still fiercely debated -- hence the MIT study in the first place. Correlation does not equal causation.

I see this in big companies all of the time. Our projects are running, on average, 100% late! So somebody looks at the data and finds that most of the time is spent testing. What do we need? Better testers, of course!

Simply because you can point to a couple of different things that track together does not mean you understand anything. And in IT, unlike Big Data, we do have to worry about causation, because we're inventing the universe every time we ship.

You always optimize complex systems from the bottom-up, never from the top-down. Otherwise you're just fooling yourself in various interesting ways. That's true no matter what the system is.

In spite of being a free market supporter, I think that command economies have been useful at times, e.g. the PRC and the USSR in their early stages. Both these countries practiced something similar to what Chile tried, in the 70's and early 80's. My theory is that before communism these countries were practicing a a very corrupt state capitalism. So a logical step before capitalism, was to rationalize state capitalism by making government decisions based on utilitarian goals instead of corruption.

EDIT: modified my request for title change after I read the original title more carefully.

Eventually, in the midst of the well-wishers there will come those souls who are not good actors, and they do seem to seem to spoil the bunch sometimes - don't they?

Given enough power, anything is worthy of unusual inspection. Similarly, the current success of Uber is no different. Regardless of whether the social, political and market narratives about Uber are currently accurate, fair or not, it is the sudden nature of company power and the driving essential questions around the nature of new economies that compel articles like these. Uber and AirBnB are challenging fundamental labor and property rights shibboleths, and demanding real consideration from people across the philosophical, commercial, and political realms.

Similarly, the 20th century spread of dictatorial socialism was incredible. No matter how much I try to sympathize with the promises of various ideologies, I always return to the premise that in all circumstances, it is the CONCENTRATION OF POWER that is the problem. It is even more disturbing for pragmatists and empiricists when they see something moving fast that is denying anyone the chance to scrutinize, test and debate the merits of the method, the motives, the means.

As a good friend says though - it is easy to make these conceptions "heuristics" and see someone's intent as globally bad. Some people's intentions can be good and means and methodology sound. Yet with data as with literacy at large, the medium by which the person't intent is made manifest is the very means by which the power will be exercised and the rules challenged. In this case, massive volumes of data are being generated that have multi-dimensional consequences for all humankind. This IS a concentration of power. Even though a witch hunt is far from a beautiful thing, the world citizen has very real reason to be concerned, and truly should be immediately concerned, about the very topics that are daily discussion here on HN. Matters of technology, privacy, rights, commerce and liberty are not weak, and neither are musings on the mating habits of whales.. from time to time.

HN is a fount of a certain kind of literacy. How does that literacy get expanded to all, and seen as a human right?

The thing is, how is it that we seemed to manage just fine without all those business analytics in the past? Somehow people were able to exercise their judgement and run the department effectively. Not that there's no room for improvement of course, but maybe you'd be better skipping straight to diagnosis/solutions rather than spending time logging and collecting indicators.