Project Cybersyn and the origins of the Big Data nation | Hacker News Reader

Project Cybersyn and the origins of the Big Data nation

newyorker.com

44 pointsblackbagboys13 comments

If this interests you, you might like the book "Red Plenty".

It alternates between novel and essay about Khrushchev-era USSR, when people on both sides of the curtain thought there was a pretty good chance that the Soviets would succeed, if not as a Socialist Utopia at least in economically overtaking the west in output and quality of life.

Also, this excellent essay about the computational aspects of Gosplan, the USSR's mostly human-powered predecessor to Cybersyn: http://crookedtimber.org/2012/05/30/in-soviet-union-optimiza... (which is where I learned about Red Plenty)

Also, see Cosma Shalizi's excellent review of "Red Plenty", which includes a mini-introduction on the application of linear programming to economic policymaking.

http://crookedtimber.org/2012/05/30/in-soviet-union-optimiza...

About 10 years ago I intended to read "Towards a new Socialism", after reading an article about the chileniean experiment that also mentioned this book. Perhaps it was actually an article about the book. I don't quite remember. I never read the book and only got reminded about it by this article now. The topic itself is quite fascinating, especially if you think about what amount of planning happens in huge companies.

You can, if you are interested in the topic, find the book at http://ricardo.ecn.wfu.edu/~cottrell/socialism_book/

As they say - the bigger the company, the more socialism there (inside the company).

This is relevant: http://en.wikipedia.org/wiki/Leonid_Kantorovich

Although not dealing so much with big data, it was still a "socialist" attempt at mathematical optimization being applied in society.

Leontief's input-output model is also interesting: http://en.wikipedia.org/wiki/Input–output_model

Related and a very good BBC documentary on the role (and tragedy) of engineering and planning in the USSR: https://www.youtube.com/watch?v=h3gwyHNo7MI

... and then one day someone read a paper about chaos theory and computational complexity and quickly realized that predicting systems with chaotic behavior is guaranteed to be impossible.

It's not impossible at all. In fact there is entire field of study devoted to the long term evolution of dynamical systems (http://en.wikipedia.org/wiki/Ergodic_theory).

And within a chaotic system there are several types of stability that can be readily predicted over a long period of time such as Lyapunov stability and asymptotic stability.

Why do you think this system would be chaotic in the first place?

Chaotic in the sense of sensitive dependence upon initial conditions? Because human actions depend upon a large number of factors influenced heavily by their environment and their life experiences. It's far harder to find systems which are not chaotic than ones which are. Even the classical 3-body system interacting under gravitation is chaotic.

And while, yes, there is definitely good work being done in studying dynamic systems, applying those things to human beings is very difficult and dangerous. We already often face situations where humans are expected to conform to a limited set of predicted responses rather than the system being seen as flawed for failing to predict all possible responses. And human beings react very, very poorly to being restricted in these ways.

I totally get what you're saying about the unpredictability of human behaviour. But I don't see the connection to this particular system. It could totally be chaotic if it was modelled as a dynamical system, I don't disagree with that, but as far as I can tell they approached it using neural networks...

Chaos isn't an intrinsic behaviour of humans, it's a property arising from specific conditions in a dynamical system. I do think there's been interesting results in trying to model human behaviour (http://www.wilmott.com/pdfs/090610_mandelbrot.pdf), but like you I'm hesitant to apply these mathematical models to us irrational humans. Anyway, I'm just being a pedantic math nerd.

P.S. Sensitivity to initial conditions alone does not imply a dynamical system is chaotic.

I only read the beginning of the article and skimmed the rest, but this seems to be a misleading article. Just because the Chileans had the idea they could use massive amounts of data and computers to achieve their socialist goals doesn't mean this was the origin of big data analysis in companies today. It doesn't take a quantum leap in creativity, imagination, or innovation to realize you can combine computing power and lots of data to glean insights into customers, or in socialism's case, societal (i.e., the government's) goals.

It isn't about customer insight, it is about control system in cybernetics sense (where the more system information reflected in observation vector, the more chances you have to generate optimal control input). While capitalist society puts a lot of belief into emergent behavior of the system, socialist is obsessed with optimal control of the system.

Reply on news.ycombinator.com