I don't think the actual order type was a secret, just that some HFT firms figured out a way to use it which gave them an advantage over other HFT firms. The firms who didn't understand it lost out.
Myth? The finance industry, where the HFT folks are near the top, pays epic salaries for the smartest people. They do this to hire the best and the brightest, and would not continue to do so if it did not work.
In the documentary, Haim relates a story where the team that he was on included PhDs in bioinformatics, physics, and mathematics. This does not look like a myth to me. The mathematical knowledge required to understand and be competitive in the HFT environment, coupled with the software development knowledge is very rare. This industry consumes the time of geniuses in order to function and pays them well for it - that this attracts dumb people and bad software developers that are occasionally hired hardly makes the existence of these geniuses a myth.
> 2) We could harness all these people to something that provides more utility. It would be great if that were true, but how do you do that?
Distributed self-organized common pool resources (http://en.wikipedia.org/wiki/Common-pool_resource).
I have found very little correlation between advanced degrees and ability to determine valuable trading strategies. In fact, the strategies that most people bemoan the most with HFT require the least amount of mathematics because they are very simple.
I've also worked outside of trading in both big institutions & start ups. The ratio of genius to average to bad seems about the same in all of these environments.
As far as the common pool resource approach, the problem with that is that it will most likely tilt the balance of power in the markets even further into the hands of a few giant institutions. Why it is popular to protect the poor investment bankers from the ravages of HFT baffles me.
Your second point is trivially answered: you let the market decide where they work.
That's not obvious to you because you conflate "provide utility" with "make money". There are lots of things that make money that provide no utility. A large part of the history of business regulation is basically restricting those so people focus on providing utility rather than making money at things that provide zero or negative societal value.
The argument you respond to is basically suggesting that HFT is not generating any value for society. If you want to demonstrate that it's false, you have to show that HFT creates value in line with its costs.
It's pretty obvious to me that social networks have some pretty dangerous attributes (decrease in privacy & personal securty, etc) to them. Do these dangerous attributes outweigh the value that twitter provides? Who decides and why?
Because most new industries aren't within the financial industry, messing with the fundamental operation of markets. Those that are have to make that argument.
http://www.chrisstucchio.com/blog/2012/hft_whats_broken.html
http://www.chrisstucchio.com/blog/2012/subpenny_rule_respons...
I'm just suggesting that using a callback API rather than polling is hardly a great example of HFT shenanigans.