How I made $500k with machine learning and high frequency trading
jspauld.com
jspauld.com
It could have easily been called "how i lost 500k with machine learning". Like gambling, it's easy to manipulate statistics to show that you did well in some period of time.
I worked for a large investment bank about 10 years ago, writing trading programs for quant traders who were market makers. The quants called guys like him "retail" investors and they gleefully picked off all those trades. It's how they made all their money.
So, everyone else, beware of making this a case study in how to make lots of money really fast. You are more likely to lose money.
Why not?
With a quick bit of R code, we can simulate his PnL over 120 days multiple times, assuming he has no skill, and see what the probability of him being up 4k/day is. I'll use a t-distribution with 3 degrees of freedom, which allows big up and down swings (again, accentuating the effect of luck).
> pnl <- c()
> for (i in 1:1000) pnl[i] <- mean(2000 * rt(120, df=3))
> mean(pnl > 4000)
0.0
This is really cool, any way you cut it.
Would you dare to "predict" the direction of this FX rate movement in the next month? Then it's a matter of calculating potential profit/loss factor and adjusting your trade value. And yes, as with any high risk investment, putting all your eggs in one bucket is not a brilliant idea. Just like taking all your savings to Vegas.
Having said that I can agree that my case is pretty unusual and that everyone should beware of attempting to do something like this. Even for myself I couldn't do it now. (There is a reason I turned my program off.)
How much did you spend before you "tuned" it? How much did you spend afterwards? What were the tax consequences of your trades? Did you make exactly 500k? Have you traded at all since then?
You mentioned that you occasionally "sat in" and took some large losing positions. Were these on purpose? Bugs? Was your exposure actually much higher than you thought? Was limiting contract size enough risk management?
With regard to tuning I may have lost $1000 or so but as I wrote in the article I was able to build a backtesting model that accurately simulated live trading. So once I had that I could basically use it to verify I had sufficient edge to make a profit after covering my commissions.
My risk exposure was very low. When I said large losing positions this meant like $600. But the bottom line is I had a daily stop loss of $3000 enforced at my broker. The most I ever lost was around $2000.
Anyway, there is not really some hidden thing that I am not telling people. It does bug me a bit that your comment is at the top given that it says I'm manipulating statistics and was actually one of the guys that the quants gleefully picked off. I think it's unlikely I traded much with other HFT systems but if I did they certainly lost money.
I cannot emphasize how important it is to understand that people who trade using price action (http://en.wikipedia.org/wiki/Price_action_trading ) are just speculating based on where they expect the price to move. It's no different than people who play Texas Hold'em online and speculate what cards others have based on betting patterns. If you get good at spotting the patterns (like this guy did) you can go on a winning streak, but when the game changes (as it did for this individual after 2009) then you either go home or go broke.
This guy found one edge in 2009. It won him 500k. Fantastic. More than any edge ever won me. But, the market has changed so much since then, with HFT becoming so prevalent (http://www.theverge.com/2012/8/7/3226187/high-frequency-trad... ) that please be careful before you follow this course. His code is unlikely to be worth much today unmodified, and when you modify it you'll realize, as I have, that when the other players have access to the order books and can jump the line you have no chance in the game in 2012.
One last nit: Please, please post recent data when you talk about projects like this. 2009-2010 is 3 years ago. Since then there was significant turmoil in the US, Asia, and the EU. How are these returns relevant for today?
You are not wrong, but what you wrote here is applicable to any success story posted on HN.
Caveat lector. Always.
When you think Facebook/Instagram, you think "Damn, those guys got lucky as hell". When you think automated trading, you think, "Hey, it can't be that hard", and start firing up your IDE and rolling out code to talk to an easily provisioned API.
Sure, it may take months to lose your shirt selling a photo service to Face/Goog/Apple. You can lose everything overnight with automated trading.
My father used to trade commodities for a living in the pit at the CME many moons ago, and when I was growing up I would be his technical side when he was trading out of our suburban Chicago home (setting up FM receiver/satellite dish/etc for real time quote data, staying up late nights with him running through trading scenarios in Tradestation on Win3.1 with data downloaded in bulk from Knight Ridder, and so forth).
Something very important I learned from him was: "The market can stay irrational longer than you can stay solvent." With a startup, you can hit bottom. In the right market, bottom is much further down than you can ever see.
I'll take it ad absurdum: You can lose everything in a second by not looking left and right while crossing the road. Or even by looking left and right while crossing the road, when someone else is driving recklessly.
It is possible to attempt HFT with not much more risk than stating a new InstaFaceGoogApple service. Put $10,000 in your margin account, and use a broker that practices proper margin checking. Tada! You're not going to lose more than $15,000 over that. (Yes, you can lose more than you put in your margin account, but not by much).
While that's more, upfront, than InstaFaceGoogApple, it is comparable to the 4 months of salary that you're going to forfeit while building the InstaFace service. And unlike most InstaFace apps, you have immediate market feedback, which can only be a good thing.
Note: Instagram did have immediate feedback from the public at large, forcing them to scale much earlier than they expected - but they did not have a feedback as to the financial value of their proposition. In fact, it wouldn't take much for instagram worth to be zero. Read, e.g. http://www.jamesaltucher.com/2011/02/my-name-is-james-a-and-... - a $100M acquisition back then is like a $300M acquisition in today's valuations; not Instagram but definitely nothing to ignore.
> With a startup, you can hit bottom. In the right market, bottom is much further down than you can ever see.
That's true. But you still have to remember that 90% of startups fail, and of those that succeed, many are just moderately successful. And yet no one keeps yelling "but most startups lose money!" at every HN story.
> When you think automated trading, you think, "Hey, it can't be that hard", and start firing up your IDE and rolling out code to talk to an easily provisioned API.
Which is what we should address, and these "it's a gamble" warning do not. When you see Suzanne Vega singing, you might think "Hey, it can't be that hard to sing". Many people do. And yet, they grow out of it, usually without trying to publish an album (and failing). This should be no different.
The point is you are lured into crossing the road, when you absolutely didn't have to.
In order to open an account with the neccessary infrastructure to engage in HFT one must be an accredited investor (typically means having a net worth of 1 million dollars or more) and the cheapest brokers typically require a minimum deposit of $500,000.
Not to mention the overall costs including hardware, co-location, market data and other vendor costs are on the order of 45-50k a month.
With $10,000 you can't even open up a normal day trading account as the law required a minimum deposit of 25k.
As long as we're quoting Keynes, let's also remember this gem from a letter he wrote to the regents of King's College about the performance of their endowment's portfolio (which Keynes managed).
"The management of stock exchange investments of any kind is a low pursuit, having very little social value and partaking (at its best) of the nature of a game of skill, from which it is a good thing for most members of our Society to be free; whereas the justification of Worlaby and Elsham lies in its being a constructive and socially beneficial enterprise, where we exercise a genuine entrepreneurial function, in which many of our body can be reasonably and usefully interested. I welcome the fact that the Estates Committee-to judge from their poker faces and imperturbable demeanour-do not take either gains or losses from the Stock Exchange too gravely-they are much more depressed or elated (as the case may be) by farming results. But it may be useful and wise nevertheless, to analyse from time to time what is being done and the principles of our policy."
Edit: Worlaby and Elsham was a farm that the endowment owned.
Didn't your father teach you about "Stop orders"
You can't have your algorithm cranking away without supervision. And to be extra sure, lots of testing and LIMITS.
Limit the amount and value of orders.
With stocks, worst case: you lose the face value of stocks in your portfolio
Derivatives: you can lose more, even 'infinite liability' (still, it's constrained by the stock market inertia)
So I guess they have a role in limiting losses (which had they not been there would be much bigger)
What pains me is just this year I've heard in 3 separate occasions for 3 separate startup businesses {industries: ['transportation', 'social', and 'mobile ads']} people propose "Let's do the Instagram strategy." It may be obvious to you and me how absurd that sounds, but there are a non-trivial number of people who blindly follow headlines.
Edit: I agree with @toomuchtodo. It's just too easy to risk with HFT that the warning is needed here more than elsewhere.
To start with, there's simple probability: knowing the odds of making you hand vs. the payoff in the pot, or the chance of winning with various starting hands. This is pretty basic but a lot of low-stakes players screw it up. If you get it right, their mistakes are your gain.
At a more advanced level, game theory comes into play, using bluffs and so on. The game is complex enough that it's not completely solved, and it's an active area of research. The University of Alberta is doing a lot of working developing poker bots using game theory. By playing a good strategy, you can prevent other players from exploiting your patterns.
Only after you've got a good grasp on all that should you really think much about exploiting a particular player's weaknesses. The Alberta guys are doing work on that part too. Exploitative play can improve your profit but also makes you more vulnerable.
For a good overview of this stuff, the book Mathematics of Poker by Ankenman and Chen is a good place to start.
I agree though that HFT is awfully competitive these days. If I had to choose between the two I'd play poker.
Poker is "a game of skill with an element of luck" and should not be confused with say, gambling on roulette or the outcome of a coin toss.
But the HFT game changes and you have to keep up. Just as a poker player from 10 years ago would not survive in the game today without adapting his style of play.
So true, and frustrating. But, the legalization of online play could bring back another boom at least for a couple of years.
At best HFT is a near zero sum game. It isn't creating value for customers. It isn't making the world a better place.
It is an unfortunate flaw of our economic system that so many smart people put so much effort into playing zero sum games with each other.
I know a very good engineer, who used to design innovative chips for 4G/LTE mobile telephony. These chips contributed to the market position of one of today's leading mobile phone manufacturers.
Today, this engineer is designing ASICs for high frequency trading (basically a specialised Ethernet switch, with all extra logic stripped out, so packets go through a few nanoseconds faster).
HFT isn't a zero sum game. It's sucking resources away from productive disciplines into an unproductive discipline, so making a net negative contribution.
Note: by spreads I mean the difference between buy and sell prices. I don't know if there is a special word for it in this context.
Each futures pit used to have hundreds of traders, who required several assistants/support and commanded a huge salary. Many firms needed multiple traders in a pit, just to be able to make sure they could provide liquidity to all possible market participants. Today, a couple strategists with a small team of programmers can cover dozens of futures markets at once.
The same principle holds across bond, FX, equity and options markets alike. HFT has supplanted a terribly inefficient market with a better one. Is it perfect or even good? Probably not, but it's magnitudes better than the traditional method.
With a deep understanding of markets and trading I fail to see why you see 'luck' as an explanatory variable is inversely correlated with the frequency of your trades (notwithstanding the effect of trading expenses)?
From what I have gleaned the following seems to be true: 1. Your algorithms worked (made money) 2. Then your algorithms did not work, but you could not figure out why
If you do not know why something stopped working it seems unlikely that you had a full understanding of why it was working in the first place. Without understanding the nature of the predictive value of the algorithm while it was working, its success seems to be good fortune.
Your algorithm could have shown a systematic correlation to any number of factors that could have created strong performance over several months. Performance would then be attributed to accidentally 'timing' a favorable market.
I know you feel differently, what am I missing?
And either way - kudos on the $500k.
Perhaps posting the source code would not be a good idea, but posting more details would be welcome so that people interested could follow their own path to automated trading.
I develop algorithmic strategies for a living, and my first reaction to reading your post was skepticism. I'm skeptical for two reasons. (1) because your methods are so unconventional in an industry where convention rules, and (2) because of the time frame of your success, which happened to be one of the more impressive market recoveries in history.
I can't tell you how many people I've worked with who fail to isolate the source of their pnl (myself included at times). This is key. It's important to benchmark your strategy against other stupid ones that you know don't have edge. When someone shows me strategies that worked in 2009 and 2010, I immediately make them prove their strategy was not the equivalent of being long equities.
Doing this will truly help isolate whether or not luck is involved. When you say that the number and size of your trades justifies the strategy's validity, that's just wrong. You could do 1000 trades in a day: buy 10 RUT futures at the beginning of the day, sell 10 at the end, and just scratch 1 lots for the other 998 trades. In a bull market like 09-10, that would have made 400k, and would have nothing to do with Machine Learning or its applications to HFT.
I make all traders benchmark their work against a series of other strategies that I know have no edge, even though they, at times, can appear to have edge.
Now, I'm not saying you didn't have legitimate edge, but you do your readers a disservice by omitting relevant stats and discussions like that.
2009-10 was more than just a huge rally, it was also a period where vol and skew were massively mispriced. I know this is high frequency, but like I alluded to, you need to make sure that what you're doing isn't replicating the pnl profile of low frequency strategies.
So, how did you perform relative to vol sellers? From the market bottom to the end of 2010, the max daily loss for a vol seller was about 3x average daily pnl, and >80% winners. So your returns do sound better, but not incredibly.
But, even if you failed to perform as well as vol selling did over the same period, that doesn't negate the strategy's validity. If returns were not correlated, then it's safe to say that you weren't just inadvertently shorting vol.
So, start there, work out a regression comparing your daily returns to someone selling vol. Do the same with moving average strategies. Mocking up a simple market making back test versus an ES beta is hard, but that too would be a something to test against. I don't expect you to do any of this, and I'm not going to bother to either. I'm just saying that a complete discussion of this subject would include that information.
I was also in this business, and there's nothing unconventional about his methods. It would, in fact, closely describe the methods of more than one shop I'm familiar with. (Except they WERE able to overcome the declines). And the 3-6 month indicator lifetime looks eerily familiar.
And these places are anything but "convention rules" - it's "creativity rules, before our competitors get creative enough".
> When someone shows me strategies that worked in 2009 and 2010, I immediately make them prove their strategy was not the equivalent of being long equities.
Assuming the OP is telling the truth, there is no equivalent "long equities" strategy that would make 1500% profit over 6 months (%3000 annualized), with a max drawdown of 20% ($2000 on $10000 - but his max drawdown was probably closer to 5% than to 20%). You are welcome to demonstrate that there is.
Sounds to me like you are doing low frequency strategies; it's a completely different ballgame than HFT. He's done 400,000 trades, half of them long, half of them short. It might have been luck, and he might have been riding something underlying the equities, but this is NOT equivalent to being long equities. He might have found a way to get non-linear leverage (rather than prediction). But that's also worth a lot of money in the right hands.
> buy 10 RUT futures at the beginning of the day, sell 10 at the end, and just scratch 1 lots for the other 998 trades. In a bull market like 09-10, that would have made 400k, and would have nothing to do with Machine Learning or its applications to HFT.
That may be (I wasn't trading in 2009-2010, and don't remember the movements or the required margins), but that would have had much higher volatility (and days with much more than $2000 loss) than the OP had. (Assuming, of course, he is telling the truth)
I'm currently building a semi-high frequency trading solution and the problem I run into is the sheer breadth of expertise you need to get it all happening. Modern chip design, low-latency, lock-free concurrent messaging, fault-tolerant system design, adaptive learning algorithms, k-means clustering and broker APIs are just a smattering of the ideas I'm trying to get across to make progress. For me, algorithm creation comes more easily than reading about and implementing a broker interface.
There is certainly armies of PhDs out there backed by big money but they exist behind heavily guarded intellectual property walls. An open source HFT/Algo/Automated trading platform that brings a hacker sensibility to this problem domain would be seriously competitive.
If your net exposure is small, but that's only because you're offsetting various positions then you're probably picking up nickels in front of the volatility steamroller & if you stay in the market long enough you'll get squashed at some point.
(And indeed, living is gambling. It's all just a matter of the risk/reward portfolie).
But jspauld has apparently made $2/trade after fees on 250,000 trading, with a very small standard deviation (I would guess less than $2/trade) - which makes it one of the best businesses one could ever have.
You can't live without gambling - by e.g. going to be a salaried employee for Yahoo rather than Google or that weird newfangled "TheFacebook" thingy back in 2004, was a gamble.
jspauld, statistically speaking, has made less of a gamble there than almost anyone else posting on HN.
> So, everyone else, beware of making this a case study in how to make lots of money really fast. You are more likely to lose money.
True. But that's true for every single success story posted on HN, reddit, or USAToday.
Most people don't understand that, when you're able to recognize patterns, playing millions of hands while never exposing more than 1% of your bankroll on any deal is not "gambling" but "printing money" (a tiny amount of money in my case compared to consulting but that is not the point).
At the same time the very fact that obviously (seen most of the posts here) most people don't understand basic bankroll management, risk management, standard deviation, expected value, variance, etc. means there are probably quite some opportunities out there to make money for those who do understand that ; )
If I tried to kludge together a definition, I might come up with something like:
>Risking the loss of something of value in exchange for the possibility of gaining something of greater value in a situation where the determining factor of losing value or gaining value is random chance
The problem with a definition like this is that, as others have pointed out, it applies to vast realms of human endeavor, from founding a company to playing the lottery. It also includes no distinction between risks with a positive expected value and risks with a negative expected value.
If a lottery has ten $1 tickets for sale and each ticket has an equal chance of winning, there is an obvious difference between the prize being $11 and $9, but buying a ticket at either price is just as much "gambling" in the common parlance.
What we really need is a word that only refers to gambling in situations with an expected value less than or equal to zero.
>If a lottery has a total of ten $1 tickets for sale and each ticket has an equal chance of winning, there is an obvious difference between the prize being $11 and $9, but buying a ticket at either price is just as much "gambling" in the common parlance.
I believe we call that gambling.
Because there's a commission on trades, and because you pay taxes on net gains but your minimum tax is zero, high frequency trading by its very nature must a loss for most players.
I was not aware that this is what defines gambling. And "no service produced" is certainly wrong by accepted economic theory - arbitrageurs provide a price discovery service for everyone; they get rewarded for exposing the inefficient prices, even though it is done through market mechanics rather than a specific customer.
OP appears to be a statistical arbitrageur - which is the same concept, except that it includes a shift in time or space (and incurs risk). You might not be interested in this price discovery service, but other people are paying for it with their wallet. (And it's mostly the market makers who pay for this with reduced profits)
> one trader's gain being another trader's loss (relative to market returns).
That's not true in investing in general - when shares have time to appreciate or depreciate, it is definitely not a zero sum game. Everyone can win, or everyone can lose, or anything in between (it all depends on your time range, and your measure of loss or profit. The "non-zero-sum" element arrives partly from companies using operating profit to buy back their own shares).
> Because there's a commission on trades, and because you pay taxes on net gains but your minimum tax is zero, high frequency trading by its very nature must a loss for most players.
That's only true if all players are hf players. If there is sufficient non-HF activity, then the zero-sum argument does not hold.
(I'm not saying that it's not a good approximation - in most time scales, in most scenarios, it is - but it is not the mathematical truth you imply it is)
Again, it's a great approximation most of time and over most time periods and asset classes, but it is NOT axiomatic in the way most people believe it is.
Remember: as long as there is a way to inject or withdraw more capital into the system (through whatever asset class, as they are all interconnected), the sum is not identically zero.
Just assume one of the stocks is a gold mining company that works efficiently. The share value rises, and the shares are redeemable for the gold, without anyone having to lose anything (except mother earth)
It sounds like you are making the argument that this is zero-sum game, but whether something is zero-sum depends on your utility function. If the players are risk averse, then a transaction like buying insurance can yield positive utility for both participants.
Many trading strategies are performing a service in similar (but more complicated) ways.
If you've really worked in that field than it's very surprising you've never heard about what professional poker players call bankroll management (and they "stole" the concept from professional traders).
The whole point is that you can --either if you gain an edge or get lucky-- win big. Very big. But you're never exposing a large part of your funds in the process.
Maybe OP had an overall "stop loss" at, say, $10K. Had he had five minus $2K days in a row at the start, he'd be out. He wouldn't be broke. He wouldn't be without a car and without a bank account. He just would have lost $10K.
But there's no upper limit as to how much you can win.
All you need is discipline and sound bankroll management.
And, yes, I've won a five-digits figure (hence not anywhere near what OP did) real $$$ at online poker. Starting from $0.01/$0.02 small blind/big blind tables and then working my way up using bankroll management.
It's assymetric. You're foolish if you think that succesful traders who won $x were as likely to lose $x. This is simply not how it works and it's very well explained in OP's article. He's detailing what his maximal daily exposure was and it was tiny compared to what he made.
With my program I didn't really have bad streaks because my P&L was averaged out over thousands of trades per day.
I played HUSNGs for a living for several years, and I could play three tables at a time (about 9 games per hour) with a 60% winrate. That's incredibly low variance--my graph over the long-term was better than a 45 degree incline.
Risk management is probably the single most important thing to understand in trading. Unfortunately, it's something that lots of people learn late, if ever.
Folks get caught up in the romantic notion of betting it all and winning big, but end up losers. Meanwhile, the consistent winners they aspire to be are exposing perhaps 0.2% of their roll at a time.
Though, one thing I think is a bit unique to trading is prevalence of folks who preach without practicing. Just about anyone with a brokerage account can rattle off the same short list of critical do's and don'ts, but very few actually follow them.
http://falkenblog.blogspot.com/2011/02/stop-loss-myth.html (with additional citations in the blog)
Except two things:
1) He didn't lose money, he made 500k.
2) If this worked reliably, you would be out of a job.
With a bit of luck and a good partner, this guy could have built a sustainable business.
The only argument in your comment that isn't your own unfounded opinion is that market makers make money from people who execute trades. But this is true by definition.
The traders who "gleefully picked off all those trades" weren't outsmarting anyone, they were simply profiting from the difference in the asking and offering price in the market. This is the role of a market maker, and actually makes it cheaper for people like OP to execute a large number of trades. So even though this comment sounds like a sensible rebuttal of the linked article, it doesn't really say anything at all.
Again, sorry for creating a negative reply and contributing to a bad tone, but I really the right thing is to call out these kinds of replies. They discourage honest sharing and discussion.
yajoe's comment is an example of how to criticize a post like this in a useful way. http://news.ycombinator.com/item?id=4748989
Given that he might convince other people to engage in high tech gambling in a less-favorable market than the one he operated in, strong words are called for in this case.
A UChicago economist and graduate student are walking across campus. The student says ... hey ... there is a hundred dollar bill on the ground! The economist scoffs and says no there isn't ... if there was one, someone must have picked it up already.
Sometimes I catch myself thinking this way. I have to remind myself that (a) markets aren't perfect, and (b) the real world has huge asymmetries in information, ideas, and perhaps willpower (by this, I mean while 100 people might think of a great idea, not all will attempt to implement it; even then, people will differ in execution).
That said, you're likely right. This trading strategy will likely lose money today :-p
In early 2000s I wrote a machine learning algorithm that beat the S&P 100 with over 1 trillion to 1 odds against it being luck. It predicted a full trading day in advance. But that was all on paper at trading firms' puny costs; unlike you I couldn't beat retail costs. It's amazing that you could do that. For that reason alone I think it's highly likely that you were a skilled monkey.
Also like you, nobody in the industry was interested in my code, even after an industry magazine watched it for 3 months and found it gave "stellar" performance. The few people I was able to discuss it with told me point blank that it was impossible to do it skillfully (efficient market theory), so they assumed it was a hoax or the algorithm was just lucky.
I think that if someone is a good programmer and has some mathematical chops and has that kind of experience daytrading, taking a shot at automated trading is probably a reasonable thing for them to do. Without all of that background, you're right, they're almost certain to lose money.
I'm thinking back to Garry Kasparov's piece in the NY Book Review a couple years back: http://www.nybooks.com/articles/archives/2010/feb/11/the-che...
He talks about a chess tournament in which it was "anything goes"...competitors could be human, computers, or humans with computers. The expected outcome was that a grandmaster using a Deep Blue-like computer would win, but the winners ended up being a couple of amateurs with three computers:
> The surprise came at the conclusion of the event. The winner was revealed to be not a grandmaster with a state-of-the-art PC but a pair of amateur American chess players using three computers at the same time. Their skill at manipulating and “coaching” their computers to look very deeply into positions effectively counteracted the superior chess understanding of their grandmaster opponents and the greater computational power of other participants. Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process.
So in HFT, how much room is there for an amateur to profit over professionals by having a sophisticated process?
HF traders are just as much hackers as anyone on HN (and there are plenty of HF traders on HN). So 'theoretically', they've already done what is being suggested here. If someone comes along and develops a winning strategy, it really shouldn't be considered as having anything to do with 'professional strategy vs novice strategies'. It would just be about one person either getting really lucky or coming up with something that is genius in its own right.
--
If there are 'professionals' and then other 'professionals' whose strategy depends on information about how other 'professionals' trade (and there is), you end up with strategies at all valid points in the sample space of possible strategies and counter strategies. Theoretically, there should be no other possible strategies. Inevitably someone will come up with one though, and the 'sample space' will grow. But its extremely unlikely that additional unique strategies are successful just because they 'counter' the strategies in the sample space. But then again, this is real life and these things aren't impossible.
I developed a fully-automated low-frequency stat arb system that I ran in 2007 based on a perhaps even simpler algorithm. It traded various equities equally to the long and short side regardless of market conditions so widespread rally or collapse was irrelevant. I logged about 20-30 trades/day - much slower.
The results, using no leverage, were +90% in a year with a worst drawdown of 2% and a Sharpe ratio of 2. Total trades were 5000+. Month-to-month the results were very consistent until the uptick rule was nixed in July 2007. August 2007 was a record winner for me, but Sept-Dec 2007 fell flat, not losing, but with greatly diminished profits and the same variation and more frequently getting slammed all-long or all-short instead of a mix that was often near-neutral. Also getting fills better than my orders then completely disappeared, as this was the beginning of the HFT middlemen - including your own brokerage. I shut it down at the start of 2008, keeping the profits intact and moving on to other priorities.
I continued to monitor the theoretical results for a couple of years but the conditions didn't return so I eventually cancelled my data feed.
I consider HFT to be any strategy where speed itself is the what gives the edge. Colocation is usually a prerequisite, though not sufficient. It's a shame HFT gets all the attention, when it's really a tiny portion of trading activity. Algo-trading in general is 70%+ of market activity in the US.
Also limiting trades isn't really adequate risk management. The tech exists to very accurately model your exposures. This is something I see underdeveloped a lot, and what separates the top trading firms from the rest.
Still I commend you creating a model, working out how to test and execute it automatically and actually trading your own money.
I really think more hackers should be actively managing their money, (in general, not like in the article). We have these amazing liquid markets, all time low spreads/commissions, products like ETFs/derivatives to accurately and cheaply execute a given strategy, and a huge increase in tech to model risk, but personal personal investing is the same as the 60s.
In Europe, HFT is mostly what OP describes, because they have reasonable control (e.g., you have to have one execution per 10 orders or pay a fine; in US exchanges, you can sometime finds 10,000 orders submitted in 3 seconds, hundreds of thousands per hour, with 10-20 executions).
> Also limiting trades isn't really adequate risk management. The tech exists to very accurately model your exposures.
That's basically what AIG did with copulas. Unfortunately, the assumptions in these models tend to break during crisis, when correlations go to one. And AIG went bankrupt.
Limiting trades, done correctly (mathematically AND legally) is the ONLY way to do risk management properly. With more assumptions, you can have "more efficient" risk management in terms of leverage (e.g., you can net S&P and RUSSELL exposure by assuming their correlation structure) - but as AIG has shown, that does not mean you are doing a better job of managing your risk.
silly example, 100 small positions could be less risky than 1 large position or, 1 long, and 1 short trade will cancel each other out and create a riskless portfolio (with 0 return).
You need to have a risk budget, account for each trade, and work out the risk for the composite portfolio. Obviously this is not fool proof, but it's a way better approximation of the real world.
Yes. And assumptions about this are bound to break at the most inopportune moment, see e.g. AIG, which I already referred to. Read about the "copula model" disaster, as your statement indicates you are unaware of its details. https://en.wikipedia.org/wiki/David_X._Li#CDOs_and_Gaussian_...
> 1 long, and 1 short trade will cancel each other out and create a riskless portfolio (with 0 return)
This is true if and only if the long and short are in the same exchange, AND exchange rules allow netting long vs. short deterministically. Otherwise, you have counterparty risk. E.g., you can be long SPY and short SP contract (in equal underlying), which would theoretically mean your only exposure is interest rate changes (and sometimes not even that!)
However, since this is in different exchanges, it might happen that during a flash crash, your SPY position will be liquidated for insufficient margin at a low price, but then the price bounces back, and you've lost money on a perfectly hedged position.
OP's model (limiting exposure and assuming the worst, if I understand correctly) is not statistically efficient use of margin, but it's way better at actually managing risk than any statistical model.
very interesting, a spike in sept-dec 2009 that roughly follows his PNL...
But, every time I've tried to actually get started, I've always found the amount of research required before being able to begin is just staggering.
It seems like the logical course of single-programmer HFT trading being:
- Find sample data
- Build your trading program using sample data
- When you're happy: connect to live API and set your trading program loose
- Iterate.
However, the first step and the third step seem like the ones which require the most research.
Is there somewhere which has a straightforward dump of timestamped market data available to download (free or not), in order to actually develop a working program?
Likewise, figuring out what to actually trade with, and which service to use is also pretty taxing.
If the easy part was building a working model either you got incredibly lucky or the model is wrong.
Symbols change, stocks split, dividends are announced etc.
The one interesting point that he glossed over is what his indicators were. He wrote, "The indicators that were most useful were all relatively simple and were based on recent events in the market I was trading as well as the markets of correlated securities."
Which doesn't really tell you much.
But for anyone coming to HFT from a coding background instead of a trading background, an explanation of one of your indicators would have been fascinating.
Worst case he runs out of capital over a period of weeks.
He can't blow up in the way that you think - but he can have large drawdowns over a period of weeks.
Markets are eventually consistent scalable systems - and that is why we prefer them over central planning. In the medium term they prices things correctly, cheaply and efficiently (decade+).
In the short term however (sub-decade) - they can't price jack.
Markets are inefficient period - if they weren't, well then P=NP and you could just put your NP-hard problems into a market and get back cheap, quick, accurate results. Oh - wait - protein folding is actually harder than that.
There are 2 major ways to make money in the markets. Value-Growth and statistical arb (often high frequency). The former (Buffett) is highly concentrated bets on the future of business (I'm value - long TSLA/GOOG/Samsung). The latter is looking for thousands of small diversified statistically significant correlations above 50% (random guesses) and trading costs between securities/price movements over short time intervals (aka statistical ghosts in the data - RenTech/Shaw).
Both work. Both work well. And will continue to do so as long as markets exist.
http://www.reddit.com/r/IAmA/comments/9s9d7/iama_100_automat...
Making $500K on $100K over 6 months is good business.
E.g. if you made 10% when overall market was up 15% for the year, you have negative alpha. [As someone could have bought index and held it through year to generate better return]
If you made 20% when market was up 10%, you have positive alpha.
That is why everyone in the investment community is 'seeking alpha'.
So the fees will be $3950 - $15,800 per day.
$15,800 - $79,000 per week.
And he surely wasn't paying retail prices - you can go down to around $0.50/future if you know what you are doing.
(In general, futures have 1:5 to 1:20 leverage)
$500 - 2,000 per day
$2,500 - 10,000 per week
$130,000 - $520,000 per year
That's just the fees. He must make that much just to break even. If you can make that much starting with $30K, you're a supermotherfucking genius.
One "tick" (minimal movement) is worth 12.5 EUR. At volume you pay 0.5 EUR, IIRC, but let's assume you pay 1 EUR in fees, everything included.
If you bought and sold at the same price, you lost 1 EUR/trade. This is the cost of business.
If you bought, and sold after a favorable 1 tick movement, (e.g. bought at 4013.0 and sold at 4013.5), you're 10.5 EUR richer - 12.5 on the difference, minus 1 for each trade (one buy, one sell).
If you bought and sold after an unfavorable one tick movement (e.g. bought at 4013.5 and sold at 4013.0), you're 14.5 EUR poorer - 12.5 on the difference, and 1 for each trade (one buy one sell).
OP averaged $2/trade over 200,000 trades; that means he had 2/3 right calls, and 1/3 wrong calls or so if he only traded dax and only had 1 tick moves.
He was very smart, but you're looking at it wrong - the fees are the cost of doing business, much like salaries are the cost of producing software. In finance, you rarely care about revenue or "notional" (which can easily run into the trillions per year for a small trader - for ~1 eur, you get 75,000 eur in notional value on the dax).
You just roll the fees up-front into your choices when thinking about it, and it all makes much more sense.
(Not trying to take away from OPs very commendable achievement - just trying to give the common perspective on how to view this)
The biggest issue is the confusion that you can apply machine learning to HF trading. HF trading sub 15min mark is more about playing the deal flow, and only the institutions have an edge on this. This is why goldman had to separate the buy and sell sides in the early 2000's. Above 15mins you are able to find an edge using time series analyses since the market is scaling invariant according to Benoit Mandelbrot and this does not apply to dealflow. Also having access to dealflow allows you to predict volatilty seconds ahead which allows you decrease your risk and increase you reward as well as handle your costs since the volatility will impact your transaction costs even if transaction costs themselves stay the same. There is just so much stuff to cover that a comment will not do justice in explaining what is wrong with this guys logic.
Survivorship bias would mean I simply got lucky. If you're going to say that you're at least going to need to look at my P&L charts and say how I could possibly achieve that much success with luck alone.
Finally, machine learning has everything to do with my success. There were hundreds of variables in my algorithm that were ALL optimized using ML. If you read the article you would know that I built an accurate model for backtesting that I used to optimize variables as well as confirm that I was going to make money before I even started live trading.
I'm pretty confident that whatever you were doing in 2004 has nothing to do with what I was doing.
If you are relying on a broker-supplied pricefeed over the internet you are far outside the real of what is traditionally understood as "High Frequency Trading".
You talked about programming hotkeys (and then automating the hotkeys) so I assumed this was running on your desktop.
Why doesn't every hacker do this to make extra money? Is it within the grasp of anybody who can program to automate trading like this?
EDIT: Sounds like it's not really for everybody. You have to own or rent a server with access to direct lines to the exchanges, or else your lag will be such that profiting from HFT is impossible. How much do these cost?
I have flirted with HFT in a hobby-like manner and it isn't the programming that will get you the money, it is the domain specific knowledge coupled with the programming. On top of that, there are quite a few risks and potential to lose a lot of money.
A profitable predictor is a much, much harder problem.
At a place like Goldman Sachs, a quant with a working predictor gets paid 5 times as much as the IT guy who makes that predictor talk to the market quickly enough.
Because, as your question implies, it is (relatively) easy to do the IT work or hire someone to do it. Not so for the predictors.
Once you have the assets and capacity to actively manipulate the price of any stock at will, the market is a VERY different animal and no longer need to be understood at all. You simply force its hand.
"I was making a lot of money but now I've stopped" is the same thing as "I was lucky until I wasn't". Making a living by gambling pretty much sucks, which is why most hackers don't do it.
(You'd think that something as complex as markets would attract hackers trying to "figure it out". The problem is that due to the changing nature of the other participants, all hacks are temporary. Makes for great blog posts, but not a long term strategy.)
2.)Finding a good predictor. Predicting up or down is easy on paper. Finding a predictor that a.)beats the spread, b.)factors in lag time to execution, and c.)factors in commission, is hard.
3.)Market regime changes. What works a few months ago isn't guaranteed to work now.
I wondered the same thing and tried to answer it for myself not that long ago.
In short, it's hard, time-consuming, stressful and costly.
1) Your code has to work going forward. Coming up with something that works in backtesting is easy, doing it moving forward while staying within your risk envelope and considering all the associated costs is hard.
2) Closely related to #1. Whatever you put together is surely going to need plenty adaptation and oversight. It's a very fluid problem, you're just one player among countless others. How much time do you have in the day? Where is your capital coming from? For most people, the source of capital is a job that they probably can't ignore while fiddling with a trading program that would have to be remarkably successful to replace that income.
3) Everything costs money. You need a certain amount of capital to start with and there are all sorts of running costs. From the basic costs of execution, to market data and eventually co-location if you get that far. The fact that the OP kicked off with an amount that's essentially the barest minimum for any kind active trading is exceptional.
4) Can you maintain discipline and continue executing on all of the above while losing money? How many people struggle to simply get up on time, control their desire to check a particular website or to get over and move on from some frustration? Even if there's automation involved, trading will test your mental fortitude.
http://www.ctsfutures.com/t4.aspx
Would be even more awesome if you could share some code on github.
(former TT employee)
What does it depend on?
If he posts the code, you're a long way from running it.
If the tools work, sell them. If they don't, tweak them, try it again, and sell them until they do.
This is risk management.
It is a software implementation of the Viable System Model (VSM), a model for autonomous systems developed by Stafford Beer. It provides structure, communications, auditing and alerting for autonomous systems.
Part of it is base code for dealing with stocks and options, treating securities positions as autonomous systems that have the scaffolding for running simulations on themselves. It's in Smalltalk and runs under Squeak and Pharo. It can be found at:
http://home.rbcarleton.com/rbc/software/smalltalk/VSA/
I won't try to advertise it as a complete solution, but it might be the start of one.
Automated trading is more on strategy, while HFT has more to do with volume and speed. With automated trading, you predict price movements. HFT involves being a liquidity provider. You don't use market technical indicators in HFT, you wait for some really huge orders.
HFT firms won't bother him. Those are dealing with an entirely different set of algorithms. He should have contacted brokers instead.
this: http://books.google.de/books?id=dobO95EBcqsC&redir_esc=y
even though its over 10 years old, is real HFT. moving correlations, windowed FFT (of bid and transaction events), microscopic operators, negative first-order autocorrelation of returns.
it is NOT about supplying liquidity or being a market maker. that's just market makers trying to say they are in on the latest trend.
Seriously though, there are some existing frameworks and products that you could check out. I haven't used this myself, it's just in my bookmarks:
http://www.rapidquant.com/features
A C++ library with python bindings: http://quantlib.org/index.shtml
And a low level library for data analysis: http://pandas.pydata.org
Oh, I forgot about another backtesting framework for python. It has a built in IDE as well.
Our backtesting engine, Zipline, is opensource - https://github.com/quantopian/zipline
Zipline was unveiled at PyData NYC, and the presentation materials are here: https://app.quantopian.com/posts/hello-from-pydata
Were you compounding the data always, or telling it to forget what was going on several months ago? Or somewhere in between?
(I'm pretty unfamiliar with machine learning, apologies if this is obvious or something)
Is this a result of bots on the other side adapting in some way to what you were doing? I would have thought you would be too small a player for them to notice.
We cannot even tell if $500k is a good risk adjusted return - we have no information on volatility, nature of the exposure or most importantly how much money he started with?
Not exactly shocked Jim Simons didn't return his email. But completely shocking that he walked away from a successful automated trading strategy... the only thing rarer than a free lunch is a man willing to walk away from one. suspect.
2. Author has stated elsewhere than he began with $10k in seed capital.
3. The "bullish" market you cite increased only about %70 in that timeframe while the author's returns were multiples of that number.
4. Author walked away from a _previously successful_ strategy that no longer produced profits. His montlhy returns went to nearly zero so he stopped.
Edited to fix numbers.
It is of course possible that once you made "real" money with your algorithm it was spotted by the other algorithms which then started working against it. (Aka exploiting it) Having talked with people in that space (hft) I was left with the impression that an insane amount of analysis was done on those trades.
A lot of people in the business would pay e.g. $5,000 for exclusive rights for something that worked this well in 2009 (with proof that it worked in 2009, e.g. verifiable broker statements), and a smaller amount (say, $5,00) for non exclusive rights.
If he claimed it still works but he wants to sell it, it is a completely different game -- because when these things work, they are cash cows.
If you want to go back to trading, you'll probably have to actively try to get a job -- at the very least, let someone who's still in the business know that you are looking. In my experience in this field, word of mouth and friends-of-friends are infinitely more successful hiring strategies, for both sides.
(Re:releasing the source - I would like to have a look at the strategy, but I would recommend against releasing anything that is even close to being useful, unless you want to spend the next year screening "where can I get a good XTAPI broker" and "I've got XTrader_PRO set up, but I'm getting error 10013, what gives?" emails).
the Nuclear Phynance message board is probably a better place to look for business offers.
2. Line
3. ...
Profit!
Other than sheer luck, the most plausible explanation for your diminishing returns is that you found a strategy that worked _at that point in time_, other people copied it (starting with your broking firm), and as that strategy became more common your ability to make money disappeared.
I work in the industry, this happens all the time. Trading strategies have a shorter half life than you may think.
I traded stocks and Forex for years and my experience says, it is not for everyone. What ever indicators,discipline or model you follow it is going to work only if you have the right intuition or luck!
Could you explain this part, specifically what do you mean by "bucket"?
"To accomplish this I tracked predicted price moves in 50 buckets that depended on the range that the indicator value fell in. This produced unique predictions for each bucket that I was then able to graph in Excel. As you can see the expected price change increases as the indicator value increases."
Could you explain this part, specifically what do you mean by "bucket".
"To accomplish this I tracked predicted price moves in 50 buckets that depended on the range that the indicator value fell in. This produced unique predictions for each bucket that I was then able to graph in Excel. As you can see the expected price change increases as the indicator value increases."
If the market dives and you quickly get into a big long position, and then it dives some more - what do you do? You can either close out your losing trade and take the loss, or hope that the market comes back up, all the while holding on to the risk of further losses.
Also, there's no guarantee that trades in the middle of a flash crash will remain valid after the crash. The exchange could nullify all trades in a certain period of time, which would completely wipe out your upside potential.
This is the most important thing: In every single "flash crash", the exchanges have retroactively canceled trades, in a rather arbitrary manner (e.g., "every trade between 16:30 and 16:38 is null and void"). There is some underlying justification, but it is also arbitrary (e.g., "anything below 3% of the price when the flash crash started", with no specific justification for the 3% number, or a well defined methodology for the time of the crash).
That could easily turn a +$100K profit into a -$500K profit, depending on circumstance.
In the Flash Crash as well as the Knight Capital incident "up/down 30% from the Previous Close" was the price collar (anything outside that was busted and anything inside stood).
Of course there is no guarantee that the same criteria will be used the next time around so caveat emptor.
edit: I found what I was looking for in the comments.
$10k loan from dad. Built it up to 30k trading manually before my automated program went live.Well done!
PS: no sarcasm intended, it truly is an excellent advert.
1. when you have a good system (even one you cannot "improve" further), you don't talk about it. 2. You don't just stop using it.
There should be more to this story.
Last time I checked, NASDAQ and NYSE don't exactly publish API's
"In 2008 I was “manually” day trading futures using software called T4. I’d been wanting some customized order entry hotkeys, so after discovering T4 had an API, I took on the challenge of learning C# (the programming language required to use the API) and went ahead and built myself some hotkeys."
If you want to make money from investing, why not do so in a socially responsible way? Invest in companies that are changing the world for the better. You might not bring home as much money, but at least you'll be able to sleep well at night.
This does not ring true.
And HFT or day-trading does bring some value to the world. It enables companies to go to the public markets and raise capital. And investors to sell their shares without having to wait too long for a buyer. But those positives come along with negatives as well.
I am not saying that this guy's trading did contribute to flash crashes! He may have successfully implemented systems to prevent that. I hope he did, and if so I'm interested to hear how.
Just kidding. It was a fascinating article. Thanks for sharing.
The level of the coders doing HFT is beyond the comprehension of this guy, added to the team of Mathematicians, Physicists and computer scientists at your average HFT firm, they probably laugh when they read this.
Good try though, it was awesome that eventually he tuned it to profitability, but there's no way in hell they'd buy that amateur software/algorithm.
Kudos though for taking on the task of learning how to code and making money with ML.
He trade other people's money, using other people's (probably employer's) account and resources, I suppose.
His employer have paid all the fees, and, took all the risks - if there is profit - it is mine, if there is a lose - it is theirs.)
The essence of trading is about having a special (insider) position of even being a market maker, who just collecting fees from every trade other people do.)
But this is just my guess.
Taking a Machine Learning and Statistics course does not make you a trader. You need something else - access to the system.
This is why my guess is that he was an employee.
Except finance is supposed to be "serious". In most serious, legitimate activities, extracting money means you provided value somehow. So, what value high frequency trading could possibly provide?
Now I do have an idea where trading could be useful. For instance, a good old merchant doing arbitrage and making a profit is pretty useful: that's how different regions can specialize, do better than they otherwise would, and ultimately lift us from hunter-gatherer tribes where 60% males die a violent death, to our civilization now. And money, as evil as it may be perceived, is to date the best organizing medium humanity ever had.
Investors also have their use: by better allocating money among companies, we could hope to give more money to those who are better at converting it to actual wealth.
But.
The OP didn't make a ton of money because his model of the the companies had an edge, but because his model of the behaviour of other traders had an edge. So I fail to see what useful information his trading put into the system. How his actions resulted in better (or worse) allocation of money between companies. How the (very serious) game he was playing was anything but zero sum.
Maybe there's an error in my reasoning. In this case, I'd happily accept downvotes, but please tell me where I have gone wrong. Not understanding why one's post is being scolded is just frustrating. Like failing an exam and being told to have a hard look at oneself. "Yeah, I get that, but what should I look for?"
I do get the value of trading, investing, and some form of arbitrage. Heck, I don't have to search for the farm to get my milk. That's a service, and it does deserve a reward. What the author did is a bit different:
> Most of the market volume was other bots that would only execute a trade with me if they thought they had some statistical edge.
I understood it meant "they would trade with me only if they think I was the sucker". And of course, he would trade with them only if he thinks they were the suckers. It's not providing a service. It's fighting in a zero-sum economy.
Now he did say "most of the market volume". So there's a fraction that's not bots, and probably also a smaller fraction that does not even play the zero sum game, but instead does some positive-sum trading with the mostly-zero-sum players. But this interface boundary seems incredibly thin, compared to the internal zero-sum behemoth. That looks like a highly inefficient use of time, energy, and brains.
And even then, I'm not sure the zero-sum game provides any service to the rest of the world: zero-sum players base their models on the behaviour of each other, not on the actual performance of companies. That would add no new information to the system. At best, that only amplifies the effect of the few that actually predict company performance. And I doubt it does it well.
There are really only two actions in a market, providing liquidity (i.e. market-making) or taking liquidity (including arbitrageurs). Stretching the shopkeeper analogy, any time you put a price tag on an item you are taking a risk and exposing yourself. You slap a price on some bananas and then there is a tropical cyclone (it happened in Australia) and the price of bananas doubles. Someone smart swoops in (call them an arbitrageur), grabs your bananas and goes to the till and you legally have to sell them to him even though they're worth double now. You also have inventory risk, if you're holding a lot of bananas out back then there is the chance that they go rotten.
Every limit order (an offer to sell, or bid to buy) has an inherent risk that you should expect to be compensated for (or you wouldn't do it), and enhances the market. The more competition amongst people posting prices the better, it just means more liquidity (you can buy or sell as many bananas as you want) and tighter prices.
The liquidity taker on the other hand is all about exploiting a misprice. Say, Intel release earnings and are down 10%, but the price of the highly correlated ARM hasn't moved yet, if you are the fastest you can take advantage of some poor person who still wants to buy ARM at the same price as before this information was known. Keep in mind here, a price can move without any trades happening (or goods changing hands). The shopkeepers can whip around and change the price tag on their bananas if they are fast enough before anyone buys them.
It is harder to justify the liquidity taker - what marginal advantage is there to having the price of ARM react to that news in 2 milliseconds rather than 20? They force the transmission of information yes, but why is it better to have it happen a bit marginally quicker? On the other hand, how can you have a realtime market without them? It keeps the providers on their toes, and there will always be someone fastest to react.
Incidentally, providing liquidity must have been the chief function of the OP's algo - 4000 trades a day doesn't work for a liquidity taker that has to cross the spread. He isn't getting 2000 mis-price signals a day to swing at, good enough to justify crossing the spread. DAX traded 120k contracts yesterday, he would have been averaging 2-3% of daily flow.
I agree that "why" doesn't matter, if you know "what". When you don't however, "why" is a useful predictor.
I understand the value of arbitrage, to some extent. In its current form, it's quite heavy, but I can imagine we're better off allowing it.
We should compensate for value, not for risk. The two are not always correlated. I know trading is risky, but that's not the point. If it doesn't create value, it shouldn't be compensated. And I suspect some forms of trading create little to no value at all.
By itself, closing a deal doesn't mean you provided value. It means the other party thinks you provided value to them. When both parties think that, either it's a win-win situation, or someone got tricked. (Delayed bids and offerings complicate this, but it's the same principle.)
If I got it right, the OP did what we could call "short term speculation". If you predict something will rise, buy from a sucker who didn't. If you predict something will fall, sell it to a sucker who didn't. And of course, pray you are not the sucker. Locally, it's totally zero-sum.
Now maybe the whole system facilitates real transactions (with non-speculators) at the boundaries? But even then, for a given volume your reward doesn't seem to be proportional to your facilitation power (which I have no idea how to compute), but to how well you manage to trick your fellow traders. I very much doubt that such twisted incentives can foster a useful, let alone efficient, system.
Now maybe that little speculator who sold you the share at $10, he goes and works the bid, he advertises that he wants to buy a share at $9. Someone fills him. It take 2 minutes of trading to work to the front of the queue (easily possible with an equity). During that time, he is exposed to the risk that the stock might spike up in price, a risk you are no longer exposed to since you got your desired trade done $1 cheaper and 2 minutes ago. Then if he succeeds he makes his $1 profit. It doesn't always work - maybe 60% of the time the price ticks up and he just breaks even. He pays exchange fees and clearing on both the in and out trades too.
In theory, you could get involved in all of this with your 500 share purchase. Try to work the bid, get a better price, etc... or if you cross you are effectively paying for a service. Every order posted in the market is a service.
How wide are bid/offer spreads these days? How can that possibly be a bad thing?
In your example, it looked like you wanted your 500 share for something else than selling them. So you're talking about "real transactions (with non-speculators) at the boundaries". So yes, those transactions are a useful service provided to you by the speculators. I'm not denying that.
On the other hand, your wording seems to imply that most transactions are of this type (non-speculator with speculator). As far as I understand the system, they are not. If the OP is to be trusted, the vast majority of transactions are between 2 speculators trying to outsmart each other. Do we at least agree on that narrow point?
My second point is that a such a transaction (between 2 speculators) is zero sum. Locally, because whatever the first speculators won, the other lost. And globally, because wherever the share is, it could still be sold to a non-speculator. Making the transaction between speculators doesn't make it any easier. I'd say it could make it harder, for the non-speculators now have to buy the share faster than the speculators (or pay extra to have traders do it for them). Seriously, where is the value in that?
Now maybe a good speculator tends to provide a good service to non-speculators. However, they are not selected by the quality of the service they provide. They are selected by their ability to rip each other off. How quality of service arises from such a cut-throat competition is mysterious to me. (Usually, we compete for quality of service directly, so the connection is obvious. Speculating is not the same thing.)
> Every single additional limit order in the market creates real value.
Bullshit. You don't cover the main case, where 2 speculators make a deal. Show me how value is created (not just transferred) in this case.
Many things in our economy can be described as a zero-sum game. Society only needs so much of any given good, and at the retailer level you aren't involved in increasing demand. Two corner stores might get into a price/advertising war, but let's argue that there is still only a fixed amount of product they can sell to a small community. So advertiser's get fat on fees (in our case, exchanges do). So what is the point you ask? Evolution, competition, it can only be good for the consumer. You still haven't disputed that by the way, the consumer wins don't they? How could they possibly have lost? It takes a tin foil hat to think that it isn't better and cheaper to invest now.
" they are not selected by the quality of the service, just ability to rip each other off" - people have a bit of a naive view that there are some sort of magical harlem globetrotter moves you can pull on an order book. The mechanics are dead simple I'm afraid. It isn't chess, not even checkers. You can't even see the other participants, everything is anonymous on the exchanges I've dealt with. Spoofing is illegal, and that is about as clever as it gets. Very little you can do to make your algo better will put it at odds with providing a better quality service.
To prove that, remember the two types of algo. If I write a liquidity provider that doesn't offer a tighter spread than the competition, I will miss out on trade flow and make little or nothing. If I write a liquidity taker that has an opinion that is wrong about the correct value of a security, someone else who is right will smash me. So effectively we are selecting for more liquidity, tighter spreads, and more accurate prices reflecting available information.
But, if I got you right, the anonymity of it all mean we cannot separate the two… hmm… Then we've got to multiply the potential utility of the trade by the (quite low) probability of it being closed by a non-speculator. Still valuable, but much less. And I'm back wondering to what extent this is worth the (collective) effort. You did change my mind a little, though. I'll need to learn more.
> You still haven't disputed that by the way, the consumer wins don't they? How could they possibly have lost?
The consumer winning does count as creating value. However there are 2 parts in the retail store competition example: the part where they compete on quality, price, diversity… and the part where they pay fat fees to the advertiser. The first part benefits the consumer, the second just add inefficiency in the loop: if both stores could only agree to not use ads, everybody would win.
When Pareto Optimum and Nash Equilibrium are at odds, life sucks.