Except finance is supposed to be "serious". In most serious, legitimate activities, extracting money means you provided value somehow. So, what value high frequency trading could possibly provide?
Now I do have an idea where trading could be useful. For instance, a good old merchant doing arbitrage and making a profit is pretty useful: that's how different regions can specialize, do better than they otherwise would, and ultimately lift us from hunter-gatherer tribes where 60% males die a violent death, to our civilization now. And money, as evil as it may be perceived, is to date the best organizing medium humanity ever had.
Investors also have their use: by better allocating money among companies, we could hope to give more money to those who are better at converting it to actual wealth.
But.
The OP didn't make a ton of money because his model of the the companies had an edge, but because his model of the behaviour of other traders had an edge. So I fail to see what useful information his trading put into the system. How his actions resulted in better (or worse) allocation of money between companies. How the (very serious) game he was playing was anything but zero sum.
Maybe there's an error in my reasoning. In this case, I'd happily accept downvotes, but please tell me where I have gone wrong. Not understanding why one's post is being scolded is just frustrating. Like failing an exam and being told to have a hard look at oneself. "Yeah, I get that, but what should I look for?"
I do get the value of trading, investing, and some form of arbitrage. Heck, I don't have to search for the farm to get my milk. That's a service, and it does deserve a reward. What the author did is a bit different:
> Most of the market volume was other bots that would only execute a trade with me if they thought they had some statistical edge.
I understood it meant "they would trade with me only if they think I was the sucker". And of course, he would trade with them only if he thinks they were the suckers. It's not providing a service. It's fighting in a zero-sum economy.
Now he did say "most of the market volume". So there's a fraction that's not bots, and probably also a smaller fraction that does not even play the zero sum game, but instead does some positive-sum trading with the mostly-zero-sum players. But this interface boundary seems incredibly thin, compared to the internal zero-sum behemoth. That looks like a highly inefficient use of time, energy, and brains.
And even then, I'm not sure the zero-sum game provides any service to the rest of the world: zero-sum players base their models on the behaviour of each other, not on the actual performance of companies. That would add no new information to the system. At best, that only amplifies the effect of the few that actually predict company performance. And I doubt it does it well.
There are really only two actions in a market, providing liquidity (i.e. market-making) or taking liquidity (including arbitrageurs). Stretching the shopkeeper analogy, any time you put a price tag on an item you are taking a risk and exposing yourself. You slap a price on some bananas and then there is a tropical cyclone (it happened in Australia) and the price of bananas doubles. Someone smart swoops in (call them an arbitrageur), grabs your bananas and goes to the till and you legally have to sell them to him even though they're worth double now. You also have inventory risk, if you're holding a lot of bananas out back then there is the chance that they go rotten.
Every limit order (an offer to sell, or bid to buy) has an inherent risk that you should expect to be compensated for (or you wouldn't do it), and enhances the market. The more competition amongst people posting prices the better, it just means more liquidity (you can buy or sell as many bananas as you want) and tighter prices.
The liquidity taker on the other hand is all about exploiting a misprice. Say, Intel release earnings and are down 10%, but the price of the highly correlated ARM hasn't moved yet, if you are the fastest you can take advantage of some poor person who still wants to buy ARM at the same price as before this information was known. Keep in mind here, a price can move without any trades happening (or goods changing hands). The shopkeepers can whip around and change the price tag on their bananas if they are fast enough before anyone buys them.
It is harder to justify the liquidity taker - what marginal advantage is there to having the price of ARM react to that news in 2 milliseconds rather than 20? They force the transmission of information yes, but why is it better to have it happen a bit marginally quicker? On the other hand, how can you have a realtime market without them? It keeps the providers on their toes, and there will always be someone fastest to react.
Incidentally, providing liquidity must have been the chief function of the OP's algo - 4000 trades a day doesn't work for a liquidity taker that has to cross the spread. He isn't getting 2000 mis-price signals a day to swing at, good enough to justify crossing the spread. DAX traded 120k contracts yesterday, he would have been averaging 2-3% of daily flow.
I agree that "why" doesn't matter, if you know "what". When you don't however, "why" is a useful predictor.
I understand the value of arbitrage, to some extent. In its current form, it's quite heavy, but I can imagine we're better off allowing it.
We should compensate for value, not for risk. The two are not always correlated. I know trading is risky, but that's not the point. If it doesn't create value, it shouldn't be compensated. And I suspect some forms of trading create little to no value at all.
By itself, closing a deal doesn't mean you provided value. It means the other party thinks you provided value to them. When both parties think that, either it's a win-win situation, or someone got tricked. (Delayed bids and offerings complicate this, but it's the same principle.)
If I got it right, the OP did what we could call "short term speculation". If you predict something will rise, buy from a sucker who didn't. If you predict something will fall, sell it to a sucker who didn't. And of course, pray you are not the sucker. Locally, it's totally zero-sum.
Now maybe the whole system facilitates real transactions (with non-speculators) at the boundaries? But even then, for a given volume your reward doesn't seem to be proportional to your facilitation power (which I have no idea how to compute), but to how well you manage to trick your fellow traders. I very much doubt that such twisted incentives can foster a useful, let alone efficient, system.
Now maybe that little speculator who sold you the share at $10, he goes and works the bid, he advertises that he wants to buy a share at $9. Someone fills him. It take 2 minutes of trading to work to the front of the queue (easily possible with an equity). During that time, he is exposed to the risk that the stock might spike up in price, a risk you are no longer exposed to since you got your desired trade done $1 cheaper and 2 minutes ago. Then if he succeeds he makes his $1 profit. It doesn't always work - maybe 60% of the time the price ticks up and he just breaks even. He pays exchange fees and clearing on both the in and out trades too.
In theory, you could get involved in all of this with your 500 share purchase. Try to work the bid, get a better price, etc... or if you cross you are effectively paying for a service. Every order posted in the market is a service.
How wide are bid/offer spreads these days? How can that possibly be a bad thing?
In your example, it looked like you wanted your 500 share for something else than selling them. So you're talking about "real transactions (with non-speculators) at the boundaries". So yes, those transactions are a useful service provided to you by the speculators. I'm not denying that.
On the other hand, your wording seems to imply that most transactions are of this type (non-speculator with speculator). As far as I understand the system, they are not. If the OP is to be trusted, the vast majority of transactions are between 2 speculators trying to outsmart each other. Do we at least agree on that narrow point?
My second point is that a such a transaction (between 2 speculators) is zero sum. Locally, because whatever the first speculators won, the other lost. And globally, because wherever the share is, it could still be sold to a non-speculator. Making the transaction between speculators doesn't make it any easier. I'd say it could make it harder, for the non-speculators now have to buy the share faster than the speculators (or pay extra to have traders do it for them). Seriously, where is the value in that?
Now maybe a good speculator tends to provide a good service to non-speculators. However, they are not selected by the quality of the service they provide. They are selected by their ability to rip each other off. How quality of service arises from such a cut-throat competition is mysterious to me. (Usually, we compete for quality of service directly, so the connection is obvious. Speculating is not the same thing.)
> Every single additional limit order in the market creates real value.
Bullshit. You don't cover the main case, where 2 speculators make a deal. Show me how value is created (not just transferred) in this case.
Many things in our economy can be described as a zero-sum game. Society only needs so much of any given good, and at the retailer level you aren't involved in increasing demand. Two corner stores might get into a price/advertising war, but let's argue that there is still only a fixed amount of product they can sell to a small community. So advertiser's get fat on fees (in our case, exchanges do). So what is the point you ask? Evolution, competition, it can only be good for the consumer. You still haven't disputed that by the way, the consumer wins don't they? How could they possibly have lost? It takes a tin foil hat to think that it isn't better and cheaper to invest now.
" they are not selected by the quality of the service, just ability to rip each other off" - people have a bit of a naive view that there are some sort of magical harlem globetrotter moves you can pull on an order book. The mechanics are dead simple I'm afraid. It isn't chess, not even checkers. You can't even see the other participants, everything is anonymous on the exchanges I've dealt with. Spoofing is illegal, and that is about as clever as it gets. Very little you can do to make your algo better will put it at odds with providing a better quality service.
To prove that, remember the two types of algo. If I write a liquidity provider that doesn't offer a tighter spread than the competition, I will miss out on trade flow and make little or nothing. If I write a liquidity taker that has an opinion that is wrong about the correct value of a security, someone else who is right will smash me. So effectively we are selecting for more liquidity, tighter spreads, and more accurate prices reflecting available information.
But, if I got you right, the anonymity of it all mean we cannot separate the two… hmm… Then we've got to multiply the potential utility of the trade by the (quite low) probability of it being closed by a non-speculator. Still valuable, but much less. And I'm back wondering to what extent this is worth the (collective) effort. You did change my mind a little, though. I'll need to learn more.
> You still haven't disputed that by the way, the consumer wins don't they? How could they possibly have lost?
The consumer winning does count as creating value. However there are 2 parts in the retail store competition example: the part where they compete on quality, price, diversity… and the part where they pay fat fees to the advertiser. The first part benefits the consumer, the second just add inefficiency in the loop: if both stores could only agree to not use ads, everybody would win.
When Pareto Optimum and Nash Equilibrium are at odds, life sucks.