Donald MacKenzie on high-frequency trading
lrb.co.uk
lrb.co.uk
Many of the systems, fiber networks, data centers, and software that I touched are mentioned here. I've never heard the term "bit fucking" before, but the explanation of having to understand what happens under the metal is very true. You have to have a good understanding of software, compilers, assembler, CPU micro architecture, network protocols, and operating systems to be able to operate in this space.
I don't really know what else to add, except that I and many others I worked with in this business think that batch auctions are the solution. Although the work is technically challenging and interesting, I certainly don't like the arms race aspect of it, and I wish the resources (natural and human) used by this sector could be better allocated. One of my big concerns was the added CO2 emissions of constantly spinning programs doing no useful work - all in the name of minimizing latency. I did some research into reducing the carbon footprint of our code by reducing idle spin cycles. Alas, I never really had the time to get it working well before I decided to pack up and move on.
Happy to answer any questions anyone may have.
But I've never seen such a match before, so who knows.
Caveat: I'm not a trader but was more on the engineering side.
If you want to get rid of the speed element (I'm not sure it's a problem) you should explicitly allow people to pay for priority. The easiest way to do this would be to allow for extremely small fractions of price. So instead of having to pay for faster lasers I could just change my pricing structure to tack on .00001 on to the price.
1) When opening a "new tick", release data about trades in last tick, and start accepting new orders. Information about orders submitted to this tick is kept secret. Allow setting order price to practically arbitrary accuracy. 2) After tick closes, match orders in _random order_. That is, it should not matter at all on at which point in time inside this tick order arrived 3) repeat
Allowing arbitrary accuracy would help turn the competition from speed or stuffing the order book with huge number of orders to "spoof" the random matching, towards competition of just providing lower spread. I understand that it would still advantageous to hold submitting trades until very end of the tick, to take into account as much outside information (other markets, etc.) as possible. To counter this, maybe the exact length of the tick could be randomized also. What do you think?
For the advantage of removing the speed advantage, you get the downside of providing advantage to people that have the loosest risk profiles and you further complicate the audit burden such that it is impossible to follow.
Think of it from the perspective of a value investor that set a market order 6 months ago. Under the current system once their price becomes top of the book they have a virtual certainty of getting filled. Under a (fair) random match they have the exact same chance of getting filled as someone who literaly just put in their order.
Naively, couldn't all the craziness about HFT be resolved by enforcing a 1-second tick on the markets?
Like, right now it appears that trades occur instantly. Instead, queue requests and have the trades occur only once a second. Or 5 seconds.
Would this work? I'm assuming that either there's no motivation to solve the problem, or there are difficulties that I'm unaware of.
---
I've been wondering this about HFT for a while and this seemed like a good place to ask. Then I went back and finished reading the article, and this solution is proposed at the end. Curious about whether it would work on not.
Primarily because regulatory capture has, until now, overruled all attempts to reign it in. The banks and hedge funds engaged in this activity wield vast amounts of political power.
>Naively, couldn't all the craziness about HFT be resolved by enforcing a 1-second tick on the markets?
Not really. That just changes the game from "who can trade a picosecond before you?" to "who can trade a picosecond before the tick?".
An almost microscopic tax on transactions would prevent it, though while having a minimal impact on all other trading activity.
The lack of live market book data means that you won't need the ultra fast connections, so at least one part of the networking arms race becomes de-weaponised.
You still have the problem of it being advantageous to bid at the end of the auction, but this could be solved by pricing. Just like the exchanges give incentives to offer prices rather than to take them, they could make bidding at the start of an auction cheaper than at the end of it.
For comparison, Morgan Stanley alone made $8B in profit in the last quarter [1]. HFT is tiny, a drop in the bucket compared to the rest of the financial industry. Further, many big players in the industry are against HFT since HFT reduces their information advantage over smaller players [2].
The danger in a 1-second tick is that a) a massive change like this is hard to do [3] and b) it would likely increase volatility. A stylized fact of dynamical systems is that continuous feedback reduces volatility, whereas discrete step sizes increase it - witness how many numerical ODE solvers can blow up even while the underlying ODE is stable.
[1] https://www.google.com/finance?q=NYSE%3AMS&fstype=ii&ei=bVYI...
[2] http://www.chrisstucchio.com/blog/2014/fervent_defense_of_fr...
[3] Bloomberg, Excel and other calculators contain mathematical errors (e.g., "weekends are sometimes treated as a business day for the purpose of interest accrual") which cannot be fixed because the change would be too disruptive.
Which is to say, even a tinier drop in the bucket.
I'm not sure what your point is here? There's more to finance than HFT? There are other, bigger rip offs out there?
Bill Ackman is another giant player who is highly vocal against HFT and is known to spend donation money to attempt to make trades go his way.
HFT is the little guy in this story, the little guy doesn't get to create regulatory environments.
http://www.reuters.com/article/2009/12/02/us-highfrequency-i...
A Senator quoted against high frequency trading at least 5 years ago. Principal Financial is also quoted.
You have an ax to grind and an incomplete understanding of how the market and HFT work. It makes you look foolish.
My position was that no large banks or hedge funds or other concentrations of political power were against it until relatively recently. Which is completely true.
An individual senator is not capable of making vast changes to the law by themselves. Your belief that this IS possible is completely absurd.
>You have an ax to grind and an incomplete understanding of how the market and HFT work. It makes you look foolish.
Frankly I don't know how you're still defending barclays for selling an service promising freedom from HFT predators and then introducing HFT traders to it so they can collect additional fees.
Your belief that these traders were dumb and did not know what they were buying and therefore deserved to be lied to by Barclays is beyond retarded. They knew exactly what they were doing. You do not.
I mentioned several giant hedge funds (orders of magnitude bigger than the entire HFT industry) that have been arguing against HFT for a very long time. Further, the first article I found on a 2 minute google search, mentions a large fund that was speaking out against HFT in 2009. This is as simple of proof as I can provide that your position that HFT is not opposed by large finance groups as untrue.
"Frankly I don't know how you're still defending barclays for selling an service promising freedom from HFT "
Not once in this whole thread have I defended Barclay's and trying to paint me with that brush is either a sign of extreme obtuseness or terrible rhetoric. What I argue is that the reason Barclay's needed to defraud their customers is that their claim (and all other dark pool's) claim that they could provide a better trading environment by excluding HFTs has been historically proven as false. Traders are not dumb, they move to the venues that provide them with the best trading environment and that has continued to be venues that allow market makers EVEN IN THE FACE OF FRAUD.
No matter how many times you paint HFT as "predators" or "sharks" or as "spearing whales" you've yet to describe a single behavior attributed to HFTs that is in any way unethical or illegal.
[edit: having to work around HN reply rules, I apologize] To the reply below about BlackRock. Do you really think it is a realistic comparison to compare BlackRocks recent profit (2013) to one of HFTs best year (2009)? At the very least you should have mentioned that it was different years.
"Were this true, dark pools would not exist in the first place"
Except that there are all manner of product offerings that don't make sense for their clients. The easiest analogy in this case is extended warranties. "If extended warranties were so bad why do so many people buy them?" Of course this is a ridiculous argument. People buy extended warranties to hedge risk that is over rated due to warranty salespersons. The exact same way dark pools are.
" There is a link I posted below in response to you that describes such a strategy"
It does no such thing. It hand waves a lot of semi-technical non specific jargon around an idea that is easy for people to get upset about. I could just as easily write an article about how multicast networks were dangerous. It wouldn't make it so.
You mentioned Blackrock, whose net income last year was $3 billion.
http://www.wallstreetandtech.com/trading-technology/the-real...?
"TABB Group estimates that annual aggregate profits of low-latency arbitrage strategies exceed $21 billion, spread out among the few hundred firms that deploy them."
>What I argue is that the reason Barclay's needed to defraud their customers is that their claim (and all other dark pool's) claim that they could provide a better trading environment by excluding HFTs has been historically proven as false.
Were this true, dark pools would not exist in the first place. Traders would have stuck to the public exchanges and paid their (lower) fees.
>No matter how many times you paint HFT as "predators" or "sharks" or as "spearing whales" you've yet to describe a single behavior attributed to HFTs that is in any way unethical or illegal.
False. There is a link I posted below in response to you that describes such a strategy (spearing the whale) in precise detail.
Thanks for using 1 number from now and another number from 5 years ago. Are you willing to concede the rest of your rhetorical high ground?
Even if we discount the market impact numbers that you have to juke, I found a senator in 2009 speaking out against HFT. Can you find one defending it?
"Were this true, dark pools would not exist in the first place. Traders would have stuck to the public exchanges and paid their (lower) fees."
Maybe if the buy side firms didn't engage in fraud to keep their clients out of public exchanges that would be true.
Your invocation of the "regulatory capture" trope doesn't even make sense in this context. The big banks and hedge funds aren't the major HFT players, and many of them are opposed to HFT.
Remind me which of those banks are calling for it to be strictly regulated or prohibited?
I've heard that some banks are now selling "protection" against HFT sharks in the form of dark pools (which is often not really protection at all - see Barclays). Bearing that in mind I can't imagine that it would hurt for them to put out a marketing message that is disdainful of the practice while still quietly engaging in it.
http://dealbook.nytimes.com/2014/05/09/goldman-under-investi...
I imagine that WSJ op ed is probably damage control.
Barclays, like other dark pool operators, knew this: that you can run a pristine dark pool without "predatory" high frequency traders, and without much trading, or you can run a useful dark pool with high-frequency traders.
http://www.bloombergview.com/articles/2014-06-26/barclays-no...
tl;dr; No HFT, no liquidity, and people who want to trade take their business elsewhere.
[edit: If Barclays didn't need to bring HFTs in to provide liquidity, why did they subsidize HFTs? Matt Levine crunches the numbers - the biggest HFTs paid Barclays about $3M assuming a wild overestimate of the amount of liquidity they take, compared to institutional investors paying $4B.]
Barclays were ripping off their clients. It wasn't even a complicated scam.
Except that is untrue historically. The reason dark pools sell that privileged access is that they can't provide liquidity any other way and their clients find it better to trade in HFT allowed environments than in ones without liquidity.
IEX is trying something new, driving liquidity via scare tactics and publicity but we still don't know how that will work out.
"Barclays were ripping off their clients. It wasn't even a complicated scam."
Barclay's lied about allowing HFT firms into their dark pool. That is fraud but it has nothing to do with bad behavior on the part of the HFT. If you are suggesting that dark pools have inherent conflicts of interest that might be a more valid argument than "HFT IS THE EVILS"
Historically there was no HFT. Historically we have managed to have liquidity without it. Historically, the thing that brought us the greatest liquidity (something that is useful but vastly overrated) in the equities markets is electronic trading.
>The reason dark pools sell that privileged access is that they can't provide liquidity
I think it's cute the way that you're pretending that they were doing their investors a favor. They were selling dark pools to investors because investors WANTED to get away from HFT. If they wanted HFT infested waters they could have traded on the public markets!
What the fuck do you think they were DOING there in the first place? They happily sacrificed a sliver of extra (and largely unnecessary) liquidity for the chance to not get ripped off.
If value investors stop being rewarded for doing good research and correctly assessing the true value of a stock they will usually withdraw from the market (c.f. market for lemons: akerlof).
It should be pretty obvious why a stock market whose values are disconnected from reality is a bad thing, and why having a pool of investors who do good real world research and get rewarded for it is a good thing.
The difference is that in Barclay's case they lied about it and thus committed fraud.
THEY WERE PAYING TO BE IN A DARK POOL IN THE FIRST PLACE BECAUSE THEY WANTED TO BE AWAY FROM HFT. There was no "they realized".
Once they got away, Barclays tried to figure out if there was any BARCLAYS could extract more profits from these dark pools than the fees they were already paid. There was: allow HFT sharks to take advantage of their customers and collect fees for letting them do so.
If what you said were true (and frankly I cannot possibly see how you could believe it) they would have ZERO reason to commit fraud and lie about selling privileged access to HFT traders.
Of course they didn't. Professional market participants don't whinge. They go to venues that provide the services they want, namely cheap liquidity. Those venues were ones that allowed HFT market makers.
When you say "they got away", you do know that those market participants moved to open exchanges where HFTs "run rampant" and stopped paying Barclay's for their service.
"allow HFT sharks to take advantage of their customers and collect payment for letting them do so."
A) what do you mean when you say HFT shark? You keep using that term, in what I assume is a pejorative but it doesn't mean anything.
B) If the point was extracting more fees why did Barclay's subsidize HFT trading? They charged HFTs an order of magnitude less than their clients, that was a central part of the complain. If it was an example of allowing advantage taking, wouldn't the fee structure be reversed? That is, if I have a chance to fleece someone, wouldn't I pay up for that instead of require payment for it?
"If what you said were true they would have no reason to lie"
Of course they had a reason to lie. They claimed that they could provide a high liquidity environment without letting market makers trade in their dark pool, despite the overwhelming historical evidence to the contrary. Admitting that they were wrong brings their whole value proposition to buy side investors into question and delegitimizes their dark pool.
If they prioritized liquidity above all else, as you appear to think they should, they wouldn't be using a dark pool in the first place. They would have been using a public exchange.
>When you say "they got away", you do know that those market participants moved to open exchanges where HFTs "run rampant" and stopped paying Barclay's for their service.
Yes, I'm sure they all left after it was revealed that Barclays was lying to them and allowing HFT predation on their trading activity.
>what do you mean when you say HFT shark?
An HFT trader that parasitically extracts value from a non-HFT traders - often institutional investors (e.g. pension funds) putting through large orders. The link I put below describing a 'whale' to you explains just one parasitic strategy.
>If the point was extracting more fees why did Barclay's subsidize HFT trading? They charged HFTs an order of magnitude less than their clients
A) Subsidizing != Charging lower fees B) Citation, please (that OVERALL fees were higher, not just fees per trade - HFT will obviously get bulk discounts).
>that was a central part of the complain.
The central part of the complaint was that the service was advertised as HFT free when, in fact, it was not. The marketing literature is all out there saying this.
No, I think they should prioritize execution costs above all else. It turns out that fees then liquidity costs are the biggest variables in determining execution cost. Public exchanges with low fees and high liquidity provide the lowest execution cost, no matter what the dark pool salesmen claim (fraudulently or not).
"Yes, I'm sure they all left after it was revealed that Barclays was lying to them"
Actually the complaint shows they were leaving before the lying started. This is what you are missing. The lying was a response to the natural market condition that execution costs are lowered in environments with HFT participants. That Barclay's had staked their reputation on that not being true is what led to the fraud.
"The central part of the complaint was that the service was advertised as HFT free when, in fact, it was not."
And that is fraud. The book should be thrown at Barcaly's but that doesn't add any insight into the arguments surrounding HFT, other than in real world conditions, even buy side clients prefer venues with HFT provided liquidity, even in the face of fraudulent marketing claims.
But the high-frequency aspect of it is separate from the algorithmic-trading aspect of it, so one can criticise high-frequency trading for its uselessness, while accepting that algorithmic trading is hugely beneficial.
It turns out that the vast majority of HFT are algorithmic market makers. Why is that? Because if they can trade more they are more profitable (if they are good at market making). This means that they can make less on each trade driving down the bid/ask spread which is good for everyone. Further, if they can reduce their risk by trading fast, then they don't have to price that risk into the spread once again making the price better for everyone.
You can't separate hft from algorithmic market making because it is a central driving premise of the industry.
Algorithmic traders compete on price: they aim to offer better prices, and narrower spreads, to liquidity consumers.
High-frequency traders compete on the speed, with which orders are placed. The parasitic traders are the ones who profit not by providing you with a service, but by taking advantage of your inability to place and remove orders as quickly as they can.
You can do both of these at the same time, but the two competitive advantages are different and separate.
Market making is what all this activity is supposed to do, because it is a service provided by the liquidity providers, through the exchanges, to the liquidity consumers.
I think the distinctions are clear enough.
If market maker A wants to compete on price and provide liquidity at $10.003/10.049, too bad. That's illegal. He must compete on speed instead.
I wrote some tutorials on mechanics of this a while back, you might find them useful:
http://www.chrisstucchio.com/blog/2012/hft_apology.html
http://www.chrisstucchio.com/blog/2012/hft_apology2.html
http://www.chrisstucchio.com/blog/2012/hft_whats_broken.html
Unless the allocation is pro-rata, but it usually isn't.
Price and speed are closely intertwined.
It's good to mix in a large dose of almost-zero-profit market making trades in with your more ethically and legally dubious trades. It helps cover your tracks - both from regulators and other market participants.
Please just come out and say what you think HFT is, how it works and why it is ethically or legally dubious, because your claims in this thread border on the hysterical.
In general, if you know important information about someone else's positions, you can sometimes use that information to profitably trade against them.
>A classic aggressive strategy involves hunting and trapping “whales.” It is a good example of aggressive HFT strategy.
I apologize for not explaining terminology used by the financial community in laymans' terms.
After reading this article your assertion is that it is unethical and/or illegal for market makers to use standard market orders that are filled in legal and standard ways as price information signals, but it is not illegal and/or unethical for large market participants to hide their order flow with specialized orders and/or venues?
It is describing a commonly used HFT trading strategy in objective terms. Your ad hominem attack on their credentials and agenda does not change this.
>After reading this article your assertion is that it is unethical and/or illegal for market makers to use standard market orders that are filled in legal and standard ways as price information signals, but it is not illegal and/or unethical for large market participants to hide their order flow with specialized orders and/or venues?
What was just described is detecting a large institutional order through the use of 'pinging' trades and then effectively taking advantage of that knowledge by trading ahead of it. That is NOT market making.
It is not strictly speaking insider trading or front running either, but it combines features of both practices.
An ad hominem attack assumes that my attack is based on an irrelevant fact about the arguer. That the arguer has no experience in, and cannot articulate a common description of the issue in question is not an ad hominem attack. Just like saying that I have no basis in describing rabbinical law and should therefore not be determining what is and is not kosher, me saying that your source doesn't know what they are talking about when it comes to electronic trading is what is important.
"What was just described is detecting a large institutional order through the use of 'pinging' trades and then effectively taking advantage of that knowledge by trading ahead of it."
What is described in that article is 1 market participant, using a well known and perfectly legal market means to send in orders that they want to trade and then extrapolating from the completion of those orders, information about another participant that is using highly exclusionary, extra-market venues (dark pools) to hide their desire to change the price of an instrument.
That you think the market participant that is most disadvantaged in this relationship should be penalized more says more about your biases than anything else in the market dynamic.
https://www.nyse.com/publicdocs/nyse/markets/nyse-arca/NYSE_...
http://nasdaqtrader.com/Trader.aspx?id=PriceListTrading2
Ask yourself: why would the exchanges pay market makers to provide liquidity?
You can make an argument that HFT market makers stop providing liquidity during strange market conditions, but they do not "remove" it.
There are several arguments against it:
1)liquidity is good(1), more liquidity is better; why artificially limit liquidity, just let people provide as much as needed at as low price as market dictates;
2)Options are good(2), limiting liquidity increases prize of certain options (as issuer needs liquidity to offer options and the less liquidity there is the more expensive issuing those becomes, the effect might be small between 1s and 0s but it's still there)
3)Certain types of arbitrage would be less efficient. Arbitrage is good(3).
(1)-the concept of liquidity is that when you have some goods you want to sell, you don't need to invest time/effort/money to sell it; that in turns frees your time to do productive stuff; the effect is obvious in situations like retail stores buying from manufacturer (so manufacturer don't need to do advertising, building stores etc. themselves so they have more resources to actually produce goods); the effect doesn't disappear magically at 1 second scale, more liquidity (faster, cheaper) is always better for overall productivity.
(2)-options allow certain ventures which normally would be too high variance (forgive the gambling term) to pursue; they are form of insurance; reducing risk is almost always good but it costs to provide; the lower the prize, the better;
(3)-arbitrage is good because the prices of goods/stocks become close to true value; we want that, we also want to pay for that as little as possible; the more effective the arbitrage the better, why stop at 1 second, when you can have cheaper/better ?
This is not the whole story but those points are clear benefits of HFT. Liquidity is cheaper today than it ever was. Spreads are the lowest as well. Some people who used to make bank charging ridiculous spreads are now not making bank - this again is great.
The arms race argument is that resources go into something counter-productive (like stockpiling nukes) instead into something productive. In case of HFT resources (at least some of the time) go into something desirable. It is a race but it is a race to improve desirable things (liquidity, arbitrage). You rarely hear argument about "arm race" in situations like that. It's usually called progress. Usually we applaud progress in situations where service provides needs to compete to offer the best/cheapest product. In case of HFT this point is often somehow overlooked.
HFT aren't really adding liquidity since they only buy when they can sell straight away. So there was already a liquid purchase there.
Arbitrage, if something is cheaper elsewhere, the 1 second tick will still bring the price into parity.
No it doesn't. Let's suppose in your 1 second time tick you have more people attempting to buy at a price than sell at the same price? Who gets to buy and who doesn't? If it is FIFO then the speed advantage is still there. If it isn't FIFO it can be applied to non 1 second ticks as well.
"HFT aren't really adding liquidity since they only buy when they can sell straight away."
This is a completely untrue statement. Lots and lots of HFT hold inventory. Their inventory periods can range from seconds to minutes to weeks.
The impact is that by being fast market makers can reduce some of their risk, which means that they can price more aggressively. If they can't reduce that risk via speed, they will reduce it in other ways, namely pricing less aggressively. This will increase price spreads and decrease liquidity.
1) Liquidity is good, but remember who it is for. Uninformed traders, who want possession of the asset itself, will not care about a one-second delay because one second is too small an amount of time in human terms. Being able to trade a fraction of a second earlier is of no benefit. Being able to trade quickly on human timescales is the great benefit of algorithmic trading.
Informed traders, such as market makers, even market-making algorithms, also shouldn't care about a one-second delay. Their contribution to the market is the price. If you are a very good market maker, and you can quote prices better than anyone else in the market, your prices will be the better ones anyway, regardless of how the market is structured. And the profits are the market's rewards for quoting good prices, not trading quickly.
I would say the general idea is that liquidity providers and liquidity consumers compete, above all, on the price, and a one-second delay does nothing to hurt their incentives.
2) Again, I think it's quite obvious that on human timescales, on which actual economic decisions are made, there is no benefit to information arriving just a fraction earlier than it would otherwise. If you think of a market's purpose as revealing the price of an asset, it really shouldn't matter if the price is revealed 1ms earlier. What matters is that it is revealed at all, and revealed correctly with as many market participants as possible.
3) Because arbitrage becomes better when it contributes information to the market, not when it contributes information to the market 1ms earlier than another arbitrageur.
I think your points are completely valid arguments for algorithmic trading. Algorithms providing market participants with better prices than humans are indeed better for everyone. But for high-frequency trading, with the emphasis on high-frequency, I don't think this is very convincing.
I think the other poster countered it well:
>>The impact is that by being fast market makers can reduce some of their risk, which means that they can price more aggressively. If they can't reduce that risk via speed, they will reduce it in other ways, namely pricing less aggressively. This will increase price spreads and decrease liquidity.
Human investors may not care about getting stuff 1ms faster but they should care about getting the stuff cheaper (or selling it for better price).
>>Again, I think it's quite obvious that on human timescales, on which actual economic decisions are made, there is no benefit to information arriving just a fraction earlier than it would otherwise.
Well, yeah, but again more stuff happens in one second than in one milisecond. It means there is more risk of more bad things happening. If there is possibility of more bad things happening the liquidity providers need to price liquidity higher to mitigate that new risk.
Only the computers care about those timescales but it's by fighting it out there that they can give you the best price.
If ISPs compete offering bigger and bigger bandwidth and faster and faster ping then it becomes pointless at some point again as only thing new competitor achieves is taking place of the incumbent without contributing significantly to what customers get. 100GB/sec or 1TB/sec, who cares. 1 second tick is similar to 100GB cap - supposedly there is no point at making the technology more efficient any more. Your argument comes down to: "The cost of providing new technology is higher than gains customer get and that is achieved by putting previous provider of the services out of business - hence waste".
It might be true or it might not. We just usually don't care much because that is one of the situations where free market/competition does well in practice. Those technological arm races have other benefits as well, you never know what other uses the technology is going to have or what kinds of scientific advancement it stimulates. One day you may have 35% faster ping from Tokyo to New York because instead of optic fibers the connection will use some of technology mentioned in the article.
The short answer is that HFT is like Wal-Mart: it's trendy and cool to complain about it, but in terms of providing better and faster inventory to ordinary customers, it works better than everything that came before.
Except Wal-Mart is hated for being (allegedly) bad for poor workers, electronic trading's victims have primarily been the Wall Street old boys' club. Why the public is convinced this is a bad thing is beyond me.
But he fucked up and bought the story the buy side scammers were selling him hook, line, and sinker.
It's such a bummer.
http://www.bloombergview.com/articles/2014-03-31/michael-lew...
The section near the end on FPGAs and programming styles was an eye-opener to me. I wonder how many other firms have taken risk checks out of their code to save some nanoseconds...
You say that like that's a good thing.
I know that a lot of rubbish has been written about HFT and algorithmic trading, but it's sad that no one seems to be able to write about its merits without starting a flame war.
A lot of the rules that markets operate by come from standards and regulations, not software design. The software design reflects the kind of markets exchanges want to set up, so it's not a software problem. The people who wrote the standards had specific markets and specific outcomes in mind when they wrote them.
That being said, I don't know if it is possible to create an ungame-able matching engine.
In trading, a quant's job is to outsmart or outspeed the other bank's team. They invest considerable brain power, effort and money into producing incremental improvements that are quickly rendered meaningless by the work of the team in the building across the street.
Another prime example is advertising, a zero-sum game where competing brands fight to undo other brands' brainwashing efforts and replace it with their own brainwashing. A waste of the time and energy of million of humans.
Some more ideas on the subject: http://www.sphere-engineering.com/blog/15-hour-work-week.htm...
Just be glad that some people's brains are bent on stuff of benefit to you, and concentrate, if you like, on what of general benefit you can do with your own brain.
1) there isn't that much money in HFT. It has an ever decreasing number of players working for an ever decreasing amount of profit. Other areas of finance such as hedge funds, traditional investment banks, and derivatives creation are much bigger and have just as many "smart" people working in them. Don't get me started on advertising...
2) HFT doesn't turn time into money. It buys & sells risk in the form of liquidity. One way that it does this more efficiently is by being able to get in and out of positions fast.
HN, predictably, has a huge bias towards the capital raising/allocation side of the financial markets, but this is not why most participants are involved. Hedging risk is another huge (probably bigger) reason the markets exist. HFT allow that risk to be hedged more cheaply and more predictably so they provide value to the rest of the market.
"Don’t touch the kernel. Don’t touch main memory … Don’t branch.’ (That third commandment means don’t fill your program with ‘if’ statements – ones with the generic form ‘if A then do B else do C’ – because they get in the way of a modern microprocessor’s capacity to do several things in parallel.)"