Watch High-Speed Trading Bots Go Berserk
technologyreview.com
technologyreview.com
Let’s piece apart some of the language here:
* “High-Speed Trading Bots Go Berserk” — doesn’t mean anything, they’re processes, they’re just following their algorithm.
* “…unnerving the financial markets.” — doesn’t mean anything, ‘markets’ don’t have nerves. In fact, I once wrote about the dangers of reification: http://blog.zacharyvoase.com/2012/03/31/reification/
* “activity becomes much, much more frenetic and erratic” — to a human observer. These algorithms don’t have concepts of ‘frenetic’ or ‘erratic’; neither does the universe.
* “…so complex that its potential systemic repercussions are literally unknowable” — suggests that systemic repercussions of anything were ever ‘knowable’. If they were, we wouldn’t have markets. c.f.: http://mises.org/daily/2608/
Don’t just read. Think. Thanks.
How convenient Nanex didn't include labels in the graphics they provided! [1] In fact, they specifically say that high frequency trading doesn't concern them, and are talking about quote spamming. Not. The. Same. The animated graph used in the article combines metrics of both, as far as I can tell, but Nanex makes it clear that High Frequency Trading is stalled or declining while High Frequency Quoting is on the rise.
Furthermore, the bit saying "activity becomes much, much more frenetic and erratic" is just... totally silly. Partly because of what you mentioned, but also because... well I don't know what else to say, it's just so utterly, embarrassingly silly. I'd wager the graph's movement isn't even nearly as 'frenetic' as the stock market itself.
In any case, it's quite clear that the author didn't even peek at the page he got the graph from.
Ugh. This kind of 'journalism' makes me sick.
[1] http://www.nanex.net/aqck/2804.HTML linked all-too-subtly in the article
Seriously? Market participants certainly have nerves, and it is perfectly clear to anyone without an ideological axe to grind that the term "market" is used here as shorthand for "market participants".
Don't just write. Think. Thanks.
The net impact on the market was negligible. If you buy on the timescale of days to weeks, you can completely ignore HFT. Here is a blog post with graphs demonstrating the market impact of various HFT events:
http://www.chrisstucchio.com/blog/2012/flash_crash_flash_in_...
Not if you have to deal with intra-day margin calls or stop losses.
The trouble with most HFT-based algos is that most of them rely on a set of common signals (regardless of how "diverse" their creators may claim them to be) and end up making the various markets and assets highly correlated.
So when stops trigger, they cause a massive cascade. This needs to be protected against at the exchange-level (edit: they do so via various mechanisms). Most bots usually don't have the sophistication to protect against tail-events.
pubsub is the hot new thing here in the valley - welcome to wall street, circa ten years ago.
asynchronous is the new synchronous? what is that - a new album by smash mouth?
if you completely ignore HFT, you'd think there would be way more programmers and mathematicians available for hire.
Can you clarify this? My interpretation is that if you discount mathematicians / programmers working in HFT, you would expect more of them to be in the market for jobs?
if you completely ignore HFT, you will ignore that there are many very qualified people working in that field, and therefore overestimate the number of those people available for you to hire.
HFT firms are solving the single problem of making their owners rich with the authoritatively, hand-wringingly repeated justification about how critically important that liquidity is to everyone and their grandmother.
i was just trying (inarticulately) to make the argument that even if the "only" cost of HFT is the massive amount of talent going into that field, it's still a huge cost.
Web development and low-level systems engineering are entirely different and neither one is particularly harder than the other. If anything, both are simply tedious.
And pubsub was old by the time wall street got a hold of it.
what HFT experience do you have that makes you think of those systems as toys?
a webpage that responds to requests in 10 milliseconds is fast. a trading engine that responds to requests in 10 milliseconds is a joke.
Not to brag, but I've built trading engines in the past year that can process 20 years worth of data in less than a ms.
Now I'm a founder at your usually derrided photos-sharing start up. Sure we could make most requests come out of the server in less than 2ms, but what of it? It's still going to take the user upwards of 1.5 seconds to load all of the content, and at least 100ms spent on the wire. Working at a consumer level startup, there's no point in optimizing the server yet when network costs are so high, and there's no point in improving the network when there's no ROI in it.
----
Answering to his question: > what HFT experience do you have that makes you think of those systems as toys?
I'd imagine that massive distributed systems are more complex to think through that HFT systems because the later are more monolithic due to speed constraints.
where i worked - a small shop with 10 engineers - we had 350 machines running in 5 different data centers in chicago, new jersey, and korea. each machine ran around 100 different processes, which were coordinated using a combination of several different pubsub systems:
- in house reliable tcp-based broker mediated (we called it pubsub3 )
- in house unrelaiable ip-multicast system (i named it blue ray because i thought that sounded cool)
- 3rd party implementation of a PGM derivative that we licensed for god knows how much
the engines subscribed to various topics on the system, processed market ticks (sent by marketdata publishers over blue ray) and fired orders at whatever instrument they were handling
the risk engines modeled risk scenarios inresponse to trading shifts, powering through the risk models our quant trader wrote in c. we had a 'calcparser' system that constructed a DAG rerpresenting the AST of the c code and distributed the DAG across several machines.
shit was nuts. i miss it.
Is there some reason that you'd need your Korean data center to constantly be talking to your US data centers.
The complexity is also in having to deal with limited resources, and factors outside of your control. Have you seen many bootstrapped HFT implementations? The budgets are just... different.
When you spend 10x or 100x hardware, get massively redundant backbone connectivity (connecting to a tier 1 core router directly, and not through an oversubscribed shared switch port that a typical bootstrapped startup would get), you end up with a much better controlled and somewhat predictable environment.
With a typical "web" (non-HFT) distributed system, you've got millions of "subscribers" aka users - and I don't think people outside of IT fully realize just how bad a typical user's [Windows] machine is messed up - and that was before they installed a couple of toolbars, adware, firewall, and AV products... And all of these subs are tying up your publishers in their own unique screwed up way, keeping sessions open, dropping packets, blocking ports...
I've designed global CDNs, and operated one for a decade - and the kind of randomness our distributed systems have to deal with are not always technical, even though we don't typically count microseconds.
As a developer, I'm familiar with the fickleness of software. What happens when we see HFT algorithms error in unison? What have we not thought about?
(TBD by your trusty government and regulators: what speed is fair and what is too fast.)
Why do you think this person didn't want or need liquidity?
And why do you feel that it's worse for a machine to sell it to him at a low price than for a human to sell it to him at a high price?
The managers seemed like your typical alpha-male..bullish, dominating and little regard for morals. Although some computer scientists that they had seemed brilliant and were genuinely interested in building safer, better algos. So i wouldn't be surprised if it's these wall st. types that are pushing the limits further even if deemed unsafe.
It's human greed. Those 'wall st. types' and those brilliant computer scientists are all in it to make as much money as possible. Those you looked up to as hero's were not corrupted by those you wish to look down on, they all went into it wanting the same thing. The environment changed allowing them to try and get it faster.
GS does bulk liquidations like this and typically takes a haircut of 2-5%, presumably based on a risk and liquidity assessment.
Safeties are paramount in the field. There's many layers of protections to avoid *ing yourself. Most of these are deployed in ways that are not terribly expensive on the 'critical path' -- that said, there are a LOT of people doing REALLY stupid things in the field. (As there are in any field, it's just more 'scandalous' when an error in a financial application occurs)
Algorithm implementation is largely trivial, and the high pricetags paid developers in the industry are usually for specific engineering skills along with the semi-requirement that developers be 'all-hands-on-deck' when the situation calls for it. (There's a bug in the new strategy...well you need to fix it before tomorrow's open etc) At least, that's how I see it.
exception-handling is something we took very seriously, because the consequeneces of a fuckup are so bad. in late 2008, our firm lost around 300k in ~2 seconds. one of the exchanges sent a signal that the interest rate was now 0, which, according to their spec, meant that something wasn't right. our system didnt' handle that properly, so the algos went nuts because hey money is free! the fuse system caught it a few seconds later, but not before we shot orders at everything in sight.
i wrote a system that handled federation and service location, and as part of the design, i constructed four state machines which were used to handle the four possible cases in the formal model we developed. i went over the state machines many times by hand, and coded unit tests for each possible (State, Input) transition, followed by a battery of stress tests.
these guys are good - but nobody's good enough to never make mistakes.
I find it a little misleading that the graph is presented as somehow being connected to market instability. If they are correlated, they haven't demonstrated the relationship clearly to me. Am I misreading this?
Also I fail to see how someone else's algorithm going nuts could impact regular Joe traders as the article also suggests. If a stock price starts plummeting because of a bug in a trading algo goes haywire, a savvy trader would look to their news sources and realise nothing has happened. Or if they are passive investors probably by the time they mobilise to panic sell it's already corrected itself.
> in 2008, a pattern starts to emerge: a big spike right at the close, at 4pm, which is soon mirrored by another spike at the open. This is the era of traders going off to play golf in the middle of the day, because nothing interesting happens except at the beginning and the end of the trading day. But it doesn’t last long.
Most of the general public and individual investors do not have clear insight into how HFTs work, or if they are beneficial or not. Most are inclined to be scared of crashes and erratic behavior traced back to HFTs. Trading off liquidity for a loss of trust in capital markets is a bad deal.
When investors lose trust in the system, it all falls apart.
Sounds like we could segment, rate, and pacakge these contracts up into insured "credit-default-swap" type-of vehicles and make money making/selling/buying/trading them.
http://www.zerohedge.com/news/presenting-rise-hft-machine-vi...
The real issue here is that -- whoever was in charge of those stock exchanges hasn't properly addressed this issue as we all can see now its been growing slowly but constantly since 2007.
I like that the side-effect would be a drastic improvement in OCR technology.
In the end maybe I don't care if these firms just transferring money between themselves but when people's retirement accounts and the fate of whole companies in their hands it seems there should be something preventing this (another meta-algo that punishes aggressive and pushy algorithm...?)
If that isn't enough to get you to test your software, nothing is. And if they're that crazy, who knows what damage they can cause.
They either need to figure out how to do it safely, or slow the heck down before they break the whole damn thing.
They are part of the infrastructure of the major markets; a company listing on NASDAQ or NYSE is required by the market to choose at least one market-maker who will be designated to always keep open buy and sell orders on the books.
As it happens, Knight does its market-making through computers; this replaces sweaty men who would stand in the "pit" of the trading floor and trade by waving fingers at each other (including, occasionally, the middle one). It looks like HFT in that computers trade quickly, but it is intended to be a liquidity-providing service to the market: there is always somebody willing to buy or sell anytime the market is open.
(In contrast, HFT does not require always-open buy and sell quotes.)
As for liquidity, since the bots only make trades with the goal of shaving off tiny little slivers of profit on millions of trades, why and how would one of those sweaty guys ever buy anything from them? Even if one wanted to, the opportunity to buy might pass in a couple of microseconds.
All in all, working in HFT or even the financial "industry" at large, is complete bullshit. No self-respecting, decent human being should do it. No matter how you look at it, your mission there is to help world-raping scumbag bankers and the like make more money. They certainly don't need it.
As for your other questions, I suggest you go read my blog posts on the topic. They answer all your factual questions.
Finance still, though? I bet it's difficult to let go of the fat paychecks.
> As for your other questions, I suggest you go read my blog posts on the topic. They answer all your factual questions.
Ah yes. I was just giddy with delight when you used Futurama character names in those examples. How could I not love everything HFT represents after that?
Your guesses as to my personality type and employment situation are just as wrong as your speculation about the nature of HFT.
That's easy for you to say. We're both human though. Humans are above all selfish, and have a hard time letting go of easy money.
So who knows, if I were some kind of math wizard living in the US, I might have ended up working for a bunch of sociopaths too. It's possible I'd be here on HN, rationalizing and defending HFT and blowing smoke up people's asses with cutesy link-bait-titled posts about it.
I didn't actually say anything about your personality type though.
We shouldn't even bother, but perhaps you'd care to point ouf my mistakes in describing "the nature" of HFT?
Unfortunately, the only time you really want liquidity is during a crisis, which is when all the HFT firms exit the market, since they're in it to make money, of course, not to act as a regulated and guaranteed market maker.
What value is there in any system where trades over "pennies millions of times per second" is provided at all to real world conditions/humans.
This is simply a tool by a few to make money in an invisible, unregulated and uncontrollable space.
This is simply not sane.
I have no clue at this point what the model should be - but HFT is something that is completely undoable for a human, though it can have impact on humans at large scale and should be thought of more critically before they can be deployed into production.
I say impose a moratorium on all HFT until a determination of their value, impact, risk etc can be done and published and debated widely.
This could, of course, be utter bullocks, and I have no way of tracking down the source, but it's at least an interesting idea.
without it, investors would not be easily able to change their minds about how much other investors think wildly complex financial instruments are worth!
think of the children!
by reducing the bid-ask spread, hft shops are adding more "certainty" to the market - but that certainty could be (and is) sometimes wrong.
in other words, we're making financial decisions faster, at a cost of adding more more false certainty.
if the economy is a computer, HFT is overclocking the shit out of the processor.
Certainly, brokers' commissions have come down significantly through price competition (empowered by automation of trading). This is generally a good thing for a free market. Some broker-dealers may disagree because it's grown harder and harder for them to be the middleman and collect fees.
[*] $0.01 is the minimum because sub-penny pricing is disallowed for 'most stocks' on 'most trading venues'.
People like you are the reason cryptographic algorithms still get banned in countries around the world.
Hav you not noticed the banking scandals in the past few years? Have you not noticed that NOTHING has been done about them?
Also, if you look lower down the thread - I stated that a moratorium on bot based HFT should be put in place until we can fully understand how to regulate these properly.
Equating me with dictatorial banning of cyptographics is ridiculous.
Read this, before jerking your knee any further. Think of it as turning on the bedroom light, showing the monster in the corner to be nothing more than a pile of clothes:
http://www.chrisstucchio.com/blog/2012/hft_apology.html http://www.chrisstucchio.com/blog/2012/hft_apology2.html
Financial markets have been moving towards zero latency trading since their inception. First, men gathered under trees or in coffee houses, within shouting distance of each other. They fought over the best spots. Then, traders became some of the earliest adopters of telegraph technology, running private lines from remote cities to the exchanges. Ticker tapes appeared in distant offices. Computerized price dissemination followed, together with electronic trading connections to the exchanges themselves. All in the name of getting information and acting on it before the next guy. It's the reason you, as an individual, can trade global markets cheaply and instantly. It helps you take control over your retirement and savings, rather than paying through the nose for some managed scheme where only a privileged few can access the markets.
Where is your arbitrary line where progress stops?
You are saying that by having a zero latency trading capability, financial markets are the sign of progress and it is an inevitability that, not only is desirable, but preferred.
That's fine, but not what I am am taking issue with is fully automated system of extremely high volume, low margin, trades that are conducted by bots and have no human interaction other than those that are profiting from them.
I admit I am not savvy enough in this area to argue about whether or not HFT provides more equilibrium to the price of a stock, but I don't see that as a wholly convincing argument as to why HFT is important and adds value.
I am skeptical of that claim, mostly due to my ignorance, so please educate me on the following: The claim that HFT rarely goes wrong such that "ordinary investors" are harmed; who then is the non-ordinary investor who is benefiting from the HFT, and what value are they providing in the markets, other than profiting from HFT.
It appears to be making the claim that HFT is a meta market that the ordinary investor shouldn't even be concerned with except in cases of extreme rarity where a flaw causes them some financial harm/risk.
Again, if this is true - what true concrete value to the world does this meta system provide? I simply cannot see it, and again, I claim awareness of my ignorance, so please explain like I am five.
Thanks!
It used to be that if you wanted to buy a book you would have to wander around town until you found a place that had chosen to stock it. If it was an obscure or specialist book you might have to drive to a different town. You might just be shit out of luck. Nowadays you search for the title on Amazon and it shows up at your door the next morning. When you are done you can sell it again on Amazon with minimal hassle. Amazon improved the efficiency and liquidity of the book market which is why they are rich, even though they are charging less overhead than the bookstores.
Your local corner store doesn't create anything. All they do is buy from manufacturers and sell to you at a markup. But you don't care about spending an extra few pennies when its 6am and you're out of milk. You just want breakfast now.
Liquidity providers make it easier to buy and sell financial instruments whenever you want. HFT enables firms to provide liquidity more efficiently and with less risk. Just like Amazon, they get rich by saving you money.