High Frequency Trading Development Kit
stoneridgetechnology.com
stoneridgetechnology.com
They're also very vague on the software architecture or framework "kit". They don't say which feeds they support or if they provide any domain-specific data structures/standard trading algorithms like VWAP (slicing an order up over the day to give you "better" fills). I do see mention of an optional Feed Parser reference design, which I can't ascertain much value from. I am guessing it is mostly a generic kit for writing on their specialized hardware, and not specific to trading. There will be a lot of pain, maybe impossible to take vendor or open source libraries for market feed handling and compile it down using their embedded libraries. You'll be stuck doing tons of TCP/IP work, fun for hacking, but relative to the latency issues in your market data/order routing, it won't help you profit. Most people buy these appliances with included feed support.
With a big budget and co-location, this may be useful in bootstrapping some really low-level processing of market data, although there are a variety of hardware and software messaging solutions that would be a good alternative in driving latency lower if you have some money, Solace (HW), Tibco, 29West, etc to name a few.
I wish there was a hobbyist solution out there, but sorry, my initial impression says this isn't it. I would like more information, but maybe there are better hobbyist efforts out there. Interactive Brokers API isn't bad, and here's a full-blown real open-source trading kit that has a similar business model to MySQL. http://www.marketcetera.com/ Note: I feel that there are a few issues with this as well in some of the technologies they chose (JMS, Eclipse RCP).
Oh wait... I get it.
It's like playing poker, if you don't know who the sucker is......
The competition is leveling out because most players have gotten the basic tricks; this is just the next logical step in evolution. It has been proved in practice that fpga based trading systems consistently edge out normal cpu based ones(atm anyway). Most likely, in the future, companies will be forced to move to such a solution or get left behind.
This sort of investment strategy doesn't seem to be creating long-term market value, it's exploiting short-term edge effects in a zero-sum strategy. The arms race to pursue ever more inhumanly fast automatic trades seems to have no net value at all yet is consuming resources to impoverish the algorithmically and connectively poorer at the expense of the richer.
Markets seem to be favoured because they allow prices to find their natural level from supply and demand. How is this facility helped by permitting trades to be completed and then reversed faster than any human can ever perceive? This is algorithmic gambling on edge effects far more than a true market.
Please help me understand this better if I'm incorrect.
(edited for clarity)
True but humans have fail-safes and kill switches too, thus this is not a valid argument against HFT in favor of human market makers. At any time, humans could step in and provide liquidity during the flash crash and during other crashes.
Humans can, and do. But algorithms can't and won't.
I'll give you the long answer when I get back from work.
HFT market makers play an important role in those transactions. Specifically, they make it cheaper to buy and sell stocks by (1) tightening the bid-ask spread and (2) providing more quantity at each price, so that the average cost of executing an order is less. Not only that, when markets become tighter, they actually enable transactions to occur that would not have happened before. In other words, previously where buyers and sellers would NOT have traded because the transaction costs of crossing the bid-ask spread were too high, those two parties can now trade. Specifically, without HFT, there would be far fewer than 1 billion shares traded by utilitarian traders on a daily basis.
To address the other point of providing liquidity "when the markets need it most", let's take a step back. When you say that "market makers should step in to provide liquidity [for society's benefit]", you're implying that there's some externality to lack of liquidity in financial markets (if so, this is yet another reason that we need HFT on a daily basis). Suppose that this is the case: there is some negative externality to society when markets are illiquid, as is oft to happen when things go crazy in the world. During those times, volatility is insanely high because the risk of being in any position is also insanely high. Remember, market makers get compensated (on average) for holding risk that you don't want. If risk is higher, naturally, the compensation should be also. This is manifested in higher costs of execution: spreads widen and the available quantity at each level decreases.
If you want to force HFT market makers, which are private corporations, to step in to provide more liquidity, then you are forcing these companies to pay for that externality. In effect, they would take on huge risk for far diminished expected returns. That doesn't make any particular sense to me. However, if society as a whole has this view that some private corporations need to pay for public externalities, then that should be a matter of regulation. But if that's the case, why pick on HFT in particular? Why not force McDonalds and Whole Foods to give food to hungry people during famines? Surely, there is an externality to the food industry NOT stepping in during periods of extended hunger, "just when people need it the most."
[1] http://georgewashington2.blogspot.com/2010/10/yes-70-of-us-e...
I think that's definitely part of it. The issue is that people don't know how much high frequency traders actually make per contract. The amount is pretty small (on average). Typically, each trade takes on "bite-sized" risk and expects "bite-sized" returns. Do that thousands of times a day, and your bite-sized returns grow to something reasonable, but not ridiculous. Certainly, it's not on the order of how much other forms of proprietary trading make.
I think the other issue is that people don't understand that all traders, HFT or not, take on risk when they enter a trade. A lot of people use terms like "skimming off the top" or "making money for nothing". This indicates to me that they don't realize why market makers should even make money (on average). It's simply risk transfer, and that has a price (on average).
You're right though, identifying bona fide trends are pretty difficult. If it were easy, everyone would be doing it and arbing those trends away.
In a zero sum game where knowledge is power, you know exactly who is buying and who is selling - before John Q. Public knows. To compound this, you can fulfill orders using immediate or cancel (IOC) orders before they hit the markets - making it impossible for others to know how much or even whether you are buying/selling.[1]
In itself, there is really not much wrong with this - market makers have been doing this for decades, possibly even centuries. And as long as there is a level playing, where everyone sees orders at the same time, there really is no problem - an invaluable service in the form of liquidity is provided.
Having the advantage of being able to front run the markets is not limited to just market making, however. Limits are one of the ubiquitous features of stock markets - the ability to say, I will buy or sell X shares of this stick up until it hits price Y. This can be exploited easily by HFT/front running - you have 30 ms [2] to probe out where the limit is with small IOC transactions. In this way, you are able to meet orders at prices that are different than the currently trading price - effectively bypassing all competition.
Just to respond to one of your statements from the linked post:
> A common misconception is that high frequency trading is like operating a money printing machine... This is false. Peruse this Forbes article, if you will - http://blogs.forbes.com/afontevecchia/2011/02/15/bernanke-pu...
I am interested in any civil responses to this. I am not professing to be any sort of expert on HFT. [3] Normally I would not bother responding to posts like this, preferring to avoid talking to anyone, but your slightly condescending tone was infuriating.
1. This may not be correct, in any case, the IOC would only be of the required size and so not provide any indication of holding size.
2. http://rusingwithronestar.blogspot.com/2009/07/goldman-effec... Cannot find any mainstream media confirmations that it is indeed 30ms, but this number seems well corroborated.
3. I have used HFT and front running as synonyms because they seem to be inescapably tied together. Why the need to have picosecond advantages if you aren't trying to beat another bottom feeder to the prey?
edit: bah, I really dislike how HN treats newlines - I can't find any way to simply start a new line without a space in between the two sentences. Comes out like a block of text at 1366x768, or ends up looking too spread out.
Explain? Who is institution C? How are they able to violate time priority?
Assuming a moderate holding of stock, you are then able to manipulate the market up and down second-to-second by selectively fulfilling buy or sell orders and so causing supply or demand to pile up temporarily.
This is also very vague...
Sources: http://www.nytimes.com/2009/07/24/business/24trading.html http://www.nytimes.com/2009/08/05/business/05flash.html
Note that the SEC has not banned flash trading; it only proposed to do so - as of so far they have not done anything.
The second point - it is the core of how market makers work to provide liquidity, with the added advantage of being able to see orders first. Simple supply/demand dictates that selling stock will drive the market down, since existing sell orders are not filled and slowly pile up at lower and lower prices. Then beginning to buy shares will raise the stock price temporarily, at which point the cycle repeats. Being able to figure out the limits on the buy/sell orders is simply icing.
--- It appears that NASDAQ probably discontinued market front running voluntarily in late '09; I wasn't aware of that. I will have to rethink my positions - HFT as such isn't wrong in my opinion, although being able to make a profit 97% of the time (and having 3 straight quarters of only profitable days) is ridiculous, as is profiting purely from having a connection that is faster than what everyone else has.
In fact, that entire paragraph is pure specious reasoning. The main point that really drove it home for me was:
Being able to figure out the limits on the buy/sell orders is simply icing.
I can gather you have not actually tried to find a statistical trend in market prices. Simply put, it's difficult to do in practice.
In a zero sum game where knowledge is power, you know exactly who is buying and who is selling - before John Q. Public knows.
Since when was it the prerogative of every market participant to get quotes at the exact same time? There is a huge cost to colocation--it doesn't come free. The fact that trading has as high barrier to entry is a moot point. Starting a biotech company takes hundreds of millions of dollars in capital. Is it unfair to John Q. Public that Amgen and Novartis have competitive advantages in that field? Of course. Is it unethical? Absolutely not. When the steam engine was invented, was it unfair for guys who drove horse buggies? Of course. The new technology replaced him because it was able to do what he did, but more efficiently. This is no different. Day traders (I'm assuming that's what you're referring to when yous ay John Q. Public) are horse and buggy drivers. It isn't the fault of high frequency traders that they are that way. If John Q. Public wants to stay competitive, he has to change what he's doing.
If, instead, you're talking about your pension fund buying 1 million shares of Microsoft, then you have to realize that HFT is actually decreasing the cost of their doing that. Tighter spreads and more liquidity lower transaction costs for everybody.
Peruse this Forbes article, if you will - http://blogs.forbes.com/afontevecchia/2011/02/15/bernanke-pu....
I find it ridiculous that the article you cite makes no mention of HFT! In fact, I'm pretty sure that those profits are from their prop trading desks that don't do HFT. I can tell you that with certainty, because I know the ballpark of how much HFT firms make, and it's nowhere near that number. Please don't cite facts that are irrelevant to your point and then carry on as if they prove your case. I perused the article, did you even read it?
Having the advantage of being able to front run the markets is not limited to just market making, however. Limits are one of the ubiquitous features of stock markets - the ability to say, I will buy or sell X shares of this stick up until it hits price Y. This can be exploited easily by HFT/front running - you have 30 ms [2] to probe out where the limit is with small IOC transactions. In this way, you are able to meet orders at prices that are different than the currently trading price - effectively bypassing all competition.
Flash orders, if they're even used by anybody anymore, must account for some negligible fraction of market volume.
3. I have used HFT and front running as synonyms because they seem to be inescapably tied together. Why the need to have picosecond advantages if you aren't trying to beat another bottom feeder to the prey?
Market makers need high speed mostly to avoid getting picked off by other aggressive HFT firms. The prey isn't day traders, it's other HFT firms. Day traders don't operate on the same timescales as HFT. Just the fact that a day trader is using a mouse to enter and exit positions already precludes that possibility. If day traders are placing limit orders into the market and are upset that they can't cancel fast enough, they need to understand that their edge isn't in cancelling quickly, but purportedly something else (predicting longer term price movement?).
I'm all for a civil debate, but please checks your facts. Most of the sources you cite are irrelevant to this discussion :(
Thank you for your contribution.
This simply isn't true. There are many HFT firms started by hobbyists/amateurs. The company I work for started this way.
HFT is a field which has many niches and relatively low barriers to entry (thought certainly not as low as renting a linode).
To give an idea of scale / endpoint, a top-tier HFT infrastructure costs on the order of $100k/day. There are probably only a half dozen total of those in the world, though.
Strikes me that it would be a great research incubator for them to understand/learn new ideas and techniques and hunt for talent.
It is a good thing that your actual trading costs are not anywhere near those at the retail brokers you have in mind. 2%/day is an outrageously huge return. $1,000 at 2% compounded 250 times (approximate number of trading days in a earth year) is over $140k; you would return 14,000% annually.
The bottom line is that it is harder than it looks.
Some of these are purely manual trading but many have software APIs.
Yes, your alpha needs to include the ability to beat transaction costs... so if you can't pay for the [relatively exceptionally cheap] cost of several years of your chosen market's historical data, maybe you're not really serious.
It's fun to play at being a financier, but let's not pretend that access to market data is the significant barrier to entry which is preventing disruption in the world financial markets.
Historical tick data (quotes and trades) for US equities will run you $20k/year, minimum. US equities runs about 30-40GB/day so you are looking at almost 10TB/year. You need to be able to access the data quickly so you probably don't want to compress it and probably do want to duplicate the data, slicing and dicing it in different ways. Call it 20TB, minimum, per year of data.
Futures and options data will run another $20k/year, each, and futures has about as much data as equities and options has an order of magnitude more. That is another 100TB/year, minimum, per year, probably closer to 200TB.
How are going to to study these data? Even if you were able to operate at disk speed (which seems unlikely unless you have lots of disks and have duplicated and distributed your data amongst them) it would take you five minutes to run against one day's worth of data, or close to a day to run a year's simulation. If you consider more than a few cores in parallel, reading from disk is going to be seriously hampered.
None of this even considers the cost of the software necessary to support this effort: contending with corporate actions, tracking halts, etc.
This all adds up to a significant financial hurdle.
Not even close, if you consider the alpha for generating value. I know it's awesome on HN to think that generating 10M over 2 years entitles one to a "50M exit."... but that's not the way things work in finance.
20K is nothing... if one thinks it is than one is playing the wrong game.
There is also plenty of anecdotal evidence that as you try and increase the amounts of capital you're putting to work in the market that it becomes tougher and tougher to maintain an edge in the markets.
Disclaimer: I sometimes consult for HFTs but we don't use FPGAs (yet).
Don't things like real time linux or soft real time patches for linux help, though ?
I don't see how it could.
The CPU essentially drops everything it's doing when a packet comes in. Have that happen often enough and there won't be time left to do anything else, unless you start dropping packets.
Folks not using FPGAs use separate ethernet cards and enable kernel bypass and can do significantly better with that.
FPGAs canonical use case is still as very fast codecs. There are exceptions, but they're really only the very simplest of arbitrage strategies. And in order to make those work you need top-of-the-line network gear and engineers as well.
I don't know if the algorithms they use can be expressed easily as state machines though.
http://www.thedailyshow.com/watch/wed-september-30-2009/cash...
To those "hobbyists", what would you look for in a solution?
It's just too slow to move it from main to video memory and back.
Also OpenCL works best when working on large pieces of data at once, with High frequency trading often you want to work on very small pieces, 50 bytes, and process them as fast as possible. Batching up the data and working on it in a SIMD type fashion just takes too long:(
(1) high technical sophistication, really hard to do (for each of the servers, network, algorithms, historical data, machine learning, etc)
(2) done behind closed doors (always interesting to get a peek)
(3) technically 'exploits' the financial system in a way that provides certain companies with immense profits
There are plenty of suppliers who offer stuff in the market, for example replacing one half of a dual CPU system with an FPGA and in general some variant of 'pick a fast interconnect and stick an FPGA at the end'
I can't see why this offers anything more to the discussion over a product announcement.