The reason for this is that the large companies won't talk to you unless you're trading at those volumes since they charge pretty high transaction fees. The smaller companies that will talk to you all have accounts with the larger companies and make money by increasing the spread between the bid and ask price they receive from the large companies. The expected return on a trade is usually somewhere in the region of 1-10 pips (where a pip is one ten thousandth of a dollar), with 10 pips being a really good return and 3-4 being more realistic. The smaller companies usually increase the spread by up to 3 pips in each direction on the price they get from the larger company which swallows up your return.
While millions of dollars per trade might sound like a lot the leverage in this market is also very high at 50-100 times meaning that with your 10k you could trade 500k-1M though on a sum that low you might get less leverage. However if you lose money on a trade you could eat substantially into your capital. So if your $1M trade goes against you by 10 pips you'd lose 1k.
I'm not ruling out making money on that market, but I'd say it's very unlikely unless you have a lot more money to start with. I am not an expert in this area though. This is just what I learned from the small amount of time I spent working with the company.
The neural network was a fairly normal feed-forward network with GA-inspired, but heavily modified learning process. It combined layers of neurons with memory and memory-less neurons. It is essential to keep the network size small, because a big network will just memorize the data you use to train it.
Also, I used several other inputs, such as the frequency of trades per minute, overnight rates for each currency, and stock market indexes for the relevant countries.
Finding historic data was the biggest problem. I used Bloomberg data from my university, but that was a pain in so many ways. There is NO free data for most of the inputs I needed. Good data from Thomson Reuters or something like starts at $5k/mo.
Overall, the model worked fairly well, generating about 10% annual returns even with a few handicaps I put to be on the safe side. This was about an year ago and the only reason I stopped working on it was that my partner in the financial startup quit. I'm sorry if some of my terminology is wrong - I haven't been working in that field since then. If you have any questions, I'd be happy to talk about this. My email is in my profile.
While they do make it up on the rest, you have to ask yourself if you can afford the costs (brokerage commissions and fees) of executing your strategy in that context (they all work for banks and hedge funds).
If you really want to get into this, you might be better off applying to a quant-based hedge fund.
A firm like that will have the capital and brokerage connections to make the game profitable.
Also, take a look at Ed Thorp's web site (http://edwardothorp.com/index.html).
He was one of the original quants and has many interesting articles on the industry.
At least an AI I would create to trade would probably not be worse than me.
It also seems you've backed off your claim that 50% of your trades are profitable (if you actually trade).
1) Limited market data: As an individual, you have limited access to market data. At best, you probably have real time top of book data but without paying for it, you almost certainly aren't getting a real time depth feed. Going off of only last trade data gives you even less to work with. In most cases free quotes are delayed making them worthless to day traders (although not longer term investors).
2) Limitations of market data: Even if you have a lot of historical data with a full order book, that is still of limited utility for simulations. The problem is that everything you do affects the market so any trade you simulate needs to be doing small enough quantity to not make your simulation unreasonable. How to deal with simulating market-affecting trades is a complex problem and the only way to truly know is to run your trades in the market.
3) For day trading, there is a lot of profit to be captured simply from the microstructure of the market (its moves back and forth, etc.). To capture much of this it helps to be fast but you won't be able to colo boxes at an exchange with only 10K.
Machine learning seems more suited to longer term investments and trading on a slower platform than making a quick profit arbing or scalping. You may want to consider this rather than day trading since not only does day trading require much larger amounts of capital, but its also extremely risky.
Also, if you plan to trade equities, enjoy all of the ridiculous SEC regulation (order marking, etc.).
Yes. The intraday movement of stocks is small enough that you'd need to be using a large fraction of your capital on every single trade in order to overcome trading costs; and based on the Kelly criterion you'd need to have unrealistically high odds of betting correctly in order to avoid going broke.
If you want more information on this, there was an IAMA on Reddit about 2-3 months ago, on a guy who claimed to do exactly this. It was incredibly informative, and went into a lot of the details that you ask for here.
http://www.reddit.com/r/IAmA/comments/9s9d7/iama_100_automat...
And $10K will almost certainly not be enough. You'll spend all your money paying commissions or bid/ask.
Be careful - much of the historical data has been pruned to eliminate some data from companies that didn't survive.
I've heard that one of the major quant houses (Shaw?) can often guess what data set you used to test your strategy by looking at the strategy.
Plus, there's a selection or fitting problem. Your strategy may have worked well on average from 1972-1999, but that doesn't mean that it will work today.
And there's a velocity problem. There are pricing errors in old data to exploit. And, if you had today's computers then, you might have been able to find them fast enough because your systems are faster than what folks had then. However, today you're competing with folks who have faster systems than you do.
And realize that this is a fanciful pursuit which is likely to lose you money, because the guys you're going up against have $ millions, ns-scale lag times, free or near-free transaction costs, and PhDs in theoretical physics.
The institutions have to hold on for longer periods and can't really "scalp" trades per se given that the bid/ask volume would collapse if they didn't distribute and "average in" slowly overtime to accumulate a position vs. buying or dumping their entire position all at once.
If you can find an algorithm that can trade only a few times a day and ride stocks that has a volatility of 1% or more then even at retail commission levels you can still do pretty well as you slowly build up your account to trade strategies that require more churn.
if you really want intraday, think about news.google.com and http://rs2007.limsi.fr/index.php/TLP:Page_12 to automatically tweak the few k base parameters (SVM, associative memory etc).
http://www.bbc.co.uk/science/horizon/1999/midas.shtml
Options, Futures and Other Derivatives, J. C. Hull, Prentice Hall Int’l
Financial Calculus, M. Baxter and A. Rennie, Cambridge University Press
but the ML part is almost irrelevant: it is not the lever in the leverage (think CDO trade).
I call bullshit. A lot of the most successful applications in machine learning are modeled using neural networks.
To regularize a neural network, you use standard machinery that controls the capacity of the machine. The scale of the weights cannot be too large (this is a famous result from IIRC Haussler in 1992) so penalize the weights using an l2 (Gaussian) prior. This reduces the model complexity and the potential for overfitting.
Other than that, I'd like to second what several other people have already said: getting a job at a firm is a good way to learn, you're outgunned in terms of brains and money, and 10k is probably too little. Now, you can do a lot of learning for 0k while you save money, and every startup is outgunned by goog, msft, but that doesn't mean there isn't room in the ecosystem. Like a startup against an established player, you could find a niche that is going unexploited. What makes trading harder is the established players don't tell you much about what they're doing, so you don't know if you are doing something new or walking into a trading strategy perfected 10 years ago by DE Shaw. With a software startup, you have more information about what your competitors are selling, what their products are, how you might differentiate yourself, what's missing from the internet, etc. Software companies promote their products and try to get public attention; trading shops try to keep as many secrets as possible.
Does anyone have any experience in doing this?
So take everything with a grain of salt-- there seems to be a lot of advice going around by people who don't have skin in the game-- especially those who say it can't be done. Unless they've put money down to test this, then their opinion should be discounted.
For those saying commissions are to high for a shorter term timeframe: trading has been commoditized. Lightspeed offers .40 per lot, and IBKR offers around that given a certain amount of trading volume. Forex brokers make money off the spread, and so the commission that you pay is only from the bid ask. Since you are trading small amounts, you will get hit on the comission structure, but if you pay per share rather than per trade it won't hurt as bad. You also might want to look into equity futures as well-- very liquid, good tax advantages, and good commission structures.
Biggest thing about small sizes is that you're subject to the pattern daytrader rule-- you can't make >3 intraday trades if your portfolio is <25k. SEC rules.
I'm a discretionary trader, and not a system trader. I am currently developing some systems to go live but they are not ready as of yet.
I also know of many who are quants, and it's possible and profitable.
http://preview.collective2.com/
Strategies are independently monitored and ranked on the site, and can be switched from simulations to real accounts when you're ready. If your strategy performs well, it will be noticed and you can get paid commissions on it (potentially helpful for raising capital).
For FX and equity data, you might as well use Yahoo finance. I run a Slicehost server that scrapes it once every minute with no problem. It's not ultra high frequency but you can't hope to compete in that space anyway, as others have pointed out. You should start building a historical database now before you do anything else, so that you'll have something ready when you need it. You probably don't need years and years of it.
I worked as a quant for a while and would support the other comments that standard ML techniques are unlikely to outperform, but you might be the exception.
This is very different from most of the stuff I read about with neural nets. In most of the other books the idea to try and predict when markets are over sold or about to change so that you can be in on the ground floor.
I'm doing a webapp about card counting at black jack and the trend following approach seems a lot more like the reality of card counting. When the deck is hot or in your favor you bet more. Most people think that card counting is a super complicated exercise in combinatorics where the player is recalculating basic strategy in real time and able to prove that certain plays are the best.
Just something to consider.
I spent a bunch of time with Neural Nets and SVM and that data. Sadly I didn't find a profitable strategy. I would love to hear about anyone else who is looking in that area.
There are also arb opportunities between the various arcane bets, but that involves speed more than ML.
I ventured into retail trading (manual) and lost around 5$. Email me if you are interested, I can give you more details (email address available in my profile).
Anyone know of any particular markets where a small timer might have a better go at it due to insufficient volume to attract the bigger players, or for any other factors?
Besides, the bigger players could easily afford to risk a few hundred grand finding out whether they can plug the existing system into the small market profitably.
D.E. Shaw, Renaissance Tech, Goldman Sachs, etc. have already hired hundreds (if not thousands) of people smarter than you to do exactly this. They have more Math, Physics, Econ, CS PhDs than you can count/
So did BSC, AIG, LEH, and MER
In order for this approach to work, you need very precise market conditions. When markets are stable, your edge is zero and you would be better off investing in some sort hedged index (minimum effort, decent result). When markets are not that stable, it is when you can anticipate trends and get a decent profit for your effort. However, when markets suffer big unexpected turns is when fortunes change hands... and it is when your algorithms give you negative advantage! While you are warrantied to keep following the numbers wherever they take you,a savvy finance human agent has a chance to smell the blood and discretely head for the exit before everyone else. So, it is your call guys. In my opinion, this will not be a viable approach until we get some type of semantic-web-ly mine-able news feeder.