Unfortunately it seem like most ML people are not really interested in trading, perhaps because it has such a bad reputation (which is IMO unjustified) - so they work on games instead :)
Unfortunately it seem like most ML people are not really interested in trading, perhaps because it has such a bad reputation (which is IMO unjustified) - so they work on games instead :)
You couldn't be more wrong on this. Stock market trading has the lowest barrier to entry of any endeavor. All you need is $1000 and Robinhood account, which you can open one on your phone in 5 min or less.
I've been following HN for a while, every time someone comes up with a trading algo or posts a link to algo, there's were hundreds of upvotes, lots of comments.
Quandl has many free, or low cost stock market/commodity datasets.
I'm not sure what you mean by a "simulator". One of the greatest challenge applying RL to stock mkt is precisely that the market itself is not a MDP.
There is a good reason trading firms pay a lot of money (sometimes millions) for fine-grained historical data from exchanges. It's not only about speed. For interesting experiments you IMO need L2 or L3 order book data, ideally somewhere on second or sub-second scales. That's not HFT (which is nano and micros), but somewhere in the "middle" - it's a different world than what you are talking about.
By simulators he means market simulators for L2/L3 data with a matching engine, latencies, queue positions, jitter, complex order types, etc. You can't simulate other market participants (at least not fully, but there are techniques to even estimate this based on live trading feedback), but there are still many things left that you can simulate in a realistic way during training and backtesting. Trading companies typically have their own high-performance simulators built in house. Some of these are incredibly complex. Good simulators can give you a huge edge and are absolutely necessary.
"outliers and events outside of the data, news" : these are precisely the stuff your models need to learn, and the fact that you consider them noise tells me most folks have no clue how to predict these "noise".