Hacker News Reader

Comment by minimax | Hacker News Reader

The problem with the paper is not overfit. They claim to have run their simulation with out of band ("live") data. The actual problem with the paper is that we have no idea if their simulator is any good, which means that their result (89% return in 50 days) could be totally bogus. In other words, we don't know if the actual bitcoin exchange would fill their orders at the same prices (if at all) as their simulator does. A decent simulator for a high-frequecy strategy like this is not trivial because you have to incorporate all the exchange behavior (documented or not) into your simulator, and then you have to validate your results by comparing your simulated results to the results of some actual trading. The fact that they spent none of the paper on the details of the simulator makes me extremely skeptical.

View on HN

People so often overlook the role that execution plays in trading. As time scales shrink the impact of execution increases, and indeed many HFT strategies are not profitable without top tier execution (both in terms of fee schedules and technology). This is also one of the places that most academics fail when analysing a trading strategy. They make typical "assume a frictionless surface" types of assumptions that break down in a real market.

To your point, high-frequency simulation is damn hard and it is highly likely that they failed here and have tainted their results with bad simulations. Most of these papers would be well served to avoid dollars and cents and simply analyse the statistical qualities of their signal to their target over time. That is the first step in this business, anyway. Simulations only happen once the signal has been through the wringer.

I see no mention of the spread, costs or even ability to put on a short position, exchange lag variability (huge issue when simulating even on modern exchanges, let alone fly-by-night bitcoin markets). Additionally, it is very easy to find trend-following signals that "work" but break down when conditions change very quickly. That seems to make up most of the prediction, along with an order book imbalance signal that might be more stable.

I think a better approach would be looking at order book features and lags vs. other markets. The costs are high on BTC markets so it would be pretty tough to overcome those though. Anyone with experience to do this is probably doing it somewhere more lucrative. I think BTC markets only trade a few million USD a day.

About 35 million USD in trades on the public bitcoin-paired markets today. About 40 million USD today in total volume for all pairs on these exchanges.

http://www.cryptocoincharts.info/coins/info

More than when I last looked, but to put that in perspective, you could trade $40mm in a few price ticks in major currency pairs and CME avg daily turnover alone is > $100bn: http://cmegroup.mediaroom.com/2013-06-07-CME-Group-Sees-Reco...

Public spot FX markets and OTC trading do huge volumes too.

Some BTC exchanges charge like 60 bps per trade. Finding a signal to overcome that cost and do enough trading to make it worthwhile would be quite difficult.

Imagine if someone learned how to legally print money and their first action was to tell the world-- then I would be extremely skeptical.

Even with live data as a validation set (and ignoring your point about execution friction), it's inappropriate to naively infer expected future return from total return during the validation run, since asset price fluctuations don't follow a normal distribution. Very high volatility events happen far more commonly than a Bell curve predicts, hence a few lucky or unlucky outliers can skew your total. The Sharpe ratio which they used to measure risk exposure also dangerously assumes a normal distribution.

These guys did it right. You first build a simulator and publish a paper and then join a hedge fund for $$$$$$ and actually try the algo.

>we don't know if the actual bitcoin exchange would fill their orders at the same prices

This is the problem with many simulated trading algorithm "success" stories. No one actually knows the effect that the simulated trades would have on the real market, or how others trading against the algorithm would react to these trades. The numbers may be somewhat close to reality if the simulated trades are very small in relation to the trading volume for the asset. However, Bitcoin markets are small to begin with, which probably makes the results in this paper unreliable.

I remember exchange delays on mtgox reaching into the minutes! This algorithm breaks with mere seconds of delay.

From the article:

>Specifically, every two seconds they predicted the average price movement over the following 10 seconds. If the price movement was higher than a certain threshold, they bought a Bitcoin; if it was lower than the opposite threshold, they sold one; and if it was in-between, they did nothing.

If they are indeed predicting the price at X+10 seconds at second X, they have more than enough time to act on that info without having to do HFT.

That assumes there is no bias to their signal returns over time, which is extremely unlikely. 90% of the moves that go their way could happen in the first second, for example.

Many HFTs hold positions for seconds or minutes, and predict prices out over similar time horizons, but they'll lose their best winners to competitors if they aren't fast and adept at executing.

Reply on news.ycombinator.com