So how should we evaluate the quality of a paper on trading AI? I mean the authors might not have access to real data, but their ideas might still be good.
In the case of trading, any paper not tackling these issues head on is not likely to be useful.
As you go to shorter time scales you get more usable data, but then you also need to deal with other issues such as latencies/jitter, market impact, complex order types, order book queues, etc. It becomes a different game.