In case anyone is wondering where they should actually get data to be competitive, I can personally speak for these two:
* Nanex NxCore
* CBOE Livevol
I haven't purchased data from here, but I have heard it well recommended from people I trust:
* TickData
* QuantQuote
There are likely others, but bear in mind that the higher the data quality, the higher the price. For most strategies you'll likely want vetted intraday data, preferably at minute resolution or lower. Tick data is better, but it's going to be huge (my drives are near 100TB). You can reformat it into custom bar sizes if you have data on the actual trades/quotes.
If you insist on data at this price point, why not just scrape Yahoo Finance? There are literally GitHub projects that obviate this for you.
What I am trying to demonstrate here is that your strategy can perform significantly differently if it is using data on different time resolutions. If you find an inefficiency that can be traded by analyzing snapshot data at 1 minute bars, that might turn out to be nothing at all when you look at the tick data. If your strategy is profitable but you don't have real insight into the trades versus just the quotes, you might have unrealistic expectations for fills, or hold times, etc...and if your strategy is profitable, your data is the first way in which you can try to prove it's not correlated with the broader market's regime.
I'm not interested in your actual strategy, I just want to understand what you're doing categorically to see if there's any way it can possibly work instead of being attributable to a bull market. You should be able to talk about the type of strategy this is without discussing the actual signal(s) key to the strategy. This is just becoming kafkaesque for me. I'm legitimately shocked no one else has asked you about this just out of basic curiosity about how you arrived at your current methodology. You mentioned Flash Boys and tick data...but HFT is not the only paradigm for which you'd use tick data.
I'm trying to give you the benefit of the doubt here but from my experience none of what you're doing makes sense. You have to understand that the reason I'm giving you so much flak for this is because you're repeatedly recommending people buy financial data from ebay, you're being cagey about why your recommendation is sound, and your recommendation is utterly alien to the way in which people professionally trade using historical data.
If your data from ebay is actually, somehow reliable, then fine; but if it isn't, recommending it for people who are looking for actionable data is irresponsible.
I mean, I'm not sure I accept the premise that there's enough information there to find significant alpha - but if you assume that he's using a different data source for that, he can test the trading performance on close prices alone.
I recall an example given in a class I took. (I may be misremembering the details, though).
Some people were trying to apply machine learning to currency trading. They had a bunch of data. They normalized the data (a common step in machine learning), and divided it into training and test and validation sets, and trained their model. Everything looked great, and they were getting excellent results on the test set.
When they went live with real money instead of making the nice profit predicted, they lost a lot of money.
Their mistake? They normalized the whole data set up front. What they should have done is split it into the training, test, and validation sets, and normalized each of those individually. Normalizing before splitting compromises the independence of the sets, biasing the learning.
(I must admit I never did quite understand this. Normalizing is optional. As far as I understand if one does an arbitrary transformation on one's data as a whole that should not actually make learning go bad, at least as long as the same transformation is applied to all the input when you go live. So when they did a normalizing step on the whole data set, why wasn't that just like doing any other arbitrary transform? There is serious dark magic here...)
You cannot normalize the entire dataset at once; the normalization contains some parity effectively allowing the algorithm to cheat and see the future... but in real life we can't see the future.
The simple example that eliminates some of the "black magic" is that normalizing against the entire set lets the algorithm know what the highest and lowest points across the entire data set are - and knowing the lowest and highest lets the algorithm buy low/sell high for all of the known data.
In real life, that's sort of like having access to the answers when you're given the questions. Being trained in that environment won't do much good.
In the United States I'd expect this to be covered by Feist Publications, Inc., v. Rural Telephone Service Co., 499 U.S. 340 (1991).
[1] https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....
[2] https://www.law.cornell.edu/copyright/cases/499_US_340.htm
https://www.unc.edu/courses/2006spring/law/357c/001/projects...
A compilation whose selection criteria is, for example, "everything traded on NASDAQ" and whose organization is "order by date" seems unlikely to me to have sufficient creativity to qualify.
1. Almost all buyers are institutional funds or individuals with the means to trade as well-informed investors. To put it succinctly, they're in it as serious business, because it's incredibly expensive. They have no incentive to make the edge they just purchased for five - six figures public.
2. These vendors go to various lengths to protect the data, including steganographic "trap streets" to identify the account the data belongs to.
3. There are significant legal liabilities inherent to releasing the data without a redistribution license. You agree to these explicitly by buying the data.
My layman's understanding is the more actors that participate on the exchanges, the more volume, the more the exchanges profit. If this is true, then making the historical trading data would encourage more entrants onto the exchanges, and the exchanges would earn far more than selling the data to a few large institutions.
I saw that the CBOE also offers 3:45pm data before the spreads widen up (sometimes drastically) when positions get unwound.