Hacker News Reader

Comment by mililani | Hacker News Reader

Good to see at least that. Personally, 11 years is not enough time to see enough things to Black Monday or the tech crash. Hell, it's not enough time to see a REAL bear monster bear market. Also, what else is included in the database? So far, it just seems that prices are the only thing included. I looked through their documentation, and I can't find things like valuations and accounting data.

View on HN

11 years just stops short of the chaos of Feb 2000, where the NASDAQ lost 50% of value over the course of the year. Every dip, and people bought more thinking it was the bottom; and then lost it all.

I understand that the limited amount of data can make for some easy false positives on trading algorithms, but how relevant do you really think market movements from over 11 years ago really apply to today?

I hear a lot of people talk about needing more than 10yr of data for backtesting, and this is something that is often overlooked. The driving forces behind market data are unobserved and extremely complex, so you simply can't control for all the variables. If you can control for at least most of them, then you are in a situation where more data is NOT always better.

The one elephant that jumps to mind is liquidity. The mechanisms by which liquidity in the markets affect the end results are absolutely key to how your algorithm's interactions with the data will pan out. With that in mind, the methods by which we come by liquidity have dramatically transformed over the years, and market makers from >10 years ago operated in an entirely different way. That's my $0.02, anyhow.

Anecdotal evidence: I have first-hard experience implementing algorithms that chugged along just fine with 10 years of data for backtesting.

Exactly. After Reg NMS and other massive regulatory changes, it's an open question as to whether the post-2007 market system is the same as it was before.

You could ask the same question about many different regulatory changes, but Reg NMS is likely to be the biggest.

Daily data from 10+ years is still relevant,intraday I have found less so,probably due to algo trading/market making reducing the time horizon of momentum based strats and the change to the one cent minimum tick size ( if we are talking about US markets)

It's not perfect, but I don't think it's useless. Historical fundamental data are difficult to come by, the only public (not just internally used at a fund) source I know of is http://www.globalfinancialdata.com/Databases/USStocksDatabas... and it costs a lot. The vast majority of algorithmic equity trading uses just pricing though.

Why is this? Why is historical data so hard to find, when in fact the stock market is one of the most computer-heavy operations? These are numerical facts, so I don't think copyright applies either. It just surprises me that the data is so inaccessible.

The "numerical facts" as you say are not copyrighted, but every data vendor I've seen does explicitly prohibit you from making their data available to anybody outside of your license. The more inaccessible the data is, the better their corner on the market, the more they can charge. In similar fashion, real-time tick data is far more expensive to come by than EOD daily data. Real-time data requires slightly more computational overhead (transmission, storage, etc.) but really they just charge much more for it because they can.

In any case, I believe the scarcity the parent was referring to was for the fundamental data, i.e. earnings, value/share, dividend yields, cash on hand, etc. This data is harder to come by compared to the standard open-high-low-close pricing data, which is what most algorithms use (although not necessarily the best algorithms ;-)

Reply on news.ycombinator.com