Hacker News Reader

So they used a LLM with knowledge cut in mid 2023 to evaluate 2023? Seems like a classic leakage problem.

From paper: "testing set: January 1, 2023, to December 31, 2023"

From the Llama 2 doc: "(...) some tuning data is more recent, up to July 2023."

Removing the "Market expert" which uses OHLCV (Open, High, Low, Close, Volume) also drops the sharpee from 5.01 to 1.88 while also increasing the max draw down to 13.29% (v.s. 9.70% for the index). I'd be very surprised if the pre training of the base model was the only source of leakage...

Reply on news.ycombinator.com