Can anyone explain how context length is tested? Do they prompt something like:
"Remember val="XXXX" .........10M tokens later....... Print val"
"Remember val="XXXX" .........10M tokens later....... Print val"
According to this it's remembering with 99% accuracy, which if you think about it is NUTS, can you imagine reading a 22x 1000 page books, and remembering every single word that was said with 100% accuracy lol
Another possible synthetic benchmark would be to present a list of key value pairs and then ask it for the value corresponding to different keys. Or present a long list of distinct facts and then ask it about them. This latter one could probably be sourced from something like a trivia question and answers data set. I bet there's something like that from Jeopardy.
Then you can ask it who is Tina's (great)^57 grandmother's twice removed cousin on her father's side?
It would have to be able to remember the context of the relationships up and down the document and there'd be nothing to key into as you could ask about any relationship.
It’s difficult to make that array longer because training time explodes.