It looks like the sparsity of the matrix is going to be a much bigger challenge than the scale.
I understand about the focus being primarily on the approach, that makes sense; how are you intending to evaluate the results files?
Sparsity is good. The sparsity is the only reason that you can keep a matrix with this many dimensions in memory.
I understand about the focus being primarily on the approach, that makes sense; how are you intending to evaluate the results files?
For any submission, I will post for a random subset of vocab words each entry's 10 related terms. I'll then ask people to vote blind.
Otherwise, it might happen that a superior result would just show words that don't even have good results, while an inferior subset would get better covered words.