Are most of the words in WordSim split into multiple BPE tokens? In that case I'd be curious if undistilled ModernBERT fares better on the task when it embeds all tokens in a word simultaneously. And whether results differ depending on how many tokens a word has.

Because alternatively, it could also be that the "no context" case is simply out of distribution for ModernBERT and it really needs some longer text to work with.