Distilling ModernBERT into a static model doesn't work

minishlab.github.io

5 pointsstephantul3 comments

Hello everyone,

here's some experiments we tried in distilling a ModernBERT model into a smaller model. Hope you enjoy!

Are most of the words in WordSim split into multiple BPE tokens? In that case I'd be curious if undistilled ModernBERT fares better on the task when it embeds all tokens in a word simultaneously. And whether results differ depending on how many tokens a word has.

Because alternatively, it could also be that the "no context" case is simply out of distribution for ModernBERT and it really needs some longer text to work with.

That's a good point! In many WordSim tasks, the words are pretty rare on purpose, so this is very likely. But it provides a good way of testing this hypothesis. Nice, we'll check that out.

Reply on news.ycombinator.com