is it possible to distill a large model into a (even) smaller MoE model, like OLMoE?
2,512 karma · joined August 27, 2007
https://www.beckman.com/liquid-handlers/echo-acoustic-techno...
All they need to do is support external RAM so they can make a machine with up to 128 or 256gb of memory. It's easy enough, but for their reasons they have just chosen not to do it yet.
We have laws for a reason.
and even intramural NIH inventions get patented and belong to the government