small players will never have a chance to push the state of the art, as whatever optimization there is will also be applied at large scale with more money
Take a leaf from Seti@Home‘s book and try to come up with a distributed, volunteer-based approach to training an open source LLM. There is already an enormous amount of suitable ML hardware on end user devices.
Huggingface actually recently did this, but I think it's for inference on their giant models like BLOOM
Good point, but perhaps a leap could take small players into territories of language models that are large enough to be useful. GPT-3 crossed that threshold
A lot of SOTA comes from small players. It just isn't the case for LLMs.