If someone legitimate put together a crowd funding effort, I would donate a non-insignificant amount to train an open model. Has it been tried before?
If someone legitimate put together a crowd funding effort, I would donate a non-insignificant amount to train an open model. Has it been tried before?
Given that the price since the original training effort has already dropped to ~$20 million, and that (a) the fundraising will take time, and (b) improvements are being made every day with regard to resource usage, you could probably get away with aiming for a much lower number.
Pulling a number out of my arse, I'd guess that training a comparable model will only cost $1-5 million in 12 months time, with the hardest part of doing so once you have the funds being acquiring the training data.
$65 million sounds pretty high though.
Worth noting, though, that it isn't just the computing budget that's missing here - it is also (and perhaps even more importantly) the high quality data to actually train the model.
People don't need to own A100s, they just need to be willing to be part of a distributed supercomputer by running a background app that downloads chunks of data, processes them, and sends the result back. The utility comes from having enough people participate (which worked quite well for SETI@home, but helping find "signals from outer space" is a little bit more interesting than "helping train an LLM")