Hacker News Reader

Comment by brhsagain | Hacker News Reader

I have a probably really stupid question but I'm going to ask it because I really don't know the answer.

What's stopping someone from starting a distributed effort to train an open source ChatGPT model, like Folding@home but for AI? Is it a technical limitation with how models are trained, difficulty of coordination, something else?

View on HN

The models are extremely large to ship around, sync, and load on consumer hardware over the internet. Maybe there are better ways to build and train a model for a distributed use case though.

The model that's used by Stable Diffusion is only around ~4 GB in size.

which would make traditional distributed training prohibitively slow.

You would have to sync the weights with residential internet speeds (10s of MB/s) instead of PCI-e Speeds (10s of GB/s).

If you want to add new stuff to Stable Diffusion - you don't have to retrain the main model from scratch. You can train it only on a few hundred images at a time and then add the resulting model as an extension or merge it into the main model. People train their models on a single celebrity or a single artist that way.

Other types of AI could be trained in a similar way.

Is this amenable to merging from thousands in one go or do you have to train -> merge -> train -> merge to not overwrite each other's trainings?

Sounds like we have our use-case for blockchain!

Training in parallel on more then one GPU is pretty hard. Distributing the training over many random computers and adding at least some kind of validation, seems to kill the performance. So it would always be easier to rend cloud GPUs.

But would it be cheaper?

That is actually irrelevant. I couldn't try out flan-t5-xl but the model flan-t5-model is like an inferior GPT but it is opensource. I don't necessarily think that the size of the model is that significant. What makes ChatGPT so impressive is that you don't have to mess around with the settings vs Flan T5.

More effort has to be put into obtaining good training data than just throwing compute at the problem.

besides the answers already given, the network is already the bottleneck on supercomputers on a local network. training that over a WAN would exacerbate that issue and make training even slower.

Hard to split the problem into small enough pieces

working on it.

together.xyz

Reply on news.ycombinator.com