Hacker News Reader

Comment by SturgeonsLaw | Hacker News Reader

Is it possible to distribute AI training like this so that people could donate their idle processing power to creating a truly open source unencumbered LLM?

View on HN

The issue with all these types of projects now is that computers are way better at idling now so there's no longer as much effectively free compute to be harvested. SETI@home made a lot more sense when CPUs consumed roughly the same power regardless of the load.

Yes. You'd have to package up slices of the training set and the prior epoch as work units and host them. Our hypothetical GPT@home clients download the training set, train just those examples, and then send back their gradients. The server then averages all the gradients together.

As far as I can tell this isn't any different conceptually from the distributed inference setup that PyTorch/Huggingface Accelerate/etc support. Of course those were built for cooperating machines owned by the same organization with lots of high-speed I/O. In an @home/BOINC setup, you need to package things up into work units that get sent out to other people's computers, and you don't get anything back until the end. So it's higher-latency and you have to worry about people cheating the system with bogus work.

I'm not sure if this would be more efficient or cheaper than just renting out a bunch of AWS spot instances with GPUs in them.

Batches need to be reasonably sized. Llms are usually trained for a single epoch. If you just average the gradients once, it will be awful.

The main problem I believe is that you would need extremely high bandwidth. For a 100B model you're looking at 400GB of gradients you need to send each batch. There's a reason why super fast networking is used.

Err gradients aren't independently additive like that right?

The Hivemind project is just that

https://github.com/learning-at-home/hivemind

It’s harder because gradient descent is usually done with near synchronous updates of the parameters across the whole model on all training hosts.

Yeah, the synchronous updates are a big deal. To get an idea of how much bandwidth this typically takes, Oracle has 1600gbps per node of interconnect with latency very low, guessing in the tens of microseconds. A really good home connection might have 1gbps of interconnect with latency in the tens of milliseconds. The big question is whether we really need all this interconnect -- GPT-JT[1] is a very promising step in this direction. The idea is that we just drop most of the gradient updates and it still works well. Unclear whether this will take off generally -- if it does it would be a huge deal, because 1600gbps of interconnect is very expensive.

[1]: https://www.together.xyz/blog/releasing-v1-of-gpt-jt-powered...

It's not clear that machine learning models consisting of weights are copyrightable in the first place. They are not the results of human authorship, but are instead essentially produced by a machine trying to optimize a bunch of values.

If you can get copyright on a photo just by pointing a camera and pressing a button you can probably get copyright on an AI.

I'm guessing you missed this discussion: https://news.ycombinator.com/item?id=35191206

No copyright on an AI if it was machine generated, which the training is. Great catch kmeisthax!

Reply on news.ycombinator.com