Hacker News Reader

Comment by mg | Hacker News Reader

    This enables you to run larger models
    than you would be able to on any single
    device.

No further explanation on how this is supposed to work?

If some layers of the neural network are on deviceA and some layers are on deviceB, wouldn't that mean that for every token generated, all output data from the last layer on deviceA have to be transferred to deviceB?

View on HN

Yes, so you would have a vector about 8k values long to be transferred on each token generated.

You could do that easily with any modern network.

That's exciting. So we could build a SETI@home style network of even the largest models.

I wonder if training could be done in this way too.

Repo author here. That's correct. The embeddings for Llama-3-8B are around 8KB-10KB. For Llama-3-70B they're around 32KB. These are small enough to send around between devices on a local network. For a SETI@home style network, latency will kill you if you go over the internet. That's why we're starting with local networks.

how does it compare to https://github.com/bigscience-workshop/petals ?

Ah yes. At first, I thought that since it is all one-way forward-only communication, latency would only affect the time to the first token.

But I guess the final output needs to be sent back to the first node before it can continue. So if there are 50 nodes with a latency of 40ms each, each token would take 2s to process.

Yeah, unfortunately the autoregressive nature of these models slows it down significantly with added device<->device latency. However, you can still max out on throughput with pipeline parallelism, where you overlap execution. See: https://pytorch.org/docs/stable/pipeline.html

For generating synthetic data you could have a SETI@Home setup if you consider each home as a node that generates some amount of data. I mean, such a setup can be built with Exo, I wouldn’t suggest including it as part of Exo.

Out of curiosity, would you ever support training or at least fine-tuning?

Yes, that’s how it works (pipeline parallelism)

Interesting. Let's do the math ...

Let's say the model has 50B parameters and 50 layers. That would mean about one billion values have to travel through the wifi for every generated token?

I wonder how much data that is in bytes and how long it takes to transfer them.

It's not the parameters that are sent, it's the layer outputs. That makes for a few thousands floats per token

Woops! I would have thought the number of neurons roughly equals the number of parameters, but you are right. The number of parameters is much higher.

The embedding size is only 8k so while the parameters are 70B. So it's a huge difference

Reply on news.ycombinator.com