There is still a very large open problem in how to federate large numbers of loosely coupled computers to speed up training "interesting" models. I've worked in both domains (protein folding via Folding@Home/protein folding using supercomputers, and ML training on single nodes/ML training on supercomputers) and at least so far, ML hasn't really been a good match for embarrassingly parallel compute. Even in protein folding, folding@home has a number of limitations that are much better addressed on supercomputers (for example: if your problem requires making extremely long individual simulations of large proteins).
All that could change, but I think for the time being, interesting/big models need to be trained on tightly coupled GPUs.
You’re only out of luck if each iteration is too compute intense to fit on one worker node, even if each iteration might be embarrassingly parallelizable, since the overhead of having to aggregate computations across workers at every iteration would be too high.
It would be great if merge-ability would exist. It would also likely apply to efficient/optimal shrinking for models.
Maybe you could dispatch tasks to train on many variations of similar tasks and take average of results? It could probably help in some way, but you'd still have large serialized pipeline to munch through and you'd likely require some serious hardware ie. dual gtx 4090 on client side.
merge-ability does exist and you can average the results.
If you have similar variants of the same task you can accelerate it more where the diff is.
You can't average on past results computed from historic base weights - it's linear process.
If you could do that, you'd just map training examples to diffs and merge them all.
Or take two distinct models and merge them to have model that is roughly sum of them. You can't do it, it's not linear process.
Let me clarify:
It's serialised, iterative, step repeating process where each step depends on output of previous one - aka linear process.
Where each step is non-linear transformation (gradient descent).
It's not distributable (over internet) task because it'd require transferring gigabytes of data (whole model weights) on each step.
To put it in other words - distributed task has massive input size and requires quick computation and tasks arrive very frequently - which means it can't be distributed over internet.
https://learning-at-home.github.io/
Getting the actual gradient descent to parallelize is more difficult because one needs to average the gradient when using data/batch parallelism. It becomes more a network speed than GPU speed problem. Or are LLMs somehow different?
But I’d love to see more federated/distributed learning platforms.