This feels like that often misinterpreted Einstein meme/qoute about humans only using a fraction of their brain power.
Is this only for inference though? could it boost training?
Hold on, you don't even need to freeze the branches completely: each node could train 1 branch on the path to its leaf node and communicate a change in the branch node to a central server, so you can distribute training without having to pre-freeze the branches. Still would need some pre-training though, and the splits would change slowly, and the attention mechanism could complicate things.
Currently distributed neural network training SETI@HOME style looks like a complete pipe dream that nobody is taking seriously. But a smart branching mechanism like this could suddenly make it possible. Folding@home reached 1.5 exaflops, which made it the world's largest supercomputer. Imagine the models we could train that way, they would far surpass whatever OpenAI or Google could train and would be public.
If this becomes true then it’s a game changer. I hope you are correct.
Does this also mean that it would be possible to train on parallel GPU-poor setup instead of needing lots of GPU memory / bandwidth on one computer?
Well, I'm assuming they'd use whatever they're given, so maybe the question should be "how much less time would training take on a 1.5 exaflops computer?"
A lot of clusters are totally homogeneous, at least within some very large domains, so for a given interconnect and a generation of GPU you know the maximum message latency, the peak sustained pflop rate, and so on but what often matters is some combination of the depreciation-cost-per-time and the watt hours per unit time, where you can sort of approximate both if you ignore the unfortunate realities, which then act as a multiplier.
For example, a problem is network issues - and not just scale - as the training sequence often involve billions of cycles of short compute-sync sequences which are bursty (e.g., all-to-all, barrier, compute, barrier, all to all, ...) but between which there isn't enough time to engage low power modes so you're burning $ due to slack and waste. This is true in different ways for a lot of training approaches.
You can approximate this, but it's so sensitive to data set size, specific training schedule, etc. that you won't be able to get the most important answer.
[1] https://github.com/learning-at-home/hivemind [2] https://github.com/bigscience-workshop/petals [3] https://chat.petals.dev/