I just thought about this, so bare in mind that I don't know much of the technical implications of this, but:
Couldn't we train a very good model by distributing the dataset along with the computing power using something similar to folding@home?
Couldn't we train a very good model by distributing the dataset along with the computing power using something similar to folding@home?
Mind you, theoretically that is a limitation of our current network architectures. If we could conceive a learning approach that was localised, to the point of being "embarrassingly parallel", perhaps. It would probably be less efficient, but if it is sufficiently parallel to compensate for Amdahl's law, who knows?
Less theoretically, one could imagine that we use the same approach that we use in systems engineering in general: functional decomposition. Instead of having one Huge Model To Rule Them All, train separate models that each perform a specific, modular function, and then integrate them.
In a sense this is what is currently happening already. Stable Diffusion have one model to generate img2depth, to generate an estimation which parts of a picture are far away from the lense. They have another model to upscale low res images to high res images, etc etc. This is also how the brain works.
But it is difficult to see how this sort of approach could be applied to very small scale, low contextual tasks, like folding@home.
- Skynet