I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.
And we’ve found ourselves hesitating on a direction because, at least for now, the primary use case in a useable env is private hosts you own in a mesh; which makes any sort of reward/ incentive structure somewhat unnecessary (for this setup).
When the public mesh becomes large enough and we get around to extending the existing “mesh governance” features is likely when we’ll add something like you’ve described.
For now, the public mesh is totally open with no restrictions or limitations.
A few questions:
1.) How does this handle privacy? If you're distributing compute this way then all actors in the compute graph will also know the sequence being computed.
2.) Any safeguards against malicious actors poisoning model activations?
P.s. for #2, you can probably do something like RAFT-styled interleaved computation. But this could get tricky unless you commit to a sharding scheme that makes it easier.
If so I can see it all dividing nicely, computation and data size wise and the only slowdown would be in search layer waiting for it's turn. If you pipelined it you could run multiple queries.
Is anyone doing best-of-n with a n stage pipeline running each query offset by one?
Numbers in this example are arbitrary. How does it actually work? What if the model’s number of layers is 33, or 34?
Is there a document explaining all constraints of this implementation?
We have a job that will take popular models, and dice them up and post on HF.
When that specific model is served, the layers will be shovelled around the available nodes on the mesh.