Show HN: Lumabri – Run Moe Models on a P2P Swarm with Colibri

Without diving into an experiment myself, it would be amazing if you could add some stats or experiment logs, if it's not too much problem and you have them.

For example:

Given model XYZ, every assuming 5 donors with a uniform 32GB each, each forward pass shunts xGB over the link. Each pass takes nMS, etc. etc. Resulting in n T/s, assuming latency of n ms.

Do you have such stats? Or perhaps I missed them in the repo?

The biggest benefit I see is to enable RAM constrained GPUs to perform inference of large parameter models with surprisingly high throughput. Because only a single expert is resident, the memory to compute ratio over the network is limited only by the activations, not the weights. For an Moe like kimi k3 where active parameters are 103B, we might expect to achieve performance limited only by ~5 effective tok/s per Tflop and ~1 tok/s per 100GB/s.

The more members of the network, the smaller your resident parameters are required to be. I’m not sure whether we can split layer inference into arbitrary chunks, but if so you’d be able to increase memory throughput by storing everything in GPU caches.

Of course, we expect latency to be relatively high, but that’s a tradeoff that's fine for certain circumstances.

I’m not sure whether there are any issues more with this idea, but it’s a fun one nonetheless :)

All the iot devices contributing to matmul but within a LAN?

second, don't use vague names of models; point to the actual repos you're testing on huggingface. There's enough diversity and specialization that even if you're smart enough to know that colibri is doing something that's particularly applicable to a type of model, it's easy to get lost in all the acronyms.

third, this looks like a fun tool to unite the diversity of random hardware people have, which is always going to win for local inference.

J/k, this looks cool :)