Mesh LLM: distributed AI computing on iroh
iroh.computer
iroh.computer
And I can't overstate how easy it was. The swarm page thing had a little "join" button and said to run "mesh-llm --auto". And I did. And it worked first try. That is such an uncommon experience I had to report back. It handled picking a model to serve, downloading it from peers, and to test it I chatted with the model I was hosting, I could see the GPU doing work, etc.
It might be more of an endorsement for iroh than mesh-llm, although I'm sure getting it to all work seamlessly took work on both sides. But to whoever spent the time and energy trying to make it seamless, consider the effort recognized!
Are we talking 1 token per second for a split model? Less?
Edit: Found a number. On the models list, Qwen 235B A22B says "MoE 235B/22B, proven at 16 tok/s across 2 nodes". They don't say what the nodes are and what network connection they have, but that's a respectable speed. Not quite comfortable for interactive use, but pretty close.
The idea is that you could take several machines without dedicated RDMA or NVLINK fabric and use them to serve a large model on hardware you own then share it with others.
I’m currently working on GLM 5.2 on my lab environment with around 10 tok/s on the same split.
We use a customized Q2 quantization that preserves sensitive tensors at Q8.
To reduce compute time per layer, we are developing a custom GLM DSA Metal graph.
While we are not yet approaching MTP, we plan to port our existing MTP implementations from versions 4.7 and 5.1 to 5.2.
Since GLM's MTP acceptance rate is very high for a single predicted token, we are exploring token prediction techniques to widen the predicted tokens and utilize parallelism for verification.
Ouch, about half of the memory bandwidth of a dedicated GPU though :/ Running LLMs on Apple hardware still doesn't make any sense to me.
Not necessarily, and I suspect there are plenty of configuration for which this isn't going to be the case. Let me explain why:
- when offloading the weights to RAM or NVMe, you need to transfer the massive weights from your slow storage to the GPU for each layer being processed for each token. And as such you are being bottlenecked by the transfer bandwidth (which is either the men bandwidth of your DRAM or the read speed of your disk)
- when using a distributed setup, the weights stay in the VRAM on each machine, the it's the GPU memory bandwidth that matters for the weights, and it's much higher than the two other bandwidth discussed above and as such the bottleneck isn't here. You need to tranfert data from a group of layers sitting on one device to the next one another device, but the amount of data is much smaller than the weights (we're talking about kilobytes of data, not gigabytes) so the network throughput isn't a limiting factor.
The limiting factor is the network latency: if you split your model between 4 devices, you'll have 3 times the network latency per token. If you're on a network with 1ms latency, that means 3ms of latency per token. Which means the theoretical upper bound for your inference speed without speculative decoding is 30tps (this theoretical limit assumes the computation itself is instantaneous).
So this is unlikely to be practical over the internet (too high of a latency) but on a local/enterprise network with speculative decoding it could totally work.
Edit: note that all of the above is about token generation, for prefill/prompt-processing the distributed setup will almost certainly win (because in this case, the network latency doesn't add up)
33 tps max token generation speed would be for 10ms of network latency in the above example.
As an aside, I saw this post mentions a public mesh, but I couldn't find any more information.
If so I can see it all dividing nicely, computation and data size wise and the only slowdown would be in search layer waiting for it's turn. If you pipelined it you could run multiple queries.
Is anyone doing best-of-n with a n stage pipeline running each query offset by one?
A few questions:
1.) How does this handle privacy? If you're distributing compute this way then all actors in the compute graph will also know the sequence being computed.
2.) Any safeguards against malicious actors poisoning model activations?
P.s. for #2, you can probably do something like RAFT-styled interleaved computation. But this could get tricky unless you commit to a sharding scheme that makes it easier.
And we’ve found ourselves hesitating on a direction because, at least for now, the primary use case in a useable env is private hosts you own in a mesh; which makes any sort of reward/ incentive structure somewhat unnecessary (for this setup).
When the public mesh becomes large enough and we get around to extending the existing “mesh governance” features is likely when we’ll add something like you’ve described.
For now, the public mesh is totally open with no restrictions or limitations.
Numbers in this example are arbitrary. How does it actually work? What if the model’s number of layers is 33, or 34?
Is there a document explaining all constraints of this implementation?
We have a job that will take popular models, and dice them up and post on HF.
When that specific model is served, the layers will be shovelled around the available nodes on the mesh.
Say a distributed inference for image processing, SDR, local weather monitoring etc. These will run on mediocre specs and produce dependable output.
Nicely done OP.
Its sort of a "P2P mesh" :-) Watch four instances of the harness running together and collaborating on checking the weather: https://www.youtube.com/watch?v=h1les1A3gcg
What I envisioned for how it works is fairly similar to this, QUIC can actually be more difficult to detect than it seems since it’s very dynamic.
AI Horde seems to be the biggest of them all. Their API speaks KoboldCPP text completion (not even chat completion). It seems that the community (or at least the active people) strongly prefer it this way because the API exposes more tunables than chat completions, which for roleplay use seems to result in better result. I don't know what else you can use AI Horde for anyway since all other use cases likely will require tool use. Just this week I was set out to improve their OpenAI bridge to support chat templates and response parsing. We'll see if I could get it deployed officially then you might be able to use it to code, although you'll have to use RP models.
I think Horde do have a lot more abuse prevention. Workers needs to have 1 week of cumulative uptime to be considered trusted to prevent brigading - users can opt into trusted workers only. Running a worker give you kudos which is required for >512 max tokens generations and also free requests get bumped to last.
Would this let me utilize that spare hardware to do anything meaningful with a self-hosted coding-focused LLM at a somewhat-decent token generating speed, or is this still a pipe dream?
We did have a version of things which used expert islands for MoE parallel alternative (I think that has been mostly scrubbed from the code). It showed early promise by having trunk+hot experts together, but as models got larger, it made both that very large on its own but also didn't seem to work as well (or we just weren't good at grouping experts), the idea being to really side step latency and route sessions to those islands. It felt like as models scaled it didn't stay smart (as diverse experts activated more than I thought they would). Could be our mistake (was an exciting possibility though - if you don't mind accepting some loss).
tldr; yes this seems a very nice enhancement for smarter batching/keeping things busy and seems like most larger models we look at are MoE!
I'd love to see some performance data.
It can be great on a local network though, especially if your workload is prefill-heavy (more text input to process than output tokens to emit).