Sharding a 70B model across 39 Intel laptops

Hey everyone, we’ve been working on a project called https://cascadia.to which allows you to shard LLMs across Intel-based machines and perform inference across CPUs/GPUs/NPUs.

To get it functioning, we split models into shards and pre-compiled them to OpenVINO IR graphs.

Since then, we’ve been working on a lot of optimizations to make sharding and running models on Intel much more performant. Most of our initial improvements are thanks to the use of speculative decoding (with some interesting workarounds there) and also micro-batching and continuous batching to improve the total throughput.

We’re still in the early days, but from our work so far:

- We ran an 8B parameter model on two Intel PCs, serving two users concurrently via iGPUs at ~43 tok/s aggregate - After adding an additional PC and user, it reached 64.67 tok/s - We’ve successfully sharded larger models (e.g. with 70B parameters), and with Cascadia inference was 3.1x faster than basic sharding - And for fun, we got 39 Intel AI PCs, hooked them up via ethernet, and sharded a 70B model across them. Benchmarks aren’t super impressive yet (~1 tok/s on CPU), but we’ve been making a ton of progress there.

If you’re wondering why we’re doing this, there’s a few reasons. There is a lot of Intel hardware out there, and dedicated AI hardware right now is expensive.

Organizations that own Intel computers will be able to pool their computing power to run models locally. For hobbyists with Intel GPU rigs or Intel Xeon Server PCs, you likely want inference to be optimized for your hardware. Cascadia is aiming to be a runtime for anyone wanting to run AI on Intel hardware and squeeze the most juice out of their machines.

Cascadia is open source (Apache 2.0) and available now.

This is alpha, so we’re open to any feedback on the architecture.

Give it a go and let us know what you think :)