The "a3b" refers to its active parameters -- unlike 27b it is a mixture of experts model, so it runs much faster, about as fast as a 3b model, but needs as much memory as a 35b model! So good for unified memory systems like macs :)

It's better for VRAM poor people. I get 4-5 t/s with 27B and 20-30 t/s with 35B A3B.

35Ba3b is usable in plain cpu inference on a fast server, the dense model is MUCH slower. On GPU the 35ba3b is still around 2x the tok/s single threaded, which can be a good tradeoff for some applications.

MoE models have less active parameters in play at any particular moment, so they perform much faster on lower bandwidth memories (like Strix Halo or DGX Spark) and also use less memory generally.

Dedicated GPU memory tends to be much faster (either DDR6 or HBM). So if you can fit a whole dense model in it, you're probably better off with that.

anecdotes: 35B-A3B does want more memory, bigger model. But if you get it running it will be faster and more enjoyable to use -- text will fly by -- due to only 3B params being active, in my experience at least.

Runs faster on bad hardware.