Comment by kamranjon | Hacker News Reader

Comment by kamranjon | Hacker News Reader

Some of the most exciting engineering work is happening in the DS4 repo - and I'm watching it almost like a sports game.

When the DSpark paper came out[1] the next day we had folks attempting to implement, working together, validating their failures. Eventually their work being synthesized into a PR[1] that admits performance is not ideal. Something antirez alluded to in one of his videos (speculative decoding is a great boon, but mostly for large labs hosting and serving many requests at once, and maybe not so effective for local inference).

There's recent work into "directional steering"[3] that has made it's way into per-session directional steering overrides thanks to audreyt[4].

There's support for the new Hy3[5] model also thanks to audreyt[6].

There's Pre-M5 optimizations[7] in the queue thanks to ivanfioravanti who also helped with some of the initial M5 optimizations.

I haven't watched a repo like this since llama.cpp and whisper.cpp in the early days (though llama.cpp is pretty exciting right now with the SYCL improvements that are flowing in for the new Intel GPUs).

The DS4 repo is a really interesting place to watch folks who heavily code with agents collaborate together in a way that seems pretty effective. I've been really enjoying it.

[1]: https://arxiv.org/abs/2607.05147

[2]: https://github.com/antirez/ds4/pull/502

[3]: https://arxiv.org/html/2406.00045v2

[4]: https://github.com/antirez/ds4/pull/148

[5]: https://hy.tencent.com/research/hy3

[6]: https://github.com/antirez/ds4/pull/523

[7]: https://github.com/antirez/ds4/pull/555

Thanks! And sorry for not yet merging many of those. The problem is, I'm dealing with tensor parallelism for the CUDA and Metal-RDMA fork right now, so was not albe to care about PR / issues for a lot of time.

I don't want to sound overly dismissive, but for quite a few practical cases pipeline parallelism with micro-batches will be a likely win over tensor parallelism. Of course this inherently comes with a requirement to take batched inference seriously for local use, at least in special cases where this doesn't put too much of a requirement on memory capacity. By comparison, tensor parallelism is probably good wrt. making memory- and KV-cache hungry models like GLM 5.2 and perhaps Kimi genuinely viable in a non-trivial local inference scenario.

Oh that’s great - I’m generally more interested in single machine use cases but the RDMA stuff is super interesting.

I do wonder if you are planning to eventually delegate some of the merging responsibilities? I see some other projects follow a similar pattern when they grow to a certain size.

> speculative decoding is a great boon, but mostly for large labs hosting and serving many requests at once, and maybe not so effective for local inference

This is somewhat of a nitpicking point, but AIUI speculative decoding is worthwhile if (1) you have viable unexploited parallelism that can speed up the "verify" step compared to plain decode, and (2) it's profitable for you to load complete model layers into memory in bulk, not just a tiny fraction of active parameters. (2) tends to be true for large hosters, but not only, e.g. if you're running inference on a large dense model (think Mistral Medium) with SSD streaming and a single session (no concurrent batching), speculative decoding can be quite great.

It's also a spectrum, e.g. some MoE models are not very sparse and comparatively few concurrent requests might already span most of the experts, such that loading a complete set of layer parameters and doing speculative decoding to increase compute intensity actually becomes worthwhile.

The ability to use steering vectors seems really powerful. I know you can build your own, but are there any collections of prebuilt steering vectors? Or even just mentions of what kinds of steering vectors people are building for their own use?

Reply on news.ycombinator.com