Hacker News Reader

Too early to tell, but this looks like it could both simplify implementation and speed-up execution of linear RNNs like RWKV, Mamba, retnet, etc.

They already use closed form summations. That's their whole point

Yes, but this uses only two plan-vanilla parallel scans, shown as a* and b* in the pdf.

In the sample code, that's two calls to PyTorch's built-in API.

That looks very efficient to me, almost like it shouldn't be possible. But I tested the sample code[a] and it works.

Not an expert on this, but I don't think I've ever seen that before. Have you?

There's a GitHub issue on the repo that explores this!

Thank you. It seems only testing will answer the question.