JohannaAlmeida

The more updated code is here sorry . In another monorepo https://github.com/JohannaWeb/Bastion/tree/main/projects/Aur...

Not sure yet just want to make it more functional rn

For a minimal browser yeah . 8kloc

Thank you so much . The next thing i want to tackle is the training bottleneck we have right now .

That will probably be another HN post when i figure it out.

Yeah RWKV is definitely related in spirit (recurrent state for long context). Here I’m combining local windowed attention with a gated recurrent path + KV cache compression, so it’s more hybrid than fully replacing attention

Yeah auto complete is an amazing use case. I needed a small model that used transformers , could fit on my weak consumer GPU .

So i needed to make fundamental arquitecture changes .Do some KV cache tricks.

And then prove the new arquitecture was faster with benchmarks and perplexity was acceptable.

Full attention O(n²): 17.96s / 5.6 tok/s

HybridAttention O(n·W + n·D): 0.35s / 286.6 tok/s