i see really cool, where i failed was trying to build my own matrix operations library, it was just too much, but using cuBLAS definitely helps, i'll look into the custom kernels you wrote they seem interesting!

did you build the backprop yourself? it is a really cool project to build and i think you can agree that it teaches you a lot of how LLMS and machine learning in general works.