In the readme you can see benchmark which everyone with different hardware is running Colibrì, and I have to say I've seen great times! I'm always doing more to improve!
What causes problems is the rewriting in this case are only read while writing is the cache! However, I'm working to improve more and more and make some parts lighter!
Absolutely yes! With nanoeuler I learned so much by testing every little detail of the project. Every little part you see has been tested and proven several times so that it could be understood and worked.
Yes, because it has many separate kernels instead of aggressive merges like PyTorch (with Torch Compile). Each pass (norm, matmul, residual, RoPE, etc.) launches its own kernel, which increases launch overhead and memory traffic. CuBLAS helps, but it's not enough to compensate.
Hi, in nanoeuler I use cuBLAS (NVIDIA's super-optimized library) for all matrix multiplications, with the tensor cores in TF32 mode. It's the same thing PyTorch uses underneath, so it's very fast.
What I've optimized (and will improve even more) and written by hand are the kernels for the other parts (like FlashAttention, which gave a nice 3x speedup), while I've delegated the large matrices to cuBLAS.
Training the 116M model on a 4070 runs well and in reasonable times. Compared to PyTorch, it's a bit slower (probably 1.5-2.5x), but nothing dramatic, especially considering it's all done from scratch without a framework and there are no other optimizations that would make it faster. I'm working on it.
Hi, thanks for the comment. Nanoeuler is starting as a study and research project that will obviously improve over time. I'll do my best to make the readme and other things more readable. Thank you very much.
Hi, the uploads are one after the other because it was a long, step-by-step research project where I tested the code on another machine. I admit that I'm slowly making up for the commits on all the projects. For Gutenberg and Shakespeare, I admit that they were the best tests I could do, but I'll always improve!
Hi guys really cool project,
I create judicex https://github.com/JustVugg/judicex maybe we can help each other to make one big software open source in a law workspace!
Creating a search engine has always been one of my dreams ever since Sergey and Larry created Google. I've done a couple of experiments but I've noticed that now with AI agents the search is changing a lot. I think it would be interesting if your search engine was a response for agents and maybe agents that help you simplify information from a website. Like not all HTML but taking the most important information. I don't know, it could be interesting. Congrats on the project anyway!
Yes, I think that having the code completely checked by AI is not a good idea, but an AI that says, "Check these," because they are noteworthy. This is my idea of a future where I hope AI is like the movie "Limitless." That is, it supports you in improving yourself and giving you greater capabilities, not in replacing you entirely.