Oh thanks bro, nope it uses the simple llama 2 rope with tetha changed to 50k to match llama 3's. I'll check your python PR, have a deeper look at the meta llama 3 & 3.1 implementation and hack together something soonish. Awesome!
Np! Oh ye my PR is just a random conversion from Python to C - haven't done C in ages though so probs won't even compile!
To be honest your pr and these notes are super helpful cos, otherwise I'd have been too lazy to read up the original implementation, but I can't merge it. Will make the fix soonish and credit it to you. I'll tell you a secret, nobody is good at C except for a few wizards like woz, Linus, jart, Knuth, Bellard and the like. I am also a total looser at C. I just love C. And also the base of this is mostly Karpathy's work and other awesome folks who are way better at everything than me.
Oh thanks! :) Oh ye Karpathy's repo is pretty sick! Well still great work on it! If you need any help on it, feel free to ask!