Really excited to see this! 2 Questions:
1. Did you try using RTD (Electra like pretraining)? Or did you skip that for reasons of compatability?
2. Why not incorporate jamba like Mamba2 alternating layers?
8 karma · joined August 3, 2020
("Shakes old man's fist at sky, and at people who seems to enjoy boilerplate code too much")