In the last week I have been researching LLM architectures, putting together all the known ways to train AIs and ended up creating something apparently new.
Long story short, the vRAM needed to train my LLM seems to be up to 5x times smaller than a standard AdamW.
I was studying the ternary weights on my own when I asked myself if it was possible to actually train an LLM with this method. Turns out, yes, it is.
I created a POC about the possibility to train a latent-free LLM, significantly saving memory for training, and shrinking the shipped model size for storage/distribution (inference speed/RAM need a dedicated kernel — not built yet.
During tests on a toy scale on babyLM benchmark (up to ≤125M), performance proved to be on par with other autoregressive models. As of today, there is no optimized kernel, so actual vRAM savings potential and speed-up are not completely fulfilled.
This is clearly still a WIP, as I am experimenting with other configurations and scale-ups. If you are curious about this, the repo is open with more technical information.
https://github.com/ValerioDolci/bitbop