Stability.ai publishes technical report on Stable LM 2 1.6B

We’ve published our technical report on Stable LM 2 1.6B; our state-of-the-art small language model trained on multilingual data. This in-depth study covers the training methods utilized for both the base and instruction-tuned versions of the model.

Leading in the OpenLLM leaderboard across multiple languages including English, German, Spanish, French, Italian, Dutch, and Portuguese, our model achieves top scores when ranked against larger sized models such as Google’s Gemma 2.

We've also introduced a multi-stage infinite Learning Rate Scheduler that enables us to adjust the model's learning speed during training, optimizing our sessions and enhancing overall performance.

Read the report here: https://arxiv.org/abs/2402.17834