This paper is very good and the results are quite impressive. It's especially good news for smaller labs and hobbyists, as these small language models take significantly fewer resources to train. To quote the paper: "the training of SLMs on TinyStories can typically be done in less than a day on a single V100 GPU." This is opposed to LLMs such as ChatGPT which can take weeks to train on hundreds or thousands of GPUs.
Basically Tiny Language Models (below 10m parameters or only one transformer block) can generate paragraphs of coherent text and reason....provided training is limited to stories that only contain words typical 3 to 4 year olds would understand.
Anyone know of a trained small language model I can download?