Why not instead of generalist models with 7b, it should specialize like “role play” model, or just code? But I just realized if the model are not generalized, it won’t understand natural language
An idea I hear often listening to talks about LLMs, is that training on a larger (assuming constant quality) and more various data leads to the emergence of grater generalization and reasoning (if I may use this word) across task categories. While the general quality of a model has a somewhat predictable correlation with the amount of training, the amount of training where specific generalization and reasoning capabilities emerge is much less predictable.
There are RP, code, etc. specialized fine tunes of some models, to get the most bank for the bunk on some small models.
You can take a general model and fine tune it for a specific task. There are various tutorials out there for creating fine-tuned models.