If you want to make the model fully generalist, feel free to train it over coding datasets (such as RL with passing unit tests as reward).
Side question, since it sounds like you were involved: how big is the impact on benchmarks of taking this 1.5B model down from fp32 to fp8 or similar? The focus on parameters alone sometimes feels like comparing house sizes by their lengths alone. And, if you were indeed involved, thanks for making all of this open and available!
Come checkout our repo at: https://github.com/agentica-project/deepscaler
This work scales up selection/routing over many models/LoRAs
However smaller specialized models looks to be the right way to handle world's complexity. Sort of mixture of experts on one level above. Orchestrating them will be another problem. Possible solution is generalists model "to rule them all".