There are lots of distilled models on huggingface that are much smaller than, say, Opus. They are distilled from Opus or Fable and show clear improvements. I do not have the budget to fine-tune a 30b or more parameter model but from my tiny model experiments, the results are quite clear. Again, I have only a couple small tests.

Have you actually fine-tuned yourself? Email categorization comes to mind and there are tons of non-LLM approaches even that will give fantastic results. How did spam filters work before LLM?

I think LLMs just made us think that is the only way. It is not.