That's why I wanted to try to understand, what am I missing about local-toy LLMs. How are they not just noise/nonsense generators?
They're bad at generative tasks. Don't have it write code or scientific papers from scratch, but you can have it review anything you've written. You can also do summaries, keyword/entity extraction, and the like safely. Any reductive task works pretty well.
On a whim, I asked Zephyr 7B (Mistral based) “what’s the name of that Ruby method that does <insert code>” and it gave me 3 different correct ways of doing what I wanted, including the one I couldn’t remember. That was a real “oh wow” moment.
So offline situations is the most likely use case for me.
Privacy.
If you use llm's on private documents you don't want others to see, you'll likely prefer local models strongly.
Local might still be there after an online service is no longer available.