My personal opinion is: LLMs give you the power of language. So far we could define rules, based on structured data, we couldn't process unstructed data that well. Now we can use LLMs to take any kind of input and either create responses to it or transform it to structured data. That is a huge leap of advance. But also, there are a million cases where it's not necessary.
On the side, I'm working for a NGO caring about sustainable finance. They have a manually gathered database, they have lots of resources, but most users couldn't care enough to actually click through everything. So offering a chatbot to make that data available seemed reasonable. It works, quite well, and still most requests are so trivial you could have just blocked them.
On my paid job, I'm working for a german radio/tv broadcast station and they're trying to involve AI in solving simple internal user issues. It seems to work quite well. We've built a RAG system based on Qdrant and LlamaIndex and it provides all available information in a format users couldn't find before - because the systems were chaotic and complciated. So in my book, that's a good use case. Users in a very complicated environment with lots of information.
I've worked with OpenAI API, Anthropic API, Azure Foundry, local models, IONOS Model Hub, etc. One thing that keeps coming up is privacy and (in Europe) GDPR-compliance. Use the capabilities of LLMs without sacrificing data that should not go into the next training round.
Anyway, I think LLMs offer a lot of possibilities, but many people tackle them from the wrong side - "what could we do with this?" instead of "what problems do we need to solve?".