Agreed heavily - and I work with these technologies regularly. There is a mania around it much like there was for crypto (though I would maintain there's more merit here than there ever was with crypto - orders of magnitude more).
There are a litany of problems that surround LLMs that are solvable, but are highly non-trivial. The present hyped demos all seem to involve mainlining the unfiltered output of LLMs directly at the user, and this approach strikes me as largely impractical, with all of the issues you've brought up and more.
Ultimately I think we will land on a few learnings, some of which are IMO pretty obvious already with existing ML:
- There are going to be lots and lots of use cases for LLMs where despite its additional complexity it won't out-perform simpler ML models, or even heuristic-based solutions. This has been a plague for AI startups forever - where the shameful truth is that the ML bits do not significantly outperform heuristic-based approaches, but the ML bits sure as useful for the hype machine. There will be more of these use cases than use cases where LLMs actually do significantly move the state of the art forward.
- In most broadly mainstream-palatable applications the LLM will need to be intermediated by many other systems - some using other ML models, and some using heuristic-based solutions. For example for search, mediating the output through a knowledge graph with reliable provenance and accuracy. Realistically there will be vanishingly few use cases that are thin-wrappers around LLMs. Real usefulness of this tech will require very heavy lifting around the core model to make it actually accurate/reliable enough to be useful on a mass scale. The many products spinning up that seem to be premised on very thin wrappers around OpenAI's API will IMO mostly not survive, both because there is no moat and because the results won't be sufficiently useful.
- Accuracy matters and will be a thorn in the side of these products for quite some time, even though it is solvable. For some use cases (mostly creative ones with a human in the loop - like creative writing or image synthesis) the lack of accuracy is livable, but for most others the propensity to hallucinate is going to be a major barrier to adoption. Right now you're dealing with an enthusiast audience that's willing to look past glaring and obvious errors in output in favor of what it could be and how far we've come, but a mainstream audience I suspect will be less forgiving. Social media virality around product failure cases will also further punish products.
Overall I think what we're looking at here is legitimately a huge leap forward, but this legitimate and well-found excitement needs to be tempered by the fact that some (most?) players in the space seem to be getting in way over their skis, and that truly productionizing this technology is going to be intensely difficult.
[edit] To be a bit less doom and gloom about this - I think the real winner here are natural language interfaces, and stepping ever closer towards computer systems that do not need to be actively learned before being used. There is IMO an over-focus on the knowledge-encoding part of LLMs (which is unreliable and prone to hallucination) and not enough focus on the language-encoding part of LLMs (which is what they actually do). For the most part voice/language-based interfaces have had limited traction because they honestly haven't been smart enough to understand user requests with a sufficient level of expressivity and fidelity. I suspect these products (and new ones) will have another crack at this problem.