I've never been worried about LLMs. I've always been worried about how people will use LLMs and how they will interpret the output of LLMs. Especially people who don't understand what LLMs are doing.
I wouldn't dare try to use an LLM for a chemistry question because I wouldn't be able to tell if it makes any sense or not. But if you're not a "tech person" and all you see is some company advertising their AIs as magical knowledge engines with disclaimer text that wouldn't pass accessibility tests, why wouldn't you assume they know their stuff? The Perplexity ads are bordering on negligent.
You need to expand your circle a bit more :-)
We can point at a media company, call out its vested interests, scream about its bias, protest in front of its office, sue it for slander and misrepresentation. We can call out individual personalities the same way. We can strive to drive the companies out of business and the personalities out of work, if we deem it necessary, and we can accumulate a paper trail that holds each one to account.
As neither individuals nor corporate entities, algorithms do not yet carry this kind of legal or public accountability even as we some start to hold them up as oracles. In most cases, failures of an algorithm are treated simply as bugs or user mistakes. Nobody is responsible for anything bad and the so the algorithm can persist and its vendor can shrug off their own responsibility by gesturing towards an perpetual development process instead of accepting consequence: "we work to make the algorithm better every day, try again tomorrow!"
Right but the previous election had Russian servers spinning up fake news websites that displayed straight up generated news. Again how do you hold them accountable? You can't the only defence against bullshit is independent thinking.
What they don't fully realize is that this is a completely different game; now the computer is just guessing, as opposed to following a deterministic algorithm to the answer.
And this misunderstanding carries the potential for pretty serious consequences, good luck getting that loan once a computer finds some arbitrary pattern and says no.
The problem isn't the people. It's the tech companies.
The tech companies are telling people that it's intelligent, and the tech companies are using it to answer people's questions as if they're presenting facts.
People are using it the way they're told.
If you advertise something as a solution, don't be surprised when people use it to solve things.