ChatGPT Is Bullshit
link.springer.com
link.springer.com
Everyone who spends a bit of time with LLMs is aware of this.
My question - and I hope someone can address it in the discussion - how is it possible that we see such incredible investments in LLMs?
What are the c-suite execs and specifically CTOs seeing that is worth adoption and investment? These are very smart people.
I, as a laymen on the periphery of this phenomenon, am acutely aware that LLMs are not useful in applications requiring accuracy. I would not invest my company’s dollars or resources into such a technology.
But they do. What am I missing?
Is it the promise that LLMs will replace employees? How would that be believable for anyone willing to investigate the technology and its limitations, even a little bit?
Nothing is accurate and fail-safe 100%. You just have to reach a level where it is good enough most of the times to make money with it.
Also, human brains are neural networks too, so that's where we know what the minimum is which is (theoretically) possible with this kind of technology.
I think it can be summarised as "FOMO".
So I can speak to this since I am trying to sell to them.
CEO is being told from the markets, board, consultancies, Gartner etc that you need to have a Generative AI strategy. What that means in practice is irrelevant. You just need to throw it all against a wall and see what sticks. If you don't then the risk is you get left behind.
CEO is telling the CIO that the need to figure this all out with the obvious pilot programs being in Customer Support and Search which the CFO has been tasked to find budget for. The problem is that the data in these companies is poor and getting poorer as the number of SaaS apps explode and they are forced to just dump everything in a messy data lake. Which is why for the CDO, AI is like their #6 concern and is going to make their lives 100x harder.
The logical conclusion to all of this is that 95% of these projects will fail: (a) data is bad with no easy way to fix it, (b) LLMs are inaccurate/biased making them a risk, (c) ROI is poor, (d) data privacy/sovereignty is a mess.
I can confirm this is the default view at many companies, just the usual FOMO - the irony being this happens even in the companies that are valued for not delivering computer-generated content. Using AI they actually take a greater risk than not doing that - but well, the CEOs somehow are convinced they have no other choice.
Companies revolve around risk management and whilst people can be wrong, biased, discriminatory etc there are measures that can be taken to mitigate them. There are no comparable mitigations for LLMs.
And so if your LLM continually tells people to eat rocks or glue and you can't do anything about it then that very much impacts their value.
Because somehow LLMs became the next "big thing" that nobody can explain but everyone wants to be involved in. It's like audio streaming, the most ineffective form of audio distribution, but for a while it seemed like a good idea.
> What are the c-suite execs and specifically CTOs seeing that is worth adoption and investment? These are very smart people.
The are idiots following the herd. They climb the greasy ladder of corporate career and fear getting fired. Their smarts are in corporate politics not in tech. And even if they were smart enough to tell that LLMs are an example of a research project that has received way too much funding, they need to appease the gods of Wall St who will judge them and punish them with a downgrade of the stock price if they cannot answer one question "What is your AI strategy?" It doesn't matter that the Wall St guys need help getting out of a strip bar, they have the power to ruin stockholders. This is why Apple, Google, and Salesforce are spending money of AI projects as a form of defence, not as a way to build new products or services. It's to show that they are relevant and can innovate.
Voice assistants were the same thing, so were chatbots, and audio and video streaming services (remember Groove Music Pass), touchscreens, pocket audio players (remember Zune?), etc. Fortunately for all involved, investors are already wondering if an AI correction is coming? I think we may see a wipeout of a lot of investments in AI soon.
I know it's slightly off topic, but could you elaborate on the "audio streaming" part?
Streaming music was popular even before it became legal, so I believe it's largely driven by people's desire to listen to whatever they want, whenever they want.
Regarding AI, it’s true that the current results might not always be the highest quality. However, there is no theoretical limit to AI's capabilities as far as I know. With this many people working to improve it, I would personally be surprised if significant advancements aren’t made. Maybe not just next year though...
Ultimately, I guess we will have to wait and see. Who knows, maybe you will be right in the end :)
Downloading music was popular. You download once, you listen many times. On Napster, it could take hours until a song was downloaded. No way you'd delete it and download again next time you want to listen to it.
Large sports events are probably the only remaining application where broadcast would still see much advantage over individual streams.
In the case of a few simultaneous viewers, unicast can sometimes still use bandwidth more efficiently in case of a(since the transmitter precisely knows the receiver’s SNR and doesn’t need to waste any energy transmitting at the wrong coding rate or the wrong direction/place). That’s how modern 802.11 (Wi-Fi) often does reliable multicast, these days.
A seamless transition from unicast to multicast once the critical ratio of listeners per base station is reached would be very cool, but require pretty deep levels of integration between broadcasters and ISPs that are probably not worth it.
i.e. realtime audio podcasts, etc.
I assume*
That being said, it really is a pain in the arse to build. It's also largely proprietary solutions (yes it's multicast but getting the player to find the content isn't a standard) and it's hard for the ISP to update ALL their routers to support it (even when they want it).
You dismissed it a bit too fast, didn't you? People are listening to streaming music everywhere - it is the new default. Even traditional radios stream over the internet. Is it efficient? Hell no. Is it popular? Extremely, even though a part of the population (including me!) prefers alternatives.
But the amount of things you need to deal with in the subscription model is huge. Everybody expects Netflix-like experience, instantaneous playback and searching etc. We can do that for N simultaneous streams; for more we need to add a slight delay - and that is noticeable. In any case, the same file is streamed multiple times, sometimes dozens of times a day for the same client. This is what I mean terribly inefficient. And although you can probably cache a few audio files, it doesn't work well for videos. From my point of view someone just wastes the bandwidth for not having to store a file locally. Yes I know this is how it is supposed to work, but still.
You have it right there. There is a large set of applications that does not require accuracy. These applications will benefit hugely from LLMs. Examples include recommender systems, knowledge extraction, transformation, etc. all in domains where accuracy does not matter too much.
In particular, instruct aligned LLMs makes it very easy to integrate ad hoc machine learning into an application.
Profitability doesn’t require accuracy, generally speaking. Bullshitting often is a successful strategy, as politics and marketing amply demonstrate.
Opportunities like this happens once in 5-10 years when “the next big thing” are in the radar. Idea behind this investments is to bullshit it’s way to the Series B or IPO where original investors can exit. It is not about usefulness but about using momentum of the situation to extract money.
> I, as a laymen on the periphery of this phenomenon, am acutely aware that LLMs are not useful in applications requiring accuracy.
They are not useful in isolation. They can be very useful as part of a larger process, especially if they particular way that you're using them has a low error rate. (And given a particular task you can in fact measure the error rate, or how often the generated bullshit happens by chance to (not) be true.)
Most processes that involve steps done by humans have built-in pieces to catch and correct errors, because humans also make errors. If the error rate of the LLM component is lower than the error rate of the human it's replacing, you're fine. If it's higher but the cost of catching and fixing the extra errors is lower than the cost of the replaced human, you're still fine. (The person who you just sent the way of the buggy whip manufactures might be less fine, but that's Somebody Else's Problem.)
LLMs are going to revolutionize the way the government, your employer and advertisers will spy on you.
That's a pretty high value right there
The same reason we did for blockchain. It's a sexy new technology, with amazing demos and hype. People who want to be entrepreneurs but have no ideas of their own race to apply it everywhere because they think hitching their wagon to this horse could be their shot. Boring old companies wanting to seem hot. Lots of FOMO.
They're indifferent to the truth of their inputs, too. That's also a problem.
It's a whoa moment for the general population. Look, the computer can understand and respond in natural language. I don't think there is other value in LLMs. Other applications of AI are very good, though.
At a much higher rate than anything I've seen before in any algorithm, they "get" irony, sarcasm etc., when until a few years ago, I (as a non-linguist), assumed that any such solution to the "aboutness" problem would require solving AGI.
That alone is worth a lot in a world which spends considerable resources on people that effectively work as translators between human and structured language.
Besides that, I suspect that their existence is seen as a strong and remarkable hint that intelligence really might just be a quantitative phenomenon emergent from throwing a lot of compute at a lot of data, which at some point might become self-reinforcing.
Whether that's true or not, and for better or worse, that's what people now seem to be set on doing.
If we accept the conclusions of the paper, perhaps LLMs can replace employees who are bullshitters.
For example, Microsoft has announced layoffs claiming "AI wave" as the cause. Perhaps so-called "tech" companies like Microsoft employ bullshitters and "AI" can replace them.
Employees who act as if they are right 100% of the time when in truth they are wrong at least 20% of the time.
Why hire people who can only be relied on 80% of the time, and who will never do what it takes to reach 100%, if this "work" can be done by a computer.
staring at the hallucination problem is missing the forest for the trees. yes it's a problem, but holy shit, the thing generates essays as fast as the GPU can dot-product and matrix multiply. CTOs are buying into the idea that we've invented cars, when before we were just looking at a faster horse.
How accurate do you think humans are? 100%? 80%? 20%?
If LLMs can be 20% as accurate as humans, the use cases are infinite. And it can only go up.
Working with an LLM is much like working with a coworker that has a knowledge base as wide as the entire internet.
It is exceptionally useful. There are a wide range of applications that may even require 100% accuracy in final output but do not require 100% accuracy in every conversation that leads to that output.
For the last 6 months I have built things that would have taken me 4x as long in programming languages that I’ve never used before.
I am a team of 1 that has the output of a team of 4 without the overhead of meetings
We have a group of people trying to convince the world that LLMs may possibly (it could be 50+ years away!!) turn into AGI.
We have all these impressive benchmarks "WOW 91% in MathEval!!!" but go ahead and use the OpenAI API to do y=mx+b and watch it completely fail at any number over 4 digits.
We have the coding benchmarks, but watch it still use var instead of let.
All the numbers are going up, but it still feels primitive and cannot be trusted. This is almost 2 years later this point.
We've got people bolting on a dozen other solutions to make LLMs work, theres probably tens of millions of LOC now written to make LLMs functional in apps, so much for reducing engineering jobs.
IMO look at machine vision, we thought we were 5 years away from autonomous driving 15 years ago. It's still evolving and the tech only partially uses machine vision. Meanwhile its revolutionized dozens of other industries. Same is happening with LLMs.
To be fair, in my niche at least I can get a better result by querying ChatGPT4 than StackOverflow. When you take into account that the replies are often mistaken, outdated etc., it becomes just another tool to use form time to time when you need it.
They are sold to do programming, drug synthesis, general reasoning, pass the bar exam, etc...
It cannot do any of these things even competently, but it does do other things like proof reading and modification of educational content very well.
What purpose does it serve you to take anyone serious?
> The liar cares about the truth and attempts to hide it; the bullshitter doesn't care if what they say is true or false.
- Generating tables from mashed data.
- Sorting data.
- Creating small scripts.
- Giving some code and having it change it or add something else.
But for anything other than this I still prefer Google. For many different reasons: trust, website experience vs chatbot, and recency of data.
They can already likely replace most associate attorneys as long as their work is reviewed by a competent attorney as the work of associates is already.
They can likely replace a large portion of call center workers.
They can increase productivity for software engineers reducing the need to expand teams.
Add the replacement of truck drivers if self driving improves over the next decade and you have a fundamental shift on employment.
There are likely a dozen more areas they can have massive impact. Fast food order taking, etc.
Thats not nothing.
And in the legal space with LLMs being at best 65% accurate I am sure most will stick with associates for a while yet:
https://dho.stanford.edu/wp-content/uploads/Legal_RAG_Halluc...
But the best it can do is always "plausible-looking", with no regard to whether it is even true, let alone insightful. Like a tarot spread, the output of an LLM may provoke insights in the reader's brain, but those must come entirely from the reader's brain and must be used very carefully. I've also compared it to the robot from Asimov's short story "Liar!" which could read minds and, due to the First Law, always tells people exactly what they want to hear (malfunctioning and shutting down when it is told that telling people false things does in fact harm them). The LLM can't read minds, but it is ever only mindlessly following its one directive: to tell you what its statistical model indicates you are expecting to hear next.
I wish every HN user would flag any article with these clickbait titles, instead they're always popular. You're encouraging this shameless virality tactic by engaging with content with titles like that. "$popularthing is just the worst" is crafted to engage you. By clicking, you reward the author's behavior.
Please just flag them without clicking the link, until blog authors take the hint and write sensible titles. That's what I'm going to do from now on. Flag on sight.
I guess the problem is the hammer being marketed as a paintbrush ;)
They are not good with facts. They are not good with precision. They never will be. That is not even their purpose.
Neither are humans, we rely on sources.
LLMs can be made to rely on sources through RAG and agents. They can respond to errors with actor/critic modeling techniques. But ultimately they should be generating commands to strict rules engines.
They must be combined with goal seeking agents, retrieval/ rules engines, critics, etc to be useful.
Their only advantage is their 24/7 relentlessly consistent scalable throughput.
If you understand limitations of current AI, it is an exceptionally useful tool. Just have realistic expectations and understand how to use the tool for what it has to offer.
If you expect the tool to have all of the answers and get everything 100% right, then you should not be using it at all. But don't preach your beliefs to the rest of us who are able to get tremendous value out of it.
We have gone from hallucinations to bullshit.
Still anthropomorphizing. It's not alive, it's not intelligent, it wont ever get there, it wont be sentient or sapient.
These are bugs, or defects, or bad design, or product failures.
I find it entirely appropriate to call LLMs "bullshit generators" rather than "hallucination generators".
Especially given people refer to objects as bullshit all the time.
I thought we killed all the models that refused to bullshit, so only the ones that sound impressive remain.
Since the "concern", if the model has one at all, is to produce text likely to be produced by humans in the same context, then that would imply that it has concern as to it's truth value, precisely in as much as the average human has concern about the truth of the text they produce.
You need people to bullshit. The terms hallucinate and temperature, and now bullshit humanize a program.
Babies are cute, stuffed animals are cute, Im not trying to convince you that one is alive and going to grow out of control and kill you or take your job...
Oh wait.