Hacker News Reader

Comment by smokel | Hacker News Reader

What alternative technology do you think is better? In other words, what is your frame of reference for labeling this "pretty terrible"?

View on HN

Given that people are already firing real human workers to replace them with worse but cheaper LLMs, I'd argue that we're not talking about a competing technology, but that the competition is simply not firing your workforce.

And, as an obligate customer of many large companies, you should be in favor of that as well. Most companies already automate, poorly, a great deal of customer service work; let us hope they do not force us to interact with these deeply useless things as well.

How many humans in your office do you think could solve the questions with better success ratio than GPT-4? I would say less than 20%.

If the primary complaint is the blues that GPT-4 wrote is not that great, I think it is definitely worth the hype, given that a year before people argued that AI can never pass turing test.

That's a false dichotomy. Language models will always confidently give you answers, right or wrong. Most humans will know if they know the answer or not, they can do research to find correct information, and they can go find someone else with more expertise when they are lacking.

And this is my biggest issue with the AI mania right now -- the models don't actually understand the difference between correct or incorrect. They don't actually have a conceptual model of the world in which we live, just a model of word patterns. They're auto complete on steroids which will happily spit out endless amounts of garbage. Once we let these monsters lose with full trust in their output, we're going to start seeing some really catastrophic results. Imagine your insurance company replaces thier claims adjuster with this, or chain stores put them in charge of hiring and firing. We're driving a speeding train right towards a cliff and so many of us are chanting "go faster!"

>Most humans will know if they know the answer or not,

No they won't.

>they can go find someone else with more expertise when they are lacking.

They can but they often don't.

>the models don't actually understand the difference between correct or incorrect.

They certainly do

https://imgur.com/a/3gYel9r

It would have been false dichotomy if I said AI could replace humans. I never claimed it anywhere. I just said AI has its usecases now that makes it worth the hype.

Meaning, in your office, at least 1 out 5 will see and understand the imminent disaster and put this on halt, before this becomes an overwhelming catastrophe. The LLM, on the other hand, will confidently insist on everything being fine. Now consider your odds on having an office to go to, in a few months. (Mind that this is systemic issue.)

LLMs might be better than random people at random tasks, and certainly memorised more trivia. But I never heard of a LLM surpassing subject experts in their field. On top, even the simplest task is unreliable - say, summarisation or translation. There is almost zero autonomy or ability to recover from unexpected situations.

Yes, that's true but experts costs money and time and GPT-4 is basically peanuts compared to that. Specially in cases where verification is easy like writing test cases or a formal email. And that is the reason enough for the hype.

I see similar comments everywhere where AI is praised, and I don't get why you need to comment this. Literally no one ever said LLM surpassed experts in their field, so basically you aren't arguing against anyone.

The problems with most company-customer interaction, for me, boil down to empowerment and integration.

To use the canonical example of "internet service support call," most issues are because the rep either can't do what you're asking (e.g. process a disconnect without asking for a reason) or because they have no visibility into the thing you're asking about (e.g. technician rolls).

I honestly think we'd be in a better place if companies freed up funding (from contact center worker salary) to work on those problems (enhancing empowerment and systems integration).

> Given that people are already firing real human workers to replace them with worse but cheaper LLMs

That's impossible, LLMs are not that good. They might be firing people and crashing service quality.

Yes, correct.

But they're only firing humans in specific cases where LLMs can compete with them. LLMs aren't equally good at all tasks.

Well, no. https://www.psychiatrist.com/news/neda-suspends-ai-chatbot-f...

https://www.cnn.com/2023/08/30/tech/gannett-ai-experiment-pa...

If the AI is a lot cheaper than a human, then it can make business sense to replace the human even if the AI is not nearly as good.

Don't forget, faster.

If it takes a whole business day to "spin up" a human for a task, and takes literally 5 seconds to call an OpenAI API, then guess what? The API wins.

What's with this zero sum mentality? "replace", "cheaper"...

We are updating our expectations very fast. We are fighting over a growing pie. Maybe the cost reduction from not having to pay human wages is much smaller than the productivity increase created by human assisted AI. Maybe it's not an issue to pay the humans. AI works better with human help for now, in fact it only works with humans, never capable of serious autonomy.

>What's with this zero sum mentality?

Capitalism baby! You must continually earn more to enrich the investor class regardless of the cost to society as a whole. Just because the pie grows in size doesn't mean those with the capitol have to share it with anyone else. Greed, unfortunately, is limitless.

Humans. After all, LLMs are designed to reason equal to or better than humans.

Humans are pretty bad at these questions. Even with the simplest questions like "Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?" I think that a lot of people will give an incorrect answer. And for questions like "Argue for and against the use of kubernetes in the style of a haiku", 99.99% will not be able to do it.

The thing with humans is they will say “I don’t remember how many syllables a haiku has” and “what the hell is kubernetes?” No LLM can reliably produce a haiku because their lexing process deprives them of reliable information about syllable counts. They should all say “I’m sorry, I can’t count syllables, but I’ll try my best anyway.” But the current models don’t do that because they were trained on texts by humans, who can do haiku, and not properly taught their own limits by reinforcement learning. It’s Dunning Kruger gone berserk.

Eh, it's not D&K gone berserk, it's what happens when you attempt to compress reality down to a single dimension (text). If you're doing a haiku, you will likely subvocalize it to ensure you're saying it correctly. It will be interesting when we get multimodal AI that can speak and listen to itself to detect things like this.

The problem isn’t just that everything is text. It’s that everything is a Fourier transform of text in such a way that it’s not actually possible for an LLM to learn to count syllables.

Again, that is just using text only.

Imagine you have a lot more computing resources in a multimodal LLM. It sees your request of count the syllables and realizes it can't do them from text alone (hell I can't and have to vocalize it). It then sends your request to a audio module and 'says' the sentence, then another listening module that understand syllables 'hears' the sentence.

This is how it works in most humans, now if you do this every day you'll likely make some kind of mental shortcut to reduce the effort needed, but at the end of the day there is no unsolvable problem on the AI side.

By "Humans", I assume you mean something like "adult humans, well-educated in the relevant fields". Otherwise, most of these responses look like they would easily beat most humans.

I think most high-school educated adults, with the ability to make a couple web searches, would do fine on all these questions. It would take the humans minutes instead of seconds because they don't have the internet memorized.

Me, Kubernetes Haikus, time taken 84 seconds:

----------

Kubernetes rules

With its smooth orchestration

You can reach web scale

----------

Kubernetes sucks

Lost in endless YAML hell

Why is it broken?

I think you're spot on here. Yes, if one's trying to compare human and GPT intelligence, then you have to define what counts as memorisation and what counts as reasoning. But what most people outside of academia are trying to do is work out whether a GPT can effectively replace a human in some time-consuming task, and to be able to do so without access to the internet is rarely an important factor.

> After all, LLMs are designed to reason equal to or better than humans.

No.

I doubt you would fully trust a LLM to replace high risk jobs such as lawyers, doctors or pilots such that when something goes wrong as it is used unattended, there is no-one held to account for it to transparently explain its own mistakes and errors.

It is just nonsense to suggest that such systems are capable of ‘reasoning’ when it pretends to do so and repeats itself without understanding their own errors.

Thus, LLMs and other black-box AIs cannot be trusted for those high risk situations over a consensus of human professionals.

Reply on news.ycombinator.com