Hacker News Reader

There are flaws but also consider PEBCAK error.

For one next time when it starts hallucinating and a gentle course correction doesn't do it, just start a new chat with a different prompt approach. Having the error in its context reinforces the same mistake and sometimes it can't get out of this loop.

PEBKAC in what context? The prompts themselves or the documentation? Because I got Active Collab running on NGINX myself.

I already did this in terms of starting new chats, I spent days on it, and consulted with half a dozen devs supposedly using it in their workflows. It's very easy to make it hallucinate.

In terms of using gpt optimally. But fair enough. If you tried it in multiple sessions trying to convert Apache rewrites to nginx and it wasn't able to do it, I guess this is another failure mode. I just found that curious because chatgpt is usually very very good at regex.

Side note is Google extra terrible lately or is there really no docs on this almost anywhere?

All I could find about it is this and the rules looked very simple, from my experience chatgpt should have got this https://activecollab.com/help/books/self-hosted-activecollab...

That's the documentation on the URL. And yeah, I was surprised too given how people had hyped it so heavily in much more complex scenarios for dev work (well, claimed to anyway, they haven't documented it in most cases of claiming to x).

> but also consider PEBCAK error.

No. It's an AI error, the person between the chair and keyboard just hit enter. If this "error" goes away when I hit the enter key a few more times and get lucky, it's not my fault.

Well, the error doesn't go away, it's a "fail state", but that still doesn't mean PEBCAK either.

It’s true that their training goal sometimes rewards them hallucinations, encouraging them do that, but that doesn’t mean it’s not reasoning. If anything hallucinations involve more reasoning because they have to make up something new that wasn’t there

It's doing what LLMs do. Next word prediction. It didn't "make it up", it incorrectly predicted the relationship of words.

It isn't reasoning about the solution to a problem either, it's running as expected in relating words to each other, that doesn't mean it has any form of understanding of the words or even what it's rendering as an output.

It does just predict the next word, yes, but it turns out that predicting the next word is a complex problem that involves solving many types of subproblems that involve relatively complex reasoning, or at least something that looks very, very much like reasoning

Is it any more reasoning than Bayes' theorem is reasoning? It's statistics and derived probability from said statistics.

> Bayesian inference is a method of statistical inference in which Bayes' theorem is used to update the probability for a hypothesis as more evidence or information becomes available.

That is very clearly reasoning

No, it's probability entered into an algorithm as a variable. This is not reasoning, this is probability, and there is no reasoning involved.

Go and program HomeAssistant (FOSS) with an automation using Bayes' theorem. Then proceed to laugh about all the fail states you encounter, thus realizing it is not reasoning, but probability. https://community.home-assistant.io/t/how-bayes-sensors-work...

It's quite literally known as Bayesian probability.

I will admit that I don't know much about Bayesian statistics, but how does something producing probability as an output mean it's not reasoning? I'm pretty sure humans do that consciously and subconsciously all the time and we call that reasoning

Because it's just an arbitrary number. It has no concept of what the numbers mean or what it means when it gets a number that it doesn't have in its existing data set (the "training", more accurately described as references), whereas a human being can make an educated guess, or adapt to the unexpected outcome and thus provide a new novel solution. It isn't thinking about the numbers, it's executing maths, as programmed.

Is a calculator reasoning? It doesn't understand what the numbers mean. It's an input-output machine. Is a sheet of paper with numbers on it the process of reasoning itself? No, of course not. Human beings apply meaning to the output, or feed that output into other things to drive processes that they've already created.

If you kick the Boston Dynamics BigDog, it will compensate for the changes in x sensors and remain upright, or get back up, ergo it can traverse a dynamic (changing) environment, such as a battlefield, or an urban area with cars, people etc. It's not reasoning. It's conditional logic based on different vars. The BigDog bot doesn't understand what it is, where it's going, it doesn't think, it's applying maths to sensor inputs and motors on a recursive loop. If it encounters a problem it hasn't been programmed for, it cannot reason a new solution.

By your logic, video game characters must use reasoning, when no, they don't. It's maths.

Is it all fascinating? Yes, absolutely. Are there uses for it? Yes, absolutely. Is it reasoning? No.

If a human brain doesn’t use maths, what does it use? It’s Magic?

> "training", more accurately described as references

Uh, no, that's not at all how the training process works. The training process very roughly mirrors evolution/natural selection except more more direct and faster.

This is a good introduction to the concept of how a computer could learn: https://www.youtube.com/watch?v=qv6UVOQ0F44

This is a more technical look at how exactly it works in more modern AIs: https://www.youtube.com/watch?v=aircAruvnKk (it's a series)

> If it encounters a problem it hasn't been programmed for, it cannot reason a new solution.

But an LLM can? It's not as good at it as a human but it can

Relevant: https://twitter.com/nearcyan/status/1632661647226462211

(if you don’t want to click it it says “referring to AI models as "just math" or "matrix multiplication" is as uselessly reductive as referring to tigers as "just biology" or "biochemical reactions"”)

> By your logic, video game characters must use reasoning, when no, they don't. It's maths.

“By your logic, if tigers are dangerous, then bananas must be too, when no, they aren’t. It’s biology.”

This has to be the most misinformed counterargument of all time. The brain certainly doesn't use maths as it is not a binary system. Information is approximated and estimated. Do some reading on chemically mediated graded responses and how neurotransmitters actually function within a synapse.

Scientific research also suggests there are quantum processes involved that we don't yet grasp, ergo a hypothetical AGI likely won't emerge without using quantum physics.

> Scientific research also suggests there are quantum processes involved that we don't yet grasp, ergo a hypothetical AGI likely won't emerge without using quantum physics.

Citation? As far as I know it has been postulated, usually by religious people, that there must be another factor at play, like quantum physics, a soul etc because we don’t understand too much about many things. We also don’t understand the emergent behaviours about LLMs or what is consciousness etc. You can religiously think or hope we are superior to the phone in your pocket, but the proof is thin and became a lot thinner a few years ago with LLMs. The mere fact we don’t know how things work or that we are not cloning our own brains or processes is not a proof at all that we won’t stumble accidentally or purposely onto something that is a reasonable simile. I believe LLMs and the emergent behaviours they show gives us some idea that we are quite stupid and arrogant; we believe we are all that, yet I replaced a team of 15 devs with gpt and we overperform more than ever.

I agree with you we don’t understand our brains, but that doesn’t mean there is no alternative or simile that works just as well rooted in silicon and LLMs are the beginning of that.

I can’t reply to your latest comment for some reason, so:

> This has to be the most misinformed counterargument of all time. The brain certainly doesn't use maths as it is not a binary system. Information is approximated and estimated. Do some reading on chemically mediated graded responses and how neurotransmitters actually function within a synapse.

Uh, what? The brain is a physical object. The way that physical objects work is dictated by the equations of physics (maths). Are you telling me that the brain doesn’t abide by physics?

And weren’t you just telling me that something that uses approximations/probability isn’t reasoning?

“chemically mediated graded responses”

That sounds like roughly the same thing as an activation function that neural networks like GPT models use.

https://en.m.wikipedia.org/wiki/Activation_function

So what is your test for when it is reasoning? It would be both reasoning and intelligence in all or most tests 25 years ago. But now we changed the definitions because ‘it cannot be, this is just stochastic mumbling’. You cannot say it’s not reasoning unless you can give me test that it doesn’t pass that humans do pass.

You in this and other comments vastly overestimate humans it seems; I dare to say that whatever test you come up with; I can pluck a random human of the street that will fail it. And chatgpt probably passes it now or soon. The tests I have seen that gpt fails on also trip over the vast majority of humanity. Not the few elite like probably you and me on HN but the rest yes; you want to say they have no reasoning power either or?

Gpt might be reasoning or it might not be; I cannot say as I don’t understand at all how I reason myself. Or how to describe what it is to reason in coherent terms. Or to test for it. But saying the substrate (meat vs silicon) makes the difference and, even though gpt actually does get a lot of calculations right even though it was not made for it and we don’t understand why that is, means something is going on. I think it will debunk most of the religious arguments we have to elevate us to something special like you and others try to do pretty soon. It seems we will find that our thinking mechanism is a few pages of Python and then trillions of complex calculations to train it. Animals like us did this over millions of years, we will repeat it in 100 years or less.

>If it can't even do an NGINX config, what use is it for anything else?

Turn it around. I'm sure there are lots of people around who couldn't do it either, even with a ridiculously high amount of time to do it

Yes, but we are discussing the claim that LLMs might exhibit superhuman smartness.

They are fallible, but they quite clearly exhibit superhuman smartness when compared to the average human.

As a thought experiment, assume the average human may be able to translate text between two human languages, or write code in two-three programming languages. GPT4 can perform those tasks on a much more diverse set of human _and_ programming languages. Is that not superhuman?

Yes, it makes mistakes. But take a hundred humans off the street and ask them to write an NGINX configuration or translate between Indian and French - how many would be able to do that? How many would be able to do that without any mistakes?

The vast majority of humans would fail your reasoning test.

And? We are talking about AI, not humans.

People on HN claim they're using it for XYZ in development, yet it can't even generate the necessary NGINX config, despite being given the URL rewrites it'll need to incorporate.

The point is that it hallucinates. It isn't that it failed, it's that despite giving it everything it needs to know, it hallucinated all kinds of things not in the documentation, not in my prompts et al.

Why? Because it's an LLM. It isn't fit for purpose in this context. A next word prediction AI (an LLM) isn't appropriate for these kinds of problems.

Is your argument that AI needs to be held to a different standard than humans? Because humans confidently hallucinate answers all the time.

To me, it feels quite intuitive that an AI trained on human knowledge would automatically learn to do the same.

Neither.

The point: it isn't that it failed. It isn't even that it "hallucinates answers", it's that it infers relationships between words that don't exist because it's an LLM. It predicts the next word. That's what it does.

Something that predicts the next word isn't an appropriate method of doing x in y of z cases, because its reliability in providing the designated function is important. Ergo, yes, LLMs may well have applications, but most of the problems that people are throwing it at are inappropriate, just like blockchain fetishism versus a database. For the overwhelming majority of problems, AI is not the answer, neither is a blockchain, neither is an NFT.

Call hallucination what it is: a fail state. It got it wrong. It didn't "hallucinate". With standard conditional logic, x yields y result. That's very useful where you want consistency and reliability, ergo, those problems are best not handled via an LLM. Why not use the appropriate tool for the job?

Deductive versus inductive versus abductive reasoning.