Comment by colonCapitalDee | Hacker News Reader

Comment by colonCapitalDee | Hacker News Reader

colonCapitalDee

Yeah, I'd buy it. I've been using Claude pretty intensively as a coding assistant for the last couple months, and the limitations are obvious. When the path of least resistance happens to be a good solution, Claude excels. When the best solution is off the beaten track, Claude struggles. When all the good solutions lay off the beaten track, Claude falls flat on its face.

Talking with Claude about design feels like talking with that one coworker who's familiar with every trendy library and framework. Claude knows the general sentiment around each library and has gone through the quickstart, but when you start asking detailed technical questions Claude just nods along. I wouldn't bet money on it, but my gut feeling is that LLMs aren't going to be a straight or even curved shot to AGI. We're going to see plenty more development in LLMs, but it'll be just be that. Better LLMs that remain LLMs. There will be areas where progress is fast and we'll be able to get very high intelligence in certain situations, but there will also be many areas where progress is slow, and the slow areas will cripple the ability of LLMs to reach AGI. I think there's something fundamentally missing, and finding what that "something" is is going to take us decades.

Yes, but on the other hand I don't understand why people think something that you can train something on pattern matching and it magically becomes intelligent.

We don't know what exactly makes us humans as intelligent as we are. And while I don't think that LLMs will be general intelligent without some other advancements, I don't get the confident statements that "clearly pattern matching can't lead to intelligence" when we don't really know what leads to intelligence to begin with.

We can't even define what intelligence is.

We know or have strong hints at the limits of math/computation related to LLMs + CoT

Note how PARITY and MEDIAN is hard here:

https://arxiv.org/abs/2502.02393

We also know HALT == open frame == symbol grounding == system identification problems.

The definition of AGI is also not well defined, but given the following:

> Strong AI, also called artificial general intelligence, refers to machines possessing generalized intelligence and capabilities on par with human cognition.

We know enough for any mechanical methods with either current machines or even quantum machines, what is needed is impossible with the above definition.

Walter Pitts drank himself to death, in part because of the failure of the perceptron model.

Humans and machines are better at different things, and while ANNs are inspired by biology, they are very different.

There are some hints that the way biological neurons work is incompatible with math as we know it.

https://arxiv.org/abs/2311.00061

Computation and machine learning are incredibly powerful and useful, but are fundamentally different, and that different is both a benefit and a limit.

There are dozens of 'no effective procedure', 'no approximation', etc .. results that demonstrate that ML as we know it today is possible of most definitions of AGI.

That is why particular C* types shift the goal post, because we know that the traditional definition of strong AI is equivalent to solving HALT.

https://philarchive.org/rec/DIEEOT-2

There is another path following PAC Learning as compression an NP being about finding parsimonious reductions (P being in NP)

Humans can’t solve NP-hard problems either, so definition of intelligence shouldn’t lie here, and these particular limits shouldn’t matter too

NP is interesting because it is about the cost of computation, and LLMs, are computation. A DTM can simulate a NTM, just not in poly time.

It is invoked because LLM+CoT requires a polynomial amount of scratch space to represent P, which is in NP.

I didn't suggest that it was a definition of Intelligence.

The Church–Turing thesis states that any algorithmic function can be computed by a Turing machine.

That includes a human with a piece of paper.

But NP is better though of the set of decision problems verifiable by a TM in polynomial time. Any TM or equivalently lambda calculus or algorithm can solve the Entscheidungsproblem, which was used by Turing to define Halt.

PAC Learning depends on set shattering, at some point it has to 'decide' if an input is a member of a set, no matter how complicated the parts are on top of that set, it is still a binary 'decison'

We know that is not how biological neurons work exclusively. They have many features like spike trains, spike retiming, dendritic compartmentalization etc...

Those are not compatible with the fundamental limits of computation we understand today.

HALT generalizes to Rice's theorm, which says all non-trivial symantic properties of programs are undecidable.

Once again, as NP is the set of decision problems verifiable by a DTM in poly time, that is why NP is important.

Unfortunately the above is also a barrier to formal definition of the class of AI-complete.

While it may not be sufficient to prove anything about the vague concept of intelligence, understanding the limits of computation is important.

We do know enough to say that the belief that AGI being obtainable without major discoveries is blind hope.

But that due to the generalization concept, which is a fundamental limit of computation.

This is the difference between the scientific approach and the engineering approach. Engineers just need results. If humans had to mathematically model gravity first, there would be no pyramids. Plus, look up how many psychiatric medications are demonstrated to be very effective, but the action mechanisms are poorly understood. The flip side is Newton doing alchemy or Tesla claiming to have built an earthquake machine.

Sometimes technology far predates science and other times you need a scientific revolution to develop new technology. In this case, I have serious doubts that we can develop "intelligent" machines without understanding the scientific and even philosophical underpinnings of human intelligence. But sometimes enough messing around yields results. I guess we'll see.

I am not so sure about that. Using Claude yesterday it gave me a correct function that returned an array. But the algorithm it used did not return the items sorted in one pass so it had run a separate sort at the end. The fascinating thing is that it realized that, commented on it and went on and returned a single pass function.

That seems a pretty human thought process and shows that fundamental improvements might not depend as much on the quality of the LLM itself but on the cognitive structure it is embedded.

I've been writing code that implements tournament algorithms for games. You'd think an LLM would excel at this because it can explain the algorithms to me. I've been using cline on lots of other tasks to varying success. But it just totally failed with this one: it kept writing edge cases instead of a generic implementation. It couldn't write coherent enough tests across a whole tournament.

So I wrote tests thinking it could implement the code from the tests, and it couldn't do that either. At one point it went so far with the edge cases that it just imported the test runner into the code so it could check the test name to output the expected result. It's like working with a VW engineer.

Edit: I ended up writing the code and it wasn't that hard, I don't know why it struggled with this one task so badly. I wasted far more time trying to make the LLM work than just doing it myself.

A tip: ask Claude to put a critical hat on. I find the output afterwards to be improved.

Do you have an example?

Reply on news.ycombinator.com