Hacker News Reader

Comment by dullcrisp | Hacker News Reader

You don’t have to believe AGI is impossible to believe we have no route to get there. The burden of proof on “we have a way to do this” falls heavily on the person proposing it. And being able to do something that wasn’t expected doesn’t imply the ability to do a specific other thing. It’s that belief that’s I’d say is perhaps more cultish than religious.

View on HN

Who’s saying we definitely have a way to do this? Very few. Nobody’s trying to prove that we definitely do right now.

But the “presumption” that we can’t is silly given our advancements

This logic doesn’t work with anything else. If I say we’re going to fly to Jupiter you’re going to ask me for specifics not just presume that I know what I’m talking about. Or any of the other infinite things I can claim to be able to do. Why is this one different?

In that scenario, you are the one claiming we will "never" fly to jupiter.

Just to close the metaphorical loop here, the current AI debate is something like:

"There's a danger we'll be able to fly to Jupiter and bring home a Jupiterian life-form that will destroy humanity. Therefore we should stop all space exploration."

The doomers take the equation with some wildly improbable step that we can't currently explain or justify, multiply the outcome of that step by "infinitely bad", and conclude that the whole thing must be dangerous. But to amp up the rhetoric, they always skip over the "wildly improbable" part.

I guess the disagreement you have with doomers is the probablility you assign to us being able to create human level AI. For the record most "doomers" don't pull a pascal's wager here. Standard EA doctrine is that if we stagnate technologically at our current level for too long (read centuries) then we will inevitably be wiped out. So they want technological advance, they just want caution.

I don't know what "most" do, but I see that form of argument all the time -- particularly from the highest profile doomers.

Personally, I think this line of argument is driven by the hype cycle more than reality. Use chatGPT or midjourney or whatever for a while, and it's pretty easy to see that we're dramatically overweighting theories of AGI risk, and dramatically underweighting stuff like "disemploying the bottom 80% of the intelligence bell curve" with technologies that automate away lots of formerly white-collar labor.

If I had to put my money on an "existential risk" attributable to AI, it would be economic strife.

It's worth saying we still haven't seen a single model trained post-hype cycle from openAI. GPT-4 was summer 2022 stuff.

I’m saying that there are very many things that I can’t prove are impossible that I don’t worry about happening. I’ve learned to live with that uncertainty.

Hmm. When the first basic black powder rockets were invented, would it have been unreasonable for someone to think that they could one day reach the moon? Of course there would be many uncertainties -- including what the moon even was. No one could say what technology would be required. But clearly there was a major paradigm shift that suddenly made it within the realm of consideration.

I imagine it might have been unreasonable to suppose that, yes. The fact that the future is unpredictable doesn’t mean that all claims about the future are a priori equally valid.

In the future where AGI takes over the world next year I might look silly for arguing with you about it, but that’s a risk I accept based on the fact that I don’t think that’s going to happen.

We absolutely have a way to get there. A sufficiently large neural network can approximate any function (the universal approximation theorem), which has been mathematically proven. LLMs are trained to predict the next word in a sequence; doing this to a high enough level of accuracy requires building an internal model of the world; the ability to reason about it. Eventually we'll have LLMs trained on visual and audio input too (and be trained to predict what happens next), so they can receive an arbitrarily large amount of training samples. If the models keep growing and being trained on more and more data, we'd expect them to become at least as intelligent (good at modelling the world) as humans.

> LLMs are trained to predict the next word in a sequence; doing this to a high enough level of accuracy requires building an internal model of the world; the ability to reason about it.

I don't see any evidence that this is true or proven in any way.

Also, the ability to brute force something doesn't mean that brute forcing it is easy or even feasible. You can apply the same logic to "calculating a private key from a public key" that you are to human intelligence. Sure, a large enough neural network can do either, but that doesn't mean that building them is actually realistic.

It has been proven in much simpler toy models that LLMs just trained on token prediction can and do build "world" models[0]. That's not the same as evidence that they must build a world model, but it's proof that at least sometimes they do so even when just being trained on prediction.

[0]https://thegradient.pub/othello/

This keeps popping up, but a sufficiently large neural network can approximate any continuous function. They can't approximate any discontinuous function. So, there is no absolute certainty that they must be able to approximate human cognition, since there is no reason to believe cognition is continuous.

Also, there is no proof and no reason to believe that any current architecture and, more importantly, that the currently known training algorithms can achieve anything close to cognition. After all, animals and humans seem to learn much much much faster (much smaller training sets) than any algorithm we have so far.

This keeps popping up, but a sufficiently large neural network can approximate any continuous function. They can't approximate any discontinuous function

Incorrect. That's what nonlinear activation functions are for.

Researchers wasted many potentially-fruitful years because Minsky and other luminaries -- people who occupied the same position of authority that LeCun occupies today -- made a huge deal about how the original perceptron and subsequent multilayer variants "couldn't learn XOR." That was almost literally how they put it. At the first sign of difficulty they abandoned the approach that was, and remains, the most promising. We have to be careful to learn from that mistake.

You have to be a lot clearer about what you mean by continuous here. An LLM technically does not produce a continuous function on the reals because its inputs are floating point numbers with finite precision. Any function on discrete inputs like this has an extension to the reals which is continuous, just imagine joining all the discrete points up with lines.

So then your claim wouldn't be about the limits of LLMs themselves, but on the limits of systems that do not take continuous inputs. The question then is do you think that humans take in continuous input?? Given that physics seems to be discrete at the low level, this suggests to me they don't, but I don't know enough to be sure.

It is quantum which isn’t exactly discrete. You receive one photon or two, not 1.5, sure, but the energy of that photon, its frequency is a real. At least I am not aware of a quantum mechanics over Q. I think the math would be hard because you loose all the convergence properties. Maybe there is some Hilbert space over computable reals, Google is not finding it but I no longer find that to be indicative of anything.

I mean continuous in the calculus sense, as in it doesn't have discontinuities, not as in continuous VS discrete.

> Given that physics seems to be discrete at the low level, this suggests to me they don't, but I don't know enough to be sure.

This is a misunderstanding of quantum mechanics. Only certain specific quantities come in discrete quanta (spin, charge, certain energy levels, etc). Other physical quantities are very much continuous - notably time and space. In fact, much of the mathematics of QM is not discretizable, it won't work if you try to make time or space discrete.

But continuous depends on the topology of the space you are working over. If the topology is discrete any function is continuous. I've found some papers that argue that simply from an error corrections standpoint the signals in the brain need to be discrete.

The whitepaper operates on mathematical numbers, though. They didn’t calculate it with floating points of n-bits.

Given a fixed level of precision for input and output, and a function on this discrete space, we can construct a continuous extension of this function to the reals. Now using the paper we know that there is a neural network with continuous weights that approximates this continuous extension to arbitrary precision. If we imagine rounding the continuous input and output to the fixed precision specified, then because of continuity of the neural network, and the fact there is a finite number of weights, we can choose a tolerance by which we can change each of the weights such that the output does not change by more than the precision of the output value. Thus we can pick a level of precision for the weights where both the weights and inputs and outputs are discrete.

There is a leap from "predicting the next word" to "modelling the world" that you are sort of glossing over here.

The idea that LLM's are actually "reasoning" rather than just performing a probability function seems a little bit...pseudo-religious. Like we've created a new life form.

Predictive models need not at all reflect what actually happens. Sometimes embracing less accurate models is actually need to enhance understanding. Epicycles in the geocentric model could be made very accurate as far as observations in the sky are concerned, but it wasn't a good world model.

That whitepaper operates with mathematical, infinite precision numbers.

It’s just a categorical error to read too much into it - it’s not really a too interesting property to be able to get closer to something forever, without some sane growth function.

Reply on news.ycombinator.com