Hacker News Reader

Comment by Lerc | Hacker News Reader

As a counterpoint, I feel like the belief that we can't get to AGI comes off as religion. It presupposes an ineffable quality that we posses that machines cannot. The argument that we have to fully understand something to build it doesn't hold water for me. I have made plenty of things where understanding the depths of what I had made took far more time and effort than building the thing in the first place.

It's always hard to predict the rate of progress, Most of the current optimism comes from how radically wrong predictions were for the capabilities of AI today. 10 years ago a lot of people would have put current AI capability as arriving well after 2050. The jump in progress may not be sustained, but it definitely places doubt on people confidently predicting slow progress.

View on HN

> I have made plenty of things where understanding the depths of what I had made took far more time and effort than building the thing in the first place.

An example of this is electricity. The battery (1800), electric motor (1821) and telegraph (1832) were all invented before the discovery of the electron (1897).

When people say we can't get to AGI, they're typically either in the camp that there's something intrinsically special about us that is incapable of being replicated, or various versions of the belief that it's too complex to be achieved within a timespan in which they would still recognize humanity.

The former can be seen as a religious belief, but the latter is simply saying that we do not understand anywhere near enough about intelligence to develop AGI.

You point to the unexpected leap in capability we've had recently, but fundamentally it isn't that different from what we have had for decades. The same fundamental unsolved limitations exist, such as of being unable to learn the way we do (eg if a child is writing a specific letter wrong, we correct that specific mistake, but that doesn't overwrite the knowledge of how to write the other letters).

As a lecture I watched a while ago had put it, we've gotten pretty good at making the icing of the AGI cake recently, but we still have no idea what the cake is supposed to even look/taste like, so we barely know what ingredients we might need to make it let alone the actual steps involved.

>not understand anywhere near enough about intelligence to develop AGI.

LLM training is essentially training them to predict what happens next. When we combine this with multi-modal training (video/audio), they're being trained to predict what happens next in the world. As we feed them more and more data and make them bigger and bigger, they'll get better and better at this task. Given that predicting what happens next requires predicting what humans do, because humans are part of the world, if they keep getting better and better then they'll be able to predict what humans would do well enough that they're capable of thinking any thoughts humans could think, because that's a requirement for predicting human behaviour. So just by training them on this one label, predict what happens next, we'd expect them to eventually develop human-level intelligence if the loss keeps decreasing.

That doesn't really follow at all.

It may result in something that is even better at faking the appearance of intelligence, but LLMs and similar stuff fundamentally lacks various features that make them not capable of becoming human level intelligences.

Humans are pretty good at faking intelligence too! It's only when you question them carefully and you yourself are an expert in the field that you can detect that they're confabulating.

Conversely, I've never seen anybody produce anything but the next word coming out of their mouth. Therefore people are glorified stochastic parrots.

You're conflating knowledge of specific subject areas with generalized intelligence, including the ability to learn and metacognition.

What do you mean by learning and metacognition? An LLM can tell you whether or not something is in their context window. Humans regularly come up with "innovative" things that they actually saw from somewhere else and just happen to forget the source.

Intelligence is like magic - every time a machine can do one thing on the list, it stops being intelligence, it's just engineering. Maybe intelligence really is just a big bag of tricks?

> fundamentally lacks

We don't actually know that yet. It may very well fundamentally have all that is necessary for intelligence but other pieces are missing.

Frontier neuroscience and functional processes for how our brains work has more in common with the fundamentals of LLMs than not, vis-a-vis minimization of prediction errors.

I don’t think “fundamentally having something” is actually meaningful here - as it’s enough to show it is Turing complete for it to have fundamentally everything. Nonetheless, it’s probably not excel that will become the next AGI.

If the loss keeps decreasing forever, then that conclusion seems to follow. But it's likely there will be plateaus and ceilings that current neural net architectures cannot surpass. We likely need more breakthroughs to achieve enough generalized reasoning, reliable predictions, alignment, and scalability within hardware constraints.

> Given that predicting what happens next requires predicting what humans do, because humans are part of the world, if they keep getting better and better then they'll be able to predict what humans would do well enough that they're capable of thinking any thoughts humans could think, because that's a requirement for predicting human behaviour.

This doesn't follow. A just as likely (or even more likely) prediction is that they will fail to predict what happens next once it requires modeling humans. It won't get any more progress even if we increase training set size or model size - it will require an architecture change of some kind.

Yes, it's completely possible that we get to some sort of limit and the scaling laws fall down, loss stops dropping even with more data, parameters, compute. But we have zero reason to believe that is the case, currently, it's only hypothetical. In terms of what we've experimentally seen so far, it just keeps getting better as you scale them up, so to find out if you're right and there's some future roadblock we don't know about we just have to keep scaling these things to see if it works. We haven't even seen loss functions start to level out - the closest argument I've seen to that is people misinterpreting the SGD curve (an artificial tapering we apply to the training process) as a fundamental architecture limitation rather than just an attempt to use resources wisely on a given training run.

Basically, training of these models currently only costs a few million dollars, peanuts compared to the budgets of CERN or ITER, and we have zero evidence that we're approaching any sort of ceiling. If we keep scaling, maybe we see some heretofore unpredicted failure that breaks scaling laws like you're predicting. But maybe we don't, and the scaling laws (which have performed incredibly accurately so far) really do hold.

Having linear scaling up to a point is a far more reasonable guess, than a forever scaling function - as per Occam’s razor.

If I have a rubbery string and try lengthening it, it will also show a linear scaling, until it catastrophically fails.

>Yes, it's completely possible that we get to some sort of limit and the scaling laws fall down, loss stops dropping even with more data, parameters, compute. But we have zero reason to believe that is the case, currently, it's only hypothetical [Edit:- and not really a good hypothetical, because there's no theoretical backing that explains current progress but predicts a future discontinuity]. In terms of what we've experimentally seen so far, it just keeps getting better as you scale them up, so to find out if you're right and there's some future roadblock we don't know about we just have to keep scaling these things to see if it works.

Also I have no idea why you think Occam's Razor supports "There will be a discontinuity in this currently accurate observed trend, near enough in the future to be relevant" when the trend is talking about the effects of an algorithm applied to data. By analogy, it's something like claiming Moore's Law won't hold because Occam's Razor suggests at some point there's a physical breakdown, except instead of saying it now you're saying it in 1978, & you're saying it about an algorithmic output rather than a physical thing with an obvious limit (atomic size). Once we see any evidence of all of such a discontinuity, we can adjust what we're doing based on that. Until then, if we want to know whether scaling laws hold we should scale the models.

First of all, there is no theory at all which explains the current progress of LLMs. It's not even a quantifiable progress - it's clearly there, but no one can say with any true precision whether GPT2 to GPT3 to GPT4 is following linear or exponential or whatever else precise function of progress in capabilities. We can clearly see that they are improving, but we have no idea how to accurately quantify this space.

Secondly, we can see that there are obvious quantifiable physical increases in the requirements of training and inference for GPT4 compared to GPT3 and to GPT2. So, we can absolutely make a physical argument: at some point, you'll need too much energy to keep scaling up GPTs for it to be feasible, even if capability would keep improving with model size.

Thirdly, the GPTs require ever more data to train, and there is a limited amount of text data in the world, of which they are already using a significant percent. So, there is also an argument that we might run out of data to improve even before we run out of physical resources.

And making an argument that Moore's law will stop at some point due to physical constraints was just as correct in 1978 as it is in 2024. Predicting when that is is obviously hard - and the same is true for LLM scaling. We may be at the peek, or there may be 100 more generations before we even approach the limits I'm contemplating. But the limits are certainly there.

This conversation started with someone saying that despite LLMs having a theoretical basis for how they could emulate human intelligence, they won't actually reach that point because the existing scaling laws won't hold. I am pointing out that we don't know that at all, scaling laws are so far fantastically accurate, and there is no sign of undershooting existing those laws so far. Therefore, it's very premature to declare that LLMs will be unable to reach any given milestone because scaling laws will hit a discontinuity before we get there. Basically:

"There isn't even a theoretical basis for how LLMs could achieve human level intelligence"

"There is, a neural network trained to predict the next token isn't restricted in the algorithms it can implement internally to achieve that goal, and eventually for loss to be low enough you need to successfully emulate human minds. We have reason to believe loss will continue to go lower because we have very accurate scaling laws which predict that & haven't been wrong so far."

"But an LLM won't be able to reach that point because the scaling laws will break down."

"Maybe, there's no evidence for that yet but we should scale them up if we want to find out for sure."

You are here -> "But in the physical extreme LLMs will break down as an architecture."

To which my response is: Yes, of course this specific architecture will break down in the physical extreme. By the time something is building computronium to run the next AI I strongly doubt it will be running matrix multiplication, let alone LLMs specifically. There is no guarantee or reason to believe that we will hit those limits before we are able to achieve human level intelligence or more; the only reason to believe it will happen is an empirically observed trend which could totally break down, so I wouldn't bet the house on it happening, but it's wrong to just rule out the outcome.

No, I am at the point where I'm saying the scaling "laws" will break down.

Especially since there are no scaling laws, it's an empirical observation based on essentially 3 examples - GPT2 to GPT3 to GPT4 (and their rough equivalents from Facebook and Google and a handful of others). This all happened in the span of 4 years or less. We don't even fully know how much larger GPT4 is compared to GPT3.

Not to mention, we already know OpenAI has spent more than a hundred millions of dollars training GPT-4, and they are quite probably selling it at a loss (each query may well cost them more in compute power than they charge for it). So, if GPT4 is ten times bigger than GPT3(.5) as sometimes reported, economics suggest they may already not be able to scale GPT5 to the same extent, and are definitely not going to be able to reach GPT7 (1000 times GPT4, or roughly 100 billion dollars). So, unless you believe compute power will go down in cost massively, or that we are just a factor 10-100 away from human-level intelligence, odds are good we won't see it from this line of AIs in the next 10-20 years.

>Especially since there are no scaling laws, it's an empirical observation based on essentially 3 examples - GPT2 to GPT3 to GPT4 (and their rough equivalents from Facebook and Google and a handful of others). This all happened in the span of 4 years or less. We don't even fully know how much larger GPT4 is compared to GPT3.

I don't think you fully understand the scaling laws. They hold for those models, sure, but also other models at the same scale, smaller scale models, models in-between those examples, etc. They're an empirically derived law, for sure, but so far they've been very accurate for prediction, not just description. We know how much the loss will drop for a given amount of compute and data.

>Not to mention, we already know OpenAI has spent more than a hundred millions of dollars training GPT-4, and they are quite probably selling it at a loss (each query may well cost them more in compute power than they charge for it). So, if GPT4 is ten times bigger than GPT3(.5) as sometimes reported, economics suggest they may already not be able to scale GPT5 to the same extent, and are definitely not going to be able to reach GPT7 (1000 times GPT4, or roughly 100 billion dollars).

I think there's a decent chance even that much money could be spent if there's appropriate returns from GPT-5 & GPT-6. More importantly:

>So, unless you believe compute power will go down in cost massively,

Why on earth wouldn't you believe that? That is what has happened consistently for decades, and unlike general purpose computing which is slowing down, the fact that we need so much compute for one relatively specific operation means we can build ASICs for them (GPUs are already a chunk of the way there) and see very large speedups. As well, at these scales energy costs are a huge part of the training costs, and energy costs are continuing to drop dramatically because of solar. Compute is also a very flexible energy load, you can build it such that it can be paused at night if necessary, which could make it a very good fit for solar energy overproduction. Basically, there's a lot of low hanging fruit in terms of cost reductions for AI training available and a long trend of the cost of compute getting cheaper, it seems silly to just assert that your position (trends won't continue, there will be a discontinuity before we reach whatever point human level performance happens to be at) is the most likely outcome. All else being equal I would expect the existing trends to continue.

My two cents on the topic. Currently, we are incapable of modeling causality effectively or at all (depends on your definition and to who you ask). LLMs seem to have it, but it's a mockery of causal reasoning. They use data created by us and so they can predict the next word following some of the inherent patterns we humans use which might look like they can reason to some extent.

In this regard something like alphago/zero (in the way I understand them at least )are better than modern LLMs albeit in a very narrow field.

The thing is that while the two camps exist among those who say we can't get to AGI, there is no real equivalent of the first camp among those who think it is possible. I'm not counting the people who say it has already (as good as) been achieved, as this claim is of the sort that should have been satisfactorily empirically verified before it is made.

When someone says that skepticism in human-level AGI probably lies in religious beliefs, it tells me that the person likely believes that they have an idea what science and philosophy are saying on the matter. But I strongly suspect that they never actually looked.

Among the small but growing number of modern idealists (the philosophy that consciousness precedes matter), a great deal are actually computer scientists who originally set out to work on AGI from a materialistic frame of reference, but bumped into the conscious machine problem. Once they more closely explored what the cognitive fields working to find answers to this problem are saying, they realized that most scientists approaching consciousness as an emergent property of matter will probably never get anywhere with this.

Why is "the hard problem of consciousness" inextricable from human-level AGI? Because intelligence is only one component of our mentation. There is a back and forth between processing and experience that make up the whole thing. A machine can have or even exceed human processing power, but it will not experience the simplest of colors. Its approach to "creating" can be random hit-or-misses, or formulaically from existing input. But without subjective experience it cannot be as intently creative as an artist trying to come up with a new genre, or as funny as a comedian that relies on the shared subtleties of feelings and emotions (conscious experience) to make original work that other humans can resonate with. It cannot have the intuition of a gifted chef who suspects that two seldom combined spices might be an unexpected but resounding success in her new dessert.

These are just opinions, there is no reason to imagine the phenomenology of qualia is so tightly coupled with creative work and/or making good conjectures. There is a lot of smoke…

What is 'subjective experience' in your view?

Multi-modal models are clearly the future, if we provide models with a robot body - sensors, motors, the ability for locomotion - would that be subjective experience?

Does gold have an 'ineffable' quality that hydrogen does not? Do cells, after 1bn years of evolution have a quality that gold does not? Does reality work one way, and not another? Are not all things unequal?

A "machine" is what, exactly? Do we take it to be an abstraction? Or is it an electrical field oscillating over silicon? Either way, you're in trouble. Abstractions have no physical properties, and electrified sand seems hardly to possess any interesting properties.

The ability for animals to adapt to their environments, by growing into them, by establishing plastic causal connections in their very bodies, grown by their environemtns... able to almost instantly move from protein expression in 1trn 1-bn-yr cellular supercomputers in each of our bodies to macro sensory-motor representation --- and back again

Is this ineffable?

Or is this extremely effable. Is rather, not the superstitious view that "everything is anything" ?

All extant, knowing, studied intelligent systems have organic properties; and radically so. Insofar as this is "ineffable" you should take that up with the animal kingdom.

I find the contrary supersitious, magical, religious, ineffable... that mere abstract patterns in arbiatrily chosen aspects of our bodies are necessary and sufficient conditions for anything at all. This would be the only case in all science. The only physical property instantiated by mere arrangement at any level. Upload our consciousness? Make it out of wood why not!

Nonesense. When I am hungry, I dream of food, when I dream of food I plan to get some, and I am angry without it. This will not be made out of sand.

Given that you need some pretty complex elements to be able to make life in the first place (not to mention complex molecules made out of those elements) I'd say yes, there is definitely an 'ineffable' quality that life has that Hydrogen does not (to short circuit your chain of reasoning), but intelligence isn't necessarily made of a particular kind of matter and that's where the analogy breaks down.

The soul, likewise, is immaterial.

Who now trades in superstition?

Everything is corporeal.

Downvoted for saying reality has physical properties? Where else but hackernews? Sorry to disappoint, but all the algorithms of CS, which do anything at all, have devices in them. ie., their critical steps are impure, and cannot be specified mathematically.

There is nothing abstract in the world, everything is concrete. Discrete mathematics is not some divine realm which is where the Mind really occurs, and the Soul is really kept.

No graph traversal algorithm cares, nor will ever. Science is the study of reality, not computer "science". There are no computational "properties". All interesting properties are of objects extended in space, and in time.

The world is physical, as described by physics; not abstract, as describe by mathematics. And so, not computable.

My mind has abstractions in it. Are you saying that mental formations are not real, not in the universe?

>Where else but hackernews?

I have seen this sentiment on every single platform I have participated in. Where else? Everywhere that humans go.

Find a more appropriate (or at least original) lamentation.

The proposition of artificial intelligence, whether at human-level or not, is that a suitably-programmed computing device (as opposed to an immaterial algorithm) could manifest the properties which we have chosen to call "intelligence" when we observe them in biological agents, and similarly for consciousness.

Particularly with your comment "no graph traversal algorithm cares, nor will ever", you appear to be trying to impute that the concept of artificial intelligence is predicated on a category error. If so, then this is a fairly common argument, but is itself predicated on a category error.

What about platonic forms

The laws of physics are corporeal?

We don't have any proof to suggest that "intelligence" grants a fundamental understanding of everything though. It can help us understand "things", do neat stuff, and build models to explain the working and structure of phenomena; However would having an IQ of 5000 mean you are now "God" or gain and understanding of everything including the mystery of existence itself?

At some point, it seems important to stop trying to understand the world through ideas and concepts and just experience it.

I really think there are more religious like beliefs on the pro AGI camp than the other way around, think about what AI promises to deliver by "visionaries" like Hinton and Kurzweil:

* Immortality.

* Resurrections.

* Nirvana / Heaven / Utopia.

For all we know, we create an AGI who becomes amused with fucking itself, or develops new games it finds interesting and spends all eternity playing them. Humans have a belief that we're going to create "god" as we think of "him". All seeing, all knowing, all powerful. It does get a bit ridiculous at times.

I sympathize a lot wit the parents view personally.

Good point, especially that intelligence doesn’t quite scale. Many (most?) of our problems are hard to solve due to lack of data/experimentation over lack of intelligence. And more intelligence won’t help with the infinite scalability of complexity.

Some folks I suppose take a religious fervor, but for my I look at how a child learns -- through the observation of cause and effect -- and know the current architectures don't support how we have husbanded animals and reared children for eons.

The inherent capacity for true self-directed learning may well be there, but it isn't hit yet in what we see.

GPTs are still neat.

I'm not sure that's true. There is reinforcement learning and its branches. I think that's part of how they do ai for chess, go, star craft etc...

Why do you think computers would most efficiently learn the way we do? They don’t do anything else (eg arithmetic as an obvious example) anything like humans normally do.

A deterministic algorithm that adjusts floating point errors for varying chipsets is more complex, agreed. So current implementations are not terribly simple for arithmetic. That highlights my point, I believe, that the only examples of actual learning, where one can make true inferences about the world without the need to collect the entirety of the internet to build transformers, we have to date are wetware, not software.

You don’t have to believe AGI is impossible to believe we have no route to get there. The burden of proof on “we have a way to do this” falls heavily on the person proposing it. And being able to do something that wasn’t expected doesn’t imply the ability to do a specific other thing. It’s that belief that’s I’d say is perhaps more cultish than religious.

Who’s saying we definitely have a way to do this? Very few. Nobody’s trying to prove that we definitely do right now.

But the “presumption” that we can’t is silly given our advancements

This logic doesn’t work with anything else. If I say we’re going to fly to Jupiter you’re going to ask me for specifics not just presume that I know what I’m talking about. Or any of the other infinite things I can claim to be able to do. Why is this one different?

In that scenario, you are the one claiming we will "never" fly to jupiter.

Just to close the metaphorical loop here, the current AI debate is something like:

"There's a danger we'll be able to fly to Jupiter and bring home a Jupiterian life-form that will destroy humanity. Therefore we should stop all space exploration."

The doomers take the equation with some wildly improbable step that we can't currently explain or justify, multiply the outcome of that step by "infinitely bad", and conclude that the whole thing must be dangerous. But to amp up the rhetoric, they always skip over the "wildly improbable" part.

I guess the disagreement you have with doomers is the probablility you assign to us being able to create human level AI. For the record most "doomers" don't pull a pascal's wager here. Standard EA doctrine is that if we stagnate technologically at our current level for too long (read centuries) then we will inevitably be wiped out. So they want technological advance, they just want caution.

I don't know what "most" do, but I see that form of argument all the time -- particularly from the highest profile doomers.

Personally, I think this line of argument is driven by the hype cycle more than reality. Use chatGPT or midjourney or whatever for a while, and it's pretty easy to see that we're dramatically overweighting theories of AGI risk, and dramatically underweighting stuff like "disemploying the bottom 80% of the intelligence bell curve" with technologies that automate away lots of formerly white-collar labor.

If I had to put my money on an "existential risk" attributable to AI, it would be economic strife.

It's worth saying we still haven't seen a single model trained post-hype cycle from openAI. GPT-4 was summer 2022 stuff.

I’m saying that there are very many things that I can’t prove are impossible that I don’t worry about happening. I’ve learned to live with that uncertainty.

Hmm. When the first basic black powder rockets were invented, would it have been unreasonable for someone to think that they could one day reach the moon? Of course there would be many uncertainties -- including what the moon even was. No one could say what technology would be required. But clearly there was a major paradigm shift that suddenly made it within the realm of consideration.

I imagine it might have been unreasonable to suppose that, yes. The fact that the future is unpredictable doesn’t mean that all claims about the future are a priori equally valid.

In the future where AGI takes over the world next year I might look silly for arguing with you about it, but that’s a risk I accept based on the fact that I don’t think that’s going to happen.

We absolutely have a way to get there. A sufficiently large neural network can approximate any function (the universal approximation theorem), which has been mathematically proven. LLMs are trained to predict the next word in a sequence; doing this to a high enough level of accuracy requires building an internal model of the world; the ability to reason about it. Eventually we'll have LLMs trained on visual and audio input too (and be trained to predict what happens next), so they can receive an arbitrarily large amount of training samples. If the models keep growing and being trained on more and more data, we'd expect them to become at least as intelligent (good at modelling the world) as humans.

> LLMs are trained to predict the next word in a sequence; doing this to a high enough level of accuracy requires building an internal model of the world; the ability to reason about it.

I don't see any evidence that this is true or proven in any way.

Also, the ability to brute force something doesn't mean that brute forcing it is easy or even feasible. You can apply the same logic to "calculating a private key from a public key" that you are to human intelligence. Sure, a large enough neural network can do either, but that doesn't mean that building them is actually realistic.

It has been proven in much simpler toy models that LLMs just trained on token prediction can and do build "world" models[0]. That's not the same as evidence that they must build a world model, but it's proof that at least sometimes they do so even when just being trained on prediction.

[0]https://thegradient.pub/othello/

This keeps popping up, but a sufficiently large neural network can approximate any continuous function. They can't approximate any discontinuous function. So, there is no absolute certainty that they must be able to approximate human cognition, since there is no reason to believe cognition is continuous.

Also, there is no proof and no reason to believe that any current architecture and, more importantly, that the currently known training algorithms can achieve anything close to cognition. After all, animals and humans seem to learn much much much faster (much smaller training sets) than any algorithm we have so far.

This keeps popping up, but a sufficiently large neural network can approximate any continuous function. They can't approximate any discontinuous function

Incorrect. That's what nonlinear activation functions are for.

Researchers wasted many potentially-fruitful years because Minsky and other luminaries -- people who occupied the same position of authority that LeCun occupies today -- made a huge deal about how the original perceptron and subsequent multilayer variants "couldn't learn XOR." That was almost literally how they put it. At the first sign of difficulty they abandoned the approach that was, and remains, the most promising. We have to be careful to learn from that mistake.

You have to be a lot clearer about what you mean by continuous here. An LLM technically does not produce a continuous function on the reals because its inputs are floating point numbers with finite precision. Any function on discrete inputs like this has an extension to the reals which is continuous, just imagine joining all the discrete points up with lines.

So then your claim wouldn't be about the limits of LLMs themselves, but on the limits of systems that do not take continuous inputs. The question then is do you think that humans take in continuous input?? Given that physics seems to be discrete at the low level, this suggests to me they don't, but I don't know enough to be sure.

It is quantum which isn’t exactly discrete. You receive one photon or two, not 1.5, sure, but the energy of that photon, its frequency is a real. At least I am not aware of a quantum mechanics over Q. I think the math would be hard because you loose all the convergence properties. Maybe there is some Hilbert space over computable reals, Google is not finding it but I no longer find that to be indicative of anything.

I mean continuous in the calculus sense, as in it doesn't have discontinuities, not as in continuous VS discrete.

> Given that physics seems to be discrete at the low level, this suggests to me they don't, but I don't know enough to be sure.

This is a misunderstanding of quantum mechanics. Only certain specific quantities come in discrete quanta (spin, charge, certain energy levels, etc). Other physical quantities are very much continuous - notably time and space. In fact, much of the mathematics of QM is not discretizable, it won't work if you try to make time or space discrete.

But continuous depends on the topology of the space you are working over. If the topology is discrete any function is continuous. I've found some papers that argue that simply from an error corrections standpoint the signals in the brain need to be discrete.

The whitepaper operates on mathematical numbers, though. They didn’t calculate it with floating points of n-bits.

Given a fixed level of precision for input and output, and a function on this discrete space, we can construct a continuous extension of this function to the reals. Now using the paper we know that there is a neural network with continuous weights that approximates this continuous extension to arbitrary precision. If we imagine rounding the continuous input and output to the fixed precision specified, then because of continuity of the neural network, and the fact there is a finite number of weights, we can choose a tolerance by which we can change each of the weights such that the output does not change by more than the precision of the output value. Thus we can pick a level of precision for the weights where both the weights and inputs and outputs are discrete.

There is a leap from "predicting the next word" to "modelling the world" that you are sort of glossing over here.

The idea that LLM's are actually "reasoning" rather than just performing a probability function seems a little bit...pseudo-religious. Like we've created a new life form.

Predictive models need not at all reflect what actually happens. Sometimes embracing less accurate models is actually need to enhance understanding. Epicycles in the geocentric model could be made very accurate as far as observations in the sky are concerned, but it wasn't a good world model.

That whitepaper operates with mathematical, infinite precision numbers.

It’s just a categorical error to read too much into it - it’s not really a too interesting property to be able to get closer to something forever, without some sane growth function.

>> The belief that we can get to AGI comes off as religion to me. It is a substitute for something we can’t really understand, and it will continue to shift the more we learn, yet always remain out of reach. There will be some true believers, and some people simply gunning for power.

> As a counterpoint, I feel like the belief that we can't get to AGI comes off as religion. It presupposes an ineffable quality that we posses that machines cannot.

That's missing the point: the argument isn't that it's theoretically impossible to build an AGI, just that humans are incapable of it.

Here's another point for the belief in AGI being a religion: it's basically a sect of the larger religion of technological progress, which (likely falsely) assumes that technology will continue to "advance" at approximately current rates until we live in a world out of a sci-fi paperback. A lot of people believe that, but frequently resort to a motte-and-bailey fallacy when challenged.

> It's always hard to predict the rate of progress, Most of the current optimism comes from how radically wrong predictions were for the capabilities of AI today. 10 years ago a lot of people would have put current AI capability as arriving well after 2050. The jump in progress may not be sustained, but it definitely places doubt on people confidently predicting slow progress.

It's worth noting that "the current optimism" is not without precedent. IIRC, there was a big boom in AI in the 70s/80s. SHRDLU was pretty impressive. But then then the promising ideas were found to be dead ends and there was a long winter.

in many ways, we already live in a world out of a sci-fi paperback.

I find it hard to fault people for thinking that we'll continue to progress at a fast pace, or at least thinking that we're a long way from a major plateau.

I think that intelligence has a “bootstrap” level - that human achieved - after which it can practically do most stuff, the bottleneck won’t be intelligence itself. E.g. our mathematics itself has fundamental limitations on what it can prove - I don’t think some machine/different intelligent entity could get further away. At most they are getting their faster.

We can not create the laws of physics, and we are fine with that aspect of immaterial reality.

Reply on news.ycombinator.com