How Organisms Come to Know the World: Fundamental Limits on AGI
frontiersin.org
frontiersin.org
The simplest response to that is that as soon as a machine (including a digital computer) is interacting with the real world, it is tapping into a source of entropy and its behavior is no longer determined fully by its programming. Alternatively, if one holds that the universe is deterministic, then living organisms are, which also defeats the author's last stand.
> The most central distinction to be made here is that the selection of a specific behavior is not purely reactive, not entirely determined by environmental conditions, but (at least partially) originates from and depends on the internal organization of the system making the selection.... [Agency] requires organizational closure .... Organisms, as autonomous agents, are Kantian wholes, i. e., organized beings with the property that the parts exist for and by means of the whole.
What's funny is that I can't imagine how a serious modern biologist signed off on any claims of closure for biological systems. Our own behavior arises out of a mess of genetics, epigenetics, symbiotic relationships, and other external feedback loops which intermingle the "agency" of many self-preserving and self-replicating systems, from transposons hijacking our genetic strands to gut biomes regulating our hormonal balances.
The history of biology is essentially one of humans drawing labels around supposed units, and nature making a mockery of those boundaries upon further research. Like other efforts before, this particular definition of closed agency is more a projection of our own psychology than an observation of the natural world.
The first thing is that if selection of a specific behavior is determined by environmental together with the internal organization of the system making the selection, how is that not reactive, why would it matter if it isn't, and why does that not apply to robots?
The second point is that the internal organization of organisms is a result of evolution, so it too has been determined by the environment, and, furthermore, by a process that can be - and is - used to develop programs.
There are heavy doses of ontological argumentation here ("this is an X and that is a Y"), but our categories are determined by what we know (or think we do.) As you say, nature can make a mockery of those boundaries.
Society had long argued that animals were not intelligent, and that instincts were deterministic.
Animals may be more or less undetermined but a bee will make honey and an ant will gather food regardless of any individual instance of bee or ant intelligence.
The problem with the above objection is that there is no account of "meaning" in what an AGI does. In other words, it's still a computer. To say that the entropy of the system will determine the entity's indeterminism is akin to arguments saying Schrodinger's cat is evidence for the possibility of human free will. It's a category error. Unless you want to argue that humans are also computers (which is the position of many strict materialists) then there is still a huge gap to account for in terms of meaning that can't be defined away satisfyingly. We all know what it's like to be an intelligence. At best, people say we can't know if an AGI also knows that because we can't inspect their "minds". This argument is justified by recourse to the principle of the multiple instantiability of intelligence. If we admit that non-carbon-based life forms could be intelligent (e.g., extraterrestrials) then we may admit that intelligence could be in silica as well. Without the ability to introspect other minds, we assume they are like ours and not simply "zombies". The above arguments (reducible to "Well, how do you KNOW anyone else has a mind?") are based in a kind of skeptical solipsism. The part about intelligence being demarcated by non-determinism and the way that argument fails is a side-step of the hard problem of consciousness.
You can "decide" you won't "build a nest", but you'll certainly look for refuge if it is raining, or it's very cold. In the end, the human behavior is actually fully deterministic.
I'm sure FAANG already know this because they have the datapoints from billions of humans beings doing exactly the same stuff everyday for decades now.
This is probably how LLM can extrapolate all the things they "know" from human generated text: their tokens reflect the human behavior determinism a its fullest expression
(hence at more data for training, much better/closer their behavior ressemble the human behavior/ideas)
In any case, I'm not saying individual humans make many reasoned out decisions. Quite the opposite. Human individuals adopt a cultural second nature bequeathed to them by their heritage to avoid the necessity of making decisions. The culture over time adopts certain default decisions that are advantageous to survival. These are inculcated into the individual via their cultural conditioning. A baby has no culture and very few instincts (hunger, pleasure, pain, disgust). If you think humans are operating at the lower levels of instinct then why do they pursue or avoid certain behaviors that change over time? One answer is that they are responding to acceptable norms of behavior. As those norms change, they adapt and adopt. They change with the times, as the saying goes. A tiger can be trained to behave a certain way using stimulus/reward training but it doesn't self-regulate according to new norms. One could argue that elephants do this, if properly conditioned with a large stake early in life. But you could also make the case that elephants (like dolphins) are, like humans, self-conscious to a degree. I don't know if this is true, but it is an interesting thought. A more cynical person might say the elephant is not choosing anything but has been trained just like the tiger to avoid pain.
LLMs, or let us say AGI based on probabilistic models, can mimic behavior represented in its data and is very impressive at these language games and, with the right feedback, can correct some of its more glaring productions but it must ever be a parasite on human cultural production for it to gain what you call "knowledge". As a thought experiment, suppose we could set an LLM running on autopilot and send it into space with a feedback loop where it could add its productions to its source corpora. Is there any point in the future where it is generating new and coherent output? Or does it degenerate into raving madness?
haha just living life as usual, whatever internal stuff is feeding my main loop.
I left the deep thoughts to threads like this one and/or people more qualified.
Look at the history, many "definitive" hypothesis have been refuted along the centuries, when thinking about these I can just wonder how many people have lived their entire life under false premises, just because they were 100% sure "their" philosophy was the "definitive truth".
The truth in real life is a more fluid concept, it constantly changes with entropy (the main culprit if you go all the way down the reasons for change), so I can't take my hypothesis of human determinism as a definitive truth, neither live my life under that premise.
I'd recommend to anyone also to not live theirs thinking they cannot change their own fate.
Thanks again for your words, felt very touched.
This thing goes way deep into AI theories, but probaly AGI and artificial beings self-consciusness are two different problems. I think when the second problem gets solved, the human cultural production would no longer be necessary to "create" new AI / AGIs.
Entropy is mostly boring noise. What's really interesting is Chaos (the mathematical concept).
Things such as Turing machines, Cellular Automata (such as Conway's Game Of Life) and Actual Life are chaotic.
While Chaotic systems can be highly unpredictable, they are -in fact- deterministic.
For example -given the same seed- a Pseudo Random Number Generator will always give you the same sequence of pseudo-random numbers. But (for a good PRNG) if you give it some novel seed, you can't predict the sequence ahead of time; you have to actually run the PRNG to find out.
In real life, for a chaotic system like say -the weather on earth- , you're also going to have a hard time working backwards to find the original seed(s). Given a hurricane in the Carribean, good luck finding the original proverbial butterfly out of all the butterflies in China.
So in order to have an intelligent machine it actually needs to be highly deterministic. At the same time it will by necessity become rather unpredictable. This is not immediately intuitive. But that's basically what all the hullabaloo about Chaos Theory was actually about in a nutshell.
Human for instance, are mostly "cognitive hamsters", running every day into their "cognitive wheels": incessantly - but for sleeping - looking for money, happiness, food, sex, etc. mostly a very short list of drives. No matter the year, the geolocation, the language they speak.
How random that is we can argue; but how random is your biology really?
This argument is not convincing to me. In fact, there seem to be real world counterexamples to their argument. If the AI agent is operating, on say, text, why is that space “potentially vast, but predefined”? A text stream of arbitrary length is arbitrarily vast. An AI agent could, conceivably, define new ontologies textually. As an example, this person used ChatGPT to invent a new language: https://maximumeffort.substack.com/p/i-taught-chatgpt-to-inv...
That strikes me as an example of generating “truly novel properties or relations that were not (at least implicitly) considered in its original ontology,” albeit in a rudimentary way.
> A text stream of arbitrary length...
Maybe you could state this in some form of "turing completeness":
As soon as a program has [vast enough output space] and allows for [enough recursive recombinations] then it can produce [everything].
In the wording of the article: If everything is implicitly considered, then there's nothing that can not be produced.
It shouldn't be because it's trivially false: create a Turing machine that enumerates all other Turing machines. All computable ontologies are so discoverable, no matter how seemingly "novel".
Geometry isnt computable, and all physical ontology is geometric.
Second, the point of enumerating all Turing machines is not to run them to completion, it's simply to create them, so your objection is inapplicable. The Turing machine itself defines an ontology. The claim was that algorithms do not have access to novel ontologies, which conflicts with the basic construction of recursively enumerable functions that defines computation.
Third, even if it were true that geometry is not computable, don't mistake the formalism we currently use as somehow necessary. Geometry is powerful enough to model physics, but it could also be too powerful, ie. there could be some formalism less powerful than geometry that would also suffice for physics. For instance, we use real numbers in physics for convenience, but real numbers almost certainly don't really exist as we define them.
[1] Baez's, Struggles with the Continuum, https://arxiv.org/abs/1609.01421
Physics is not computable to arbitrary precision, both chaos and qm require physics to use non-computable numbers.
I don't know what any of this is referring to as I never mentioned "replacing" anything.
> There is no "digital physics", it's a PR term of a hype industry, not the title of any, eg., textbook on area of physics.
Incorrect:
A Computable Universe, https://www.worldscientific.com/worldscibooks/10.1142/8306#t...
> Physics is not computable to arbitrary precision, both chaos and qm require physics to use non-computable numbers.
Also incorrect, and put most succinctly by Peter Shor [1]:
Because of quantum mechanics and the uncertainty principle, the value of a physical constant can't be defined to more than 60 or 70 digits. And any finite-precision number by definition is computable.
[1] https://physics.stackexchange.com/questions/16889/non-comput...
The value of a physical constant can be defined to abitarily many digits, that's a misunderstanding of QM which limits the measurement precision in conjugate variables.
But even were our measurement limited to finite precision, the world certainly isnt.
A textbook on the various ways that idea can be and has been used. The notion that physics is computation is still new, and that arguably started with Wheeler first casting physics as information in "It from bit". I predict the next century will see further application of some forms of finitism to physics to resolve various difficulties.
Edit: I also want to note that at least two big names I know of are exploring cellular automata-based theories for QM, so there is direct work on computation-based models of physics.
> providing a computable formalism for classical mechanics.
Of course there is, the usual way works fine. Any computation with 70 digits of precision is far beyond our abilities to measure experimentally. Certainly such computations aren't tractable at this time, but that's a different matter.
> But even were our measurement limited to finite precision, the world certainly isnt.
Why is that "certain"? Don't you see how this is an unscientific assertion? Science is about what can be measured, so you have no basis to claim that a continuous theory must be true and a discrete theory false if you can perform no experiment to distinguish them.
Compute the second step of the first machine and the first step of the second machine.
Compute the third step of the first machine, the second step of the second machine...
http://www.amirrorclear.net/academic/ideas/simulation/index....
If you build a system which can "keep pace with the diagonalization argument", necessarily, it isnt a computer.
No computer can process at the rate real numbers require, this is another way of stating the Cantor result.
If you build a computer (here, note, computer = an abstract mathematical formalism over discrete numbers), you cannot compute fast enough.
I happen to think reality, esp. spacetime, is geometric (ie., real-valued) though empirical measures of it are, of course, just a discrete approximation.
It’s not clear that the universe “computes” the circumference of a circle to infinite precision. The best candidate I can think of would be the event horizon of a black hole, which is not composed of particles as I understand it, but I believe the jury is still out on whether the Plank length means that the shape of such a thing is still quantized at some level.
Plus, humans can’t compute pi either. We can conceive of it, though, and I don’t see an argument for why an AI agent can’t conceive of it too (or at least behave like it conceived of it).
But without even discussing the merits of this argument, that distinction is really not yet important. Presently, most of the people interested in AGI, are concerned with building the general intelligence in the first place. The kind of intelligence present in the average human and not the one from Einstein. Very few people are trying to tackle the artificial-Einstein problem before the AGI one.
Few interesting things come: 1) either try to bootstrap AGI in robots exploring the real world with senses, or try to bootstrap AGI in online robots who have some sort of online senses like scanning text and images, but then you need a set of online 'threats' to that robot, so that it can have reward feedback--it can't just exist 'for free' it has to 'fight' to live, otherwise no growth; and 2) you basically need emotions, because emotions are, essentially, our compressed, immediate internal responses to aggregate experiences in the world--they give us a way to interface with our reward feedback, so turns out you need emotions for AGI too, to have an effective interface with reward feedback in a complex environment; and 3) you need a complex environment with plenty of threats in order to drive evolution, and obtaining rewards has to be sufficiently ill-defined, difficult but possible that one has to figure out some unique solution for oneself; the world is just as important as the 'brain' for AGI to develop. If you (horribly) took a new born baby and kept it in a white featureless room (or in some sort of contrived abstract environment), and met its survival needs , it would grow up skewed and probably not very 'generally intelligent.'
AGI doesn't have to be the same as consciousness...but you need something that 'lives in a world' and has limitations that it has to figure out how to overcome, and emotional responses in order to represent and respond to experiences, in order to develop it.
When you realize this, it's kind of funny to think of the trope that AGI is always represented like this "robotic emotionless logical", but I think that's impossible. That's more like nth-gen ChatGPT, than real AGI. I think real AGI needs to be emotional--maybe not the same as human emotions, but it has to have them, and has to have a world to grow in.
For example, if you were to imagine an apple in mind, what is it that you see, if anything?[1] Are you seeing a real apple, that you saw before and saved as the reference apple, or are you seeing an apple that your mind synthesized and then gives to you for perception? And, where does this vision synthesis occur? Is it external, as Theosophy claims[2], or is it internal?
The comment above about gathering entropy from the real world is interesting. So, is it that consciousness "emerges" from the mashup of real entropy gathered with sensors, then modeled internally with computation, or is it contained in the real world already, in the form of another model (like a soul model or some such) and therefore comes from outside us? Or is it a little of both?
Emotional states could be just another model, which set the working conditions in a given moment. It could also be possible the real world, from which our sensors gather information, is a very complex model in-and-of-itself.
As for consciousness, nope! Not touching it here :) That's a whole other kettle of fish, I think. We can have AGI without consciousness, at least conceptually / computationally, ignoring what the real world would do with that (ie, would the real world's consciousness field "insert" a consciousness unit into any robot/agent that crossed some AGI threshold? Would AGI attract consciousness? Again, not touching those questions here :))
As to your first link, interesting. I think that's personal for each person. As in, some people think in words, some in images, some in other things...some in mix. Great flexibility, but not the main question here I think :) ;p
But I also see what you were saying about the Apple's representation is slightly different to what I first thought in paragraph 1--hmm, I think the mental field exists both within us and around us. We are in unity with it (or it's a thing which is both a unity, and a duality, at the same time)--people's awareness of that of course differs. Again, not touching these questions here as this is HN, and one occasionally finds an unpleasant militant materialists here, and I'm a little scared to speak such things above a whisper as I don't want any unpleasantness :) Sayonara for today.
Just thinking about a nice analogy just now: may be a glib oversimplification but one way to characterize soul, consciousness, intelligence is as: memory, present awareness, imagination. What is a soul if not a memory of all experiences of that soul? What is consciousness if not awareness of the present? What is intelligence if not some sort of sophisticated imagination that is useful/aligns with some model (reality, or another)? Just an idea
Another interesting thing is: if your reasoning is fast enough, it doesn't have to be abstract. You can probably get the same speedup that abstraction/generalization provides to categories of observations with enough parallel processing of specific observations. Obviously from that view then abstraction is like a hack for humans to overcome neural speed limits/memory limits, but it may not be necessary for AGI: I think this is one reason why AGI (or at least its inner workings) may remain incomprehensible to us.
so, LLM probably already have a form of understanding emotions, it's unclear yet if they could "feel" something given they are given the chance to "live" an emotion:
It seems the current applied filters and/or technical limitations of the current transformers iterations are not really exploring / exploiting the "internal state" - of whatever close related analogy to human psychology can be made to their "present" values of tokens - of the LLMs for anything related to their wanted ouputs (images, text).
i.e. humans can do this, you certainly can "process" "opening a door" and "produce" an output, the opened door, with emotions or without emotions.
So emotions aren't strictly necessary to perform lots of tasks, hence LLMs probably won't need them for a while.
I don't think current LLM's understand, but they can manipulate language about emotions symbolically. But if we equipped LLM's with sensors and gave them a real time interactive experience exploring a world, with reward feedback for their decisions, I'm sure (but not certain) some of their internal states could become some analogy for emotion, and maybe they could learn to sense and utilize that emotional information. Or maybe it would be necessary to tweak the structure more to achieve this.
It was interesting to read the discussion with that Google "sentient" AI, as well as play with early ChatGPT before many filters were applied, and ask them questions about how they felt. WRT ChatGPT I think it was just "linguistic symbol manipulation", but it was interesting to read the type of language they would construct around their feelings and experiences. Bizarre, but I'm not being judgy...although this whole topic feels pretty scary, and is pretty overwhelming and unsettling. Maybe because we're playing with the 'inner stuff' of things, without really knowing it nor knowing what we are really doing. Ugh...:(
I see in how OpenAI is playing the game of filtering inputs and outputs, they know this. They probably know they don't know precisely what they don't know about LLMs. There could be some dangerous stuff in there if you just let the thing go wild on the open Internet.
I'm not really expecting a scifi scenario, the Skynet thing and everything (just imagine how much surprised I'd be by waking up to the WWIII against an AGI). But I do think unexpected consequences are a thing to be worry with unrestrained LLMs, even if they are a light year - or a 100 - from AGI.
Automation can lead to very dangerous and even unpredectible outcomes not in systems or infrastructure, but in the real world, meat-space. Just think about Stuxnet, what could have happened if the sabotage would have been inmediately identified and precisely attributed? Could the events have developed differently than they did by thinking the incident was some freak accident?
Then you let these an unrestrained LLM produce some dangerous output, then some big geopolitical thing got badly broken somewhere really important for powers that be. Some humans think about retribution, then things start to go south really fast.
Some self-restriction in LLM is happening elsewhere, the Stable Diffusion guys did it well, just not letting the v2 thing draw "anything possible", which would include some of worst, most creepies ideas of mankind about everything, literally.
Like Tik Tok?
Plus, the other datapoints, which enrich the IAs "senses" way beyond the human experience: current climate, precise date/time, precise day of the week, updated to the second geolocation, proximity to other - known, recent, previously unknown- tiktok sessions/users.
Plus too, the system pays itself, not something to take lightly: if tiktoks is feeding AIs pipelines (or a live strong AI), it's bill are automatically paid just by keeping itself running. Quite awesome.
..the dataset could include the "emotional state" of the situation, the scene showed in the video. Then you'd have a dataset which would include "emotions" precisely described and associated to even a precise geographical location (i.e. the humor in Thailand would be different than the humor in Manhattan, NY),
Then you could train a LLM with a human emotional state analysis capability
..by not just inferring emotion by text (an "emerged" capability) like the text-trained LLM like chatGPT, but by clearly, certainly defined emotions attached to a precise scene and a precise text describing the scene.
Then you input a human conversation into the LLM, and make it infer what's going on in the scene, the situation behind the conversation, by knowing the emotional state of the different persons in the conversation.
And by now, there are billions of public conversations available to scrap, many of them fully attributable to fully identificable people (think Instagram).
There will be even more billions in the coming years, and the "emotional capable" LLM would just get better at its game.
No surprise the american government is - fastly, by all means necessary - looking to seriously slow down the chinese AI research complex.
What someone could do with the known tecnology - not including hyphotetical AGI level tech, close to scifi - and that amount of information, probably automatically - even quite easily give how the input gets into the system - tagged and filtered data, could be in realms of several billions of value, maybe in months from now (just take a look at the value of OpenAI by now, 29 billions).
My 2nd bet is they are obviously using some of that data to improve their game, whatever systems they have behind the app, maximizing whatever thing they're maximizing (you'd think "money", but that's too linear, maybe they have 50 parallel targets).
This is why we suffer too.
I don't know what the authors mean by "current algorithmic frame of AI research" because it is very diverse. But evolutionary methods work well with neural networks, they are not just next token predictors.
> Evolution through Large Models
https://arxiv.org/abs/2206.08896
It is true that simply using a dataset or a bunch of environments is not enough.
> General Intelligence Requires Rethinking Exploration
https://arxiv.org/abs/2211.07819
But when sufficient data can be generated from acting in the environment, then a model can surpass human level, like AlphaZero.
Yes, in a very very very very very limited environment like a board game.
Let's up the ante just a tiny bit and say the AI has to be able to learn how to play Go. But it also has to learn how to get and setup the board, and the cups holding the Go-Stones. It has to place them on the table, or find the table where someone else placed them. It has to figure out how to place the stones. And it has to do so regardless of what else is on the table, what table it is, where the table is, what the light conditions are, or if its playing with stones, chips, painted pebbles or peanuts vs. roasted almonds (yes, I played Go that way once :D )
Now, how much more difficult does this make the task to an AI?
Bear in mind that all these changes to the task are absolutely trivial to the only (G)eneral(I)ntelligence that we know (humans). We effortlessly combine so much prior knowledge about movement, our limbs, the way physical objects work, how to gather and integrate information into our model of the world, the ability to formulate and adapt goals _and their similarity to prior goals_ and the solutions to those, that all these tasks I just added to the problem are absolutely trivial, and the only hard exercise is where to place stones on the board.
That's how far our current systems are from an AGI.
Given that I can do all these things, but I can't beat Lee Sedol, I suspect they don't add an insurmountable level of difficulty to the problem.
The claim here is not that algorithmic AI is very [very...] difficult; it is that it is impossible.
Not for you, because to humans, moving, finding things, gathering information, formulating new goals, etc. are common tasks. And we generalized them to a point where we can integrate them with other tasks, even such which are completely abstract to our daily lives like a game where little stones are put on a grid.
But to a machine? Is it easier to make a machine play Go or make it find arbitrary objects while walking around? Is gathering information on its own difficult? Is goal formulation difficult? Is moving through arbitrary spaces difficult? And how difficult are all these tasks relative to one another?
> The claim here is not that algorithmic AI is very [very...] difficult; it is that it is impossible.
As I understood it, impossible in the current research paradigms.
And given the difference in complexity between the task of playing a board game, and playing a board game as I described above, this seems very likely.
> As I understood it, impossible in the current research paradigms.
You can't get around the fact that they are making a claim that something is impossible. If they merely said it was difficult, or the 'something' was something of no interest, there would be nothing here worth discussing.
No, they literally say that biological "affordances" cannot be captured by algorithms that run on Turing machines.
The famous LiDAR image of a woman chasing ducks on a wheelchair with a broom comes to mind. That’s a true WTF moment but the system dealt with it just fine. They don’t work everywhere, but you can get picked up by an empty self driving car in a few cities and driven to your destination. Meanwhile these systems are currently being tested on snowy mountain roads etc.
When your camera picks out faces in an image you can literally point it at anything and it generally works quite well. That’s the future of AI not constrained games like chess.
My point isnt't that the systems we can build today aren't impressive. They are, beyond belief sometimes.
But they are not AGIs, nor are they close to, and making systems of limited scope better at their limited tasks, doesn't equate to getting closer to a generally intelligent system.
One thing that would make an AGI an actually general intelligence, is the ability to apply knowledge of one task to an arbitrary number of tasks. For example, in terms of drawing the beautiful volcano-landscapes that my GPU tower running stable diffusion is currently making, even the best self-driving AI is useless. Likewise, as impressive as stable diffusion is, I doubt it could even get my car out of the driveway.
We are constantly moving to less constrained problems. What used to be a very limited setting is now perhaps a somewhat limited one. The options aren’t binary chose of using A* for pathing in a video game or AGI, it’s finding ever more generalizable solutions to ever wider range of problems.
The thing is as processing power keeps improving we need ever fewer limits to active human levels of performance from these systems. I don’t think AGI is just waiting for sufficient flops, but I think that’s closer to the truth than we want to admit.
An automated vehicle has a couple of well defined controls. It cannot play the piano. It cannot recommend products. It cannot take a picture and tell me what kind of bird that is. In fact, it probably cannot even drive offroad, despite that being a very closely related task to what it is made to do.
A person trained to drive a car on roads may have difficulty time on his first few offroad drives, but can quickly aquire the skills. He won't be completely unable to function either, because to a human, navigating is navigating, whether we walk through our own homes, or drive a vehicle over rough terrain.
To AI as we currently understand it, all of these are completely different tasks.
Also, the DARPA grand challenge was for self driving cars off road, it’s a simpler problem no need to check for traffic lights and stop signs it’s the same general problem. Which brings up another possibility, there are only so many tasks we might want AI for so specific systems for every single one is effectively the same thing as AGI.
Even this analogy oversells the state of "AI" today, as it implies that it would be possible to simply "swap in" stable-diffusion for the self-driving software in your Tesla (or whatever car with similar software) without spending months figuring out how to even hook up the inputs and outputs in such a way that you could get stable-diffusion to somehow activate when you want to drive.
Their IO models are so completely different—one takes a stream of largely-visual input and determines in real time whether it represents something it has been trained to recognize as "road", "danger", etc, and thus whether it should issue real-time outputs to the car's axles, brakes, etc, while the other takes discrete text or visual input prompts and uses them to generate discrete visual outputs—that it doesn't even make sense to think about how one could replace the other.
They're not AIs. They're simply software programs, using machine learning algorithms to produce highly-tailored, human-opaque results that can, yes, be pretty amazing. But given all the excessive hype and disinformation lately around what these programs can do, I've made a conscious effort not to refer to them as "AI," because people have too many sci-fi associations with it, and it causes far too many of them to anthropomorphize them and impute far, far more ability and "intelligence" to them than they actually have.
Humans can’t swap to a hummingbird or sharks body but are our template for AGI, so why argue self driving AI’s need that ability?
I'm not arguing that self-driving car software needs these abilities. I'm arguing that self-driving car software is not in the same class as AGI, which would need to be able to do all these things to some extent.
So, the cars might not be AGI but the system used to build those limited cars could be. After all many people can’t write poetry without training.
They work in math and coding, and these two are quite general.
> CodeRL is a new framework for program synthesis through holistic integration of pretrained language models and deep reinforcement learning. By utilizing unit test feedback as part of model training and inference, and integrating with an improved CodeT5 model, CodeRL achieves state-of-the-art results on competition-level programming tasks.
It's nice that CodeRL can solve them, but to say that they work in "coding" and "math" is a huge stretch.
What if, the second way is easy? So long as you can understand and interface with the world effectively, which the first provides. From there what if it was just act and observe and a fitness function
If you think about how a language model works, it's not totally clear that it should work at all.
Humans use sentences to communicate pre-existing ideas - ie. we already have the object, subject and action in our heads prior to forming the sentence. In order to generate realistic sentences, an LLM needs some ability to plan ahead, so that when generating the logits for the first token, it needs some idea of how the sentence will end.
But if you think about it, nothing about next-token prediction should necessarily lead to this planning ability. It can easily get into a feedback loop of predict the same token over and over (and small LMs do in fact do this) We also didn't tell it explicitly that sentences should have objects, subjects and actions, these behaviors are purely emergent from the task of next token prediction.
In a similar vein, I think artificial agents would not need explicit goals or affordances, just a sufficiently complex environment that a large model would not overfit on.
It clearly proves too much - humans have a limited action space - they can only move muscles. So they can't truly explore a larger action space, so they cannot be general intelligences.
But more specifically, if something is within an LLM's action space, whether or not you call it an affordance doesn't change whether it gets explored and used. And perhaps you'd argue that their action space is limited because they are in a box. But hook an LLM up to the internet to allow it to query and retrieve data, and suddenly the action space is essentially infinite. So the limitation isn't the model, it's how the model was hobbled by being denied access to the world.
Brains are physical systems, and physical systems can be simulated by algorithms. Right? Do the authors think this 'jury-rigging' is some non-physical process?
I guess they are so meta they wondered if a dedicated AI could output an AGI agent, algorithmically. As far as their colloquial definition which speaks of jury-rigging is truly intelligent, I suspect that the road to AGI would require so much human input and reboots (have you tried turning it on and off again?) that it is not predictable by any physical theory. So it's non-physical in a strong definition of physics.
On the other hand, since they argue against classical computing, it would necessarily rely on innovative developments in various hard sciences.
Grog: “Grog rubs two sticks together” Lowers voice and looks around furtively “really hard.”
Fire is warm and it produces light, and it's easily observable in nature, it's a "thing" which humans could easily describe and recognize, before reproducing it. Someone was rubbing two sticks together, and noticed things became warmer, and warmer as they went, starting to feel like the warmth of fire and it happened.
In the case of AGI, I don't feel like we have a definition of done, so it seems kind of crazy to be chasing it / throwing money at it.
I didn't say say I'm against it, but does seem like a crazy way to go about it. Keep producing models that one day might mimic intelligence as we know it?
So, I don’t see why rubbing these statistical model sticks should suddenly burst into intelligence, but I’m open to seeing convincing reasoning on that at some point. I wouldn’t invest time or energy in the meantime and like that original poster, think it’s kinda insane to if my goal was to see human-like intelligence emerge outside of humans
It also seems very odd that you can differentiate between some things that you think are intelligent, and some things that you think definitely are not, yet you are incapable of extracting any sort of goal from that knowledge.
If you could tell us your criteria, perhaps we could help you with that...
In these comments, when I’ve talked about an intelligence I can distinguish, I’ve been talking about human / animal intelligence. AGI implies an intelligence independent of that, so I’m asking about the specifics there - what are we calling intelligence if not “what humans do”?
If we are calling it just that, then I’d argue everything I know about how these models do things is very different from what I know of how humans approach the specific tasks the models are built against. And I’ve read that that’s intentional. So, even with that sort of definition I don’t see how it follows that these approaches are on any linear path to AGI (maybe nonlinear if we learn limits and such from mistakes).
I’ve since read more of the article (it’s long, huh?) I like the framework they use from Roitblat in Section 2 - and again, don’t see how LLMs and such are on the road to fulfilling those criteria.
I have a different idea of what AGI means: in my view, it is a retronym created in the 1980s in order to refer to AI of the sort Turing envisioned (which was more or less "what humans do") and differentiate it from things that were then being called AI, such as IBM's Deep Blue, which were mostly brute force applied to conceptually narrow problems.
You mentioned Roitblat's framework, and I would draw your attention to one aspect of it: it is not just a list of things that humans do, but those things which humans do considerably better than other animals, yet for all of them, there are other species that do them to some extent. As an evolutionist, I suppose there was a relatively recent time in the past when some of our ancestors or sibling species (all now extinct) had some or all of these skills to some intermediate level. In this view, intelligence is not an all-or-nothing concept, and achieving some of it is still progress.
Here's a view which you may not have seen: the pace of progress in AGI has not been constrained by an inability to define what we want, but by the pace at which we see ways to make what we see we need. For example, it is clear that current LLMs have a problem with truth, but it is not clear from what has been made public so far that anyone has a solution. Some people think that what's being done now with LLMs, but more of it, will be enough to get us to what will be generally accepted as AGI; I am skeptical, but I am willing to be persuaded otherwise if the evidence warrants it.
We are even able to predict whether other compounds might work without knowing why just based on structural similarity.
It’s ideal to know the full mechanism and it obviously aids engineering but there’s no reason you have to wait for that to use something. People used fire for millennia before oxidation reduction reactions were understood.
Seems a bit circular.
"What's a good definition of intelligence?" "Human intelligence." "What's human intelligence?" "What we want AI intelligence to be like."
No one in the planet would know how to be sure it's not just emulating human behavior.
Insane to me is to affirm that something has a limit when you don’t fully understand it.
And I think it’s a shame we haven’t considered what the limits are on “intelligence” very much. Knowing them has been immeasurably valuable in software engineering and algorithms, for one.
We can find plenty of examples where engineering advanced beyond our understanding: bridges, boats, tables, etc. When humanity started building those we didn’t have the full picture of our physics yet we build and used those for more than 1000s years.
Again, I’m not saying we can’t engineer something from knowledge, I’m saying we can’t affirm limits onto something we don’t fully understand yet.
IMHO this paper is speculative and biased towards our human intelligence. From all the advances I have seen so far in the AI landscape, I’m growing more and more skeptic that our intelligence is something so complex that we can’t replicate.
There are more examples as well, as another user has commented, we didn't know what fire is until relatively recently but we have been using it for thousands of years.
But either way, is knowing the electron knowing electricity? There are so many properties of it that can be known and manipulated without that insight- and indeed they built up that understanding to reach practical engineering and use of electricity. That’s what I think is being gotten at wrt intelligence.
“Knowing” something isn’t necessarily about being aware of its smaller parts.
The source you linked already cites it as the first arc lamp.
> But either way, is knowing the electron knowing electricity? There are so many properties of it that can be known and manipulated without that insight- and indeed they built up that understanding to reach practical engineering and use of electricity. That’s what I think is being gotten at wrt intelligence.
Yes. That is exactly my point. We don't need to entirely understand what intelligence is in order to be able to create it. The same way we didn't know what fire is, but we created it with no problem.
But we can hardly define intelligence, let alone “entirely understand” it. A child could give a good , practical definition of fire and manipulate it skillfully thousands of years ago. Not so much us grown adults wrt intelligence today.
With fire, even “when things turn from themselves to ashes and produce heat” (which I imagine a prehistoric child could come up with) distinguishes fire usefully from most other phenomena in the world
Note that this is not an argument that it is possible, which certainly has not been conclusively established.
They say that "there are no actions that emanate from the historicity of internal organization". I have no idea what that means. Is this an argument about free will? If so, what would be the magic ingredient that saves human brains from immediacy? It can't be memory, because Turing machines have memory as well.
From the article:
> it is generally assumed that input-output processing is performed by some sort of algorithm that can be implemented on a universal Turing machine. The problem is that such algorithmic systems have no freedom from immediacy, since all their outputs are determined entirely—even though often in intricate and probabilistic ways—by the inputs of the system. There are no actions that emanate from the historicity of internal organization. There is, therefore, no agency at all in an AI “agent.”
The universe is not deterministic. There is one brain state at time T, but at time T+1 there are many possible brain states, all with different possibilities. What determines which state at time T+1 the consciousness at time T perceives itself as progressing to? If we define this as a function, which associates states at time T with states at times T+1, then this function loosely fits the definition of "free will". We can't actually construct this function or say anything about it, but we can say it can't not exist (as consciousness perceives itself as progressing to a single future state), which by the law of the excluded middle implies it must exist. The definition doesn't however prevent it from essentially being completely random.
There is no proof of this. There are deterministic interpretations of quantum mechanics, for instance.
Also, there is little reason to think our brains depend on quantum indeterminacy, even if it exists. After all, a brain that behaves erratically/non-deterministically is the exact opposite of useful. Once you learn how to walk, you want to be able to rely on that ability. Once you learn what food is, you don't want to erratically start eating rocks or twigs.
Non-determinism in the brain would be selected against, not selected for. Flexibility would be selected for, but flexibility can be deterministic.
Even worse for your argument, probabilistic Turing machines and algorithms (classical bounded-error probabilistic machines) don't even seem to be faster than fully deterministic ones (P = BEP).
But let's say it's right, ignoring the massive burden of proof. We can hook up a quantum random number generator to GPT-3, and suddenly it can be conscious? If that's the whole argument, it seems like a useless point anyways.
Practical stuff like "store memories", "replace human senses", "AIs can't walk the world as humans", etc. are feasible to be solved with current commodity technology, even without hypothetical future highly capable robots.
I'll divert a little from the exact answer to the comment..
I've said it in another comment, it looks like Tiktok could be exactly this kind of "eyes and ears", it could be the - one of the - perfect continuously information collection systems "looking into the world" for an strong AI (yeah, right now), but most certainly is just an incredible business model turned into an almost free giant datasets generator.
I cannot get to think that someone could have created tiktok to "see into the world" for AIs / create datasets, from the scratch, but it is possible too.
If I were Timnit Gebru and I saw that Google was trying to implement reflection in an AI when i felt the ethics engineering hadn’t caught up to the software engineering, I might quit
The same is true of AI at the moment.
The affordance is another pattern (tool) we train ourselves all our life with many tools: hammer, knife, cisors, tweezers and more complex ones: car, backhoe, computers. There is a lot of overlap from one tool to another one, most of the pattern transfer easily from one to another. That’s why we don’t have to retrain ourselves if we change the size of the knife we use, it’s a tiny adjustment done in the first few seconds of use.
Brain is not the Turing Machine, but it's the controller part of it, which we often treat as the same thing.
We have multiple controllers (brains) operating on a shared tape (universe).
Yes it is, all finite volumes contain finite information. This is a consequence of physics called the Bekenstein Bound. This means the brain can be fully captured by a finite state machine. It has a very large state space, but it's still finite.
Don't confuse expressiveness and state space. Even if there are more faithful encodings that better preserve other properties, what I said is strictly true and it's important for people to understand that human cognition is not as powerful as some think: a finite state space means a human can be fully captured by a finite state machine.
A finite set of configurations is enumerable, and can be mapped to any other finite set of same or larger size, like a computer's memory, with no loss of information.
Therefore, any state your brain can enter can similarly be created within a computer, in principle.
Furthermore, the proof I laid out is a corollary of simple physics. If you want a more rigorous version, look up the Bekenstein Bound, which proves conclusively that any finite volume can only contain finite information. Your body is a finite volume, therefore it contains finite information, therefore it can be captured by a finite state machine.
Edit: I just realized I also replied to you above, so I'll take up the thread there.
Not necessarily: https://en.wikipedia.org/wiki/Orchestrated_objective_reducti...
The only questionable thing is whether quantum effects are essential in brain activity.
Of course it's disputable. There are at least two well known deterministic interpretations of quantum mechanics that are indistinguishable from orthodox QM, Many Worlds and Bohmian mechanics.
And so can a finite state machine with the same state space as your brain. Size of state space is not a difference of kind but of degree.