The Myth of AI Omniscience: AI's Epistemological Limits

> To sum up: because LLMs are fundamentally limited to i) using our vocabulary, ii) “understanding” concepts in the ways we do, and iii) “talking” in the ways we do, then, at best, a LLM can only mirror back to us the order we have carved, the truth we have honed.

That conclusion is not correct. Generative models can combine existing concepts in novel ways which have never been considered before. This is most easily seen with text-to-image models. For example the teddy bear swimming in an Olympic pool [1] or many other surreal examples [2] were not part of the training set. So as long as an answer can be expressed as a combination of primitive concepts, LLMs can generate them.

[1] https://imagen.research.google/main_gallery_images/teddy-bea...

[2] https://imagen.research.google/

LLMs don't learn to simulate or mimic, that's just a byproduct. They learn to predict the training corpus. There is absolutely nothing about the act of prediction that necessitates an upper bound of intelligence on the corpus itself.

https://www.pnas.org/doi/full/10.1073/pnas.2016239118

They found representations on fundamental properties of proteins such as secondary structure, contacts, and biological activity in an LLM only trained to predict protein structure sequences. No folding, nothing explicit in the corpus itself. Yet those truths manifest as a necessity of accurate predictions. Prediction is not bound by what the data explicitly shows.

>A language model’s vocabulary is limited to the words that exist within the model’s training texts, which means a LLM can only refer to objects and relations that we humans have already discerned, named, and written about.

This is not true and is easy enough to test.

The article is claiming that advanced AIs cannot become more intelligent than humans, essentially because LLMs cannot become more intelligent than humans.

LLMs aren’t the be all and end all of AI though. They’re an impressive but inherently limited stepping stone, with a very constrained scope of applicability and capabilities. There is no reason to suppose that future, much more advanced architectures can’t surpass humans in reasoning ability.

Advanced AI doesn’t have to be omniscient fir us not yon be able to predict it’s capabilities, it just needs to be smarter than us. By definition if something is smarter than you, it’s not possible for you to anticipate and predict its behaviour. Therefore in principle it is not possible to predict whether that behaviour will be to your liking. That’s the problem.

> But the ways in which a LLM can “talk about” the universe (and everything it contains) are limited to the ways in which humans have previously talked about the universe.

This is often said, but it isn't so.

The task of predicting the next token in human speech really well requires immense intelligence — potentially far more intelligence than possessed by the original speaker! Imagine yourself engaging in the task of listening to someone who isn't that smart speak and then trying to figure out what they'll say next — in doing so, you might make all sorts of extrapolations about the person, their motivations, their manner, their dialect, etc — calling on all sorts of internal models that you've built up about people over time. This is what models are being trained to do when we train them on predicting tokens.

There are concrete examples of models inventing new ways of thinking that are not described in their training set. For example, when training a transformer from scratch to perform addition mod P (and having no training data other than examples of addition mod P), the transformer was able to discover the use of discrete fourier transforms and trigonometric identities [1]. As we can see, neural nets can build all sorts of internal mental models that no one explained to them beforehand. These internal mental models can then be elicited and used for other purposes by e.g. fine-tuning.

I think a good mental model for transformers/neural nets is that they're automatic scientists. They figure out ways of modeling things in order to predict the output from the input — which is what scientists do! As part of this, they can de-facto discover new theories, and come to rely on the theories that prove useful in their prediction task.

Also, not all tokens in the training set are from human speech, so models are being trained to model all manner of data-generating processes.

[1] https://arxiv.org/pdf/2301.05217.pdf

>OpenAI, the company behind ChatGPT, recently announced a massive investment into research on “superalignment,”

The head of the “superalignment” project at OpenAI recently appeared on the AXRP podcast talking about the Automated Alignment Researcher they’re developing.

The host asked him how they were going to align the AAR itself, and if they can do that, what would be left for the AAR to do, but he didn’t seem to understand the question.

Not encouraging signs.

People are still desperate to maintain humankind at the top of the intellectual pyramid. It's tiresome and a waste of energy. There are definitely limits to intelligence, nothing will ever be smart enough to predict the future perfectly. But to believe that humans represent the peak of what's possible, is just hubris and nothing else.

he lost me at "a language model’s vocabulary is limited to the words that exist within the model’s training texts, which means a LLM can only refer to objects and relations that we humans have already discerned, named, and written about."

This is trivially demonstrated to be a false statement, as GPT is capable of synthesizing entirely novel words based on very little input guidance.

This is plainly false. With reinforcement learning, an objective function and the ability to generate random solutions it's possible for AI to stumble upon sequences/strategies that greatly outperform human state of the art.

I found this article a bit fluffy and lacking on technical substance. If we want to entertain the limits of AGI, we need to look at concepts like Solomonoff induction, which were conceived of and researched in the 1960s (and apparently mostly forgotten about since then). Given an input sequence of bits, what is the best (i.e., “oracle”) posterior distribution possible given a domain prior and the universal prior? How would a resource-limited approximation widen this posterior distribution? What are the practical limits to achieving the shortest Turing machines that reproduce the input sequence? These are interesting questions that should be looked into if we want to discuss AGI in the epistemological limit.

Also, I tend to get frustrated with predictions by “AI experts” about what a future with AGI will look like. We don’t have any world experts on AGI currently, for the simple reason that it doesn’t exist yet. We have experts (or I’d prefer to say “first movers”) on particular types of machine learning based upon neural networks, but that’s about it. A true AGI would almost be unpredictable by definition, so human attempts to forecast the future in this regard are mostly sensationalism or a projection of some kind of personal agenda (cough cough Marc Andreessen).

Rather than viewing generative AI as a form of artificial intelligence, I posit that it should be seen as an automated tool for tapping into human cultural, linguistic, and empirical knowledge. Data and computation are two sides of the same coin. The 'intelligence' in AI is embedded within the data, with the computational model serving as a tool to access and express this inherent intelligence.

I would argue for a change in perspective towards AI, one that recognizes LLMs as powerful tools for accessing the vast wealth of human cultural knowledge rather than viewing them as a separate form of intelligence. We must carefully consider critical ethical considerations about control, access, and trust that will become increasingly relevant as these tools become more integrated into our everyday lives.

This paradigm shift carries some implications:

LLMs will not achieve superintelligence: Although these models can process information quickly and access a wide range of knowledge, they lack the superior reasoning or inference abilities that would classify them as superintelligence.

LLMs as an extension of human thought: These models can automate and amplify human capabilities but do not introduce new abilities beyond what is already present in human thought processes.

LLMs as mirrors of human culture and knowledge: These models reflect the recorded artifacts of human language, art, and culture. They can make the inherent intelligence within these artifacts accessible, providing a vast information resource.

Implications for the future: Access to this "memetic matrix" of human knowledge will become a fundamental part of being human as these tools become more integrated into our lives, bringing up issues of ownership, access, and the potential for misuse.

Thought consolidation and control of inference engines: There's a potential risk that control of inference engines by a small number of companies could lead to a consolidation of thought that threatens democratic governance. I propose a diversity of federated or self-hosted inference tools as solutions to mitigate this risk.

The necessity for trust and individuality: As these tools become more influential in our lives, maintaining trust in our individual thoughts and avoiding the uncritical acceptance of synthesized ideas from sources with opaque motives will become increasingly important.

Synthetic Inference relies on a vast cultural commons: We cannot allow these commons to be closed off and owned by a few big companies. This resource is to approximate totality of all human knowledge, language, and culture. It belongs to all of humanity. Training data must be open, free, and available for examination.

I think it's safe to say that we should disregard any substack/medium posts, especially when it comes to AI.

LLMs have obvious limitations and a big contributor is the training data and the way they are trained in a supervised way. Although that is not as much of a limitation as with most other AIs because of the "in-context learning" ability.

But that doesn't mean that future LLMs couldn't conceivably learn abstractions than are at a higher level or even perhaps inaccessible to humans in number of layers or complexity.

I think the obvious type of superintelligence that is only a few years away is what I call hyperspeed AI. LLMs are a very specific application which we will be able to accelerate greatly. Within a decade the output speed will be dozens of times faster than human thought.

Because they will have robust reasoning and extreme "thinking speed", these AIs will probably be connected to industry and military applications. Humans will need to be removed from the loop because waiting for a human decision means the competitors' AIs race ahead the equivalent of days or weeks.

If this is fine then it will create a precarious situation for humans where they are only nominally in control and things like advanced AI "viruses" might become very dangerous.

It's also certain that researchers will continue to develop new types of AIs that incorporate more animal (such as human)-like capabilities.

> Future LLMs, regardless of model architecture, are fundamentally constrained in this way, by virtue of the fact that they are trained on human-written texts. Therefore, the outputs of a LLM, at best, reflect our current understanding of the universe and nothing more.

That constraint holds only for pure LLMs. There is a secondary source of learning - external feedback to the model. When a LLM is part of a larger system, it can integrate feedback to improve itself, like AlphaGo Zero. A LLM+compiler could self-optimize by observing compilation errors and program outputs. A LLM+game could self-improve by conditioning actions on the score. Humans could create a preference dataset and model for RLHF to refine the LLM based on human feedback. A LLM+robot could learn from past experiences to improve its planning.

Similarly, humans need tools to conduct research. We can't do pure research without labs and experiments. LLMs need external confirmation the same way we do. The outputs of a pure LLM merely reflect our current understanding of the universe. But coupled with real-world feedback, LLMs have the potential to learn and discover new knowledge.

Putting aside the question of what LLMs, specifically, are capable of, I feel the author's central argument has already been shown to be false, through events outside of AI.

In a nutshell, the argument is that, as AI will have just the knowledge that is available to us, expressed in the languages we use, we will be able to anticipate and put a stop to it doing anything catastrophic.

Exactly the same argument, however, could be made over computer and communications security. If anything, those trying to keep things secure have the advantage in knowledge, at least initially, yet zero-day exploits keep surfacing, some of them at fundamental levels such as processor architecture. Nothing truly catastrophic has happened yet, but it cannot be said to be impossible.

The flaw in the argument is that one can easily know all the bare facts without seeing all the implications but someone - or something - else could do so.

"Therefore, the outputs of a LLM, at best, reflect our current understanding of the universe and nothing more." Others have reflected on the problems with this statement, and I agree, LLMs can be prompted to navigate the training corpus and create novel output, countering this claim. However, these criticisms neglect to credit the external intelligence inherent to the prompts themselves. AutoGPT and its ilk have yet to create convincing agency: for the time being the current state of AI is a mere extension of human intelligence and will not be making lofty discoveries without direct human interaction. LLMs are a form of human co-intelligence.

Reading through there is a heavy fault in logic. Statically reasonable limitations to llm output based on training material, ignores hallucinations. And hallucinations ignore the sheer chance of new emergent information by odds.

Why do we need anything close to superintelligence or "omniscience" (wtf) before we get profound changes to how humanity operates. It's really an obsession!

Instead compare to the average human, at most. Compare to refrigeration? Compare to fire? Compare to computers? And that's even if the bar is placed at the singularity. Human progress as we have known it so far is not founded on superhumans but on a lot of work by a mix of merely visionary, hard working, and just plain average people. Just speeding that an order of magnitude changes everything.

This focuses on the argument that some hyper-sophisticated future LLM will not discover the secrets of the universe on its own. That's plausible, but beside the point when it comes to the potential dangers of A.I. Not all future A.I.s will be LLMs. Understanding the universe isn't the only way to pose an existential threat.

This is an old discussion with much deeper insights already available than I can parrot here, but one obvious counter just to this argument is that even an LLM can be an existential threat, depending what it can convince you it knows.

I'm interested in seeing how brains change as AI usage increases, in the same way many of us don't attempt to memorize phone numbers, what skills will we offload to 'AI', will we be aware that it's happening? Perhaps more interesting, what new adaptations or adjustments may occur as a result of this augmentation. I hope that our brains ability to synthesize new thoughts both extrapolated and 'from thin air' will remain a useful skill regardless.

Why would AGI be bound to human made models? Why can it no develop its own?

LLMs are trained on human-written texts, therefore their output can only reflect our current understanding - am I reading this right? Hopefully not as that would be insultingly wrong.

Anything with started with "the myth" arguing about something that is at the limits of knowledge without a clear answer is unavoidably biased by a person with big ego.

Yay, we need more of this before OpenAI finishes hypnotizing the global south (lol).

«xAI, a venture whose ambitious aim is to “understand the true nature of the universe.”»

C'mon, we already have the answer it's 42

It seems silly to even mention epistemology when current AI has no Intelligence at all yet.