How will AI learn next?
newyorker.com
newyorker.com
Given this, the problem isn't just that there's not enough new content. It's that an ever-increasing fraction of the content in the public sphere will be generated by these models. And can the models detect that they are ingesting their own output? If they get good enough, they probably can't. And then they'll get worse.
This could have a strange impact on human language / communication as well. As these models are increasingly trained on their own output, they'll start emulating their own mistakes and more of the content we consume will have these mistakes consistently used. You can imagine people, sometimes intentionally and sometimes not, starting to emulate these patterns and causing shifts in human languages. Interesting times ahead...
Where it isn’t grounded, as in endless online conversations with like minded people (closed loop feedback) about informally abstracted (poorly specified constraints) and emotion invoking (high reinforcement) topics, people go batshit too.
So the more AI models actually practice what they know in objective environments, the more likely that output->input feedback will inform introspection toward self-improvement, and less like an iterative calculation of the architecture’s resonant frequencies or eigenvalues.
Thank you for this observation.
LLM Learning from LLM -> Drift
Human on Human echo chamber -> batshit crazy.
I've often thought human echo chambers were their own phenomena. Something about the brain and tribalism from evolution.
I never thought of it in terms of AI training data.
As LLMs train on data produced by other LLMs, they will drift.
And this drifting is the same phenomena as when humans get in an echo chamber. If each person hears what the others are saying, and spits it out in some form, and the others hear it, and also spit it back out in some form, this turns into a drifting in understanding just like an LLM. ("idea telephone game")
Technically, it isn't just in echo chambers. It is all humans, at a lot of different scales, from small to large groups. Countries, and cultures are echo chambers at larger scales.
Like how the concepts in philosophy, as they become more abstract, they kind of twist back on themselves, and become re-invented. And as they get more abstract get accused of just 'playing with words'. Just like an LLM can just 'play with words'?
The difference is eventually humans have to relate to 'real objects'.
So even if the word for 'apple' drifts over time and between groups, eventually you can still relate the words back to the real 'apple'.
Humans are grounded in the reality of 'objects' in space.
But. I tend to think this is temporary. As LLM's are linked to things like AlphaGo, and drone flight systems. They will also have to deal with real 'objects'. Maybe that will then lead to more grounded reasoning.
Unfortunately, our society does not understand this. If we did, we would value the opinions of auto mechanics far more than those of lawyers.
And value science (understanding it is simply our accumulated tools for finding harder truth, not a priesthood of “the truth”), more than pandering & populist politicians, and tribal media & online personalities.
Well, except for floating signifiers, which make up an increasing part of our vocabulary.
It's kind of interesting to think we might all be contributing to the training data of AGI not just through generating content but also what we choose to share or upvote.
Why don't humans suffer from this problem, then? Humans have been creating art (and content) that imitates nature and society for thousands of years now, and yet we have little problem (the exception are possibly things like crop circles) to recognize what is a natural phenomenon and what is generated culturally.
I think it's wrong to assume that this is a problem with intelligence in general, rather than just a feature (stupidity) of the current models.
They do, but the problem is poorly stated. The problen isn't “If your models are influencing the generation of data that they are later retrained on”, its “If output produced by your models with inadequate filtering for quality/accuracy is dominating the data that they are later retrained on.”
Humans definitely experience the same problem: we see it when societies become closed, inward-looking, and locked into models of information filtering that don't involve accuracy or fitness for purpose other than confirming an in-groups priors. There are some examoles in stagnation of large societies, but probably the clearest examples are cults that socially isolate their membership and degrade into increasingly bizarre behaviors terminating in mass suicide, murder/suicides, etc.
LLM’s experience self-reinforcing degradation more acutely because they are individually less capable than humans and they are less diverse.
It's really the paperclip maximizer argument again. "Superintelligent AI" will supposedly change the world, but will lack enough common sense to understand a simple instruction with ethical constraint. Same here, the crappy AI content will somehow take over the world, despite the fact that most humans can actually mostly recognize original content from a generated one.
The art that humans created doesn't replace nature, just like tons of kitsch don't pose a problem for new artists. In the same way, crappy AI content is not gonna replace good content, just augment it, and any decent AI (which has good enough common sense to be called "intelligent") will be able to tell the difference. Nobody's gonna care about crappy content in a few years, we are all gonna just throw it away, and this process happens continuously.
(1) accuracy or fit for purpose, not natural, is the issue.
(2) evaluation, not having a discussion, is the issue.
(3) LLMs can definitely do evaluation, and probably can also have discussions and decide to, if given a proper prompt and access to ground truth in the first case, and if also used in a properly constructed agent framework, for the second.
(4) if you did all that, and used LLM decisions and not human decisions to guide retraining, I suspect with current LLMs you’d actually have made the problem worse not better, because you’d reduce the influence of higher-capacity and more diverse humans on the training cycle and increase the influence of less diverse, lower capacity LLMs.
In theory. In reality what you get is the Climate Change "debate".
Can we be sure that we don't? For example, depending on perspective our language is either evolving towards clarity or devolving towards primitivity, compared to XVIII/XIX century. The same could be argued about other aspects of society and culture.
How much of your taste to the opposite sex's (if you're straight) physical appearance is cultural and how much is in your genes?
We have such a huge problem with this, that (with apologies) I cant help but think that you also don't know.
If I look out of the window, what do I see that's natural? There's less nature in my view than most, because I'm in a city: ignoring the concrete, glass, steel, and tarmac, the traffic and the clothing, I see… humans, a tree and some mosses and grasses, the clouds above.
Can humans even be said to be in a natural state or not, in contexts like this? I don't know. We domesticated ourselves, and it was our nature to do so, and also our nature to make the tools and clothing that led to our loss of body fur and other divergences from the other primates.
But what I can say is that the tree was planted (if it was cultivated or selectively bred to this state, I wouldn't know); and a third of the CO2 in the air I breathe is from human actions, influencing the atmosphere and precisely when and which clouds I see (no contrails visible today, which would be more explicitly artificial).
If I look a little further afar, I find entire hills in this city made by piling up rubble from the second world war and then covering it in topsoil, the only indication of which is the large signpost present to tell all of this[0]; and there are other hills both here and elsewhere that are made from trying to turn landfill sites into something less unpleasant to be around, with varying degrees of effectiveness in their disguises.
If I think back to my childhood bedroom in a suburban home: there was a lawn (placed there by humans, then kept short with an unnatural Flymo) with two apple trees (cultivated and planted by humans), a vegetable patch and a herb garden (each plant likewise cultivated and placed by humans), surrounded by wooden fences (cut and placed).
In the distance there was a row of trees, which might have been self-seeded or planted (I wouldn't know), enshrouding a small victorian folly covered in vines, and separating us from a series of fields (unnatural) where horses (selectively bred) were being stabled (unnatural structures); far beyond them was the ruin of an ancient tower destroyed centuries ago[1] — clearly built, but ask yourself: while stones are natural, are those specific larger stones on the corners, naturally like that?
In a more abstract sense, if I look at foods in the supermarket, some will say "made from natural ingredients": if that thing is meat, such a claim ignores the selective breeding of the animal (and the conditions they were raised in, which would be a separate sticker saying "free range", though even then that's not like being wild). And even then, if it's made from ingredients plural, that's not natural either: bread doesn't grow in wheat fields, sushi rolls don't grow in rice paddies. Even if is a single ingredient, there's often processing involved: wheat (already selectively bred) has to be sorted from chaff, then ground to become flour. Even "mineral water" probably has had something done to it, even assuming there's not some small fine-print on the label saying "from a municipal source" (or whatever the clause is that means "actually just tap water").
Pretty soon ChatGPT will be able to do a lot of training by iterating on its own output, such as by writing code and analyzing the output (including using vision systems).
Here's an interesting thing I noticed last night. I have been making a lot of images that have piano keyboards in them. DALL-E 3 makes some excellent images otherwise (faces and hands mostly look great), but it always messes up the keyboards, as it doesn't seem to get how black keys are in alternating groups of two and three.
But I tried getting chatgpt to analyze an image, using its new "vision" capabilities, and the first thing it noticed was that the piano keys were not properly clustered. I said nothing about that, I just asked it "what is wrong with this image" and it immediately found that. What if it could feed this sort of thing back in, using similar logic to Alpha Zero?
That's just a tiny hint of what is to come. Sure, it typically needs human generated data for most things. It's already got thousands of times more than any human has looked at. It will also be able to learn from human feedback, for instance a human could tell it what it got wrong in a response (whether regular text, code, or image), and explain in natural language where it deviated from what was expected. It can learn which humans are reliable, so it can minimize the number of paid employees doing RLHF, using them mostly to rate (unpaid) humans who choose to provide feedback. Even if most users opt out of giving this sort of feedback, there will be plenty to give it new, good information.
With natural language you don't have that unaided feedback evaluation metric. Especially when given idioms, domain specific terms, etc.
This is slow and hardwork because you need to process some text, evaluate and correct that data, retrain and repeat with the next text. You also need to check and correct the existing data, because inconsistencies will compound any errors.
In short, they're paying contract workers for quantity, not quality; they don't have time to do independent research or follow up on citations. Unsurprisingly, the LLM optimizes for superficially convincing bullshit.
How do you know this?
Just taking a wild guess, but I'd think a company with billions of funding, and a ton of people trying to find flaws in what they are producing, would have some processes in place to incentivize quality as well as quality.
What you are suggesting is that a company that produces a product that is based on balancing trillions of floating point numbers makes core business decisions in the most simplistic black and white terms. "Hey, lets just go with a one and a zero on this." Bizarre assumption.
Maybe I'm just good at prompting, and I'm not trying to trick it, but I don't see this "superficially convincing bullshit." Can you show me a chat where you have sincerely prompted it and gotten something that matches that description?
I often see responses that are better than I could have given even if given hours to research and compose them, and I'm a pretty good writer and researcher.
Here, since I'm asking you to share one where it fails as you say it does by creating "superficially convincing bullshit", I'll share several where it succeeds.
https://chat.openai.com/share/523d0fec-34d3-40c4-b5a1-81c77f...
https://chat.openai.com/share/e09c4491-fd66-4519-92d6-d34645...
https://chat.openai.com/share/233a5ae2-c726-4ddc-8452-20248e...
https://chat.openai.com/share/53e6bda1-fe97-41ce-8f5c-89d639...
https://chat.openai.com/share/19f80ea9-e6be-4ac3-9dd4-7ea15c...
https://www.washingtonpost.com/world/2023/08/28/scale-ai-rem...
These jobs are overwhelmingly paid by task, which puts a lot of pressure to go fast.
I assert the entire "hallucination" phenomenon is a side effect of these practices. When ChatGPT makes up a fake fact with fake sources to back it up, it's largely because such lies are rated very highly by the underpaid humans who aren't incentivized to follow up on sources.
It seems like with billions of investment, they could figure that out. It's commonly discussed as an extremely difficult problem to solve and the most important problem to solve in the most talked about industry on the planet. I'm having a problem believing that its something that is so easy to solve.
Are you suggesting that even with that much money, they have to do things the way things are "overwhelmingly" done, as opposed to being able to say "hey, we need it done this way instead, because it's important and we can pay for it."
It just seems pretty bizarre to think that the highest of high tech, that is massively funded, doesn't have the clout to just fix that in a heartbeat, if that's really where the problem is.
Source: https://huyenchip.com/2023/05/02/rlhf.html#demonstration_dat...
With billions in funding, and bad results causing bad press etc, you think that OpenAI would not have given this a bit of consideration?
> you think that OpenAI would not have given this a bit of consideration?
Those are just assumptions though. The issue is not “this was labelled as a shoe, but it’s a car”, the issue is about depth vs superficially, which is harder to verify. See also https://www.theverge.com/features/23764584/ai-artificial-int... for a well-sourced article on the subject.
RLHF typically employees humans, and that can be time consuming in itself, but less time consuming than creating content. And their efforts can be amplified. If they are actually rating unpaid humans, that is, users, who are willing to provide feedback and are also prompting the system. Plenty of people are happy to do this for free, and some of it happens, just as a byproduct of them doing what they're already doing, creating content and choosing, which comes out good and which one doesn't. Every time I am working through a coding problem with chatGPT, and it makes mistakes and I tell her about those mistakes, it can be learning from that.
People can also come up with coding problems that can run and test itself on. As a simple example, I imagine it's trying to write a sorting algorithm. It can also write a testing function simply tests that it is correctly sorted. They can also time its results, they can count how many steps had to do in that sense it can work just like Alpha zero, where there is an objective goal, which is to do it with the least clock cycles, and there's a way to test whether and how well it is a achieving that goal. While that may be a limited number of programming problems that that works for, by practicing on that type of problem it will presumably get better at other types of problems, just like humans do.
This is exactly what large language models do, they find a way to objectively test their writing ability, which is by having them predict words and things that they've never seen before. In a sense it's different from actually writing new creative content, but it is practicing skills that you need to tap into when you are creating new content. Interestingly, a lot of people will dismiss them as simply being word predictors, but that's not really what they're doing. They're predicting words when they're training, but when they're actually generating new content, they're not "predicting" words (you can't predict your own decisions, that doesn't make sense), they are choosing words.
there are still tons of potentially valuable applications with clear objective: win stock market, create new material or design to maximize some metrics, etc.
https://developers.google.com/machine-learning/gan/discrimin...
I don't want to downplay those applications, but the killer breakthrough that the breathless world imagines and has wanted since Turing first talked about this is accurately modeling physical reality. "Invent a better engine" and what not. Without being physically embodied and being able to conduct experiments in the real world, you can't bootstrap that, short of simulating physics from first principles, which is not computationally feasible. You're inherently relying on some quorum of training material produced by embodied sources capable of actually doing science to be factually accurate.
The last bit doesn't seem to be true. There's quite a lot of indication that the computation can distinguishing hallucinations. It just has no incentive to communicate this.
GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975
Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334
Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221
Also even if we're strictly talking about text, there is still a ton of data left to train on. We've just barely reached what is easily scrapable online and are nowhere near a real limit yet. And of course, you can just train more than one epoch. That said, it's very clear quality data is far more helpful than sheer quantity and sheer quantity is more likely than not to derail progress.
Right now it would be pretty easy to simply take ChatGPT output, feed it back in in a different thread (or even to a different model, such as Claude), and ask it which items in the response should be fact-checked, and also just to point out any that seem obviously wrong.
The former should be really easy to do, it doesn't have to know whether it's right or wrong -- it just has to know that it is a checkable fact. For instance the well known case of a lawyer citing a non-existent case from ChatGPT, it could say "this case should be fact checked to see that it is real and says what I said it said". Based on my experience with chatGPT (GPT-4 especially), this should be well within its current capabilities. (I'm going to try an experiment now.)
They could probably start having it do this behind the scenes, and check its own facts and learn from it so it learns when it is likely to hallucinate and learn to pick up on it. Even if for safety reasons it's not going out and hitting the web every time you're asking a question, it could be giving you a list at the end of the response of all the things within the response that you might want to check for yourself, maybe suggesting Google searches you should do.
Surely humans using it would find great value in knowing the model's confidence or whether it thinks its confabulating or not.
These services are created to give the best product to users, and so wouldn't this be a better product? Therefore there is incentive. Happier users and a product that is better than competitors.
Why would the computation care about any of that ? I'm talking about incentive for the model.
Sounds like it is a choice of the model creators then if they could instruct their testers to reward quality.
Its responses sure seem like it does.
I’d be happier with its responses if it was honest about when it was not confident in its answer.
Why would "making users happy" incentivize for truth ?
Seems like common sense to me.
Who’s asking the chat bot questions not looking for or wanting a truthful answer a lot of the time?
If the model understood or captured “human interest” at all in its training this should be pretty fundamental to its behavior.
In Douglas Adams Hitchhikers Guide to the Galaxy, this is (somewhat) fixed by giving the AIs emotion ..
It might also need to be able to distinguish between Knightian uncertainties and probabilities when there is nothing to base things on.
Ah, the old aphorism, don't put anything on the web you don't want Google to take.
My read is that genius never owned the copyright to the lyrics in the first place.
I understand with how training is done right now that more data makes things scale really well without having to come up with new concepts, but it seems completely obvious that better processing of already available knowledge is the way to make the next leaps. The idea is that, what is keeping me from having expert level knowledge in 50 different fields and using that knowledge to draw entirely new connections between all of them, in addition to understanding where things go wrong, is not lack of freely available expert level information.
And yet, GPT4 barely reaches competency. It feels like computers should be able to get much more out of what is already available, specially when levering cross discipline knowledge to inform everything.
https://www.youtube.com/watch?v=cz-3WDdqbj0&list=PLX9JDz3uBp...
I mean why ? It took millions of years of optimization for humanity to get to the competence level they're currently at. If you think you're "starting from scratch", you really aren't. Keep in mind LLMs can use significantly less data (but still a lot) when you're not trying to force feed the sum total of human knowledge.
So should they be able to get more out of it ? or is this par the course for NNs?
Our genomes have ~600MB, so where are you claiming that humans transmit millions of years of language optimization between generations?
It also contains a remarkable amount of compression. Even if you assume that a genome contains a complete template for a human (it likely doesn't) the fact that the compressed version is 600MB doesn't really relate to the unpacked amount of information. Especially since the compression has seen millions of years of optimization.
They're not saying we store the weights, they're saying that we store the "architecture" that you overlay the weights on with training.
This either says we need better models, not more data.
Or, the human ability to be multi-modal augments our ability to perform language tasks in which case, we need to pump LLMs with much more image and video input than we currently do.
GPT has no such predisposition.
GPT needing lots of training data doesn't mean we need a better architecture. You would expect it to have a lot of training because humans have a lot of training too, spanning millions of years..
Human training begins at birth.
Evolution might result in better architecture and init(inductive biases), but that's a separate thing than training.
No reason beyond compute we couldn't do something similar. Ie find good architectures by evaluating them using multiple random weights, and evolve those archigectures that on average gives the best results.
Then over time add a short training step before evaluating.
Is this true? My understanding is that people are born with many pre trained weights. Was the evolutionary convergence of those weights not itself training?
No, inductive biases are not training.
I'm saying that better models(ie: better inductive biases) or non-language data is needed to advance LLMs and somehow we've arrived at "evolution is training." I'm not sure how that's relevant to the point.
And the more inductive bias we've shoved into models, the worse they've performed. Transformers have a lot less bias than either RNNs or CNNs and are better for it. Same story with what preceded both.
ChatGPT today is like a backhoe compared to a team of human with shovels. You still need a person who knows how to operate it, and their skills are different from those who dig with shovels. A bad backhoe operator is worse than any number of humans with shovels.
Pretty soon it will be able to learn by running its own code and testing it by looking at its output, including with its "vision."
That is very interesting. I can't think of a single time the Google built-in LLM has worked for me, let alone surprised and delighted me with a technical answer. I'm sure it's great at a lot of things, but it's not a replacement for SO yet.
ChatGPT not only gets my specific problem but can produce workable code in many cases.
A couple of speakers from Microsoft at the MPPC 2023 this week indicated that OpenAI’s models were not merely exposed to e.g. poetry, programming, etc. and learned those fields.
Rather they were saying that the model is more of a composite of skills that were specifically trained, building on word identification, sentences, grammar, ultimately moving on to higher order skills.
Perhaps this isn’t a secret (or perhaps I misunderstood), but it means the model’s ability to perform self-directed learning is much less than I previously though.
If you would judge GPT4 to be a competent programmer your experience is wildly different from mine. (I am not sure why you felt the need to put a "basic" in there in reference to what I wrote, since that is not what I wrote).
It’s good, don’t get me wrong, but if you go deep it’s usually incorrect somewhere.
Here's the b+tree implementation it gave me. I haven't checked if it's right. But, I was just curious what it'd come up with.
https://chat.openai.com/share/a582aa43-cca8-426a-a4de-f45fdb...
But this is ultimately impossible right? That’s the one thing I really hate about what is happening right now with ChatGPT.
I can’t tell you how many people are worried about their future because of AI because I don’t know the exact number, but I know I am worried about it because it can already do so much, and I fail to see a scenario in which attribution alone is going to make things better.
Writing and digital art more than code, but not even code is safe. It is merely safe by the extent that OpenAI is willing to drip feed its future releases.
The bad outcome isn't that we put weavers out of a job, but that we're stuck weaving our clothes by hand forever. Same thing for art and coding.
I'd actually be fine weaving clothes by hand a few hours a month if it meant I had no other work to do. But at the moment there are still people who don't own enough if any clothes and work far more than I do. And the question is if the people building AI can be trusted to liberate those people from slavery.
All the money the rich make by selling their robot products to who? Each other? Certainly not to the destitute workers, those don't have money.
I agree that there is a risk that a minority will hoard the new resources for themselves, only it's not this simple.
So how can you generate data at level N+1 when you have a model at level N?
You amplify the model - give it more tokens (CoT), more rounds of LLM interaction, tools like code executor and search engine, you use retrieval to bring in more useful context, or in some cases you can validate by code execution.
But there is a more general framework - by embedding LLMs in larger systems, they act as sources of feedback to the model. From the easiest - a chat interface, where the "external system" is a human, to robotics and AI agents that interact with anything, or simulations. We need to connect AI to feedback sources so it can learn directly, not filtered through human authored language.
From this perspective it is apparent that AI can assimilate much more feedback signal than humans. The road ahead for AI is looking amazing now. What we are seeing is language evolving a secondary system of self replication besides humans - LLMs. Language evolves faster than biology, like the rising tide, lifting both humans and AI.
GPT-4 cannot play a good game of tic tac toe. But it can play passable chess. This is a good point to ponder.
It can.
https://chat.openai.com/share/75758e5e-d228-420f-9138-7bff47...
Building an AI that can actually learn the way humans learn instead of slightly nudging the output in one direction with countless examples would be a major leap forward, I would guess. I have no good idea how far we are away from that, but it seems not the easiest thing to do with the way we currently build those systems. Or maybe the way we currently train these models turns out to be good enough and there is not much to be gained from a more human like learning process.
Let's just say that they often aren't.
We will call them Psykers.
The Machine God has blessed them with the ability to take existing knowledge and fill the void.
No RAG. No vector databases. Pure willpower and biologics combined with the blessings of the Machine God.