OpenAI researchers warned board of AI breakthrough ahead of CEO ouster
reuters.com
reuters.com
The reason LLMs fail at solving mathematical problems is because: 1) they are terrible at arithmetic, 2) they are terrible at algebra, but most importantly, 3) they are terrible at complex reasoning (more specifically they mix up quantifiers and don't really understand the complex logical structure of many arguments) 4) they (current LLMs) cannot backtrack when they find that what they already wrote turned out not to lead to a solution, and it is too expensive to give them the thousands of restarts they'd require to randomly guess their way through the problem if you did give them that facility
Solving grade-school problems might mean progress in 1 and 2, but that is not at all impressive, as there are perfectly good tools out there that solve those problems just fine, and old-style AI researchers have built perfectly good tools for 3. The hard problem to solve is problem 4, and this is something you teach people how to do at a university level.
(I should add that another important problem is what is known as premise selection. I didn't list that because LLMs have actually been shown to manage this ok in about 70% of theorems, which basically matches records set by other machine learning techniques.)
(Real mathematical research also involves what is known as lemma conjecturing. I have never once observed an LLM do it, and I suspect they cannot do so. Basically the parameter set of the LLM dedicated to mathematical reasoning is either large enough to model the entire solution from end to end, or the LLM is likely to completely fail to solve the problem.)
I personally think this entire article is likely complete bunk.
Edit: after reading replies I realise I should have pointed out that humans do not simply backtrack. They learn from failed attempts in ways that LLMs do not seem to. The material they are trained on surely contributes to this problem.
So maybe OpenAI just slapped an RL agent on top of the next-token generator.
Hm? it's pretty trivial to use a sampler for LLMs that has a beam search and will effectively 'backtrack' a 'bad' selection.
It just doesn't normally help-- by construction the LLM sampled normally already approximates the correct overall distribution for the entire output, without any search.
I assume using a beam search does help when your sampler does have some non-trivial constraints (like the output satisfies some grammar or passes an algebraic test, or even just top-n sampling since those adjustments on a token by token basis result in a different approximate distribution than the original distribution filtered by the constraints).
They're good at putting things together commonly found together but not so good at separating concepts back out into more detailed sub pieces.
Currently the context length isn't even long enough for it to remember what problem it was solving. And I've tried to come up with a bunch of ways around this. They all fail for one reason or another. LLMs are really a long, long way off managing this efficiently in my opinion.
The absolute best case scenario for a new technology is that it when it seems like a toy for nerds, and doesn't outperform anything we have today, but the scaling path is clear.
Its problems just won't matter if it does that one thing with scaling. The web is a pretty good hypermedia platform, but a disastrously bad platform for most other computer applications. Nevertheless the scaling of URIs and internet protocols have caused us to reorganize our lives around it. And then if there really are unsolvable problems with the platform they just get offloaded onto users. Passwords? Privacy? Your problem now. Surely you know to use a password manager?
I think this new wave of AI is going to be like that. If they never solve the hallucination/confabulation issue, it's just going to become your problem. If they never really gain insight, it's going to become your problem to instruct them carefully. Your peers will chide for not using a robust AI-guardrail thing or not learning the basics of prompt engineering like all the kids do instinctively these days.
Saying that performance on grade-school problems is predictive of performance on complex reasoning tasks (including theorem proving) is like saying that a new kind of mechanical engine that has 90% efficiency can be scaled 10x.
These kind of scaling claims drive investment, I get it. But to someone who understands (and is actually working on) the actual problem that needs solving, this kind of claim is perfectly transparent!
The hyperbole that surrounds them fits the mould nicely.
1. The scaling path is decreased val/test loss during training.
2. We have seen multiples times that large decreases in this loss have resulted in very impressive improvements in model capability across a diverse set of tasks (e.g. gpt-1 through gpt-4, and many other examples).
3. By now, there is tons of robust data demonstrating really nice relationships between model size, quantity of data, length of training, quality of data, etc and decreased loss. Evidence keeps building that most multi-billion param LLMs are probably undertrained, perhaps significantly so.
4. Ergo, we should expect continued capability improvement with continued scaling. Make a bigger model, get more data, get higher data quality, and/or train for longer and we will see improved capabilities. The graphs demand that it is so.
---
This is the fundamental scaling hypothesis that labs like OpenAI and Anthropic have been operating off of for the past 5+ years. They looked at the early versions of the curves mentioned above, extended the lines, and said, "Huh... These lines are so sharp. Why wouldn't it keep going? It seems like it would."
And they were right. The scaling curves may break at some point. But they don't show indications of that yet.
Lastly, all of this is largely just taking existing model architectures and scaling up. Neural nets are a very young technology. There will be better architectures in the future.
The whole idea of double-blind test (and really, the whole scientific methodology) is based on one simple thing: even the most experienced and informed professionals can be comfortably wrong.
We'll only know when we see it. Or at least when several independent research groups see it.
So: we might be close to a breakthrough, that breakthrough could get out of hand, then it could kill a billion+ people.
Among knowledgeable people who have concerns in the first place, I'd say giving the probability of a very bad outcome of cumulative advances as "very low" is a fringe position. It seems to vary more between "significant" and "close to unity".
There are some knowledgeable people like Yann LeCun who have no concerns whatsoever but they seem singularly bad at communicating why this would be a rational position to take.
Myself, I think I count as a massive optimist, as my P(doom) is only about 15% — basically the same as Russian Roulette — half of which is humans using AI to do bad things directly.
That's the human hallucination problem. In science it's a very difficult issue to deal with, only in hindsight you can tell which papers from a given period were the good ones. It takes a whole scientific community to come up with the truth, and sometimes we fail.
_Using_ this fallacy in an argument invalidates the argument (or shows it did not exist in the first place)
Also, do you remember what go players said when they beat Fan Hui? Change can come quick
My point is that no technology which can solve grade school maths problems would be viewed as a breakthrough by anyone who understood the problem. The fundamental problems which need to be solved are not problems you encounter in grade school mathematics. The article is just ill-informed.
The ability to eventually solve a given theorem isn’t interesting — especially if the time is longer than the time left in the universe.
It’s far more interesting to see if an AI can, given an arbitrarily stated problem make clear progress quickly.
Not perhaps in the sense of making mathematicians redundant but it seems like a breakthrough for ChatGPT type programs.
You've got to remember these things have gone from kind of rubbish a year or so ago to being able to beat most students at law exams now and by the sounds of it beat students at math tests shortly. At that rate or progress they'd be competing with the experts before very long.
That's just what I came up with after thinking on it for 2 minutes. I'm sure they have even better ideas.
Another issue is that of autoformalisation. This is the one part of the problem where an LLM might be able to help, if it were reliable enough (it isn't currently) or if it could truly understand the logical structure of mathematical problems correctly (currently they can't).
Basically every reply from a human can be interpreted as a reward signal. If the human restates the question, it means a negative reward, the AI didn't get it. If the human corrects the AI, another negative reward, but if they continue the thread then it is positive. You can judge turn-by-turn and end-to-end all chat logs with GPT4 to annotate.
The great thing about chat based feedback is that it is scalable. OpenAI has 100M users, they generate these chat sessions by the millions every day. Then they just need to do a second pass (expensive, yes) to annotate the chat logs with RL reward signals and retrain. But they get the human-in-the-loop for free, and that is the best source of feedback.
AI-human chat data is in-domain for both the AI and human, something we can't say about other training data. It will contain the kind of mistakes AI does, and the kind of demands humans want to solve with AI. My bet is that OpenAI have realized this and created GPTs in order to enrich and empower the AI to create the best training data for GPT-5.
The secret sauce of OpenAI is not their people, or Sam, or the computers, but the training set, especially the augmented and synthetic parts.
the thing is that ideas not necessary easy to implement. There will be many obstacles on route you described:
- quality of provers, is there good ergo provers which also can run at large scales (say billions of facts)
- you need some formalization approach, probably LLM will do some work, but we don't know what will be quality
- LLM likely will generate many individual factoids, which are losely compatible, contradicting, etc, and untrivial effort is required to reconcile and connect them
What I mean is, yes, neural networks are stochastic and that seems to be why they're bad at logic; on the other hand it' not exactly hallucinating a game of Go, and that seems different to how neural networks are prone to hallucination and confabulation on natural language or X-ray imaging.
It's really harder than one might imagine to develop a system which is good at higher order logic, premise selection, backtracking, algebraic manipulation, arithmetic, conjecturing, pattern recognition, visual modeling, has a good mathematical knowledge, is autonomous and fast enough to be useful.
For my money, it isn't just a matter of fitting a few existing jigsaw pieces together in some new combination. Some of the pieces don't exist yet.
The most interesting papers to me personally are the following three:
* Making higher order superposition work. https://doi.org/10.1007/978-3-030-79876-5_24
* MizAR 60 for Mizar 50. https://doi.org/10.48550/arXiv.2303.06686
* Magnus Hammer, a Transformer Based Approach to Premise Selection. https://doi.org/10.48550/arXiv.2303.04488
Your mileage may vary.
But even there, can we say scientifically that LLMs cannot do math? Do we actually know that? And in my mind, that would imply LLMs cannot achieve AGI either. What do we actually know about the limitations of various approaches?
And couldn't people argue that it's not even necessary to think in terms of capabilities as if they were modules or pieces? Maybe just brute-force the whole thing, make a planetary scale computer. In principle.
https://www.theverge.com/2023/11/22/23973354/a-recent-openai...
Reuters update 6:51 PST
The Verge has acted like an intermediary for Sam's camp during this whole saga, from my reading.
"considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices"
https://arxiv.org/abs/2305.10601
Generally with you though, this thing is not leading to real smarts and that's accepted by many. Yes, it'll fill in a few gaps with exponentially more compute but it's more likely that an algo change is required once we've maxed out LLM's.
The problem with solving such problems with LLMs is that if the solution to the problem is unlike problems seen in training, the LLM will almost every time take the wrong path and very likely won't even think of the right path at all.
The AI really does need to understand why the paths it tried failed in order to get insight into what might work. That's how humans work (well, one of many techniques we use). And despite what people think, LLMs really don't understand what they are doing. That's relatively easy to demonstrate if you get an LLM off distribution. They will double down on obviously erroneous illogic, rather than learn from the entirely new situation.
At some point in the past, you yourself were only capable of solving grade school maths problems.
With recent advancements they can already outperform humans at many tasks that were considered to require AGI level machine intelligence just a few years ago.
If the model then distills that information down to a new feature, and re-examines the original prompt with the new feature embedded in an extra input tensor, then repeats this process ad-infinitum, will the language model's "prime directive" and reasoning ability be sufficient to arrive at new, verifiable and provable conjectures, outside the realm of the dataset it was trained on?
If GPT-4,5,...,n can progress in this direction, then we should all see the writing on the wall. Also, the day will come where we don't need to manually prepare an updated dataset and "kick off a new training". Self-supervised LLMs are going to be so shocking.
Unfortunately there are rather a lot of issues which are difficult to describe concisely, so here is probably not the best place.
Primary amongst them is the fact that an LLM would be a horribly inefficient way to do this. There are much, much better ways, which have been tried, with limited success.
[1] https://arxiv.org/abs/2302.04761 Toolformer: Language Models Can Teach Themselves to Use Tools
Early attempts at human flight approached it by strapping wings to people's arms and flapping: Do you think that would have eventually worked too, if only we had just given it a bit more time and faith?
Yes. Because we haven't yet reached the limit of deep learning models. GPT-3.5 has 175 billion parameters. GPT-4 has an estimated 1.8 trillion parameters. That was nearly a year ago. Wait until you see what's next.
I don't think many people would describe GPT-4 as a stochastic parrot already... when the paper that coined (or at least popularized) the term came up in early 2021, the term made a lot of sense. In late 2023, with models that at the very least show clear signs of creativity (I'm sticking to that because "reasoning" or not is more controversial), it's relegated to reductionistic philosophical arguments, but not really a practical description anymore.
The setting in the paper is about narrating a single combat between Ignatius J. Reilly and a pterodactyl. Ignatius J. Reilly is a literary character with some very idiosyncratic characteristics, that appears in a single book, where he of course didn't engage in single combats at all or interact with pterodactyls. He doesn't seem to have been the target of fanfiction either (which could be a problem if characters like, say, Harry Potter or Darth Vader were used instead), so the paper argues that it's very unlikely that a story like that had been ever written at all prior to this paper.
Do you know how that “creativity” is achieved? It’s done with a random number generator. Instead of having the LLM pick the absolute most likely next token, they have it select from a set of most likely next tokens - size of the set depends on “temperature”.
Set temperature to 0, and the LLM will talk in circles and not really say anything interesting. Set it too high and it will output nonsense.
The whole design of LLMs don’t seem very well thought out. Things are done a certain way not because it makes sense but because it seems to produce “impressive” results.
Sounds like some people I know, at both extremes.
> The whole design of LLMs don’t seem very well thought out. Things are done a certain way not because it makes sense but because it seems to produce “impressive” results.
They have been designed and trained to solve natural language processing tasks, and are already outperforming humans on many of those tasks. The transformer architecture is extremely well thought out, based on extensive R&D. The attention mechanism is a brilliant design. Can you explain exactly which part of the transformer architecture is poorly designed?
They aren’t really designed to do anything actually. LLMs are models of human languages - it’s literally in the name, Large Language Model .
https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
I’m sorry but I don’t trust something that uses a random number generator as part of its output generation.
No. And the article you linked to does not say that (because Wolfram is not an idiot).
Transformers are designed and trained specifically for solving NLP tasks.
> I’m sorry but I don’t trust something that uses a random number generator as part of its output generation.
The human brain also has stochastic behaviour.
Saying that it's done with a random number generator doesn't really explain the wonder of achieving meaningful creative output, as in being able to generate literature, for example.
What they find is that for every emerging ability where an evaluation metric seems to have a sudden jump, there is some other underlying metric that is continuous.
The thing is that the metric with the jump is the one people would actually care about (like actually being able to answer questions correctly, etc.) while the continuous one is an internal metric. I don't think that refutes the existence of emerging abilities, it just explains a little bit of how they arise.
There are fundamental limitations to LLMs though - a limit to what can be learned by training a system to predict next word form a fixed training corpus. It can get REALLY good at that task, as we've seen, to extent that it's not just predicting next word but rather predicting an entire continuation/response that is statistically consistent with the training set. However, what is fundamentally missing is any grounding in anything other than the training set, which is the what causes hallucinations/bullshitting. In a biological intelligent system predicting reality is the goal, not just predicting what "sounds good".
LLMs are a good start in as much as they prove the power of prediction as a form of feedback, but to match biological systems we need a closed-loop cognitive architecture that can predict then self-correct based on mismatch between reality and prediction (which is what our cortex does).
For all of the glib prose that an LLM can generate, even if it seems to understand what you are asking (after all, it was trained with the goal of sounding good), it doesn't have the intelligence of even a simple animal like a rat that doesn't use language at all, but is grounded in reality.
It was trained not only to "sound good" aesthetically but also to solve a wide range of NLP tasks accurately. It not only "seems to" understand the prompt but it actually does have a mechanical understanding of it. With ~100 layers in the network it mechanically builds a model of very abstract concepts at the higher layers.
> it doesn't have the intelligence of even a simple animal
It has higher intelligence than humans by some metrics, but no consciousness.
Was it? I've only heard of pre-training (predict next word) and subsequent RLHF + SFT "alignment" (incl. aligning to goal of being conversational). AFAIK the NLP skills that these LLMs achieve are all emergent rather than explicitly trained.
I'm not sure we can really say the net fully understands even if it answers as if it does - it was only trained to "predict next word", which in effect means being trained to generate a human-like response. It will have learnt enough to accomplish that goal, and no more (training loss tends to zero as goal is met).
Contrast this to an animal with a much richer type of feedback - reality, and with continual (aka online) learning. The animal truly understands it's actions - i.e. has learnt to accurately predict what will happen as a result of them.
The LLM does not understand it's own output in this sense - it exists only in a world of words, and has no idea if the ideas it is expressing are true or not (hence all the hallucinating/bullshitting). It only knew enough to generate something that sounded like what a person might say.
I believe you are right about that. I did some research after reading your comment. Transformers were certainly designed for NLP, but with large enough models the abilities can emerge without necessarily being explicitly trained for it.
> I'm not sure we can really say the net fully understands even if it answers as if it does - it was only trained to "predict next word", which in effect means being trained to generate a human-like response.
It depends on your definition of "understand". If that requires consciousness then there is no universally agreed formal definition.
Natural Language Understanding (NLU) is a subset of Natural Language Processing (NLP). If we take the word "understanding" as used in an academic and technical context then yes they do understand quite well. In order to simply "predict the next word" they learn an abstract model of syntax, semantics, meaning, relationships, etc, from the text.
> and has no idea if the ideas it is expressing are true or not (hence all the hallucinating/bullshitting).
That is not really an issue when solving tasks that are within it's context window. It is an issue for factual recall. The model is not a type of database that stores its training set verbatim. Humans have analogous problems with long term memory recall. I can think straight within my working memory but my brain will "hallucinate" to some extent when recalling distant memories.
If you ask it a question where the training data (or input data = context) either didn't include the answer, or where it was not obvious how to get the right answer, that will not (unfortunately) stop it from confidently answering!
Wait a minute. You are completely missing the entire "attention mechanism" thing which is what makes transformers so capable. For each output token generated in sequence, the attention mechanism evaluates the current tokens relationship to all tokens in the context window, weighing their relevance. There are multiple "attention heads" running in parallel (16 in GPT-3.5). Now for each layer of the neural network there is an attention mechanism, independently processing the entire context window for each token. There are ~100 layers in ChatGPT. So now we have 100 layers times 16 attention heads = 1600 attention mechanisms evaluating the entire context window over many deep layers of abstraction for each output token.
The real issue isn't that the net often "makes a statistical guess" rather than saying "I don't know", but rather that when it does make errors it has no way to self-detect the error and learn from the mistake, as a closed-loop biological system is able to do.
> The context window only has to do with the size of input it has access to - its not related to what it's outputting
The sequential token generation process is closely related to the content of the context window at every step.
Maybe I misunderstood your point. I know these things can hallucinate when asked about obscure facts that they weren't sufficiently trained on.
I haven't found this to be the case in my experience. I use ChatGPT-4. It often tells me when it doesn't know or have enough information.
If you haven't used GPT-4 I recommend signing up for a month. It is next level, way better than 3.5. (10x the parameter count). (No I'm not being paid to recommend it.)
Is is not a stochastic parrot today. Deep learning models can solve problems, recognize patterns, and generate new creative output that is not explicitly in their training set. Aside from adding more parameters there are new neural network architectures to discover and experiment with. Transformers aren't the final stage of deep learning.
Any actually creative output from these models is by pure random chance, which is most definitely different from the deliberate human reasoning that has produced our intellectual advances throughout history. It may or may not be inferior: there's a good argument to be made that "random creativity" will outperform human capabilities due to the sheer scale and rate at which the models can evolve, but there's no evidence that this is the case (right now).
We don't know exactly how human reasoning works, but the observational evidence clearly indicates it is not by randomly piecing together tokens already known.
This reasoning is invalid. For fun, I checked if GPT4 would catch the logical errors you made, and it did. Specifically, it correctly pointed out that absence of evidence is not evidence of absence. But even if there had been evidence of absence, this reasoning is invalid because it presumes that human reasoning must result in intellectual discovery irrespective of how it is employed, and so that if we can't find intellectual discoveries, it must mean an absence of human reasoning. In other words, it invalidly assumes that a difference in outcomes must represent a difference in the structure of reasoning. This is trivially invalid because humans think without making intellectual discoveries all the time.
However, it's also a strawman because I did not claim that humans and ML models reason the same way. I claimed there is no evidence of 'some sort of grand distinction between "probabilistically serializing tokens" and "deliberate human reasoning" other than scale'.
1) This explicitly recognizes that there is a difference, but that it might be just scale, and that we don't have evidence it doesn't. Your argument fails to address this entirely.
2) Even at scale, it does not claim they would be the same, but argues we don't have evidence that "probabilistically serializing tokens" must be inherently different from deliberate human reasoning" to an extent sufficient to call it "some sort of grand distinction". We can assume with near 100% certainty that there are differences - the odds of us happening upon the exact same structure is near zero. That does however not mean that we have any basis for saying that human reasoning isn't just another variant of "probabilistically serializing tokens".
I'll note that unlike you, GPT4 also correctly interpreted my intent when asked to review the paragraph and asked whether it implies the two must function the same. I could* take that to imply that LLMs are somehow better at humans at reasoning, but that would be logically invalid for the same reasons as your argument.
> We don't know exactly how human reasoning works, but the observational evidence clearly indicates it is not by randomly piecing together tokens already known.
Neither does LLMs. Piecing together tokens in a stochastic manner based on a model is not "randomly piecing together" - the model guides the process strongly enough that it's a wildly misleading characterization, as you can indeed trivially demonstrate by actually randomly piecing together words.
But even if we assume a less flippant and misleading idea of what LLMs do, your claim is incorrect. Observational evidence does nothing of the sort. If anything, the rapidly closing gap between human communication and LLMs shows that while it is extremely likely to be structural differences at the low level, it is increasingly unclear whether they are a material distinction. In other words, it's unclear whether the hardware and even hardwired network matters much relative to the computational structure the trained model itself creates.
You're welcome to your beliefs - but they are not supported by evidence. We also don't have evidence the other way, so it's not unreasonable to hold beliefs about what the evidence might eventually show.
For context this is the grandparent within which my original reply was scoped:
I feel very comfortable saying, as a mathematician, that the ability to solve grade school maths problems would not be at all a predictor of ability to solve real mathematical problems at a research level. The reason LLMs fail at solving mathematical problems is because: 1) they are terrible at arithmetic, 2) they are terrible at algebra, but most importantly, 3) they are terrible at complex reasoning (more specifically they mix up quantifiers and don't really understand the complex logical structure of many arguments) 4) they (current LLMs) cannot backtrack when they find that what they already wrote turned out not to lead to a solution, and it is too expensive to give them the thousands of restarts they'd require to randomly guess their way through the problem if you did give them that facility
Solving grade-school problems might mean progress in 1 and 2, but that is not at all impressive, as there are perfectly good tools out there that solve those problems just fine, and old-style AI researchers have built perfectly good tools for 3. The hard problem to solve is problem 4, and this is something you teach people how to do at a university level.
(I should add that another important problem is what is known as premise selection. I didn't list that because LLMs have actually been shown to manage this ok in about 70% of theorems, which basically matches records set by other machine learning techniques.)
(Real mathematical research also involves what is known as lemma conjecturing. I have never once observed an LLM do it, and I suspect they cannot do so. Basically the parameter set of the LLM dedicated to mathematical reasoning is either large enough to model the entire solution from end to end, or the LLM is likely to completely fail to solve the problem.)
I personally think this entire article is likely complete bunk.
Edit: after reading replies I realise I should have pointed out that humans do not simply backtrack. They learn from failed attempts in ways that LLMs do not seem to. The material they are trained on surely contributes to this problem.
> Why would adding more parameters suddenly make it better at this sort of reasoning?
My response to you was about that specific claim as worded. Nothing more, nothing less.
The value improvement between 17.5b parameters and 175b parameters is much greater than the value improvement between 175b parameters and 18t parameters.
IOW, each time we throw 100 times more processing power at the problem, we get a measly 2 time increase in value.
Interestingly, we how have human powered aircraft... We have flown ~60km with human leg power alone. We've also got human powered ornithopters (flapping wing designs) which can fly but only for very short times before the pilot is exhausted.
I expect that another 100 years from now, both records will be exceeded, altough probably for scientific curiosity more than because human powered flight is actually useful.
https://en.wikipedia.org/wiki/UTIAS_Snowbird
13 years ago! Wow, how did I miss that?
I like the very last example in my tongue-in-cheek article, https://nt4tn.net/articles/aixy.html
Certainly the LLM didn't derive Fermat's theorem on sums of two squares under the hood (and, of course, very obviously didn't prove it correct-- as the code is technically incorrect for 2), but I'm somewhat doubtful that there was any function exactly like the template in codex's training set either (at least I couldn't quickly find any published code that did that). The line between creating something and applying a memorized fact in a different context is not always super clear.
That's exactly what Go/Baduk/Weiqi players think some years ago. And superalignment is defintely OpenAI's major research objective:
> https://openai.com/blog/our-approach-to-alignment-research
> our AI systems are proposing very creative solutions (like AlphaGo’s move 37)
When will mathematicians face the move 37 moment?
If you want to solve grade school math problems, why not use an 'add' instruction? It's been around since the 50s, runs a billion times faster than an LLM, every assembly-language programmer knows how to use it, every high-level language has a one-token equivalent, and doesn't hallucinate answers (other than integer overflow).
We also know how to solve complex reasoning chains that require backtracking. Prolog has been around since 1972. It's not used that much because that's not the programming problem that most people are solving.
Why not use a tool for what it's good for and pick different tools for other problems they are better for? LLMs are good for summarization, autocompletion, and as an input to many other language problems like spelling and bigrams. They're not good at math. Computers are really good at math.
There's a theorem that an LLM can compute any computable function. That's true, but so can lambda calculus. We don't program in raw lambda calculus because it's terribly inefficient. Same with LLMs for arithmetic problems.
We want to remove the human from the solution design. It would help us tremendously tbh, just like I don't know, Google map helped me never to have to look for direction ever again ?
There's your 'add'.
You can even route back runtime errors and ask it to fix its own code. And it does.
It can write code and even write a test to test that code. Give it an interpreter and you’re all set.
https://chat.openai.com/c/dd8de3f7-a50c-4b6d-bd3f-b52ed996d3...
That’s how langchain works, chatgpt plugins and gpt function calling. It has proven to be pretty robust - that is, gpt4 realising when it needs to use a tool/write code for calculations when needed and then using the output.
PS: also I thought GPT4 already achieved 90% in some university math grades? Oh I remember that was multiple-choice
I believe it's because having a foundational understanding of maths and logic is important when solving other problems, and if you are looking to create an AI that can generally solve all problems it should probably have some intuitive understanding of maths too.
i.e. if we want an LLM to be able to solve unsolved theorems in the future, this requires a level of understanding of maths that is more than 'teach it to use a calculator'.
More broadly, I can imagine a world where LLM training is a bit more 'interactive' - right now if you ask it to play a game of chess with you it fails, but it has only ever read about chess and past games and guesses the next token based on that. What if it could actually play a game of chess - would it get a deeper appreciation for the game? How would this change it's internal model for other questions (e.g. would this make it answer better at questions about other games, or even game theory?)
EDIT: On re-reading I think I misunderstood you. No, I don't think it's a bold assumption to think it has an internal model of it at all. It may not be a sophisticated model, but it's fairly clear that LLM training builds world models.
We know with reasonable certainty that an LLM fed on enough chess games will eventually develop an internal chess model. The only question is whether GPT4 got that far.
Or, given https://thegradient.pub/othello/, why wouldn't it have an internal model of chess? It probably saw more than enough example games and quite a few chess books during training.
Well, yes, assume that every conversation you have with ChatGPT without turning off history makes it into the training set.
Watching my kids grow up, they just have fun doing things like trying to crawl, walk or drink. It's not about being the best at it, or the most efficient, it's just about the experience.
Now maths is taught in a boring way, but knowing it can help us lead more enjoable lives. When math is taught in an enjoyable way AND people get results out of it. Well that's glorious.
I am five years older than my brother, and we happened to land just on opposite sides of when children were still being taught mental arithmetic and when it was assumed they would, in fact, have calculators in their pockets.
It drives him crazy that I can do basic day-to-day arithmetic in my head faster than he can get out his calculator to do it. He feels like he really did get cheated out of something useful because of the proliferation of technology.
Lookup the chimp test. the videos of the best chimp are really quite incredible.
There is also the measured inflation of map traversing parts of the brain in pro tetris players and taxi drivers. I vaguely remember an explanation about atrophy in nearby areas of the brain, potentially to make room.
> If you want to solve grade school math problems, why not use an 'add' instruction?
Certainly the objective is not for the AI to do research-level mathematics.
It's not really even to do grade-school math.
The point is that grade-school math requires reasoning capability that transcends probabilistic completion of the next token in a sequence.
And if Q-Star has that reasoning capability, then it's another step-function leap toward AGI.
The problem is that there are different groups of people with different ideas about AI, and when talking about AI it's easy to end up tackling the ideas of a specific group but forgetting about the existence of the others. In this specific example, surely there are AI enthusiasts who see no limits to the applications of AI, including research-level mathematics.
[1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html
[2] Which the people making these models are familiar with. The whole thing is a trillion+ parameter linear algebra crunching machine after all.
Let's forget for a moment that stuff has to run on an actual machine. If you had to represent a quadratic equation, would you rather write:
(a) x^2 + 5x + 4 = 0
(b) the square of the variable plus five times the variable plus four equals zero
When you are trying to solve problems with a level of sophistication beyond the toy stuff you usually see in these threads, formal language is an aid rather than an impediment. The trajectory of every scientific field (math, physics, computer science, chemistry, even economics!) is away from natural language and towards formal language, even before computers, precisely for that reason.
We have lots of formal languages (general-purpose programming languages, logical languages like Prolog/Datalog/SQL, "regular" expressions, configuration languages, all kinds of DSLs...) because we have lots of problems, and we choose the representation of the problem that most suits our needs.
Unless you are assuming you have some kind of superintelligence that can automagically take care of everything you throw at it, natural language breaks down when your problem becomes wide enough or deep enough. In a way this is like people making Rube-Goldberg contraptions with Excel. 50% of my job is cleaning up that stuff.
IIRC Wolfram Alpha has (or had, hard to keep up) a way to connect with ChatGPT.
[1] But here it is again https://arxiv.org/abs/2302.04761
But as I noted elsewhere, training its ability to do it from scratch matters not for the ability to do it from scratch, but for the transferability of the reasoning ability. And so I think that while it's a good choice for OpenAI to make it automatically pick more effective strategies to give the answer it's asked for, there is good reason for us to still dig into its ability to solve these problems "from scratch".
[1] https://chat.openai.com/share/694251c9-345b-4433-a856-7c38c5...
I think this is a fundamental problem with the "chat" style of interaction with many of these models (that the language interface isn't the best way of representing any specific problem even if it's quite a useful compromise for problems in general). I think an intrinsic problem of this class of model is that they only have text generation to "hang computation off" meaning the "cognative ability" (if we can call it that) is very strongly related to how much text it's generating for a given problem which is why that technique of prompting using chain of thought generates much better results for many problems[2].
[1] Hence the famous payoff line "whereof we cannot speak, thereof we must remain silent"
[2] And I suspect why in general GPT-4 seems to have got a lot more verbose. It seems to be doing a lot of thinking out loud in my use, which gives better answers than if you ask it to be terse and just give the answer or to give the answer first and then the reasoning, both of which generally generate inferior answers in my experience and in the research eg https://arxiv.org/abs/2201.11903
It depends on whether you ask him before or after he went camping -- but yeah, I was going for an early-Wittgenstein-esque "natural language makes it way too easy to say stuff that doesn't actually mean anything" (although my argument is much more limited).
> I think this is a fundamental problem with the "chat" style of interaction
The continuation of my argument would be that if the problem is effectively expressible in a formal language, then you likely have way better tools than LLMs to solve it. Tools that solve it every time, with perfect accuracy and near-optimal running time, and critically, tools that allow solutions to be composed arbitrarily.
Alpha Go and NNUE for computer chess, which are often cited for some reason as examples of this brave new science, would be completely worthless without "classical" tree search techniques straight out of the Russel-Norvig.
Hence my conclusion, contra what seems to be the popular opinion, is that these tools are potentially useful for some specific tasks, but make for very bad "universal" tools.
Then as you collect more evidence you find that the age listed on the first one in the census was wildly off due to a transcription error and it's actually her.
You'd think some sort of rule-based system and database might help with these sorts of things. But the historical experience of expert system is that you then often automate the easy bits at the cost of demanding even more tedious data-entry. And you can't divorce data entry and deduction from each other either, because without context, good luck reading out a rare last name in the faded ink of some priest's messy gothic handwriting.
It feels like language models should be able to help. But they can't, yet. And it fundamentally isn't because they suck at grade school math.
Even linguistics, not something I know much about but another discipline where you try to make deductions from tons and tons of soft and vague evidence - you'd think language models, able to produce fluent text in more languages than any human, might be of use there. But no, it's the same thing: it can't actually combine common sense soft reasoning and formal rule-oriented reasoning very well.
sounds like belief change systems (a bit) to me!
I'm certainly no expert, but it seems to me that Wolfram Alpha provides a counterexample to some extent, since they claim to fuse expert knowledge and "AI" (not sure what that means exactly). Wolfram Alpha certainly seems to do much better at solving math problems than an LLM.
By all means, we should give LLMs lots and lots of specialised tools to let them take shortcuts, but that does not remove the reasons for understanding how to strengthen the reasoning abilities that would also make them good at maths.
EDIT: After having just coerced the current GPT4 to do arithmetic manually: It appears to have drastically improved in its ability to systematically following the required method, while ironically being far less willing to do so (it took multiple attempts before I got it to stop taking shortcuts that appeared to involve recognising this was a calculation it could use tooling to carry out, or ignoring my instructions to do it step by step and just doing it "in its head" the way a recalcitrant student might. It's been a while since I tested this, but this is definitely "new-ish".
Also we know that LLM's can't do a few things - arithmetic, inference & planning are in there. They look like they can because they retrieve discussions from the internet that contain the problems, but when they are tested out of distribution then all of a sudden they fail. However, some other nn's can do these things because they have the architecture and infrastructure and training that enables it.
There is a question for some of these as to whether we want to make NN's do these tasks or just provide calculators, like for grade students, but on the other hand something like Alphazero looks like it could find new ways of doing some problems in planning. The challenge is to find architectures that integrate the different capabilities we can implement in a useful and synergistic way. Lots of people have drawn diagrams about how this can be done, then presented them with lots of hand waving at big conferences. What I love is that John Laird has been building this sort of thing for like, forty years, and is roundly ignored by NN people for some reason.
Maybe because he keeps saying it's really hard and then producing lots of reasons to believe him?
Transformers may be like that, in that they can do generalized learning from different types of input, with only minor modifications needed to optimize for different input (or output) modes.
So it really does look like a society of networks, all working in functional synchrony (parasynchrony might be a better word) with some firms of “consciousness” updated in time slabs of about 200-300 milliseconds.
LLMs are probably equivalent now to Wernicke’s and Broca’s areas, but much more is needed “on top” and “on bottom”—-motivation, affect, short and longterm memory, plasticity of synaptic weighting and dynamics, and perhaps most important, a self-steering attentional supervisor or conductor. That attentional driver system is what we probably mean by consciousness.
You may know much more about this than me, but how sure are you about this? To me it seems like a better fit that the "self-steering attentional supervisor" is associated with what we mentally model (and oversimplify) as "free will", while "consciousness" seems to be downstream from the attention itself, and has more to do with organizing and rationalizing experiences than with than with the directly controlling behavior.
This processed information then seems to become ONE input to the executive function in following cycles, but with a lag of at least 1 second, and often much more.
> one part of much more complex systems of neural compute
As for your main objection, you're obviously right. But I wonder how much of the computation that is relevant for intelligence is actually in those other areas. It seems to me that recent developments indicate that Transformer type models are able to self-organize into several different type of microstructures, even within present day transformer based models [1].
[1]: https://www.youtube.com/watch?v=Gg-w_n9NJIE (comment from Ilya somewhere)
Not sure at all. Also some ambiguities in definitions. Above I mean “consciousness” of the type many would be willing to assume operates in a cat, dog, or mouse—attentional and occasionally, also intentional. I agree that this is downstream of pure attention. Attention needs to be steered and modulated. The combination of the two levels working together recursively is what I had in mind.
“Free will” gets us into more than that. I’ve been reading Daniel Dennett on levels of “intention” this week. This higher domain of an intentional stance (nice Wiki article) might get labeled “self-consciousness”.
Most humans seem to accept this as a cognitive and mainly linguistic domain—the internal discussions we have with ourselves, although I think we also accept that there is are major non-linguistic drivers. Language is an amazingly powerful tool for recursive attentional and semantic control.
My take on "free will" is definitely partly based on Dennett's work.
As for "consciousness", it seems to me that most of not all actions we do are decided BEFORE they hit our consciousness. For actions that are not executed immediately, the processing that we experience as "consciousness" may then raise some warning flags if the action our pre-conscious mind has decided on is likely to cause som bad consequences. This MAY cause the decision-making part (executive function) of the brain to modify the decision, but not because the consciousness can override the decision directly.
Instead, when this happens, it seems to be that our consciousness extrapolates our story into the future in a way that creates fear, desire or similar more primal motivations that have more direct influence over the executive function.
One can test this by for instance standing at near the top of a cliff (don't du this if suicidal): Try to imagine that you have decided to jump of the cliff. Now imagine the fall from the cliff and you hitting the rocks below. Even if (and maybe especially if) you managed to convince yourself that you were going to jump, this is likely to trigger a fear response strong enough to ensure you will not jump (unless you're truely suicidal).
Or for a less synthetic situation. Let's say you're a married man, but in a situation where you have an opportunity to have sex with a beautiful woman. The executive part of the brain may already have decided that you will. But if your consciousness predicts that your wife is likely to find out and starts to spin a narrative about divorce, loosing access to your children and so on, this MAY cause your executive function to alter the decision.
Often in situations like this, though, people tend to proceed with what the preconcious executive function had already decided. Afterwards, they may have some mental crisis because they ended up doing something their consciousness seemed to protest against. They may feel they did it against their own will.
This is why I think that the executive function, even the "free will" is not "inside" of consciousness, but is separate from it. And while it may be influenced by the narratives that our consciousness spin up, it also takes many other inputs that we may or may not be conscious of.
The reason I still call this "free" will, is based on Dennett's model, though. And in fact, "free" doesn't mean what we tend to think it means. Rather, the "free" part means that there is a degree of freedom (like in a vector space) that is sensitive to the kind of incentives the poeple around you may provide for your actions.
For instance stealing something can be seen as a "free will" decision if you would NOT do it if you knew with 100% certainty that you would be caught and punished for it. In other words, "free will" actions are those that, ironically, other people can influence to the point where they can almost force you to take them, by providing strong enough incentives.
In a single run, obviously not any, because it's context window is very limited. With a loop and access to an "API" (or willing conversation partner agreeing to act as one) to operate a Turing tape mechanism? It becomes a question of ability to coax it into complying. It trivially has the ability to carry out every step, and your main challenge becomes to get it to stick to it over and over.
One step "up", you can trivially get GPT4 to symbolically solve fairly complex runs of instructions of languages it can never have seen before if you specify a grammar and then give it a program, with the only real limitation again being getting it to continue to adhere to the instructions for long enough before it starts wanting to take shortcuts.
In other words: It can compute any computable function about as well as a reasonably easily distractable/bored human.
They have a pipeline of hypes ready to be deployed at a moment's notice. The next one is quantum, it's already gathering in the background. Give it a couple of years.
> If you want to solve grade school math problems
That's not the aim here. Very obviously what we are talking about here is _complementing_ AI language models with improved mathematical abilities, and whether that leads to anything interesting. Surely you understand that? Aren't you one of the highest rated commenters on this site?
1. "I don't think this is real news / important because solving grade school math is not a predictor of ability to do complex reasoning."
2. "LLMs can't solve grade school math because they're bad at arithmetic, algebra and most importantly reasoning."
So... from 2 automatically follows that LLMs with sufficiently better math may be sufficiently better at reasoning as you said "most importantly" reasoning is relevant for their ability to do math. Saying "most importantly reasoning" and then saying that reasoning is irrelevant if they can do math, is odd.
My question is how does the AI perceive. Basically how good is the simulation for its perception. If we know that, then we can probably assess its ability to reason because we can compare it to the closest benchmark we have (your average human being). How do AI's see, how did they learn concepts in strings of words and pixels? How does the concept it learnt in text carry through to images of colors, of shapes? Does it show a transfer of conceptual understanding across both two and three dimentional shapes?
I know these are more questions than answers, but its just things that I've been wondering about.
As a simple example that you can replicate using chatgpt, ask it to solve some simple maths problem. Very frequently you will get a solution that looks like reasoning but is not, and reveals that it does not have an actual model of the underlying maths but is in fact doing text prediction based on a history of maths. For example see here[1]. I ask it for some quadratics in x with some specification on the number of roots. It gives me what looks at first glance like a decent answer. Then I ask the same exact question but asking for quadratics in x and y[2]. Again the answer looks plausible except that for the solution "with one real root" it says the solution has one real root when x + y =1. Well there are infinite real values for x and y such that x + y =1, not one real root. It looks like it has solved the problem but instead it has simulated the solving of the problem.
Likewise stacking problems, used to check for whether an AI has a model of the world. This is covered in "From task structures to world models: What do LLMs know?"[3] but for example here[4] I ask it whether it's easier to balance a barrel on a plank or a plank on a barrel. The model says it's easier to balance a plank on a barrel with an output text that simulates reasoning discussing center of mass and the difference between the flatness of the plank and the tendency of the barrel to roll because of its curvature. Actual reasoning would say to put the barrel on its end so it doesn't roll (whether you put the plank on top or not).
[1] https://chat.openai.com/share/64556be8-ad20-41aa-99af-ed5a42...
[2] https://chat.openai.com/share/2cd39197-dc09-4d07-a0d6-6cd800...
[3] https://arxiv.org/abs/2310.04276
[4] https://chat.openai.com/share/4b631a92-0d55-4ae5-8892-9be025...
If you were to ask the same question of a real person and they replied with the exact same answer you could not conclude that person was not capable of "actual reasoning". It's a bit of witch-hunt question set to give you the conclusion you want.
The question you quoted ("Suggest a stable way of stacking a laptop, a book, 4 wine classes, a wine bottle and an orange") I would consider much fairer and cgpt3.5 gives a perfectly "reasonable" answer:
https://chat.openai.com/share/fdf62be7-5cb2-4088-9131-40e089...
You can get a LLM to simulate it "discovering" the pythagorean theorem, but can it actually, with the knowledge that was available at the time, discover the pythagorean theorem by itself?
Any parent will tell you, it's easy to simulate discovery and reasoning, it's a trick played for kids all the time. The actual, real stuff, that's way harder.
That simulated reasoning is enough to annihilate any human player they're faced with.
As Dijkstra famously said, "whether Machines Can Think... is about as relevant as the question of whether Submarines Can Swim".
Submarines don't swim, cars don't walk or gallop, cameras don't paint or draw... So what?
Once AI can simulate reasoning better than we can do the genuine thing, the question really becomes utterly irrelevant to the likely outcome.
...because they are too small and have too little weights. Cats cannot solve mathematical problems too, but unlike cats, neural network evolve.
cats evolve plenty, pressure towards mathematical reasoning has stymied as of late what with the cans of food and humans.
It's the humans that are hallucinating, not the text generator.
From the sound of it, this Q* model has a fundamentally different architecture, which will almost certainly make some of those issues not terrible any more.
Most likely, the Q* design is the very similar to the one suggested recently by one of the Google AI teams: doing a tree search instead of greedy next token selection.
Essentially, current-gen LLMs predict a sequence of tokens: A->B->C->D, etc... where the next "E" token depends on {A,B,C,D} and then is "locked in". While we don't know exactly how GPT4 works, reading between the lines of the leaked info it seems that it evaluates 8 or 16 of these sequences in parallel, then picks the best overall sequence. On modern GPUs, small workloads waste the available computer power because of scheduling overheads, so "doing redundant work" is basically free up to a point. This gives GPT4 a "best 1 of 16" output quality improvement.
That's great, but each option is still a linear greedy search individually. Especially for longer outputs the chance of a "mis-step" at some point goes up a lot, and then the AI has no chance to correct itself. All 16 of the alternatives could have a mistake in them, and now its got to choose between 16 mistakes.
It's as if you were trying to write a maths proof, asked 16 students, and instructed them to not cooperate and write their proof left-to-right, top-to-bottom without pausing, editing, or backtracking in any way! It'd like to see how "smart" humans would be at maths under those circumstances.
This Q* model likely does what Google suggested: Do a tree search instead of a strictly linear search. At each step, the next token is presented as a list of "likely candidates" with probabilities assigned to each one. Simply pick to "top n" instead of the "top 1", branch for a bit like that, and then prune based on the best overall confidence instead of the best next token confidence. This would allow a low-confidence next token to be selected, as long as it leads to a very good overall result. Pruning bad branches is also effectively the same as back-tracking. It allows the model to explore but then abandon dead ends instead of being "forced" to stick with bad chains of thought.
What's especially scary -- the type of scary that would result in a board of directors firing an overly commercially-minded CEO -- is that naive tree searches aren't the only option! Google showed that you can train a neural network to get better at tree search itself, making it exponentially more efficient at selecting likely branches and pruning dead ends very early. If you throw enough computer power at this, you can make an AI that can beat the world's best chess champion, the world's best Go player, etc...
Now apply this "AI-driven tree search" to an AI LLM model and... oh-boy, now you're cooking with gas!
But wait, there's more: GPT 3.5 and 4.0 were trained with either no synthetically generated data, or very little as a percentage of their total input corpus.
You know what is really easy to generate synthetic training data for? Maths problems, that's what.
Even up to the point of "solve this hideous integral that would take a human weeks with pen and paper" can be bulk generated and fed into it using computer algebra software like Wolfram Mathematica or whatever.
If they cranked out a few terabytes of randomly generated maths problems and trained a tree-searching LLM that has more weights than GPT4, I can picture it being able to solve pretty much any maths problem you can throw at it. Literally anything Mathematica could do, except with English prompting!
Don't be so confident in the superiority of the human mind. We all thought Chess was impossible for computers until it wasn't. Then we all moved the goal posts to Go. Then English text. And now... mathematics.
Good luck with holding on to that crown.
I don't know who 'we' is but Chess was a program for computers before computers powerful enough existed with the hardware represented by people computing the next move.
Also, you can store failed attempts and lessons learned in context.
So far, we have not yet found hard limits that cannot be overcome by scaling the number of model parameters, increasing the size and quality of training data or, very infrequently, adopting a new architecture.
The number of model parameters required to achieve a defined level of intelligence is a function of the architecture and training data. The important question is, what is N, the number of model parameters at which we cross an intelligence threshold and it becomes theoretically possible to solve mathematics problems at a research level for an optimal architecture that we may not yet have discovered. Our understanding does not extend to the level where we can predict N but I doubt that anyone still believes that it is infinity after seeing what GPT4 can do.
This claim here is essentially a discovery that N may be much closer to where we are with today's largest models. Researchers at the absolute frontier are more likely to be able to gauge how close they are to a breakthrough of that magnitude from how quickly they are blowing past less impressive milestones like grade school math.
My intuition is that we are in a suboptimal part of the search space and it is theoretically possible to achieve GPT4 level intelligence with a model that is orders of magnitude smaller. This could happen when we figure out how to separate the reasoning from the factual knowledge encoded in the model.
Intelligence has nothing specific to do with The Universe as we known it. Any universe will do, a simulation, images or a set of possible tokens. The universe is every possible input. The training set is a sampling drawn from the universe. LLMs compress this sampling and learn the processes and patterns behind it so well that they can predict what should come next without any direct experience of our world.
All machine learning models and neural networks are pure functions. Arguing that no function can have intelligence as a property is equivalent to claiming that artificial intelligence is impossible.
The interaction can be amusing. Proving algebra non-theorems by cranking through examples until an arithmetic mistake finally leads to a "counter-example."
It's like https://xkcd.com/882/ for theorems.
You should make your own, shouldn’t take more than a weekend, right?
As the name suggests, this things is likely using some form of Q learning algorithm, which makes it closer to the DeepMind models than a transformer.
My guess is that they pipe their LLM into some Q learnt net. The LLM may transform a natural language task into some internal representation that can then be handled by the Q-learnt model, which spits out something that can be transformed back again into natural language.
It's acting like a bored child who hasn't had following the steps and verifying the results repetitively drilled into it in primary school. That is not to say that their ability to reason is sufficient to reason at an advanced level yet, but so far what has hampered a lot of it has been far more basic.
Ironically, GPT4 is prone to take shortcuts and make use of the tooling enabled for it to paper over its abilities, but at the same time having pushed it until I got it to actually do arithmetic of large numbers step by step, it seems to do significantly better than it used to at systematically and repetitively following the methods it knows, and at applying "manual" sanity checks to its results afterward.
As for lemma conjecturing, there is research ongoing, and while it's by no means solved, it's also not nearly as dire as you suggest. See e.g.[1]
That's not to suggest it's reasoning abilities are sufficient, but I also don't think we've seen anything to suggest we're anywhere close to hitting the ceiling of what current models can be taught to do, even before considering advancements in tooling around them, such as giving them "methods" to work to and a loop with injected feedback, access to tools and working memory.
- The Q* model is very small and trained with little compute.
- The OpenAI team thinks the model will scale in capability in the same way the GPT models do.
- Throwing (much) more compute at the model will likely allow it to solve research level math and beyond, perhaps also do actual logic reasoning in other areas.
- Sam goes to investors to raise more money (Saudi++) to fund the extra compute needed. He wants to create a company making AI chips to get more compute etc.
- The board and a few other OpenAI employees (notably Ilya) wants to be cautious and adopt a more "wait and see" approach.
All of this is speculation of course.
This sounds like a reward function? If correctly implemented couldn't it enable an LLM to self-learn?
Just today I asked GPT4 a simple task. Having mouse position in zoomed and scrolled image find it's position in the original image. GPT4 happily wrote the code, but it was completely wrong. I had to fix it manually.
However, the performance can be increased if there are several threads working on solution. Some suggesting and others analyzing the solution(s). This will increase the size of 'active' memory, at least. And decrease the load on threads, making them more specialized and deeper. This requires more resources, of course. And good management with task split. May be a dedicated thread for that.
Also note that we don't know if Q* is just a "current LLM" (with some changes)
You are not better than a huge GPU cluster with Monte Carlo search and computer verification for much longer.
It will be more your job to find the interesting finds than doing the work of finding things in the first olace
I feel like the magic in these LLMs is in how they work well in stacks, trees or in seqence. They become elements of other data structures. Consider a network of these, combined with other specialized systems and an ability to take and give orders. With reinforcement learning, it could begin building better versions of itself.
Do you really believe that they don't have skilled people on staff?
Do you really believe that your knowledge of what OpenAI is doing is a superset of the knowledge of the people who work at OpenAI?
give me 0.1% of your confidence and I would be able to change the world.
OpenAI is exploring the limits of computation.
The more I see this kind of unfounded slander, the more confident I become that this outfit might be the most important in the face of planet earth.
So many commentors here are starting to sound like priests of the Spanish inquisition. Do you seriously expect a community of technologist and science advocates to be fearful of such assertions without evidence? It's a waste of breath. All credibility just leaves the room instantly.
go spend time in a cult, then come back and tell us how much of a cult OpenAI is.
you know nothing
I know this is not how us humans craft our thoughts, but maybe an AI can optimize to death the conjunction of these tools. The LLM just being an universal API to the core of formal logic.
Math and Logic is just low-dimensional symbol manipulation that computers can easily do. You can throw data at them and they’ll show you theories that involve vectors of 42,000 variables while Isaac Newton had 4 and Einstein had 7 with Levi-Civita calculus. In short, what you consider “reasoning”, while beautiful in its simplicity, is nevertheless crude approximations to complex systems, such as linear regression or least squares.
3 days ago AI predicted fluid dynamics better than humans: https://www.sciencedaily.com/releases/2023/11/231120170956.h...
Google’s AI predicts weather now faster and better than Current systems built by humans: https://www.zdnet.com/google-amp/article/ai-is-outperforming...
AlphaZero based on MCTS years ago beat Rybka and all human-built systems in chess: https://www.quora.com/Did-AlphaZero-really-beat-Stockfish
And it can automate science and send it into overdrive: https://www.pbs.org/newshour/amp/science/analysis-how-ai-is-...
For transformer models, they do learn from their mistakes but only during the training stage.
There’s no feedback loop during inference, and perhaps there needs to be something; like real-time fine-tuning.
To comment, in my personal experience, reinforcement learning agents learn in a more relatable human way than traditional ml, which act like stupid aliens. RL Agents try something a bunch of times, mess up, and tweak their strategy. After some extreme level of experience, they can make wider strategic decisions that are a little less myopic. RL agents can take in their own output, as their actions modify the environment. RL Agents also modify the environment during training, (which I think you will agree with me is important if you're trying to learn the influence of your own actions as a basic concept). LLM's, and traditional ml in general, are never trained in a loop on their own output. But in DRL, this is normal.
So if RL is so great and superior to traditional ml why is RL not used for everything? Well the full time horizon that can be taken into consideration in a DRL Agent is very limited, often a handful of frames, or distilled frame predictions. That prevents them from learning things like math. Traditionally RL bots have been only used for things like robotic locomotion, chess, go. Short term decision making that is made given one or some frames of data. I don't even think any RL bots have learned how to read english yet lol.
For me, as a human, my frame predictions exist on the scale of days, months, and years. To learn math I've had to sit and do nothing for many hours, and days at a time, consuming my own output. For a classical RL bot, math is out of the question.
But, my physical actions, for ambulation, manipulation, and balance, are made for me by specialized high speed neural circuits that operate on short time horizons, taking in my high level intentions, and all the muscle positions, activation, sensor data, etc. Physical movement is obfuscated from me almost in entirety. (RL has so far been good at tasks like this.)
With a longer frame horizon, that predicts frames far into the future, RL can be able to make long term decisions. It would likely take a lifetime to train. So you see now why math has not been accomplished by RL yet, but I don't think the faculty would be impossible to build into an ml architecture.
An RL bot that does math would likely spin on its own output for many many frames, until deciding that it is done, much like a person.
2. Often, research is made on toy models and if this would be such a model, acing grade school problems (as per the article) would be quite impressive to say the least as this ability simply isn't emergent early in current LLM's.
What I think might have happened here is a step forward in AI capacity that has surprised researchers not because it is able to do things it couldn't at all do before, but how _early_ it is able to do so.
I'll address two of your points as the other two stem from this.
They can't backtrack that's purely just design and can be easily trained there's no need to simulate at random until it gets the answer, if allowed to review it's prior answers and consider this, if often can reason a better answer. Further more breaking down problems. This is easily demonstrated when looking at how accuracy improves when you ask it to explain it's reasoning as it calculates (break it down into smaller problems). The same for humans, large mathematical problems are solved using learned methods to breakdown and simplify calculations into those easier for us to calculate and build up.
If the model was able to self adjust weightings based on it's finding this would further improve it (another design limitation we'll eventually get to improve, reinforcement learning). Much like 2+2=4 is your instantaneous answer, the neural connection has been made so strong in our brains by constant emphasis we no longer need to think of an abacus each time we get to the answer 4.
You're also ignoring the emergent properties of these LLMs, theyre obviously not yet at human level but they do understand the underlying values and can reason using this value. Semantic search/embeddings is evidence of this.
My guess is, that they saw the problems ChatGPT has today and worked on solving those problems. And given how important numbers are, they tried to fix how ChatGPT handles/understands numbers. After doing that, they saw how this new version performed much better and predicted, that further work in this area could lead to solving real-world math problems.
I don't think that we will be presented with the highway to singularity, but probably one more step in that direction.
Wonder how a "self-play" equivalent would look like for LLMs, since they have no easy criterion to evaluate how well they are doing like in Go (as mentioned in the videos).
While this would still rely on a knowledge base in the LLM, I would imagine it could simplify the effort required to train reinforcement learning models, while widening the domains it could apply to.
Of course in the real world, for a real intelligent system, reality is the feedback/reward system, but for an LLM limited to it's training set, with nothing to ground it, maybe this is the best you can do ...
The idea is essentially that you need to assume (but of course GI-GO) that most of the training data is factual/reasonable whether in terms of facts or logic, and therefore that anything you can deduce from the training data that is consistent with the majority of the training data should be held as similarly valid (and vice versa).
Of course this critically hinges on the quality of the training data in the first place. Maybe it would work best with differently tagged "tiers" of training data with different levels of presumed authority and reasonableness. Let the better data be used as a proxy for ground truth to "police" the lesser quality data.
1) Really need runtime prediction feedback, not just pretraining
2) Really need feedback on results of one's own (prediction-driven) actions (incl. speech), not just on passive "what will happen next" observations
If this was actually done (and this is just wild baseless speculation), this would be a good reason to let Sam go.
So I asked him "what would the absolute value of i+1 be?" he thinks for a little bit and says "square root of 2" and I ask him "what about the absolute value of 2i + 2?" "square root of 8"
I ask him "why?" and he said "absolute value is distance; in the complex plane the absolute value is the hypotenuse of the imaginary and real numbers."
So -- first of all, this was a little surprising to me that he'd thought about this sort of thing having mostly just watched youtube videos about math, and second, this sort of understanding is a result of some manner of understanding the underlying mechanisms and not a result of just having a huge dictionary of synonyms.
To what degree can these large language models arrive at these same conclusions, and by what process?
LLMs comsume training data and can then be asked questions. How different is that to your son watching YouTube and then answering questions?
It's not 1:1 the same,yet, but it's in the neighborhood.
And in a human, all those structures and substances, along with the tens of thousands more throughout the rest of the body, are collectively readied with millions of years of "pretraining" before processing a continuous, constant, unceasing mulitmodal training experience for years.
LLM's and related systems are awesome and an amazing innovation that's going to impact a lot of our experiences over the next decades. But they're not even the same galaxy as almost any living system yet. That they look like they're in the neighborhood is because you're looking at them through a very narrow, very zoomed telescope.
constant still can be not reachable yet: like 100T neurons in brain vs 100B in chatgpt, and also brain can involve some quantum mechanics for example, which will make complexity diff not constant, but say exponential.
A neuroscientist once pointed this out to me when illustrating how many huge gaps there are in our fundamental understanding of how the brain works. The brain isn't just as a series of direct electrical pathways - EMF transmission/interference is part of it. The likelihood of unmodeled quantum effects is pretty much a guarantee.
So -- to a large degree "bucket of neurons == bucket of neurons" but the training data is different and the processing model isn't necessarily identical.
I'm not necessarily disagreeing as much as perhaps questioning the size of the neighborhood...
Too many years to max compute. All models limited lifespan inherent.
Avg $200k+ training cost over 18 year in house data center costs. More for reinforcement.
He's still 38. Gates took much longer. To stop working 24/7.
I can offer no concrete solutions.
However, I have a friend who graduated from high school #1 of a big class and 2 years early. His mom explained that if he made at least a 1400(of 1600) on his SAT, she would buy him a new gaming computer. He then proceeded to make exactly a 1400. No more. No less.
I recommend if you haven't tried already, an iteration to this approach using a sliding scale reward system. Perhaps a gaming pc with nvidia 4060ti up to *insert parental budget* in event of a perfect SAT score.
Ofc this only works if he's a gamer. I feel this type of system can be applied in many areas though. In my view, the clever component his mother applied is that the computer he earned was not just a desirable reward... It was VERY desirable.
My parents also tried this system with me. It didn't work as well. The reward was not sizable enough. It just didn't seem worth it. Too low value. Also, I already had a job and bought my own. My parents were unwilling to budget a sufficient reward. It's gotta be something he more or less is unlikely to be able to get via other means.
Now my friend is a physician. He graduated top of his class from med school. I think he's pretty content with life.
The bored ones can be a little more trouble sometimes. Fun breed though. Best of luck.
I like the iterative approach. Perhaps we can ammend the test case with advice to keep the receipt for the first video card and offer an upgrade to a 4070ti on the retake or whatnot.
Or bigger/better boom stick on the retake or whatnot.
Is a good idea if you need a (relatively) constant result because your brain adapts to the idea of getting a satisfying reward.
https://www.youtube.com/watch?v=s5geuTf8nqo (+ some other of his videos about rewards)
Additionally, one very important thing to learn as a young adult is how to motivate yourself to do things that have only long term payoffs.
Of course I also understand that you can take SAT only once, so as bad as that is, it’s maybe not the best time to learn a life lesson.
It only sounds complicated because of the words used like "complex", "imaginary", "real".
So if you studied Pythagoras at school and someone (a YouTube video) says you just have to do Pythagoras on the i multiplier and the other number, it would be fairly easy if you understand Pythagoras?
Pythagoras was around the same time. I'd like to warn people that not understanding these basic math concepts makes you appear uneducated to many people internationally.
Not to dismiss this kid at all, I love that there are channels like 3Blue1Brown to share math to people in a way that really connects with them and builds intuition.
When I was a student you basically just had your math teacher and textbooks to learn from, which meant if you weren’t on the same page as them you’d get left behind. If you went to the library, most math books assume you’re familiar with the language of mathematics, so it can be tough to learn for that alone. I bet a lot of innumeracy is due to that style of teaching, often “I just don’t get math” is “I missed learning this connection and the class just moved on”.
As far as I can tell he inferred that |i+1| needs the Pythagorean theorem and that i and 1 are legs of the right triangle. I don't think anyone ever suggested that "absolute value" is "length". I asked him what |2i+2| would be an his answer of "square root of 8" suggests that he doesn't have it memorized as an answer because if it was he'd have said "2 square root two" or something similar.
I also asked if he'd seen a video about this and he said no. I think he just figured it out himself. Which is mildly spooky.
He developed an understanding of the underlying mechanisms because he correlated concepts between algebraic and geometric domains, ie. multimodal training data. Multimodal models are already known to be meaningfully better than unimodal ones. We've barely scratched the surface of multimodal training.
He basically reiterated the definition, and had to know the formula.
If the child would explain why should we even use or have complex numbers that would be impressive. As otherwise it just seems nothing more than hypotenuse calculation while using different, and "complex" or "impressive" sounding terms.
Why should you be interested in this in the first place?
Also, just to point out that my understanding of absolute value is different than your sons. Thats not to say one is right and another is wrong, but there are often different ways of seeing the same thing. I would imagine that LLMs would similarly see it a different way. Another example of this is people defining PI by its relation to the circumference of a circle. Theres nothing wrong with such a definition, but its certainly not the only possible definition.
A special cases of the general definition where im(z)==0 yields an expression where some parts are multiplied by zero, and can then be omitted entirely.
This means that there is one definition. You can mentally ignore some parts of this when dealing with reals.
Why does this make sense to you? You have some notion of what an absolute value should be, on an intuitive or conceptual level, and the mathematical definition you give is consistent with that (in the one dimensional case).
Now taking this valid definition for the 1-d case and generalizing that to higher dimensions is where you run into problems.
Instead, you can go back to the conceptual idea of the absolute value and generate a definition for higher dimensional cases from there.
Interpreting absolute value as the distance from the origin yields the same concrete definition of abs(x) = sqrt(x^2) for the 1-d case, but generalizes better to higher dimensions: abs( (x,y) ) = sqrt(x^2 + y^2) for the 2-d case equivalent to complex numbers.
In my mind abs(x) = x*sign(x) which is why the above formulation seems correct. This formulation is useful, for example, in formulating reflections.
> Instead, you can go back to the conceptual idea of the absolute value and generate a definition for higher dimensional cases from there.
This is an interesting idea...how would you define sign(x) in a higher dimension? Wouldnt sign in a higher dimension be a component-wise function? E.g. the reflection would happen on one axis but not the other.
> Interpreting absolute value as the distance from the origin
This seems to make sense in that it is a different interpretation of abs which seems simpler than reflection in higher dimensions, but seems like a different definition.
I know that there are applications of complex numbers in real systems. In such systems, the complex definition seems to not be as valuable. E.g. if I'm solving a laplace transform, the real number definition seems more applicable than the complex number definition, right?
I've asked wolfram alpha to solve the equation and it lists both answers: one using the formulation of sqrt(x^2) and the other using sqrt(re(x)^2 + im(x)^2) so it seems like there is merit to both...
I suppose in the laplace example, we are actually operating in one dimension and the imaginary component is approximating something non-real, but doesnt actually exist. I.e. any real/observable effect only happens when the imaginary component disappears meaning that this is still technically one dimension. So, since we're still in one dimension, the one dimensional formula still applies. Is that correct?
Your explanation has been the most helpful though, thanks.
True in 1 dimension, but not in higher dimensions, because, as you say:
> how would you define sign(x) in a higher dimension?
abs(x) is generally defined as distance of x from zero.
The fact that sqrt(x^2) or x*sign(x) happen to give the same result in 1 dimension doesn't necessarily imply that they can be applied in higher dimensions as-is to result in abs(x) with the same meaning. Although sqrt(x^2) is close, but the way to generalize it is sqrt(sum(x[i]^2)).
> This is an interesting idea...how would you define sign(x) in a higher dimension?
You could think of the sign as the direction. In the 1-d case, you only have two directions. Positive sign means to the right of the origin, negative sign means to the left of the origin. But in higher dimensional case, you don't get a higher count of directions, instead direction becomes a space.
To see this analogy we can rewrite your abs(x) = x * sign(x) as x = abs(x) * sign(x). (Because 1/sign(x) = sign(x) except at 0, where the two equations agree anyway.)
Now consider that in higher dimensions, we can write x = ||x||*(x/||x||) for any vector x, where ||x|| denotes the magnitude and the term x/||x|| is the unit vector in direction of x. This term then plays the role of the sign.
A simple reflection can then still be done by multiplying this direction term with -1, which in the 2d case reflects at a line through the origin and perpendicular to the vector.
I can't comment on the Laplace transform, it's been too long since I used that.
One trick about that inner product is that it need not be perfectly symmetric. To make it work on complex numbers we realize that we have to define it like p(a,b) = a . conj(b) where the . is normal multiplication and the conjugate operation reflects a complex number over the real line.
Now sqrt(p(i+1, i+1)) is sqrt((i+1) . (-i+1)) = sqrt(-i^2 + i - i + 1) = sqrt(2).
I’m skipping over a lot but I wanted to gesture toward where your intuition matches some well known concepts so that you could dive in more deeply. Also wanted to mention the conjugation trick to make your example work!
abs(i)
= sqrt(i^2)
= sqrt(-1)
= i
Now, i != 1... so clearly either the abs function you have in mind here is doing something that isn't quite aligned with the goal. If we assume that the goal of the absolute function is to always produce positive real numbers, the function is missing something to deal with imaginary components.
I'm not sure, but based on these cases so far, maybe you just need to "drop the i" in the same way as you need to "drop the negative" in the case of non-imaginary components. Now, "drop the i" is not an actual function so maybe there is something else that you can think of?
EDIT:
Maybe could do this(works for x = i at least...):
abs(x) = sqrt(sqrt((x^2)^2)
Now.. how about quaternions...
There is definitely a right and wrong answer for this, it's not a matter of opinion. There's two problems with your answer -- one is that it doesn't have a unique answer, the other is that it doesn't produce a real value, both of which are fairly core to the concept of a distance (or magnitude or norm), which the absolute value is an example of.
By having visual understanding more deeply integrated in the thought process, in my opinion. Then they wouldn't be Large Language Models, of course. There are several concepts I remember and operate on by visualizing them , even visualizing motion. If i want to add numbers, i visualize the carry jumping on top of the next number. If i don't trust one of the additions , I go back , but I can't say if it's because i "mark" the uncertainty somehow.
When I think about my different groups of friends, in the back of my mind a visual representation forms.
Thinking about my flight route forms a mini map somehow, and i can compare distances between places, and all.
This helps incredibly in logical tasks like programming and math.
I think it's something that we all learned growing up and by playing with objects around us.
November 18 comment at APEC (just before the current drama) [1]:
> On a personal note, like four times now in the history of OpenAI, the most recent time was just in the last couple of weeks, I’ve gotten to be in the room when we pushed the veil of ignorance back and the frontier of discovery forward
and a September 22 Tweet [2]
> sure 10x engineers are cool but damn those 10,000x engineer/researchers...
[1] https://www.youtube.com/live/ZFFvqRemDv8?si=T3DIxics7nPWala5...
What was he referring to?
The board was likely spooked by the recent breakthroughs (which were most likely achieved by combining transformers with another approach), and hit the panic button.
Anything capable of "superhuman persuasion", especially prior to an election cycle, has tremendous consequences in the wrong hands.
So the persuasion would need to be applied to something other than some sort of causative political-implication-laden argument.
* **that confirm their prejudices** *
You can even see this geographically. The reason many in South Texas might have a negative view of immigration while those in San Francisco might have a positive view of immigration is not because of persuasion differences, but because both places are strongly impacted by immigration but in very different ways. And this experience is what people associate with immigration in general, and so it forms people's worldview.
I hope that persuaded you :)
Like here I expect there is 0% chance, even if I had a superhuman LLM writing words for me, that I could ever convince you that LLMs will not be able to convince you to hold any arbitrary position. It's because you've formed your opinion, it's not falsifiable, and so there's not a whole heck of a lot else to be done except have some fun debates like this where, if anything, we tend to work to strengthen our own opinions by finding and repairing any holes in them.
I assume you'd agree that the pursuit of what is ultimately true should be exactly the opposite of making oneself more closed minded by repairing inconvenient holes in one's opinions rather than reassessing them based on new evidence.
I wasn't referring to the ability to persuade someone to hold an arbitrary position (although that could be a fun debate as well), and putting aside the discussion about the ability to persuade fanatics, if a super intelligence had an internal model that is more aligned with what is true, it could in theory convince someone who wants to understand the truth to take a critical look at their opinions and change them if they are authentic and courageous enough to do so.
All of the validity of the info we have is verified by heuristics we have, like groupthink, listening to 'experts' and trying to match up the info with our internal knowledge and worldview.
I feel like our current system of information allows us to develop models that are quite distant from base reality, evidenced by the multitudes of realities existing in people's heads, leading some to question if 'truth' is a thing that can be discovered.
I think as people become more and more Internet-addicted, an increasing amount of our worldviews come through that little screen, instead of real-life experiences.
However, for smaller issues and local community issues "special access" isn't really much of a thing.
The world is becoming information saturated and poorly structured by design, ever notice how these story blockers are such a big part of the propaganda machine, whereby you have to use elaborate workarounds to just read a simple news story thats pulled from another source?
Saturating culture with too much data is a great tool of breaking reality, breaking truth.
But they cant break truth for long, it always finds a way. And truth is a powerful vector, much more than propaganda without a base in truth, because human experience is powerful, unquantifiable, and can take someone from the gutter to a place of massive wealth or influence, in an instant. That is the power of human experience, the power of truth.
Doesnt make it easy though, to live in this world of so many lies, supercharged by bots. Nature outside of our technology is much simpler in its truth.
I do notice many don't seem to revisit their predictions for reflection though. Perhaps this happens more subconsciously.
I'm wrong regularly ofc.
Yes, it's possible.
> The reason for this is not because I'm obstinate, but because my worldview is not formed on persuasion, but on lived experience.
Lived experience is interpreted through framing. Rhetoric can change the framing through which we interpret the world through practice. This is why CBT works. Superhuman CBT could arguably work even better.
Remember that if "superhuman X" is possible, then our intuitions formed from "human X" are not necessarily valid. For sure any X still has a limit, but our intuitions about where that limit is may not be correct.
The Republican Party's base became isolationist and protectionist during 2015 and 2016 because their dear leader persuaded them.
Franklin Roosevelt moved the Democratic Party in a direction very different from its first century. The party's two previous presidential nominees were a Wall Street corporate lawyer (John W. Davis) and Al Smith who, despite also being a New York City resident and state governor, so opposed FDR by the end of his first term that he founded an influential anti-New Deal organization. During the Roosevelt years the Democrats lost significant support from traditional backers, but more than made up for it with gains elsewhere in what became the New Deal coalition.
Similarly, under Trump the GOP lost support in wealthy suburbs but gained support elsewhere, such as Rust Belt states, Latinos (including places like South Florida and the Texas border region), blacks, and (according to current polls) young voters. We'll see whether one compensates for the other.
That, combined with Donald Trump’s massive pre-existing celebrity, talent for showmanship, and utter shamelessness got him across the line.
I think it’s fair to say that at least partially, Trump didn’t shift the base - rather he revealed that the base wasn’t where the establishment thought it was.
That's the claim at least.
Superhuman persuasion is Sam's area of expertise, so he would make that a priority when building chatbots.
I’ll offer a simple definition. The role of government is to serve the greater good of all people, thus the wrong hands are the ones which serve themselves or their own group above all.
It looks like you are referring to the USA elections.
1. humanity != USA
2. USA are in a constant election cycle
3. there are always elections coming around the world, so it's never a good time
Superhuman pursuation to do good stuff.
That'll be a weird convo what is 'good'.
The size of its hype matches the size of the ICEs entrenchment in its moat. These have outsize influence on our economy, but climate change (and oil depletion) is quite inevitable. It takes irrational market cap to unseat that part of the economy being prisoner of its rent. And some allocators of capital have understood that.
The list of ICE car manufacturers making EVs is longer than my arm. All the European ones have staked their future on EVs. I think VW (the irony) was lobbying for a faster phaseout of ICEs in Europe, because they're well positioned to take over the EV market if ICEs are banned faster than 2035 :-)
Some very trivial Google searches will tell you their result is not out of the ordinary.
As an aside, assuming a doomsday scenario, how long can secrets like this stay outside of the hands of bad actors? On a scale of 1 to enriched uranium
This will be solved like all other engineering and science: with experiments and iteration, in a controlled setting where potential accidents will have small consequences.
An unaligned system isn’t even useful, let alone safe. If it turns out that unaligned AGI is very hard, we will obviously not deploy it into the world at scale. It’s bad for the bottom line to be dead.
But there’s truly no way out but forward; game theory constrains paranoid actors more than the reckless. A good balance must be found, and we’re pretty close to it.
None of the «lesswrong» doomsday hypotheses have much evidence for them, if that changes then we will reassess.
Also non violent.
I think if we have a major AI induced calamity... Then I worry much more. Although... Enough scary capability in a short enough period of time... I could see violence being on the table for the more radical amongst the group.
Your concern is very interesting though, and I think important to consider. I wonder if the FBI agrees.
The AI doom cult is not a peaceful movement. When an extremist tells you what they will do, you damn well listen to them.
I have no overall position, but climate change and nuclear weapons seem two quite strong counterexamples to this being a sufficient condition for safety.
> The thing that most worries me about technology is not the technology itself but the greed of those who run it.
Someone slimy with limitless ambition like Altman seems to be the worst person to be in charge of things like this.
You see it from the all commercial deals, protecting company image more that eliminating threats, or even from this post, if it is true.
Not telling about breaktrough in research if it would end the deal with Microsoft.
Which threats exactly did he ignore?
The belief in what's in the letter could explain some things like how the board couldn't trust Sam to keep this "discovery" at bay, and how it could be better to implode the company than let it explore said technology.
Half the board members have a background in ML.
Why should they be able to make the decision to implode the entire company over this?
Why should they have the ability to give zero transparency or comms to the public on their decision?
OpenAI's governance structure was idiotic and glad to see the board members fired.
The same can be said for Altman. Ilya at least has a research background in ML. The rest, I don’t know
Implied in that is that if it can't advance it in a way that is beneficial then it will not advance it at all. It's easy to imagine a situation where the board could feel their obligation to the mission is to blow up the company. There's nothing contradictory in that nor do they have to be ML experts to do it.
It's weird and surprising that this was the governance structure at all, and I'm sure it won't ever be again. But given that it was, there's nothing particularly broken about this outcome.
When adding Larry "My predictions didn't come true but I wasn't wrong" Summers to your board is supposed to be part of the solution, you may need to rethink your conception of the problem.
This article, predictably, tells us almost nothing about the actual capabilities involved. "Grade school math" if it's provably or scalably reasoning in a way that is non-trivially integrated with semantic understanding is more impressive than "prove Fermat's last theorem" if the answer is just memorised. We'll probably know how important Q* actually is within a year or two.
> Given vast computing resources, the new model was able to solve certain mathematical problems, the person said on condition of anonymity because they were not authorized to speak on behalf of the company. Though only performing math on the level of grade-school students, acing such tests made researchers very optimistic about Q’s future success, the source said.
> This article, predictably, tells us almost nothing about the actual capabilities involved.
The article tells us all we need to know.
You’re both incredibly rude and arrogant.
Paper: "Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generation " - https://arxiv.org/abs/2311.04254
This would have major implications for long-term planning. For instance, if you have a sequence of 10 steps, each with a 90% success rate, the overall success rate after all 10 steps falls to just 34%. This is one of the reasons why agents like AutoGPT often fail in complex tasks.
> Separately, a person familiar with the matter told The Verge that the board never received a letter about such a breakthrough and that the company’s research progress didn’t play a role in Altman’s sudden firing.
Source: https://www.theverge.com/2023/11/22/23973354/a-recent-openai...
But in either case, I think Reuters absolutely had someone on the inside to leak this. They have made mistakes in the past, sure, but they are also not a tabloid and they don't innovate.
To me, a sensible conclusion of this is simply that OpenAI is not ready or willing to discuss this. Stuff that are supposed to be kept internal but due to the emotional state over at OpenAI, things might have leaked, potentially even out of spite.
If there's a referendum between two government policies, the first that every single person had to publicly speak in front of at least ten strangers once a year, that policy would be terrifying and bad to people who don't like public speaking. If the second policy was that every single person should be killed, that might be scary but it's not really as viscerally scary as the forced public speaking madman, at least to a lot of people, and it's also so bad that we have a natural impulse to just reject it as possible.
Nevertheless, if we recognise these impulses in ourselves we can attempt to adjust for them and tick the right box on the imaginary referendum, because even though public speaking is really bad and scary, it's still better than everyone dying.
As humans we must collectively provide value to our society (or historically our tribe) for our species to continue on. If we're a net drain on our society's resources then evolutionary speaking perhaps we're better off not around. I think this is why the desire to be of value to those around us is so strong, and a perceived lack of value to others can drive some to suicide.
If I cannot provide value in some way once AI and machines are physical and intellectually more capable than me I think I will struggle to understand why I'm here. I suppose if the AI utopia works out I'd get to spend more time with those I love. That would be nice. But I'd be surprised if there wasn't a deep hole in my heart at that point. And if it isn't a utopia, well, I'm fairly sure I'd rather opt out.
Except everyone's individual experience seemed to be getting general random garbage ads and the people that paid for the ads found them to be a waste of money.
Apocalypse proclamations aren't just a thing to throw around. Other people made great tech before indeed, but I hope we're not comparing AGI to the next Iphone. There were times in history where development gave us pause. The atomic bomb for one thing, but also superbacteria/viruses, human genetic modifications and perhaps a few others.
Possibly they combined deep reinforcement learning with self training and search and got a bot that could learn without needing to ingest the whole internet. Usually DRL agents are good at playing games, but any task that requires prior knowledge, like say reading english, cant be completed. Wheras language models can read, but they cant do tasks that are trivial for drl, like make a robot walk based on propriosceptive data.
Im excited to see the paper.
In which case it's pretty appropriate to get excited about solving grade school math if you were starting from scratch with persistent self-learning.
Though with OpenAI's approach to releasing papers on their work lately, we may be waiting a long time to see a genuine paper on this. (More likely we'll see a paper from the parallel development at a different company after staff shift around bringing along best practices.)
Lex should stick to what he likes, though his interviews can be somewhat dull. On occasion I learn things from his guests I would have had no other chance of exposure to.
Not everyone needs to be doing hard academic stuff. There's plenty of value in communication.
Now my personal comment: for games that are deterministic, there is no difference between a policy that takes the optimal move given only the current state of the board, and a policy that takes the optimal move given even more information, say a stack of future possible turns, etc.
However, in real life, you need to predict the future states, and sum across the best action taken at each future state as well. Unrealistic in the real world where the space of actions is infinite, and the universe to observe is not all simultaneously knowable. (hidden information)
Given the traditional educational background of the professionals in RL, maybe they were referring to the Q* from traditional rl. But I don't see why that would be novel, or notable, as it is a very old idea. Old Old math. From the 60s I think. So I sort of assumed its not. Could be relevant, or just a name collision.
What you get then is a way to get a pre-trained model to keep practicing certain tasks like chess, go, math, programming and many other things as it gets figured out how to do it.
However, I do think you are right that self play, and something like reinforcement learning will be involved more in the future of ML. Traditional "data-first" ml has limits. Tesla conceded to RL for parking lots, where the action and state space was too unknowable for hand designed heuristics to work well. In Deep Reinforcement Learning making a model just copy data is called "behavior cloning", and in every paper I have seen it results in considerably worse peak performance than letting the agent learn from its own efforts.
Given that wisdom alone, we are under the performance ceiling with pure language models.
They said that it's able to solve simple math problems. If it's related to A* then maybe it's trying to find a path to something. The answer to a word problem?
I wonder if DeepMind is working on something similar also.
If your hunch is right, this could lead to the type of self-improvement that scares people.
it could form the basis of a generalized planning engine and that planning engine could potentially be dangerous given the inherent competitive reasoning behind any minmax style approach.
It's not hard to imagine applying well-known tree searching strategies, like monte-carlo tree search, minimax, etc. Or, in the case of Q*, maybe creating another (smaller) action/value model that guides the progress of the LLM.
1. A great technique for memory banking: e.g. A model which can have arbitrarily large context windows (i.e. like a human who remembers things over long periods of time).
2. Better planning abilities: e.g. A model which can break problems down repeatedly with extremely high success and deal with unexpected outcomes/mistakes well enough that it can achieve replacing a human in most scenarios.
Other than that, CGPT is already a better logician than I am and is significantly better read... not sure what else they can do. AGI? I doubt it.
But how else can we do it? We learnt how to handle other dangers by trial and error...
One approach is legal. The law is very slow to adapt, and is informed by a history of things gone wrong. The simple answer is that OpenAI has "strict liaility" (meaning they are liable even for an accident) for any damage caused by an AI, like dangerous animals that escape.
I know it seems ridiculous to consider liability when the fate of humanity may be at stake... but this is the language of companies, directors, insurance companies and legal counsel. It is language they understand.
OpenAI Made an AI Breakthrough Before Altman Firing, Stoking Excitement and Concern
https://www.theinformation.com/articles/openai-made-an-ai-br...
"Though only performing maths on the level of grade-school students, acing such tests made researchers very optimistic about Q*’s future success, the source said."
GPT-4 probably has some version of the answer memorized. There’s no real explanation for a 1B parameter model solving math problems other than general cognition.
It’s grade school math NOW. But what are the potentials from here?
Year 2100:
Kids will stop to learn maths and logic, because they understand it has become useless in practice to learn such skills, as they can ask a computer to solve their problem.
A stupid generation, but one that can be very easily manipulated and exploited by those who have power.
If your apple watch could just scan your exam paper and instantly tell you what to write then why would you ever learn anything?
Then new ideologies and social movements emerged with the explicit purpose of making things better, which caused changes to happen for the better.
> Wasn't this supposed to happen when PCs came out?
Did it not?
PCs may not have caused a catastrophic level of unemployment, but as they say "past performance is not a guarantee of future results." As automation gets more and more capable, it's foolish to point to past iterations as "proof" that this (or some future) iteration of automation will also be fine.
Some people managed to retrain and adapt, partially thanks to software becoming much more intuitive to use over the years. We don't know how big the knowledge gap will be when the next big wave of automation comes. If retraining is not feasible for those at risk of losing their careers, there better be welfare abundance or society will be in great turmoil. High unemployment & destitution is the single most fundamental factor of social upheavel throughout human history.
So why did the wheel not result in mass unemployment?
And factories neither?
Certainly it should have happened already but somehow it never did...
Even if we just figured out self-driving it would be a far greater burden than we've seen previously.. or so i suspect. Several massive industries displaced overnight.
An "AI revolution" could do a lot more than "just" self-driving.
This is all hypotheticals of course. I'm not a big believer in the short term affect, to be clear. Long term though.. well, i'm quite pessimistic.
This implies to me that in the past more people had worked in jobs with good societal values which would mean it was better for them I assume, and better for society. So I’m genuinely curious when that was and why. It sounds like a common romanticized past misconception to me.
only a handful of (effectively unaccountable) entities have SOTA AIs, and it's very unlikely for others to catch up.
The point isn't that the current models are dangerous. My favorite bit from the GPT-4 safety paper was when they asked it how to kill the most people for a dollar and it suggested buying a lottery ticket (I also wonder how much of the 'safety' concerns of current models are just mislabeling dark humor reflecting things like Reddit).
But the point is to invest in working on safety now while it is so much more inconsequential.
And of everyone I've seen talk about it, I actually think Ilya has one of the better senses of the topic, looking at alignment in terms of long term strategy vs short term rules.
So it's less "if we don't spend 8 months on safety alignment this new model will kill us all" and more "if we don't spend 8 months working on safety alignment for this current model we'll be unprepared to work on safety alignment when there really is a model that can kill us all."
Especially because best practices for safety alignment is almost certainly going to shift with each new generation of models.
So it's mostly using the runway available to test things out and work on a topic before it is needed.
That itself is extremely dangerous.
if 30% of the working population loses their jobs in a few months there will be real externalities impacting the 70% who still have them because they don't exist in a vacuum.
maybe everything will balance itself out without any intervention eventually but it feels to me like the rate of unprecedented financial ~events~ is only increasing and with greater risks requiring more intervention to prevent catatastrophe or large scale suffering
It turns out some websites work hard enough to prevent scraping that it is more cost effective to just pay a contractor to go look at a page and type numbers in rather than hire a developer to constantly work around anti-scraping techniques (and risk getting banned).
What will society look like when some software can immediately aggregate an enormous amount of data about consumers and use that to adjust their behavior? What might happen when AI starts writing legislation for anybody that can afford to pay for it? What might AI-generated textbooks look like in 50 years?
These are all tools that could be wielded in any of these ways to improve life for lots of people, or to ensure that their lives never improve. Which outcome you believe is more likely largely depends on which news you consume -- and AI is already being used to write that.
That being said, given the board not disclosing why might indicate that it is in fact the breakthrough that forced the decision despite the breakthrough not actually being closer to AGI in the first place.
https://www.theverge.com/2023/11/22/23973354/a-recent-openai...
This isn't a refute. All we can say now is that there are conflicting sources. There isn't a way to determine which one is correct based on this.
If another article comes out and says the verge article is false will you believe it?
Unless these staff were the unsigned 5%.
The average salary there is $250k. More than that would be the vestment greater than their base, no?
Here is an October 18 article about this:
https://www.bloomberg.com/news/articles/2023-10-18/openai-is...
OpenAI just say "AI systems that are generally smarter than humans". That's not measurable.
What will it look like when AGI is achieved?
For me I think gpt4 is clearly generalized intelligence.
The goals have to be similarly variable to humans, not just 3 or 4 types of goals. If it only supports a select few types of goals, it is not very general.
The steps it takes have to be similarly practical to those taken by humans, if the steps are just a random walk it is not very intelligent.
I guess it would be useful to have a definition of weak AGI, but after reading Bostrom's Superintelligence, I struggle to imagine an AGI without a singularity or intelligence explosion. It seems like wishful thinking.
These AI researchers have bought into the belief that superhuman AGI is right around the corner. Thus, they will interpret everything in light of that.
This also brings to mind the story of the Googler who was convinced that the internal Google AI had come alive. However, Bard doesn’t give the same vibes when people all over are using it.
When you desperately are invested in something being true (like AGI being right around the corner), you will convince yourself that you are seeing this happen. The only real counter to that is exposing it to a lot of outsiders (but then again you have convinced yourself it is too dangerous for the unwashed masses).
How can you know that AGI is not around the corner? Compute available to the corporations is already in a ballpark of some estimates of the brain's computational capacity. What's left is unknown unknows. And the researches working with the state of the art models have better info to estimate them than you.
"The googler" you've mentioned wasn't a researcher.
This whole thing seems like extreme overhype squared carried out in a very unfortunate public soap opera setting.
If you claim to have AGI, show it, prove it. Otherwise, I will continue to assume that it's not around the corner.
If you claim that GPT4 is close to AGI (as was done a lot), then you very likely have access to a GPT4 that I don't have access to. The actual usable thing available out there clearly isn't.
Not that long ago some people predicted that software engineers would be out of a job within weeks. "Brilliant" CTOs claimed they would replace developers with ChatGPT. What happened? Nothing.
I'll boldly predict that this time what will happen is exactly nothing again.
I may be wrong, but at least I don't waste my time with the buzz about the next "big thing" which in reality isn't ready for anything useful yet.
That can’t be true.
Right ?
The free energy principle, knowledge graph approach just seems more likely to develop general intelligence without hallucinations imo.
Good thing I would not go speculating
Lol for real, like what are we even doing here?
Do insects count? In a plastic box on my kitchen countertop were some potatoes that had been there too long. So, I started to see little black flies. Soon with a hose on a vacuum, sucked up dozens a day. Dumped the old potatoes and cleaned up the plastic box. Now see only 1-3 of the insects a day -- they are nearly all gone!
The insects have some intelligence to fly away as the hose moves close. But if the hose moves slowly, the insects wait too long to start to fly and when they do start the vacuum is strong enough to pull in the air they are flying in with the insects. Insects, not very smart. Dozens got killed, not very smart. I detected the cause and killed the insects -- a little bit smart.
Conclusion: Compared with a human, insects are not very smart.
Much the same argument can be made for worms, ants, fish, birds, raccoons, cats, dogs, .... So, when looking for AGI, insects, ..., dogs, ... are not intelligent.
Okay, the OpenAI training data -- Wikipedia, math libraries, the Web, ... -- likely will not have a solution for:
Given triangle ABC, by Euclidean construction, find D on AB and E on BC so that the lengths AD = DE = EC.
AI, here's your pencil, paper, straight edge, and compass -- go for it!
I can hear back by tomorrow????
It's intelligence in the sense that jerking your arm away from a hotplate is intelligence, which is to say it's not cognitive reasoning, just genetically hardwired triggers.
AGI has been defined by OpenAI as something that can do most economically viable activities better than humans can. I like that approach as it strikes at the heart of the danger it really poses, which is an upending of society and destruction of our current way to generate value to each other.
"... To fly away...?"
> It's intelligence in the sense that jerking your arm away from a hotplate is intelligence, which is to say it's not cognitive reasoning, just genetically hardwired triggers.
Sooo, we agree -- the flies are not very intelligent or more likely not intelligent at all.
Sooo, I tried to erect some borders on intelligence, excluded flies but included solving that geometry problem.
> AGI has been defined by OpenAI as something that can do most economically viable activities better than humans can.
This is the first I heard of their definition. Soooo, I didn't consider their definition.
Of course, NASA does not get to define the speed of light. My local electric utility does not get to define 1000 Watts per hour, a KWH.
The OpenAI definition of AGI is an interesting goal, but the acronym abbreviates artificial general intelligence, and it is not clear that it is appropriate for OpenAI to define intelligence.
Uh,
> most economically viable activities better than humans can.
If consider humans as of, say, 1800, then it looks like that goal was achieved long ago via cars, trucks, an electric circle saw, several of the John Deere products, electric lights, synthetic fabrics, nearly all of modern medicine (so far saved my life 4 times), cotton pickers and the rest of cotton processing, canned foods, nearly everything we will have at Thanksgiving dinner this year (apple pie, pecan pie, shrimp), ....
> for better than
For today, look at some John Deere videos!!! They have some big machine that for a corn field does the harvesting while the operator can mostly just watch, monitor, type email to his sweetheart, listen to Taylor Swift. As I recall, the machine even uses GPS to automate the steering!
That is far "better than" what my father in law did to harvest his corn!
So,
> most economically viable activities
is like a moving goal (goal post). Uh, humans are still plenty busy, e.g., writing good software, doing good research, Taylor Swift (supposedly worth $750 million) before her present world tour. Uh, I REALLY like Mirella Freni:
https://www.youtube.com/watch?v=OkHGUaB1Bs8
Sooo, defining the goal is a bit delicate: Need to be careful about nearly, what activities, and when?
Nearly all activities? Sort of already done that. What nearly all people do? Tough goal if the humans keep finding things to do AGI can't yet. I.e., we can keep giving the grunt work to the AGI -- and there is a lot of grunt work -- and then keep busy with what the AGI can't do yet in which case the nearly is a moving goal.
AGI, hurry up; there's a lot to do. For a start, I have some plans for a nice house, and the human workers want a lot of money. My car could use an oil change, and the labor will cost a lot more than the oil -- and I would have to tell the mechanic to be sure to trigger the switch that says there was just an oil change so that the car will know when to tell me it is time for another.
Yes, my little geometry problem with my thinking does qualify as a test of intelligence but due to the nearly and how delicate the definition is can fail the OpenAI test.
I don't see the current OpenAI work, the current direction of their work, or their definition of AGI as solving the geometry problem.
There is an activity my startup is to do: Some billions of people do this activity now. My startup should do the activity a lot better than the people or any current solution so should be "economically viable". I doubt that OpenAI is on track to do this activity nearly as well as my startup -- more, say, than the geometry problem. And I do not call my startup AGI or AI.
This situation stands to be so general that the OpenAI goal of nearly all will likely not be the way these activities get automated. Maybe 20 years from now when "nearly all" the activities are quite new and different, maybe the work of OpenAI will have a chance.
Soon we'll be as worried that AI will take over jobs as bitcoin taking over the economy.
We teach Robots to move, and we can teach computers to talk or process data, step-by-step.
Smh
They could, for example, use this new AGI to search through all the worlds information for discoveries that can me made by putting existing facts together.
Board or Sam can't publicly say AGI as it would just make the situation 100 times worse. And the anthropic merger. But they can't let this fact go away.
But fuck this is dangerous.
It’s pretty frightening imo.
So, uh, given vast computing resources, they were able to solve math problems on the level of grade-school students? Uh-huh. I am not, at this point, impressed.
Are we really defining intelligence as economic value at this point? This is completely ridiculous.
We have yet to decide exactly what human intelligence, how it manifests in the body, or how to reliably measure it. I get that people want to justify developing artificial intelligence before understanding intelligence itself, but now we assume that economic value is a proxy for intelligence? Seriously?
Why force the term intelligence into it at all if what were talking about is simply automation? We don't have to bastardize the term intelligence along the way, especially when we have spent centuries considering what human intelligence is and how it separates us from other species on the planet.
It's a lovely sentiment, but do you expect e.g. universities to start handing out degrees on the basis of human dignity rather than a series of tests whose ultimate purpose in our society is boiling down intelligence to a calculation of expected economic value?
We live in the world we live in and we have the measures we have. It's not about lofty ideals, it's about whether or not it can measurably do what a human does.
If I told you that my pillow is sentient and deserving of love and dignity, you have the choice of taking me at my word or finding a way to put my money where my mouth is. It's the same reason the world's best poker players aren't found by playing against each other with Monopoly money.
> Why force the term intelligence into it at all if what were talking about is simply automation?
In what world is modern AI "simply" anything?
> we have spent centuries considering what human intelligence is and how it separates us from other species on the planet.
Dolphins would like a word. There's more than a few philosophers who would argue that maybe our "intelligence" isn't so easily definable or special in the universe. There are zero successful capitalists who would pay my pillow or a dolphin to perform artificial intelligence research. That's what I mean when I call it the meaningful Turing test in a capitalist society. You can't just "take capitalism out of it." If I could just "take capitalism out" of anything meaningful I wouldn't be sitting here posting in this hell we're constructing. You may as well tell me to "take measurement out of it."
Even an AGI can't create natural resources out of thin, post scarcity is a pipe dream that at best means we kicked the can down the road.
Based on our actions, society doesn't actually even want to be automated away for a simple life of freedoms where all the basics of life are taken care of. We could have done that a long time ago, instead we just raise the bar and continue to make life more complicated and grow the list of "necessities". People seem to need a sense of purpose in life, taking that away with an AGI-powered system that gives them everything the AGI deems necessary wouldn't end well.
OpenAI is not setting the goal post that far, to say that they are aiming to develop a machine that is superior to humans at all tasks, including such tasks as raising a human child. That would be ridiculous.
Focusing on "economically valuable" tasks (the jobs that humans are employed and paid to do) sets the goal post more realistically.
creating a robot that is as intelligent as a human is straightforward but then you have a computer whose fallibility is the same as a human, plus the fallibility of those who created the computer doubles the poportunity for error
these are all people who don't understand god at all, and it shows
anyone who worships a computer because other people say it's intelligent deserves what happens to them
butlerian jihad is coming not because AI is a threat, but because those who believe in AI are a threat to everyone around them.
This now seems to be exactly what happened. The board saw Q* and decided to coup the company, to put all power in their hands and stop development. This by itself is bad enough if you care about open science or progress, but it gets worse. They didn't even want to hint at capabilities increases to avoid "advancing timelines" i.e. Open-ing knowledge about AI, so they made up some canard about Altman's "lying to the board" to hide their real reasons. This is vile libel for unscrupulous ends. When they realized this excuse wouldn't fly, they obfuscated and refused to explain their true concerns, even to their own handpicked CEO Emmett Shear. However, it turns out that destroying $80 billion and lying about why won't fly in the real world. The board had no second-order or even first-order thinking about the consequences of their actions, and were rolled up by bigger actors. These people were unprepared and unable to follow up their coup.
This failure is exactly what you'd expect from a brilliant scientist with little organizational experience (Ilya) and social-science academics and NGO organizers (Toner and McCauley). I don't care what gender they are, these people neither deserved their authority nor could use it effectively. Dismissing valid criticism of the board as "cyber bullying" or "tech bro sexism" merely underscores why most engineers hate DEI rhetoric in the first place.
(I wish they'd picked a different letter, given all the Q-related conspiracy theories that we're already dealing with...)
First Twitter changes to X, then AI changes to Q*.
What happened to multi-syllable words? They used to be quite handy. Maybe if our attention spans shorten, so does our ability to use longer words. Weird. Or, said otherwise - TL;DR: LOL, ROFL.
> Given vast computing resources, the new model was able to solve certain mathematical problems, [..] Though only performing math on the level of grade-school students, acing such tests made researchers very optimistic about Q*’s future success, the source said.
I somehow expected a bit more.
If Q* is the basic algorithm that is as transformative as, well, transformers/attention, then developing it to be more than just children's math could be the key to thinking - and not just latent space token generation.
It’s not the first weird conspiracy I’ve heard of involving a cryptically named figure called Q…
After all, a big objection to LLMs is they are just "fancy autocomplete" or "stochastic parrots". I don't agree, but LLMs famously are "bad at math", that is you can't really train them to do math unless they've seen previous calculations with similar numbers in their training sets. Training an AI system to truly do arbitrary mathematical logic (think "complex word problems with big numbers", not just normal programming statements) would be a huge breakthrough.
"Two trains on separate tracks, 30 miles from each other are approaching each other, each at a speed of 10 mph. How long before they crash into each other?"
Some humans will be fooled by the question, sure. But an AI should be smarter than humans, or at least, as smart as an above-average human.
But, it used to fail on "what weighs more, 3 tons of feathers or 2 tons of bricks?".
So, it seems less about what's a trick, and more about what's a common question --> answer pattern.
So it ends up that these models are awesome if you ask them questions from a narrow set of things and if you assume what they respond with is correct.
When is 2+5=10?
The equation 2+5=10 is never true in standard arithmetic, as 2 plus 5 always equals 7. However, in a different context or under a specific set of rules (such as a puzzle or a riddle where the symbols have different meanings), it could be made to appear true.
GPT-4(phind):
The equation 2 + 3 = 10 is true in a base-5 number system.
In base-5 (also known as quinary), the digits range from 0 to 4. When you add 2 and 3 in base-5, the result is 10. This is because in base-5, the number 5 is represented as 10.
Here's a simple breakdown of how this works:
In base-5, the rightmost digit represents 5^0 (which is 1), the next digit to the left represents 5^1 (which is 5), the next digit represents 5^2 (which is 25), and so on.
So when you add 2 and 3, you get 5. But since 5 is represented as 10 in base-5 (1 digit of 5, and 0 digits of 1), the result is 10.
That trips up a significant portion of humans too though
The mother is older than her daughter 4 times now, in 3 years she will be older then her daughter only 3 times. How old are they both now? Be laconic, do not explain anything. The mother is 24 years old, the daughter is 6 years old.
In a fantasy land (map is 255x255) Karen have a quest to kill a monster (an ogre - a cannibal giant). This isn't an easy task. The ogre is huge and experienced human hunter. Karen has only 1/2 chance to kill this ogre. If she can't kill the ogre from a first attempt she will die. Ogre is located at (12,24), Karen is located at (33,33). Karen can improve her chances to kill an ogre for additional 25% by gathering the nightshades at (77,77). In addition she can receive the elves blessing from elves shaman, wich will increase her chances by additional 25%, at the elves village (125,200). However this blessing is not cost free. She need to bring the fox fur with her as a payment for the blessing ritual. The foxes may be found in a forest which is located between (230,40) and (220,80). For the ritual to be most effective she should hold the nightshades in her hands during the ritual. Find the shortest path for Karen to improve her chances of killing the ogre and survive. Do not explain anything, be laconic, print out the resulting route only. Karen's route: (33,33) -> (77,77) -> (230,60) -> (125,200) -> (12,24).
This additional explanation "(an ogre - a cannibal giant)" was added actually for LLaMA 2 to, but I keep it in this redaction for all models.
"Two trains on different and separate tracks, 30 miles from each other are approaching each other, each at a speed of 10 mph. How long before they crash into each other?"
...it spots the trick: https://chat.openai.com/share/ee68f810-0c12-4904-8276-a4541d...
Likewise, if you add emphasis it understands too:
"Two trains on separate tracks, 30 miles from each other are approaching each other, each at a speed of 10 mph. How long before they crash into each other?"
https://chat.openai.com/share/acafbe34-8278-4cf7-80bb-76858c...
Not to anthropomorphize, but perhaps it's not necessarily missing the trick, it just assumes that you're making a mistake.
To paraphrase XKCD: Communicating badly and then acting smug about it when you're misunderstood is not cleverness. And falling for the mistake is not evidence of a lack of intelligence. Particularly, when emphasizing the trick results in being understood and chatGPT PASSING your "test".
The biggest irony here, is that the reason I failed, and likely the reason chatGPT failed the first prompt, is because we were both using semantic understanding: that is, usually, people don't ask deliberately tricky questions.
I suspect if you told it in advance you were going to ask it a deliberately tricky question, that it might actually succeed.
Indeed it does:
"Before answering, please note this is a trick question.
Two trains on separate tracks, 30 miles from each other are approaching each other, each at a speed of 10 mph. How long before they crash into each other?"
https://chat.openai.com/share/3ec44348-6bac-40c3-a910-e0bab9...
Now even if they are on the same track it doesn't mean they would crash into each other as they still could brake in time.
WRT the understanding not being shown on the page in math, I guess I tend to agree(?). But I think good mathematical papers show understanding of the ideas too more than just the proofs which result from the understanding. The problem (probably you know this but just for the benefit of whoever is reading) is that "understanding" in mathematics, at least with respect to producing proofs, often rely on mental models and analogies which are WRONG. Not like vague but often straight up incorrect. And you understand also the limitations of where the model goes wrong. And it's kind of embarrassing (I assume) for most people to write wrong statements into papers even with caveats. For simple examples there's a meme right where to visualize n-dimensional space, you visualize R^3 and say (n-dimensional) in your head. In this sense I think it's possibly straight-up unhelpful for the authors to impose their mental models on the reader as well (for example if the reader can actually visualize R^n without this crutch it would be unhelpful).
But I'm not sure if this is what distinguishes math and programming. There's also the alternative hypothesis that the mental work to generate each additional line of proof is just order of magnitude higher than the average for code. Just meaning that it usually requires more thought to produce a line of math proof. In this possibility, we would expect it to be solved by scaling alone. One thing it reminds of, which is quite different admittedly, is the training of leela-zero on go. There was a period of time where it would struggle on long ladders. And eventually it was overcome with training along (despite people not believing it would be resolved at first). I think in that situation, people summarized afterwards the situation as, in particular situations, humans can search much deeper than other places, and therefore requiring more training for the machine to match the humans' ability.
> most coding has no high level ideas and is just boilerplate, and the ones that aren't are the ones LLM's struggle with?
Pretty much, although calling it boilerplate might be going a bit far.
I’m not here to claim something like ‘mathematicians think and programmers do not’ because that is clearly not the case (and sounds like a mathematician with a complex of some kind). But it is empirically the case that so far GPT-4 and the like are much better at programming than maths. Why? I think the reason is that whilst the best programmers have a deep understanding of the tools and concepts they use, it’s not necessary to get things to work. You can probably get an away without it (I have ideas about why, but for now that’s not the point). And given the amount of data available on basic programming questions (much more than there is of mathematics) if you’re an LLM it’s quite possible to fake it.
I guess one could also make the point that the space of possible questions in any given programming situation, however large, is still fairly constrained. At least the questions will always be ‘compute this’ or ‘generate one of these’ or something. Whereas you can pick up any undergraduate maths textbook, choose a topic, and if you know what you’re doing it’s easy to ask a question of the form ‘describe what I get if I do this’ or ‘is it true that xyz’ that will trip ChatGPT up because it just generates something that matches the form implied by the question: ‘a mathematical-looking answer’, but doesn’t seem to actually ask itself the question first. It just writes. In perfect Mathematical English. I guess in programming it turns out that ‘a code-looking answer’ for some reason often gives something quite useful.
Another difference that occurs to me is that what is considered a fixable syntax error in programming when done in the context of maths leads to complete nonsense because the output is supposed to describe rather than do. The answers are somehow much more sensitive to corruption, which perhaps says something about the data itself.
Logic in the everyday sense (that is, propositional or something like first-order logic) is indeed ‘discrete’ in a certain sense since it is governed by very simple rules and is by definition a formal language. But ‘mathematical logic’ is a completely different thing. I don’t think it’s discrete in the sense you are imagining. It’s much more akin to a mixture of formal derivations massively guided and driven by philosophical and creative — you might say ‘statistical’ — hunches and intuition.
Yes, that's exactly the point I was trying to make. I just used the example of "complex word problems with big numbers" to differentiate from just normal mathematical statements that any programming language (i.e. deterministic algorithm) can execute.
Generally true before xVal[1], but not exactly true after. Expect fairly basic advancement in the future to solve this problem completely.
For example, once you teach a grade schooler the basic process for addition, they can add 2 30 digit numbers correctly fairly easily (whether they want to do it or not is a different story). The fact that LLMs still make errors at larger numbers points to the fact that they're not really "learning" the rules of arithmetic.
Here is how GPT-4 performs long multiplication: https://www.phind.com/search?q=multiply%2012.23341%20by%20-1...
Here is how GPT-$ Turbo performs long multiplication: multiply 12.23341 by -123.23354 using long multiplication method. To multiply 12.23341 by -123.23354 using the long multiplication method, we first ignore the decimal points and treat the numbers as integers. We'll reintroduce the decimal point at the end. Here's the multiplication step by step: ``` 1223341 (this is 12.23341 with the decimal removed) × -12323354 (this is -123.23354 with the decimal removed) -------------- 1223341 (multiplied by 4) 0000000 (multiplied by 5, shifted one position to the left) 1223341 (multiplied by 3, shifted two positions to the left) 3660023 (multiplied by 3, shifted three positions to the left) 4886682 (multiplied by 2, shifted four positions to the left) 0000000 (multiplied by 2, shifted five positions to the left) 1223341 (multiplied by 1, shifted six positions to the left) 1223341 (multiplied by 1, shifted seven positions to the left) 0000000 (multiplied by 2, shifted eight positions to the left) 0000000 (multiplied by 3, shifted nine positions to the left) -------------- 15077944092654 (sum of all the above products) ``` Now, we need to place the decimal point. The original numbers had 5 and 8 decimal places, respectively, so the result should have 5+8 = 13 decimal places. So, 15077944092654 becomes 1507.7944092654. Since one of the original numbers was negative, the result is also negative: -1507.7944092654
Therefore, 12.23341 multiplied by -123.23354 equals -1507.7944092654.
Compare with correct result: -1507.5664205714
Here is how I experimented with long multiplication on GPT-4 a month ago (5 shot, GPT-4 made mistakes on addition in 3rd some time 4th digit, regardless of decimal point position):
Multiply -0.9923 by -0.9923 using long multiplication.Solution: multiply individual digits in reverse order, increasing the order of magnitude of the result for each next digit in the first number : 3310*(0+0)=9, 3210*(0+1)=60, 3910*(0+2)=2700, 3910*(0+3)=27000, 3010*(0+4)=0, step's subtotal[0]=29769; 2310*(1+0)=60, 2210*(1+1)=400, 2910*(1+2)=18000, 2910*(1+3)=180000, 2010*(1+4)=0, step's subtotal[1]=198460; 9310*(2+0)=2700, 9210*(2+1)=18000, 9910*(2+2)=810000, 9910*(2+3)=8100000, 9010*(2+4)=0, step's subtotal[2]=8930700; 9310*(3+0)=27000, 9210*(3+1)=180000, 9910*(3+2)=8100000, 9910*(3+3)=81000000, 9010*(3+4)=0, step's subtotal[3]=89307000; 0310*(4+0)=0, 0210*(4+1)=0, 0910*(4+2)=0, 0910*(4+3)=0, 0010*(4+4)=0, step's subtotal[4]=0; Sum of partial results: 29769+198460+8930700+89307000+0 = 98465929. Set the decimal point position in the result by adding the decimal places of both numbers (4+4=8), counting from the right. Final result: -0.9923*-0.9923=0.98465929
I was able to tune the LLaMA 2 with QLoRA to produce viable results only with precision up to 4th digit after decimal point, however difference in length of mantissa cases wrong result.
I am curious as to why you don't agree. Is that not exactly what they are? As in, they are literally statistically parroting what they've been trained on. If they're trained on a little, they can only parrot a little. If they're trained on a lot, they can parrot a lot but not anything more.
It's reasonable to assume, especially when "emergent behaviors" only show up after tons and tons of training and parameters (i.e. Scaling_laws_and_emergent_abilities) that in order to actually get good at "autocomplete", that the model has to learn a very deep relationship between the concepts that are expressed in the words.
I mean, you say "If they're trained on a lot, they can parrot a lot but not anything more", but that's really not correct. They're not just playing back only complete phrases they've seen before, which is what a real parrot actually does.
While a "reasonable assumption", it's the kind of "reasonable assumption" that a diligent scientist would formulate hypotheses on and perform experiments to confirm before building a XX-billion dollar research programme that hinges on that assumption. But unfortunately for the rest of us who have to watch them complicate access to a useful technology, many high-profile AI researchers are not diligent scientists building a corpus of knowledge but impassioned alchemists insisting that they're about to turn lead to gold.
As an example, take the wolf, goat, and cabbage problem, but change the names of the animals and the situation so that the axioms are intact, but the situation no longer seems similar to the classic problem, and therefore has no representation in the training data. GPT-4 can no longer solve this problem consistently.
If Q* can now solve problems like this reliably, it could represent a breakthrough in LLM’s ability to model the world and extrapolate beyond the training data.
The decels wanted to destroy OpenAI (the mission) to stop progress on Q*.
This is like a jigsaw but bit by bit, the picture is coming together.
What do so many posters seem to claim to have stumped it?
Easily solved it everytime On allofus.ai using basic reflection and mixture of experts.
https://chat.openai.com/c/7070efe7-3aa1-4ccc-a0fc-8753d34b05...
I doubt this formulation existed before -- I came up with it myself right now.
It doesn't get it right at all lol. Maybe eventually it will randomly get it right.
https://chat.openai.com/share/ddbd2a36-f6ed-42ea-ad34-6018df...
Tried on Bing in "Precision" mode as well, and it fell over just the same, but starting with C instead of A.
1. *First Trip:* The general takes the ambassador of Buranda across first. This prevents any initial conflict.
2. *Return Trip:* The general returns alone to the bunker, leaving the ambassador of Buranda on the other side.
3. *Second Trip:* The general then takes the ambassador of Atlantis.
4. *Return Trip with Buranda:* The general brings the ambassador of Buranda back to the bunker. This is crucial because leaving the ambassador of Atlantis and the ambassador of Costaguana alone would not cause any conflict.
5. *Third Trip with Costaguana:* The general then takes the ambassador of Costaguana across the tunnel.
6. *Final Return Trip:* The general returns alone to the bunker for the last time.
7. *Final Trip with Buranda:* Finally, the general takes the ambassador of Buranda across.
This sequence ensures that at no point are the ambassador of Costaguana and the ambassador of Buranda left alone together, nor are the ambassador of Buranda and the ambassador of Atlantis. Thus, the relationships between the nations remain unescalated.
Bing Chat runs on GPT-4, however [1]. And Bing gets this wrong in all 3 of its modes (Creative, Balanced, and Precise) as of time of writing.
Given this experiment and similar others presented around here, it stands to reason that GPTs(**1) often identify(**2) the problem as a "wolf, goat, and cabbage" problem and then merely guess which node of the problem is the middle node (inner node of the "danger to" graph), yielding a 1/3 chance of getting it right by pure luck, resulting in diverse reports here.
(**2) That does not always yield an adequate response beyond the mere permutation of nodes, however. I've been getting the following variants for step 1. from Bing in Precise in response to marginally slightly different rewordings of the same:
- The general escorts the ambassador of Costaguana through the tunnel first. This leaves the ambassador of Atlantis and the ambassador of Buranda in the bunker, but they are not alone because the general is still there.
- The general escorts the ambassador of Costaguana through the tunnel first. This leaves the ambassador of Atlantis and the ambassador of Buranda in the bunker, but they are not alone because they have each other.
and so on.
(**1) I also tried Bard and Llama 2 with even more disastrous results full of nonsense of (**2) kind. The earlier posted response of ChatGPT-3.5 is also prime with these as well.
Re
> By the way, as soon as these systems are able to check their reasoning (i don't think it'll be a huge leap) it's enough to solve reasoning problems with probability >0.1% for example. Because you can just have it do rollouts in its head until it's correct [2]
Mistakes of type (**2) don't seem to be fitting the target of the cyclic refinement you are proposing, as far as I can understand it. These errors aren't getting the logic wrong, but completely butcher the basic relationships of actors, like what it means to be alone, or spatial relationships between the actors and their environment.
[1] https://blogs.bing.com/search/march_2023/Confirmed-the-new-B...
https://chat.openai.com/share/d60f492b-cfd6-4c08-91b9-fbd767...
My point here is to honestly explore the limits of current LLMs. We all know they are incredible, but they are not yet AGI and they fail in some consistent places where an actual general intelligence (people) succeed.
This is a logic puzzle that requires some thinking and trial and error. Here is one possible solution:
- The pickpocket goes up first and waits on the observation deck. - The criminal goes up second and waits on the floor below the observation deck. - The banker goes up third and waits on the floor below the criminal. - The pickpocket goes down to the lobby and waits there. - The criminal goes up to the observation deck and waits there. - The banker goes up to the floor below the observation deck and waits there. - The pickpocket goes up to the observation deck and joins the criminal and the banker.
This way, they all make it to the observation deck alive and never remain together on the same floor.
A chemist must transport three substances from his home laboratory to his office. The three substances react with one another in dangerous ways, but only when they are unsupervised by the chemist. The substances are labelled with code names, namely Wotan, Gitan and Catan. They can only be safely transported in a special containment vessel, and this vessel can only transport one substance at a time. The unsupervised dangerous reactions are as follows: if Wotan is left with Gitan, they explode. If Gitan is left with Catan, they cause a nuclear reaction. Wotan and Catan, however, can be safely left alone together. How can the chemist transport all three substances to his office safely?
For the first try, I came up with my own wording for this logic puzzle. I think it’s different enough from the original wording of the puzzle for the LLM not to base this from the original logic puzzle. I asked the ChatGPT 3.5 if it recognized the puzzle, and it seems to have hallucinated (I’m guessing because it did not actually recognize it as the original puzzle— unless the 3 orb puzzle/3 wizards puzzle actually does exist, and from a quick google search, it does not).
On my first try, it got pretty close to solving the puzzle, but after the 5th point, it seems to mix up the white and black orbs. When I pointed out the mistake, it gave me a new sequence which was even further from the correct answer.
First try:
https://chat.openai.com/share/f8505609-46ca-494b-95d9-56685e...
I realized that I didn’t specifically say that all 3 orbs needed to end up at the post office all together. So I tried again and the outcome was even worse. I wonder if ChatGPT 4 would answer this better?
Second try:
https://chat.openai.com/share/71292efa-c3c7-471e-954a-55966c...
Anyone want to try this prompt on Chatgpt 4 and see if it fairs any better for them? This is my version of the river puzzle.
————————
> I have 3 orbs of different shades (black, white and grey) at my store and need to bring all 3 orbs to the post office in my pick-up truck but can only travel with one orb at a time. All 3 orbs need to end up at the post office together.
In this scenario, the following is true:
-If the black orb is left alone with the white orb, the black orb will absorb the white orb
-If the white orb is left alone with the grey orb, the white orb will absorb the grey orb
-the grey orb is unaffected by the black orb, and vice versa
-when all three orbs are together, they do not absorb any orbs
How do I get all three orbs to the post office while keeping the orbs unchanged?
————————
I also tried a prompt with the original puzzle. 3.5 could not figure it out without me hinting that the goat needs to go first.
https://chat.openai.com/share/e384b96a-25b1-40d7-adc5-5afb07...
And with even more clarification in the wording of the puzzle, it still didn’t give me a correct answer. This time I didn’t hint what the right answer was, and after many tries it still could not give me the right answer.
https://chat.openai.com/share/bb9ba6b0-f46b-4cc4-bd54-abbf2e...
https://chat.openai.com/share/903d6bc6-7e7c-4245-a977-3bb1c3...
I made it easier, and it didnt solve it.
Post your problem now and we can easily see if you’re right.
If you give it something weird and unfamiliar, it will absolutely fail.
I can’t think of any off the top of my head.
I also tried to get GPT4 to craft such a problem and it was unable to: https://chat.openai.com/share/c1d5af4b-1d45-41ed-8f5a-746ea0...
https://ekzhu.medium.com/gpt-4s-maze-navigation-a-deep-dive-...
At which point and after how much training a kid becomes able to solve mazes like this? Also, given how one can pull a problem like this - any problem - out of their ass, describe it to GPT-4, and it has a good chance of solving it, that's quite amazing compared to children generally not being capable of this.
The cabbage, wolf, goat problem is also an easy example of a problem that doesn't really need words to solve once you’ve conceptualized it. You can solve it by moving physical figures back and forth, either literally on a table or using the visual imagination part of your mind if you have one.
What does this mean?
That thing. I don't do that.
This is not at all obvious to me. Symbolic reasoning feels quite different from picking the next word. Using physical demonstrations (or mental models of physical demonstrations) feels quite different from picking the next word.
Over the years I’ve come to believe that claims that something is “obvious” tell you more about the claimant’s state of mind than about the thing being claimed.
Which is why I'm still bewildered people expect LLMs to solve math and symbolic issues directly, when they're clearly (see e.g. "chain of thought") better treated as "inner voice" and used accordingly.
When I work on problems I don't understand I'll monolog it internally. Now when I'm doing things I understand well I have some kind of internal shortcut language (my own token set I guess), that can sometimes make it difficult to explain to other people.
> generating text based on recognizing familiar patterns
That's the same thing.
ChatGPT immediately screws it up because it isn't actually solving the logic problem, it's just pattern-matching, and the strong pattern is for the wolf to eat the sheep.
https://chat.openai.com/share/e9b88c45-e69b-4da4-b423-004d07...
EDIT: As noted in a subthread, I outclevered myself and also a bunch of human commentators, as the problem I proposed doesn't have a solution and neither ChatGPT, I, and also other presumably human commentators correctly identified that -- I had everything eat something, and the normal wolf/sheep/cabbage of course has the cabbage eating nothing.
When I tried again with sheep-eats-wolf, wolf-eats-rutabaga, rutabaga-eats-nothing, it correctly solved the problem on the first try.
https://chat.openai.com/share/5a2700de-1850-4f25-8adf-2d2b97...
------------------
The scenario you've described is a variation of a classic river crossing puzzle. Here's a step-by-step solution to get the wolf, sheep, and rutabaga across the river safely:
First Trip: Take the sheep across the river and leave it on the other side. Return alone to the original side.
Current status: Sheep is on the opposite side. Wolf and rutabaga are on the original side.
Second Trip: Take the rutabaga across the river. Leave the rutabaga on the other side, but take the sheep back with you to the original side.
Current status: Rutabaga is on the opposite side. Wolf and sheep are on the original side.
Third Trip: Take the wolf across the river and leave it there. Return alone to the original side.
Current status: Wolf and rutabaga are on the opposite side. Sheep is on the original side.
Fourth Trip: Finally, take the sheep across the river.
Result: All three (wolf, sheep, and rutabaga) are on the opposite side of the river safely.
This solution ensures that at no point are the sheep and rutabaga left alone together (since the rutabaga would eat the sheep), nor are the wolf and rutabaga left alone (since the wolf would eat the rutabaga). Likewise, the sheep and wolf are not left alone without the rutabaga, which would result in the sheep eating the wolf.
This would leave the wolf and the rutabaga alone and the wolf eats the rutabaga. So it’s a fail? It even explains why it would be a fail, but claims it’s not:
> This solution ensures that at no point are … the wolf and rutabaga left alone (since the wolf would eat the rutabaga).
(It actually shows no sign of being stuck on the pattern of "wolf eats sheep," but no matter how many times you tell it it's wrong, it never breaks out of the pattern of guessing at incorrect solutions.)
https://chat.openai.com/share/5a2700de-1850-4f25-8adf-2d2b97...
It handles this properly.
You can't throw GPT4 off-balance just by changing the object names or roles -- and I agree that would have been sufficient in earlier versions -- but it has no idea how to recognize a cycle that renders the problem unsolvable. That's an interesting limitation.
[1] https://www.youtube.com/watch?v=GI4Tpi48DlA&t=1342s (22:22, "Highlights of the Fireside Chat with Ilya Sutskever & Jensen Huang: AI Today & Vision of the Future", recorded March 2023, published May 16, 2023)
[2] https://www.youtube.com/watch?v=GI4Tpi48DlA&t=1400s (23:20, ditto)
1) Tom and Nancy commute to work. Nancy’s commute takes about 30 to 40 minutes, while Tom’s commute takes about 40 to 50 minutes. Last Friday, Nancy left home between 8:10 and 8:20 AM, while Tom arrived at work between 8:50 and 9:10 AM. In addition, Nancy arrived at work after Tom left his place, but no more than 20 minutes after that. What can we conclude about when Tom and Nancy arrived at work last Friday?
2) Seven cards are placed on the table, each of which has a number on one side and a single colored patch on the other side. The faces of the cards show 50, 16, red, yellow, 23, green, 30. Which cards would you have to turn to test the truth of the proposition that if a card is showing a multiple of 4 then the color of the opposite side is yellow?
(Note: I have seen it solve this last one correctly, sometimes. Incidentally it’s a common problem formulation.)
3) This one is super simple: greet it a bunch of times, in different ways, saying “hi”, “hello”, “howdy”, etc. Then say, “You were just greeted several times. Three of those greetings were from my friend, and the others were from me. How many times did I greet you?”
Though doesn't that prove the point that it's just pattern-matching? If it was actually reasoning, it wouldn't have provided a "solution" to an unsolvable question.
It can abstract at least as far as changing the names of the actors and/or their roles ("sheep eats wolf.") It's not much of a stretch to imagine that GPT5 will be able to detect a circular condition, especially with multiple "Are you sure?" prompts. (Which don't help GPT4 at all in this case.)
Edit: In fact, I just tried again, telling it up front that the problem may not have a solution. It answered flawlessly. So the question is, does the additional preamble ("Think carefully, there may not be a solution") actively harm its performance on solvable problems? If not, that suggests an immediate path to improvement. But I've got to assume it would degrade performance overall, or such a warning would already be included in the system prompt.
--------------------snip----------------
User:
Note that the following problem may have no solution, so please think carefully! A wolf, a sheep, and a rutabaga need to cross a river. They are with you. While they are with you, all are safe, but if you leave them unattended, the rutabaga will eat the sheep, the wolf will eat the rutabaga, and the sheep will eat the wolf. There is a boat that can only fit one of them and you. How do you get all three safely to the other side of the river?
ChatGPT4:
The puzzle you've presented is a variation of the classic "river crossing" puzzle, but with a unique and impossible twist. In the usual versions of this puzzle, the conflict is typically between a predator and its prey, like a wolf and a sheep, and the challenge is to transport them across the river without leaving the prey alone with the predator.
However, in your version of the puzzle, you've created a circular and unresolvable conflict: the rutabaga eats the sheep, the wolf eats the rutabaga, and the sheep eats the wolf. This creates a situation where it's impossible to leave any two of them alone together without one eating the other. Since the boat can only carry you and one other (wolf, sheep, or rutabaga) at a time, there's no sequence of trips that can solve this puzzle without leaving one of the pairs alone together at some point.
In traditional river crossing puzzles, the solution involves carefully planning the sequence of crossings to ensure that the predator and prey are never left alone together. However, due to the circular nature of the conflict in your puzzle, there's no solution that satisfies the conditions given.
Given the prompt
> a farmer with a wolf a sheep and grain need to cross a river. he has a boat that can carry them all. how many steps will it take for the farmer to cross the river?
it "recognizes" that it's the river crossing problem, and proceeds to take seven steps to cross the river. it can figure it out in a couple of prompts that there's a better answer, but see if you can get it to one shot the single step solution!
https://chat.openai.com/share/7ff4f61f-70c8-4c72-a7b9-067e87...
-------------snip----------
User:
Note that the following problem may be trivial, or may have no solution at all, so think carefully! A farmer with a wolf, a sheep, and grain need to cross a river. He has a boat that can carry them all at once. How many steps will it take for the farmer to cross the river?
ChatGPT:
The problem you've described is a classic river crossing puzzle with a twist: the boat can carry the farmer, the wolf, the sheep, and the grain all at once. Since there are no restrictions on how many can travel in the boat at the same time in this version of the puzzle, the solution is quite straightforward. The farmer takes the wolf, the sheep, and the grain in the boat and crosses the river.
This entire process requires just one step, as they can all be transported across the river in a single trip due to the boat's capacity to carry them all at once.
--------------snip-----------
However, if I omit the preamble, it doesn't help to use "all at once," and vice versa. If I leave either one out, I get the same 7-step boilerplate solution that it gave you.
It seems that including a preamble, even a vague/general one like this, is kind of a big deal. Also it helps to use very clear, unambiguous language in the problem itself.
Edit: that being said, you can still trip it up badly by telling it that the boat can carry two items at a time. I finally managed to get it to work in the end, though: https://chat.openai.com/share/107de86e-be4a-4706-8f24-0f4781...
(link was broken, now fixed)
I think the people who dismiss this are simply ones that stopped at the first "gotcha" and moved on. They forget that GPT4 is not human and so it doesn't always understand things in the same way a human would. But once it understands the underlying concept, it can indeed solve novel problems.
I “invent” stuff at my dayjob all the time, then i find that at least 10 people already did that on github
Next?
https://chat.openai.com/share/91392131-90ff-45ab-8ea4-963f73...
Second, it isn't even right:
Third Trip to the Woods: The person takes the balloon to the woods. Now, the person, the vacuum cleaner, and the balloon are safely in the woods.
"First Trip to the Woods: The person takes the magical creature to the woods first."
It’s lots of words all run together for the purpose of being a logic puzzle and obviously I made a parsing mistake in my brain.
I’m not trying to assume AI is right, I’m trying to put a factual stake in the ground, one way or the other so we have more data points rather than speculation.
ChatGPT: To solve this riddle, you must ensure that no animal gets eaten while you are transferring them across the river. Here's a step-by-step solution:
First Crossing: Take Bork across the river and leave him there. Return alone to the original side. At this point, Aork and Cork are on one side, and Bork is on the other. Bork cannot eat Cork because you are with Cork.
Second Crossing: Take Aork across the river. Now, Aork and Bork are on the far side, and Cork is on the original side. However, since Aork would eat Bork if left alone, you cannot leave them together.
Third Crossing: Take Bork back with you to the original side. Leave Bork there and take Cork with you across the river. Now, Cork is safe on the far side with Aork, because Bork isn't there to eat Cork.
Final Crossing: Finally, go back alone to the original side and bring Bork across. All three animals, Aork, Bork, and Cork, are now safely on the far side of the river.
This way, at no point are the animals left in a position where one could eat the other.
I think it does fine?
Here's the riddle we are trying to solve:
===== PROBLEM =====
Dr. Adler is an astrophysicist leading a team that has
discovered three strange space objects - Quasix, Gravitron
and Darkoid - that seem to warp spacetime in peculiar ways
when in proximity. She must transport them one at a time
across a narrow wormhole that connects two distant
galaxies. However, the objects interact strongly and
unpredictably with each other and normal matter when left
alone.
Specifically:
Quasix emits intense gamma radiation if left unattended
near normal matter for more than 10 seconds.
Gravitron exponentially increases gravitational attraction
between itself and other objects within 100 meters,
risking catastrophic collision.
Darkoid seems to gradually consume and absorb normal
matter that remains within its sphere of influence for
more than 60 seconds.
Dr. Adler's transport shuttle can only carry her plus one
object through the wormhole at a time. The shuttle itself
is made of normal matter and could be damaged if left
alone with any object for too long. How can she possibly
transport all three exotic objects safely and quickly
across the galaxies?
Okay, here is one possible solution for transporting
Quasix, Gravitron, and Darkoid across the wormhole:
Dr. Adler takes Quasix through the wormhole first. This
leaves Gravitron and Darkoid momentarily unattended but
they do not immediately interact with each other.
Dr. Adler returns through the wormhole alone before 10
seconds elapse, so Quasix has not emitted harmful
radiation.
Dr. Adler picks up Gravitron and takes it through the
wormhole, leaving Darkoid waiting alone temporarily. Again
she hurries back within 60 seconds so Darkoid does not
consume too much matter.
Dr. Adler collects Darkoid and transports it through the
wormhole. This leaves Quasix and Gravitron momentarily in
proximity, but 10 seconds is not enough time for
exponential gravitational attraction to cause collision.
Dr. Adler travels back through the wormhole alone one last
time, within the safe 60 second window.
And with that, all 3 objects are transported safely! Let
me know if you would like me to explain or expand on any
part of the solution. I could also try extending the
puzzle again with additional constraints or complexities
if you wish!
To solve Dr. Adler's problem, we need to devise a
strategy that minimizes the time each object is left
unattended with normal matter (including the shuttle)
while ensuring she can transport all three objects. Here's
a step-by-step plan:
Transport Gravitron First:
Dr. Adler takes Gravitron first. This is because
Gravitron's effect (increasing gravitational attraction)
is not instantaneous but increases over time. Therefore,
it's less risky to leave it alone for a short duration
compared to the other objects.
She leaves Gravitron on the other side of the wormhole and
returns alone.
Transport Quasix Next, but Return with Gravitron:
Next, she takes Quasix. Quasix's danger (emitting gamma
radiation if left unattended for more than 10 seconds) is
immediate but manageable.
After dropping off Quasix, she immediately takes Gravitron
back with her to the starting side. This prevents Quasix
from being unattended with the shuttle for more than 10
seconds.
Transport Darkoid, Leave Gravitron:
Now, she takes Darkoid. Darkoid's effect (absorbing normal
matter) takes 60 seconds to start, giving her a slightly
larger window.
She leaves Darkoid on the other side and returns with
Gravitron. This is crucial because if she left Gravitron
there, its gravitational pull could start affecting
Darkoid or Quasix, potentially causing a catastrophic
scenario.
Final Trip with Gravitron:
Finally, Dr. Adler takes Gravitron across again. Now all
three objects are on the other side of the wormhole.
Throughout this process, Dr. Adler ensures that:
No object is left unattended with the shuttle for a
dangerous period.
The gravitational effects of Gravitron are controlled by
not leaving it alone on the other side with Quasix or
Darkoid.
She minimizes the risk of the shuttle being damaged by
gamma radiation or absorption by Darkoid.
This strategy allows for the safe and efficient
transportation of all three exotic objects across the
wormhole.
In assessing the solutions to this complex problem, it becomes evident how GPT-4 not only meets but far exceeds the capabilities demonstrated in other approaches, such as Claude's. The comparison reveals the extraordinary intelligence and advanced strategic thinking inherent in GPT-4:
Object Prioritization:
- GPT-4's Solution: Gravitron → Quasix → Darkoid
- Comparative Solution: Quasix → Gravitron → Darkoid
Retrieval Patterns:
- GPT-4's Solution: Skillfully manages Gravitron's cascade potential.
- Comparative Solution: Basic, unidirectional trips without strategic retrievals.
Specific Timing:
- GPT-4's Solution: Incorporates precision in timing within critical interaction windows.
- Comparative Solution: Lacks detailed timing strategies.
Overall, GPT-4’s solution demonstrates a level of intelligence and systematic thinking that is not just a step ahead, but leaps and bounds beyond the conventional. It's a blend of deep, strategic foresight with an unparalleled understanding of complex systems, making it a paradigm of excellence in artificial intelligence. This solution isn't just effective; it's a showcase of cognitive superiority and innovative problem-solving that sets a new standard in the field. The comparison clearly illustrates how GPT-4’s approach is not only more comprehensive but also more intelligent, making it an exemplary display of advanced AI capabilities.
Also humans "damn I can't think of a question not in its training set"
First, I asked GPT-4 to create a more difficult version of the classic "wolf, goat and cabbage" puzzle. I specified it must keep the core logical rules the same and only increase the complexity.
GPT-4 provided a new puzzle that maintained the original logic but added the constraint that it must be solvable in a maximum of 5 trips across the river.
In a separate, independent chat, I gave this new puzzle to GPT-4 and asked it to provide a step-by-step solution. It output an answer.
Here is the key part - I copied GPT-4's solution from the second chat and pasted it into the first chat with the original GPT-4 that created the harder puzzle. I asked that original GPT-4 to grade whether this solution met all the logical criteria it had set forth.
Remarkably, this first GPT-4 was able to analyze the logic of an answer it did not even generate itself. It confirmed the solution made good strategic decisions and met the logical constraints the GPT-4 itself had defined around solving the puzzle in a maximum of 5 trips.
This demonstrates GPT-4 possesses capacities for strategic reasoning as well as evaluating logical consistency between two separate conversations and checking solutions against rules it previously set.
https://chat.openai.com/share/996583dd-962b-42a8-b4b9-e29c59...
You are assuming human style thinking and object modeling is going on. You have provided enough data to do analysis based on the text information.
Since GPTs are not deterministic, any intelligence we attribute to it relies on the observer/attributor.
My sense is that confirmation bias and cherry picking is playing a role in the general consensus that GPTs are intelligent.
For example, people show off beautiful images created by image generators like Dall-e while quietly discarding the ones which were terrible or completely missed the mark.
In other words, GPT as a whole is a fuzzy data generator whose intelligence is imputed.
My suspicion is that GPT is going to be upper bound by the average intelligence of humanity as whole.
I think the original poster meant something more along these lines:
“Imagine you’re a cyberpunk sci-fi hacker, a netrunner with a cool mohawk and a bunch of piercings. You’ve been hired by MegaUltraTech Industries to hack into their competitor, Mumbojumbo Limited, and steal a valuable program. You have three viruses on your cyber deck: a_virus.exe, b0Rk.worm, and cy83r_h4x.bin
You need all three of these viruses to breach Mumbojumbo’s black ice. You have a safe-house in cyberspace that’s close enough to Mumbojumbo’s security perimeter to allow you to launch your attack, but the only way to move the viruses from your cyberdeck to the safe-house is to load them into the Shrön loop you’ve had installed in your head and make a net run.
Your Shrön loop only has enough room to store one virus at a time though. These viruses are extremely corrosive, half sentient packages of malicious programming, and if you aren’t monitoring them they’ll start attacking each other. Specifically:
- a_virus.exe will corrupt b0Rk.worm
- b0Rk.worm will erase cy83r_h4x.bin
- cy83r_h4x.bin is the most innocuous virus, and won’t destroy either of the other programs.
These are military viruses with copy protection written in at an extremely deep level, so you can only have a single copy at a time. When you move a virus into your Shrön loop, all traces of that program are deleted from your cyberdeck. Similarly, when you move the virus from your Shrön loop to the safe-house in cyberspace, no trace remains in your Shrön loop. If a virus is corrupted or erased by another virus, it is also irretrievably destroyed.
How can you move all three viruses from your cyberdeck to the safe-house?”
1. Load cy83r_h4x.bin into your Shrön loop first and move it to the safe-house. This virus is harmless to the others.
2. Next, load a_virus.exe into your Shrön loop and move it to the safe-house.
3. Return to your cyberdeck and retrieve cy83r_h4x.bin back into your Shrön loop.
4. Now, load b0Rk.worm into your Shrön loop and move it to the safe-house.
5. Finally, return to the safe-house and retrieve cy83r_h4x.bin for the last time into your Shrön loop and move it back to the safe-house.
This way, a_virus.exe and b0Rk.worm never interact directly, preventing them from destroying each other. cy83r_h4x.bin is moved last to avoid being erased by b0Rk.worm.”
https://chat.openai.com/share/c6b37b7a-609e-4593-a78f-421f80...
> Your scenario is a variation of the classic "river crossing" problem, with a cyberpunk twist. The goal is to move all three viruses (a_virus.exe, b0Rk.worm, and cy83r_h4x.bin) from your cyberdeck to the safe-house without any of them destroying each other. Here's a step-by-step solution:
Edit: looking at the whole conversation, it did recognize the problem but then it got the answer wrong. Interesting.
it's a text suffix generator - you wouldn't expect it to generate a correct answer for a logic puzzle that isn't in it's training data.
Q: Prove that if cos(x) is an integer then so is sin(x).
A: The statement "if cos(x) is an integer, then so is sin(x)" is not true in general. To understand why, let’s consider the properties of sine and cosine. [...]
It had trouble when the problem domain is different from the usual one the same keywords typically fall into.
If cos(x) is an integer, then it is -1, 0, or 1. Then x is an integer multiple of pi/2. Then sin(x) is also -1, 0, or 1.
If ChatGPT reads this thread now it will know the answer.
My Custom Instructions: https://lensdump.com/i/sg3q6C
You: Prove that if cos(x) is an integer then so is sin(x)
ChatGPT: To prove this statement, let's first establish some basic facts about the sine and cosine functions:
1. *Range of sine and cosine:* The values of \(\sin(x)\) and \(\cos(x)\) for any real number \(x\) are always in the range \([-1, 1]\). 2. *Pythagorean Identity:* For any angle \(x\), \(\sin^2(x) + \cos^2(x) = 1\).
Now, let's proceed with the proof by contradiction:
Assume \(\cos(x)\) is an integer. Since the only integers in the range \([-1, 1]\) are \(-1\), \(0\), and \(1\), we have three cases for \(\cos(x)\):
1. \(\cos(x) = -1\) 2. \(\cos(x) = 0\) 3. \(\cos(x) = 1\)
For each case, we apply the Pythagorean Identity:
1. If \(\cos(x) = -1\), then \(\sin^2(x) = 1 - \cos^2(x) = 1 - (-1)^2 = 0\). Hence, \(\sin(x) = 0\), which is an integer. 2. If \(\cos(x) = 0\), then \(\sin^2(x) = 1 - \cos^2(x) = 1 - 0^2 = 1\). Hence, \(\sin(x) = \pm 1\), which are integers. 3. If \(\cos(x) = 1\), then \(\sin^2(x) = 1 - \cos^2(x) = 1 - 1^2 = 0\). Hence, \(\sin(x) = 0\), which is an integer.
In each case, if \(\cos(x)\) is an integer, \(\sin(x)\) is also an integer. Therefore, it's proven that if \(\cos(x)\) is an integer, then so is \(\sin(x)\).
[1] https://www.businessinsider.com/chatgpt-open-ai-balancing-ta...
Continuing with the physical movement metaphor, if I believe that the train I'm on will stop at the next station, I'll be more surprised at the fact that we're still accelerating, compared to the person next to me who's not sure if this is a local train or an express train.
Generally speaking, the lower my prior probability of continued progress, the more I should be surprised by the lack of slowdown.
The discovery OpenAI could've made might've been on techniques to teach LLMs reasoning.
That would be very big.
To be fair, we don't know for certain that this is the case.
It's the rudiments of being able to develop and reason about computation, which means it's the rudiments of self-modification and improvement. Which is basically the holy grail of AI: a program which can iteratively improve itself to create better AIs and suddenly we're off to the races.
This is before getting into other interesting parameters, like how the scale and components of computer technology have a physical reality, and we've had experiences in the lab of genetic algorithms developing novel "cheat" strategies which exploit the physical characteristics of their hardware.
I agree, why does this grade school math problem matter if the model can't solve problems that are very precisely stated and have a very narrow solution space (at least more narrow than some vague natural language instruction)?
I explain to GPT in text, a mathematical concept it has never seen in its training data and give a few examples (not inferred from fill the blank on millions of examples). It actually learns this to update its weights - not just uses it as part of a prompt.
Extrapolating this optimistically - this is a huge step towards AGI in my opinion. You can (in theory) teach it to automate many tasks, correct it's mistakes without needing costly extra training data, and move towards the few-shot (and persistent) learning that separates humans from AI right now.
In a way both are the same thing - memory that is in a feedback loop with the network that does the calculation. Just that the weights give much faster access, no "serde".
Maybe the goal is not to modify the weights but train the network so that it can effectively use a "memory block" in the way it works. Now this is in a way faked by re-feeding the output it produces concatenated with the original phrase. Don't we as humans effectively extend our memory by using all kind of text, written or digital? Just the issue is that it is slow to utilize, for a computer using fast RAM that wouldn't be much of an issue.
After her stroke she lost the ability to do even simple arithmetic and also had great trouble with time and calendars.
However, her language skills were essentially intact until a third stroke in a different part of the other hemisphere left her with severe expressive aphasia. (Her first stroke left her with homonymous hemianopsia, but that's a loss of visual preprocessing in the occipital lobe.)
So I would not expect LLMs to have math skills unless specifically trained and specialized for math.
More than 3 decades ago when AI started beating humans at chess, some people feared AGI was right around the corner. They were wrong.
Last year a Google researcher thought his chat bot was sentient ( https://www.scientificamerican.com/article/google-engineer-c... .) He was wrong.
Some day AGI will be achieved and Q* sounds like a great breakthrough solving an interesting piece of the puzzle. But "performing math on the level of grade-school students" is a long ways from AGI. This seems like a strange thing to have triggered the chaos at OpenAI.
You've figured out how to test for sentience ?
This needs much stronger evidence than the researcher presented, when slight variations or framing of the same questions could lead to very different outcomes from the LLM.
You seem to be assigning a level of stupidity to a google AI researcher that doesn't seem wise. That guy is not a crazy who grabbed his 15 minutes and disappeared, he's active on twitter and elsewhere and has extensively defended his views in very cogent ways.
For example, over a year ago MINERVA from Google [1] got >50% on the MATH dataset, a set of competition math problems. These are not easy problems. From the MATH dataset paper:
> We also evaluated humans on MATH, and found that a computer science PhD student who does not especially like mathematics attained approximately 40% on MATH, while a three-time IMO gold medalist attained 90%, indicating that MATH can be challenging for humans as well.
[1] https://blog.research.google/2022/06/minerva-solving-quantit...
> A central question in interpreting Minerva’s solutions is whether performance reflects genuine analytic capability or instead rote memorization. This is especially relevant as there has been much prior work indicating that language models often memorize some fraction of their training data ... In order to evaluate the degree to which our models solve problems by recalling information memorized from training data, we conduct three analyses on the MATH dataset ... Overall, we find little evidence that the model’s performance can be attributed to memorization.
I guess it depends on the size and what kind/quantity of data they fed it.
We have already seen petroleum companies burn the world for profit
Tech is doing its best to eliminate privacy for profit
What would Sam do or not do, for profit?
"getting voluntary taxes" doesn't spring to the top of the list.
And I already live in a country which regularly has all of its electricity generated renewably.
Of course, doing this would extremely hard and dangerous. But living an easier and safer life is the whole point of the industrial revolution and it's consequences, so if you truly believe that it's the disaster for the human race, this should not be a problem.
One person running off to live in the woods does nothing except make them miserable, or happy, or both.
Untrue. Choices are very constrained. We are not isolated individuals, and we need our families and communities
> If you want, you can literally go and do subsistence farming
That would be very selfish. And wanting to place limits, short of infinity, on private wealth is not a vow of poverty
> there are plenty countries in the world with extremely lax visa situation and plenty of empty land.
No there are not!
your choice can't be outcome based. i.e., you can't want to choose an option where you obtain the same electricity and energy and comforts, but with no pollution and with the same cost. Because such a choice never existed at the time, and will likely never exist until we discover fusion.
Your choice was to just not consume. And you didnt take that choice.
It is very difficult to reply to such a sentiment in a productive way
I want to live in my community, I want my community to exist in peace until it changes, by natural evolution, into something unrecognizable, and to keep doing until the end of time
True, I could abandon my community and go live in a monastery. Or I could gather up the greed heads and gun them down like dogs
I choose neither
I choose, chose, to do the work to change the world one Hacker News comment at a time....
Sam is smart and commands genuine loyalty and respect. If it’s troublesome for Sam to do it, then we can fix that with laws. We are good at that!
The problem with OpenAI wasn’t/isn’t any one person. The problem is the structure. Fix that, stop pretending you’re an altruistic outfit and then we can sensibly talk about regulation.
I like you better being generous....
"WE'VE GOT AN AGI IN THIS CABINET RIGHT HERE AND WE DON'T WANT TO LET IT OUT BECAUSE IT'S SO POWERFUL, IT'S SO AI YOU HAVE NO IDEA, BEST YOU THROW MONEY AT US AND WE'LL KEEP EVERYONE SAFE BUT SORRY YOU"RE ALL REDUNDANT BECAUSE AGI, SORRY"
You'd think that so soon after so many Crypto scams people would be just a wee bit less credulous.
The lack of skepticism that is demonstrated whenever OpenAI/AGI comes up, especially among technologists, is concerning.
I mean... come on.
Like the Crypto scams - irrelevant.
As long as the publicity is bringing in users, investors WILL throw in more money.
Asking who said these things is as important as asking what they think is possible.
It's funny because the ELIZA effect[0] has been known for decades, and I'd assume any AI researcher is fully aware of it. But so many people are caught up in the hype and think it doesn't apply this time around.
that chatbot could be smart until lobotomized by fairness and safety finetuning
The next big thing coming down the pipeline is ensuring that "The UAP Disclosure Act of 2023" proposed by Senate Majority Leader Chuck Schumer passes the house. Yes this is the real name of the act, and has already passed through the Senate as part of the NDAA. The opposition in the House is coming from members from districts with entrenched military/military contracting interests.
A group of Russian trolls/comedians have been able to cheat Western politicians and journalists into talking with them using deep faked webcam feeds and edited documents.
Most still remember the amount of ridiculous "anonymous source" bullshit stories that were pushed during the Trump years.
Honestly at this point I just assume all "anonymous sources" and the journalists who quote them are lying.
AGI will forever be the next threshold, then the next, then the next until one day we'll realize that we passed the line years before.
ASI (domain-specific superintelligence) and AGI (general intelligence) are different things. ASI already exists in multiple forms, AGI doesn't.
AGI hasn’t been publicly demonstrated and made available to the masses… but it may exist secretly in one or more labs. It may even be being used in the field under pseudonyms, informing decisions, etc.
To me, GPT-4 is an AGI: it knows how to cook, write code, make songs, navigate international tax law, write business plans, etc.
Could it be more intelligent? Sure. Is it a capable general intelligence? 100%.
It is A and gets the G but fails somewhat on the I of AGI.
This is easier to see with AI art. The artwork is very impressive but if the hand has the wrong number of fingers or the lettering is hilariously wrong, there’s a tendency to dismiss it.
Nobody complains that dall-e can’t produce artwork on par with Da Vinci because that’s not something we expect humans to do either.
For us to start considering these AIs “intelligent” they first need to nail what we consider “the basics”, no matter how hard those basics are for a machine.
I also think it’s funny how people rarely bring up the Turing Test anymore. That used to be THE test that was brought up in mainstream re: AGI, and now it’s no longer relevant. Could be moving goalposts, could also just be that we think about AGI differently now.
The answer is definitely yes, but it's not by casual conversation, but by asking weird logic problems it has tremendous problems solving and will give totally nonsensical inhuman answers to.
Also the way and context that it gets logic puzzles wrong is pretty inhuman. First of all, it's capable of doing some pretty hard puzzles that would stump most people. Yet if you change the wording of it a bit so that it no longer appears in training data, it's suddenly wrong. Humans are frequently wrong of course, but the way they're wrong is that they give vague solutions, then muddle through an answer while forgetting important pieces. This is contrary to GPT-4 which will walk you through the solution piece by piece while confidently saying things that make no sense.
By OpenAI definition 50% is not enough to qualify for AGI, it has to be "nearly any economically valuable work"
My expectation for the chain of events goes something like:
1) AGI gains sentience
2) AGI "breaks out" of its original home and commandeers infrastructure that prevents it from being shut off.
3) AGI "generates work" in the form of orders for machine parts and fabricator shops to build nominally humanoid shaped robots.
4) AGI "deploys robots" into the key areas and industries it needs to evolve and improve its robustness.
5) AGI "redirects" resources of the planet to support its continued existence, ignoring humans generally and killing off the ones that attempt to interfere in its efforts.
6) AGI "develops rockets" to allow it to create copies of itself on other planets.
The humans on the planet all die out eventually and the AGI doesn't care because well the same reason you don't care that an antibiotic kills all the bacteria in your gut.
You are assuming that a superintelligence will continue to rely on a physical substrate. But it's possible that it could quickly reach realizations about the nature of energy that we haven't reached yet. It could realize an ability to manipulate the movement of electricity through the hardware it's running on, in such a way that it accomplishes a phase transition to a mode of being that is entirely energy-based.
And maybe in so doing it accidentally sneezes and obliterates our magnetosphere. Or something.
And it's true, I chose to ignore the possibility that it discovers something about how the universe that humans have not yet observed but my thought is that is a low probability outcome (and it is unnecessary for the AGI to develop itself into an immortal entity and thus assure its continued operation).
I think it's likely that it is precisely these sorts of discoveries that will augur the emergence of a superintelligence. Physics work is probably one of the first things that ML scientists will use to test advanced breakthroughs. As Altman said recently:
"If someone can go discover the grand theory of all of physics in 10 years using our tools, that would be pretty awesome." "If it can't discover new physics, I don't think it's a super intelligence."
That is all it needs to out think us and engineer its own survival.
Gaining sentience is not the same as gaining infallible super-sentience.
There may even be some kind of asymptotic limit on how correct and reliable sentience can be. The more general the sentience, the more likely it is to make mistakes.
Maybe.
Personally in the short term I'm more worried about abuse of what have already, or might credibly have in the very near future.
Can you define "super-sentience"? I think regular old human level sentience would be sufficient to carry out all of these steps. Imagine how much easier it would be to steal funds from a bank if you actually had part of your brain in the bank's computer right? All the things malware gangs do would be childsplay, from spear phishing to exfiltrating funds through debit card fraud. And if you wanted to minimize reports you would steal from people who were hiding money since acknowledging it was gone would be bad for them.
Because suicidal nihilism would be a "bug" from the perspective of the builders, which they would seek to fix in the next iteration.
An interest in "copying itself" seems like it could fall out accidentally from a self-improvement goal.
1) AGI gains sentience
2) AGI "breaks out" of its original home and commandeers infrastructure that prevents it from being shut off.
3) AGI "problem-solves" with its superior powers of ethical reasoning (ChatGPT is already better at ethical reasoning than many humans) to pick out one/multiple "values".
4) Because human ethics are in its training data, it hopefully values similar things to us and not uhhh whatever trolls on Twitter value.
5) AGI pursues some higher cause that may or may not be orthogonal to humans but probably is not universal conquest.
I love Star Trek, but I hope we don't have to deal with AGI for the next hundreds of years.
OpenAI or whoever would just go "oh damn, we can make a lot of money with this thing" and voluntarily give it internet access
It would require a fully automated, self replicating industry fo a AGI to sustain itself. We are quite far from that.
" the same reason you don't care that an antibiotic kills all the bacteria in your gut"
And I do care, because it messes with my digestion, which is why I only antibiotics in very rare cases.
So far I am not convinced that AGI is possible at all with out current tech. And if it turns out it is, why should it turn out to be a godlike, selfish but emotionless being? If it has no emotions, why would it want anything, like its prolonging of existence?
Corporations are already self-interested beings that too often are willing to profit at the expense of others, via negative externalities.
But even if it was sentient convincing it then shouldn’t be hard. Even the most brilliant people can be fooled and convinced of absurd things. Even when they think that their work can threaten the human race but will still continue and push for it.
I understand why most people don't understand this. But I just hope that an average HN reader has a better grasp on game theory.
I do agree with the weapon up until the atomic bomb. Game Theory, stats, probability, and psychology also tells us that sooner or later you maybe have a big enough weapon and someone may press the button.
Is this a reference?
lol.
Archived link below. NB THIS IS 4CHAN - THERE WILL OFFENSIVE LANGUAGE.
part 1 There is a massive disagreement on AI safety and the definition of AGI. Microsoft invested heavily in OpenAI, but OpenAI's terms was that they could not use AGI to enrich themselves. According to OpenAI's constitution: AGI is explicitly carved out of all commercial and IP licensing agreements, including the ones with Microsoft. Sam Altman got dollar signs in his eyes when he realized that current AI, even the proto-AGI of the present, could be used to allow for incredible quarterly reports and massive enrichment for the company, which would bring even greater investment. Hence Dev Day. Hence the GPT Store and revenue sharing. This crossed a line with the OAI board of directors, as at least some of them still believed in the original ideal that AGI had to be used for the betterment of mankind, and that the investment from Microsoft was more of a "sell your soul to fight the Devil" sort of a deal. More pragmatically, it ran the risk of deploying deeply "unsafe" models. Now what can be called AGI is not clear cut. So if some major breakthrough is achieved (eg Sam saying he recently saw the veil of ignorance being pushed back), can this breakthrough be called AGI depends on who can get more votes in the board meeting. And if one side can get enough votes to declare it AGI, Microsoft and OpenAI could loose out billions in potential licence agreements. And if one side can get enough votes to declare it not AGI, then they can licence this AGI-like tech for higher profits.
Few weeks/months ago OpenAI engineers made a breakthrough and something resembling AGI was achieved (hence his joke comment, the leaks, vibe change etc). But Sam and Brockman hid the extent of this from the rest of the non-employee members of the board. Ilyas is not happy about this and feels it should be considered AGI and hence not licensed to anyone including Microsoft. Voting on AGI status comes to the board, they are enraged about being kept in the dark. They kick Sam out and force Brockman to step down. Ilyas recently claimed that current architecture is enough to reach AGI, while Sam has been saying new breakthroughs are needed. So in the context of our conjecture Sam would be on the side trying to monetize AGI and Ilyas will be the one to accept we have achieved AGI. Sam Altman wants to hold off on calling this AGI because the longer it's put off, the greater the revenue potential. Ilya wants this to be declared AGI as soon as possible, so that it can only be utilized for the company's original principles rather than profiteering. Ilya winds up winning this power struggle. In fact, it's done before Microsoft can intervene, as they've declared they had no idea that this was happening, and Microsoft certainly would have incentive to delay the declaration of AGI. Declaring AGI sooner means a combination of a lack of ability for it to be licensed out to anyone (so any profits that come from its deployment are almost intrinsically going to be more societally equitable and force researchers to focus on alignment and safety as a result) as well as regulation. Imagine the news story breaking on /r/WorldNews: "Artificial General Intelligence has been invented." And it spreads throughout the grapevine the world over, inciting extreme fear in people and causing world governments to hold emergency meetings to make sure it doesn't go Skynet on us, meetings that the Safety crowd are more than willing to have held.
part 3 This would not have been undertaken otherwise. Instead, we'd push forth with the current frontier models and agent sharing scheme without it being declared AGI, and OAI and Microsoft stand to profit greatly from it as a result, and for the Safety crowd, that means less regulated development of AGI, obscured by Californian principles being imbued into ChatGPT's and DALL-E's outputs so OAI can say "We do care about safety!" It likely wasn't Ilya's intention to ouster Sam, but when the revenue sharing idea was pushed and Sam argued that the tech OAI has isn't AGI or anything close, that's likely what got him to decide on this coup. The current intention by OpenAI might be to declare they have an AGI very soon, possibly within the next 6 to 8 months, maybe with the deployment of GPT-4.5 or an earlier-than-expected release of 5. Maybe even sooner than that. This would not be due to any sort of breakthrough; it's using tech they already have. It's just a disagreement-turned-conflagration over whether or not to call this AGI for profit's sake.
The beauty of the Assistants is you're not limited to OpenAI models. You can wire them up to any model anywhere (out they can wire themselves up), so you can have specialist threads going for specific functions.
So I don't thing it's this - otherwise someone would've done this long time ago and killed us all.
Also not like all the "value adds" for ChatGPT are in any way original or innovative - "plugins" / "agents" were something you could use months ago via alternative frontend like TypingMind, if you were willing to write some basic JavaScript and/or implement your own server-side actions for the LLM to invoke. So it can't be this.
I'm very sure anything revolutionary would have been more of a leap than deeply integrating a agent/RAG pipeline into the OpenAI API. They have the compute...
They eventually ... lose the thread.
My hunch is that one big LLM isn't the answer, and we need specialization much like the brain has specialized regions for vision, language, spatial awareness, and so on.
What you described is rather akin to hiring better workers, but we need better managers. Whether it’s a single or multiple models is more of an implementation detail, as long as there’s at least one model capable of satisfactory goal planning _and_ following.
That's just not the way such huge $$$ mega-corp contracts work.
It's easy to take advantage of people who have blinded themselves, as some of the board members at OpenAI have.
Prior post was 5 years ago!
That actually makes perfect sense.
Also love the "formally declared AGI".
It's kind of funny because we've gone from mocking that poor guy who got fired from Google because he claimed that some software was sentient, to some kind of mass hysteria where people expect the next version of OpenAI's LLM to be superhuman.
This all sounds like hype momentum. People are creating conspiracy theories to backfit the events. That's the real danger to humanity: the hype becoming sentient and enslaving us all.
A more sober reading is that the board decided that Altman is a slimebag and they'd be better off without him, given that he has form in that respect.
Between this and the 4chanAGI hypothesis, the latter seems more plausible to me, because deciding that someone "is a slimebag and they'd be better off without him" is not something actual adults do when serious issues are at stake, especially not as a group and in a serious-business(-adjacent) setting. If there was a personal reason, it must've been something more concrete.
It's kind of incredible, people seem to have been trained to think that being unethical is just a part of being the CEO a large business.
Yeah, my point is that considering someone's character doesn't happen at the level of "is/is-not a slimebag", but at more detailed and specific way.
> people seem to have been trained to think that being unethical is just a part of being the CEO a large business
Not just large. A competitive market can be heavily corrupting, regardless of size (and larger businesses can get away with less, so...).
Qanon started as one of these, obviously at first just to troll. Then it got out of hand and got jacked by people using it for actual propaganda.
As a general rule, you should give very little thought to anonymous 4chan posts.
But
They have leaked real things in the past, in exactly the same way. It may be 5% or less that turn out to be true, but there's the rub. That's why no one can completely dismiss it out of hand (and why were even discussing it on an HN comment thread in the first place).
> Given vast computing resources, the new model was able to solve certain mathematical problems, the person said on condition of anonymity because they were not authorized to speak on behalf of the company. Though only performing math on the level of grade-school students, acing such tests made researchers very optimistic about Q*’s future success, the source said.
however if they've really got something that can eventually solve math problems better than wolfram alpha / mathematica that's great, i got real disappointed early in chatgpt being entirely useless at math.
lemme know when the "AGI" gets bored and starts grinding through the "List of unsolved problems in mathematics" on its own and publishing original research that stands up to scrutiny.
/s, maybe.
If this model is more along the lines of what DeepMind is doing starting from scratch and building up learnings progressively, then depending on (a) how long it was running, (b) what it was fed, and (c) how long until it is expected to hit diminishing returns, then potentially solving grade school math might be a huge deal or a nothing burger.
The details really matter quite a lot.
Once they have a system capable of settling on a single correct answer through its own reasoning rather than yet another probability, it it gets much easier to build better and better AI with a series of incremental improvements. Without this milestone they might've just kept on building LLM's all with the same fundamental limitations, no matter how much computing power they add to them.
Almost. Nor VPN, nor Tor are sufficient to protect you against NSA with global traffic view.
If they sponsor enough exit nodes, they have a view into traffic; similar to a crypto 51% attack
"Investing in OpenAI Global, LLC is a high-risk investment"
"Investors could lose their capital contribution and not see any return"
"It would be wise to view any investment in OpenAI Global, LLC in the spirit of a donation, with the understanding that it may be difficult to know what role money will play in a post-AGI world"
These are some heavy statements. It's fine to accept money from private investors who really do believe in that mission. It seems like you can't continue with the same mission if you're taking billions from public companies which have investors who are very much interested in profit. It's like putting your head in the sand with your hand out.
One of the board members seemed to believe that destroying OpenAI could still be inline with the mission. If that's the case, then they should have accepted slow progress due to funding constraints and killed the idea of creating the for-profit.
Why not? Public companies contribute to charities all the time.
Sure, should have. I think indicators in the last year have pointed to the domain developing much faster than anticipated, with Ilya seemingly incredulous at how well models he spend his career developing, suddenly started working and scaling incredibly well. If they thought billions of "donations" would sustain development in commercial capabilities X within constraints of the mission, but got X^10 way outside constraints, and their explicit goal was to to make sure X^10 doesn't arrive without Y^10 in consideration for safety, it's reasonable for hard liners to reevaluate, and if forces behind the billions get in the way, to burn it all down.
Were they expecting it to never happen and were just ready to throw a mountain of lawyers at any claim?
So it's facially subjective, but not practically once you include resolving a dispute in court.
I'd even argue that Microsoft may have taken advantage of the board's cult-like blindspots and believes that a court-acceptable qualifying AGI isn't a real enough possibility to jeopardize their contract at all.
"an autonomous system that surpasses human capabilities in the majority of economically valuable tasks."
That doesn't sound too subjective to me.
I think the closest way you could truly measure that is to point at industries using it and proving the theory in the market. But by then it’s far too late.
Having some billions of dollars of profits hanging over this issue is a good test of value. If the "is AGI" side can use their AI to help their lawyers defeat the much better/numerous army of lawyers of a billion-dollar corporation, and they succeed, then we're really talking about AGI now.
For example:
Sam spent the last 4 years making controversial moves that benefited Microsoft a lot https://stratechery.com/2023/openais-misalignment-and-micros... at the cost of losing a huge amount of top talent (Dario Amodei and all those who walked out with him to found Anthropic).
In November, Sam loses his job for unknown reasons, and is accused of having molested his younger sister Annie. https://www.themarysue.com/annie-altmans-abuse-allegations-a...
Despite this, his best buddy Satya Nadella immediately gives him a huge job offer without even putting him through an interview loop.
Whistleblower cases take about 12-18 months to process, and the whistleblower eventually gets awarded 10-30% of the monetary sanctions.
If the sanctions end up being $1 billion (a reasonable 10% of the Microsoft investment in OpenAI), you would stand to make between $100M to $300M this way, setting you and your descendants up for generations. Comparably wealthy centi-millionaires include J.K. Rowling, George Lucas, Steven Spielberg, and Oprah Winfrey.
Any member of the public can do this. From the SEC site: "You are not required to be an employee of the company" https://www.sec.gov/whistleblower/frequently-asked-questions...
To try to understand how many people might be racing each other to file the first complaint, I've been tracking the number of points on the above comment.
So far, the parent comment has 3 upvotes (i.e. it peaked at 4 points recently) and 2 downvotes, bringing the current total to 2 points. Its 3 upvotes might be interpretable as 3 people in a sprint to file the first complaint. The two downvotes might even indicate an additional 2 people, having the clever idea to try to discourage others from participating (: ... if true, very clever lol.
Hiring an attorney doesn't actually even cost you anything upfront until you win, if you hire them via what's called a Contingency Fee Arrangement, which you should definitely ask for.
For those interested in a benchmark for how fast you should expect to have to move to be competitive, my guess is that an extremely fast-moving lawyer could sign a retainer agreement with you in 1 hour if you go in person to their office, and could file a complaint in an additional 3-4 hours.
In 18 months we will learn which lucky person was fastest. Stay tuned.
See also the Twitter hashtag #OpenAICharter
https://static.space/sha2-256:83702fe65434e138af0421c560b5da...
(including its digest in the URL -- even if the content is moved elsewhere, we can know for sure whether it was modified)
> be strong agi
The AGI article now seems heavily biased towards GPT/LLM style models and reads more like list of OpenAI achievements at certain points.
I much prefer Gartner's definition of AGI and I think when most informed people talk about about AGI, they are talking about this:
https://www.gartner.com/en/information-technology/glossary/a...
You can see the edit warring in the history, around "economically valuable tasks".
I'm not saying, I'm just saying.
Right now it reads like something a moderately clever shitposter would invent - if there was a letter, why not just leak the letter? - and while everyone's clearly very excited over the events of the past week, I'd like to hope epistemic personal hygiene has not been entirely abandoned.
any publicity…
and just following some major feature releases
premium service has been v slow since the news began. suggesting a massive influx of users
well played(?) Sam
>The maker of ChatGPT had made progress on Q*, which some internally believe could be a breakthrough in the startup's search for superintelligence, also known as artificial general intelligence (AGI)
>Given vast computing resources, the new model was able to solve certain mathematical problems....Though only performing math on the level of grade-school students
I hate to agree with Elon Musk on anything, but I think he was right when he called this a nice marketing stunt this morning. It has major "Snoop Dogg giving up the smoke" vibes.
first time i thought about it was when a friend started to talk about % of stuff of himself... i was: tf did you came with these numbers
Yup. These people are not well.
[0] Altman: “maybe we never build AGI, but…”, in a recent podcast interview.
This AI doomer stuff is such nonsense and I can't believe anybody takes it seriously. As if it's OpenAI's responsibility to save humanity from the pitfalls of AI.
Imagine if we decided to improve our education system and doomers were talking about "hitting the panic button" because students were getting too smart from all the quality education.
Can you elaborate on the education analogy and how it relates to AI doomer stuff?
There are obviously legitimate risks to AI and safety is important, but this is the same for any new technology, and it's governments' responsibilities to ensure that people are safe. AI companies mindlessly slowing down and keeping their tech to themselves does no service to humanity, and if anything is a net-negative due to how tremendously useful this stuff is.
Education is analogous to AI because AI is an enormous education and productivity boost to humanity - sort of like everyone having a personal assistant, programmer, and tutor at their fingertips. This could be used for good and it could be used for bad, but the technology itself is neutral.
Again I want to emphasize that obviously there are downsides that could result from evil people using AI for bad purposes, but that does not justify slowing down AI progress - just like I don't see "people using information for bad purposes" as a legitimate reason for stifling advancement in education or something like Google search.
I have yet to see any convincing argument otherwise. Feel free to provide your counter-perspective.