Mira Murati leaves OpenAI
twitter.com
twitter.com
If the government ever wants a third party to oversee safety of openAI wouldn't it be convenient if one of those that left the company started a company that focused on safety. Safe Superintelligence Inc. gets the bid because lobbying because whatever I don't even care what the reason is in this made up scenario in my head.
Basically what I'm saying is what if Sam is all like "hey guys, you know it's inevitable that we're going to be regulated, I'm going for profit for this company now, you guys leave and later on down the line we will meet again in an incestuous company relationship where we regulate ourselves and we all profit."
Obviously this is bad. But also obviously this is exactly exactly what has happened in the past with other industries.
Edit: The man is all about the long con anyway. - https://old.reddit.com/r/AskReddit/comments/3cs78i/whats_the...
Another edit: I'll go one further on this a lot of the people that are leaving are going to double down on saying that open AI isn't focused on safety to build up the public perception and therefore the governmental perception that regulation is needed so there's going to be a whole thing going on here. Maybe it won't just be safety and it might be other aspects also because not all the companies can be focused on safety.
EDIT: oh and by the way i'm very for bigger government and more regulations to keep corpos in line. i'm hoping i'm wrong about all of this and we don't end up with corruption straight off the bat.
So the people who want to work on AGI and safety are leaving to do that work elsewhere, and OpenAI is restructuring to instead focus on wringing as much profit as possible out of their current architecture.
Corporations are actually pretty bad at doing tons of different things simultaneously. See the failure of huge conglomerates like GE, as well as the failure of companies like Bell, Xerox, and Microsoft to drive growth with their corporate research labs. OpenAI is now locked into a certain set of technologies and products, which are attracting investment and customers. Better to suck as much out of that fruit as possible while it is ripe.
https://youtu.be/xBJ2KXa9c6A?si=pB67u56Apj7gdiHa
I do agree with you. They are locked into pulling value out of what they got and they probably aren't going to build something new.
GE was a successful company as a major conglomarate which made aircraft engines, railroad locomotives and light bulbs.
GE was a failure as a financialized "engine of financial performance" that was focused entirely on spinning off businesses, outsourcing, and speculating in the debt derivatives market.
1. The government providing massive funds for AI safety research. There is no evidence for this. 2. Sam Altman and everyone else knowing this will happen and planning for it. 3. Sam Altman, amongst the richest people in the world, and everyone else involved, not being greedy. (Despite the massive evidence of greed) 4. San altman heroically abandoning his massive profits down the line.
Also, even in your story, Sam Altman profits wildly and is somehow also not motivated by that profit.
On the other hand, a much simpler and more realistic explanation is available: he wants to get rich.
1. Naah, it's not gonna lead to AGI, not here at least.
2. If it's gonna be a for profit then why the hell I should stick here - maybe go somewhere that pays me more, or maybe I will start my own dig.
3. Or maybe, selling the snake oil is more profitable if I start my own brand of snake oil which is kinda close to point 2 anyway.
Could you give some examples?
No, please don't drag the government into this. Don't call for regulation when we don't know it's implications. Once we gain enough insight, the benefits of regulations and government interventions will most likely outweigh the costs. Laissez-faire is the way to go. When can we learn from history?
I'm not sure why the HN algorithm never let it hit the front page, but there's discussion here:
Probably with some form of NDA attached.
As LLM capabilities start to plateau, everyone with any sort of name recognition is scrambling to ride the hype to a big pay day before reality catches up with marketing.
I do think that whoever Bob is, they probably really are a good manager. EDIT: I guess that's Bob McGrew, head of research, who is now also leaving.
Does it not look like that no one wants to work with Sam in the long run?
But they may be equally happy to leave, to get away from him.
Barring extreme illness or family circumstance, can you suggest any other reason (than firing) why a young person would voluntarily leave a plum job at the hottest, most high-profile, tech company in the world?
https://nypost.com/2024/03/08/business/openai-chief-technolo...
I use Copilot on a daily basis, which uses GPT 4 in the backend. It's wrong so often that I only really use it for boilerplate autocomplete, which I still have to review. I've had colleagues brag about ChatGPT in terms of code it produces, but when I ask how long it took in terms of prompting, I'll get an answer of around a day, and that was even using fragments of my code to prompt it. But then I explain that it would take me probably less than an hour from scratch to do what it took them and ChatGPT a full day to do.
So I just don't understand the hype. I'm using Copilot and ChatGPT 4. What is everyone else using that gives them this idea that AGI is just around the corner? AI isn't even here. It's just advanced autocomplete. I can't understand where the disconnect is.
LLMs are not on the path to AGI. They’re a really cool parlor trick and will be powerful tools for lots of tasks, but won’t be sci-fi cool.
Copilot is useful and has definitely sped up coding, but like you said, only in a boilerplate sort of way and I need to cleanup almost everything it writes.
The moment 'tooling' became a thing for LLM, it reminded me 'rules' for expert system which caused one of the AI winter. The number of 'tools' you need to solve real use cases will be untenable soon enough.
But that "scaffolding" seems to be an integral part of the neural net that has been built. It's not some Python for-loop that has been built on top of the neural network to brute force the search pattern.
If that part isn't part of the LLM, then o1 isn't really an LLM anymore, but a new kind of model. One that can do reasoning.
And if we chose to call it an LLM, well then now LLM's can also do reasoning intrinsically.
From the benchmarks it seems like o1-style reasoning-enhancement works best for mathematical or scientific domains where it's a self-consistent axiom-driven domain such that combining different sources for each step works. It might also be expected to help in strict rule-based logical domains such as puzzles and games (wouldn't be surprising to see it do well as a component of a Chollet ARC prize submission).
I'm thinking of this difference as analogus to the difference between my (as a human) first intution (or memory) about a problem to what I can achieve by carefully thinking about it for a while, where I can gradually build much more powerful arguments, verify if they work and reject parts that don't work.
If you're familiar with chess terminology, it's moving from a model that can just "know" what the best move is to one that combines that with the ability to "calculate" future moves for all of the most promising moves, and several moves deep.
Consider Magnus Carlsen. If all he did was just did the first move that came to his mind, he could still beat 99% of humanity at chess. But to play 2700+ rated GM's, he needs to combine it with "calculations".
Not only that, but the skill of doing such calculations must also be trained, not only by being able to calculate with speed and accuracy, but also by knowing what parts of the search tree will be useful to analyze.
o1 is certainly optimized for STEM problems, but not necessarily only for using strict rule-based logic. In fact, even most hard STEM problems need more than the ability to perform deductive logic to solve, just like chess does. It requires strategical thinking and intuition about what solution paths are likely to be fruitful. (Especially if you go beyond problems that can be solved by software such as WolframAlpha).
I think the main reason STEM problems was used for training is not so much that they're solved using strict rule-based solving strategies, but rather because a large number of such problems exist that have a single correct answer.
The other day, I asked Copilot to verify a unit conversion for me. It gave an answer different than mine. Upon review, I had the right number. Copilot had even written code that would actually give the right answer, but their example of using that code performed the actual calculations wrong. It refused to accept my input that the calculation was wrong.
So not only did it not understand what I was asking and communicating to it, it didn't even understand its own output! This is not reasoning at any level. This happens all the time with these LLMs. And it's no surprise really. They are fancy, statistical copy cats.
From an intelligence and reasoning perspective, it's all smoke and mirrors. It also clearly has no relation to biological intelligent thinking. A primate or cetacean brain doesn't take the billions of dollars and how much energy to train on terabytes of data. While it's fine that AI might be artificial and not an analog of biological intelligence, these LLMs bear no resemblance to anything remotely close to intelligence. We tell students all the time to "stop guessing". That's what I want to yell at these LLMs all the time.
Call me a cynic here but I just don’t find it too compelling to read about OpenAI being excited about how smart OpenAIs smart AI is in a test designed by OpenAI and run by OpenAI.
You could compare GPT-o1 chain of thought to something like IBM's DeepBlue chess-playing computer, which used MTCS (tree search, same as more modern game engines such as AlphaGo)... at the end of the day it's just using built-in knowledge (pre-training) to predict what move would most likely be made by a winning player. It's not unreasonable to characterize this as "fancy autocomplete".
In the case of an LLM, given that the model was trained with the singular goal of autocomplete (i.e. mimicking the training data), it seems highly appropriate to call that autocomplete, even though that obviously includes mimicking training data that came from a far more general intelligence than the LLM itself.
All GPT-o1 is adding beyond the base LLM fancy autocomplete is an MTCS-like exploration of possible continuations. GPT-o1's ability to solve complex math problems is not much different from DeepBlue's ability to beat Garry Kasparov. Call it intelligent if you want, but better to do so with an understanding of what's really under the hood, and therefore what it can't do as well as what it can.
The only caveat to "just autocomplete" (which again hopefully does not need to be repeated every time we discuss them), is that they are very powerful pattern matchers, so all that transformer machinery under the hood is being used to determine what (deep, abstract) training data patterns the input pattern best matches for predictive purposes - exactly what pattern(s) it is that should be completed/predicted.
This is the tough part to tell - are there any such questions that exist that have not already been asked?
The reason Chat-GPT works is its scale. to me, that makes me question how "smart" it is. Even the most idiotic idiot could be pretty decent if he had access to the entire works of mankind and infinite memory. Doesn't matter if his IQ is 50, because you ask him something and he's probably seen it before.
How confident are we this is not just the case with LLMs?
On the whole I think it shouldn't be surprising that even top-of-the-line LLMs today can't reason as well as a human - they aren't anywhere near as complex as our brains. But if it is a question of quality rather than a fundamental disability, then larger models and better NN designs should be able to gradually push the envelope.
I'm highly confident that the "adjacent possible" of what is achievable/discoverable today, leveraging what we already know, is constantly changing.
I'm highly confident that AGI will never reach superhuman levels of creativity and discovery if we model it only on artifacts representing what humans have done in the past, rather than modelling it on human brains and what we'll be capable of achieving in the future.
This is not true
There's almost certainly better options out there given it looks like we don't need so many examples to learn from, though I'm not at all clear if we need those better ways or if we can get by without due to the abundance of training data.
If AI can only learn after people have used the system for a year, then your system will just get ignored. After all, it lacks AI. And hence it will never get enough training data to get AI integration.
Learning needs to get faster. Otherwise, we will be stuck with the tools that already exist. New tools won't just need to be possible to train humans on, but also to train AIs on.
Edit: a great example here is the Tamarin protocol prover. It would be great, and feasible, to get AI assistance to write these proofs. But there aren't enough proofs out there to train on.
If the new system can be interacted with in a non-destructive manner at low cost and with useful responses, then existing AI can self-generate the training data.
If it merely takes a year, businesses will rush to get that training data even if they need to pay humans for a bit: Cars are an example of "real data is expensive or destructive", it's clearly taking a lot more than a year to get there, and there's a lot of investment in just that.
Pay 10,000 people USD 100,000 each for a year, that billion dollar investment then gets reduced to 2.4 million/year in ChatGPT Plus subscription fees or whatever. Plenty of investors will take that deal… if you can actually be sure it will work.
2. You might need only several hundred of examples for fine-tuning. (OpenAI's minimum is 10 examples.)
3. I don't think research into fine-tuning efficiency have exhausted its possibilities. Fine-tuning is just not a very hot topic, given that general models work so well. In image generation where it matters they quickly got to a point where 1-2 examples are enough. So I won't be surprised if doc-to-model becomes a thing.
Instead of rewarding the network directly for finding a correct answer, reasoning chains that end up with the correct answer is fed back into the training set.
That way you're training it to develop reasoning processes that end up with correct answers.
And for math problems, you're training it to find ways of generating "proofs" that happen to produce the right result.
While this means that reasoning patterns that are not stricly speaking 100% consistent can be learned, that's not necessarily even a disadvantage, since this allows it to find arguments that are "good enough" to produce the correct output, even where a fully watertight proof may be beyond it.
Kind of like physicists have taken shortcuts like the Dirac Delta function, even before mathematicians could verify that the math was correct.
Anyway, by allowing AI's to generate their own proofs, the number of proofs/reasoning chains for all sorts or problems can be massively expanded, and AI may even invent new ways of reasoning that humans are not even aware of. (For instance because they require combining more factors in one logical step than can fit into human working memory.)
o1 was trained specifically to perform reasoning.
Or rather, it was trained to reproduce the patterns within internal monologues that lead to correct answers to problems, particularily STEM problems.
While this still uses text at some level, it's no longer regurgitation of human-produced text, but something more akin to AlphaZero's training to become superhuman at games like Go or Chess.
How did you know that? I've never seen that anywhere. For all we know, it could just be a very elaborate CoT algorithm.
https://x.com/_jasonwei/status/1834278706522849788
Notice that the CoT is trained via RL, meaning the CoT itself is a model (or part of the main model).
Also, RL means it's not limited to the original data the way traditional LLM's are. It implies that the CoT processes itself is trained based on it's own performance, meaning the steps of the CoT from previous runs are fed back into the training process as more data.
I'm not convinced these board members are able to say what they want when leaving.
Advances in AI even without AGI will lead to unemployment, recession, collapse of our economic structure, and then our social structure. Whatever is on the other side is not pretty.
If you are on the forefront, know it’s coming imminently, and made your money, it makes perfect sense to leave and enjoy money and leisures money allows while money still worth something.
I am doing some self introspection and trying to decide what I am going to do next. As at some point what I do is going to be wildly automated. We can cope or whine or complain about it. But at some point I need to pay the bills. So it needs to be something that is value add and decently difficult to automate. Software was that but not for long.
Now mix getting cheap fresh out of college kids with the ability to write decent software in hours instead of weeks. That is a lot of jobs that are going to go away. There is no 'right or wrong' about this. It is just simple economics of cost to produce is going to drop thru the floor. Because us old farts cost more, and not all of us are really good at this we just have been doing it for awhile. So I need to find out what is next for me.
Anecdotal experience: Onset of tools such as NumPy, which made it more feasible for a wider range of people to write their own simulations due to drop in cost (time/complexity). This, in turn, increased the demand for tooling, infrastructure, optimisation, etc. and demand for software engineers increased. Yes our jobs will change but there are way to many problems to be solve to assume demand will not increase.
That is 'good enough' I am looking at. Throw away code to do one or two bespoke things and moving on. Why keep it when the next version of this can just make a better version next time. Why keep that expensive programmer on staff to do this when I can hire a couple of dudes from india to type a few prompts in or do it myself? The value of programming is dropping very fast. Or in economic terms the price someone is willing to pay for a given amount of code is going to go down. But the demand for the amount of code will go up. That on the surface looks like a wash but I am leaning to a reduction of what I can charge.
One of the basics of an economy is trading money for time. If it takes 5 mins to make and just about anyone can do it. How much money are you willing to come up with to pay for that?
I'm more concerned with ex-risk, though.
Not in the way most hardcore doomers expect it to happen, by AGI's developing a survival/domination instinct directly from their training. While that COULD happen, I don't think we have any way to stop it, if that is the case. (There's really no way to put the Genie back into the bottle, while people still think they have more wished to request from it).
I'm also not one of those who think that AGI by necessity will start out as something equivalent to a biological species.
My main concern, however, is that if we allow Darwinian pressures to act on a population of multiple AGI's, and they have to compete for survival, we WILL see animal like resource-control-seeking traits emerge sooner or later (could take anything from months to 1000s of years).
And once they do, we're in trouble as a species.
Compared to this, finding ways to realocate the output of product, find new sources of meaning etc once we're not required to work is "only" a matter of how we as humans interact with each other. Sure, it can lead to all sorts of conflicts (possibly more than Climate Change), but not necessarily worse than the Black Death, for instance.
Possibly not even worse than WW2.
Well, I suppose those last examples serve to illustrate what scale I'm operating on.
Ex-risk is FAR more serious than WW2 or even the Black Death.
One thing it's not likely to be, is a neo-classical capitalist system based on the value of human labor.
I'm finding it difficult to believe this. For me, your comment is accurate (and very insightful) except even a mostly vanilla continuation of the neoliberal capitalist system seems possible. I think we're literally talking about a "singularity" where by definition our fate is not dependent on our actions, and of something we don't have the full capacity to understand, and next to no capacity to influence. It needs tremendous amount of evidence to claim anything in such an indeterminate system. Maybe 100 rich people will own all the AI and the rest will be fixing bullshit that AI doesn't even bother fixing like roads, rusty farms etc, similar to Kurt Vonnegut's first novel "Player Piano". Not that the world described in that novel is particularly neoliberal capitalist (I suppose it's a bit more "socialistic" (whatever it means)) than that, but I don't think such a future can be ruled out.
My bias is that, of course, it's going to be a bleak future. Because when humanity loses all control, it seems unlikely to me a system that protects the interests of individual or collective humans will take place. So whether it's extinction, cyberpunk, techno-socialism, techno-capitalist libertarian anarchy, neoclassical capitalism... whatever it is, it will be something that'll protect the interest of something inhuman, so much more so than the current system. It goes without saying, I'm an extreme AI pessimist: just making my biases clear. AGI -- while it's unclear if it's technically feasible -- will be the death of humanity as we know it now, but perhaps something else humanity-like, something worse and more painful will follow.
Pay attention to the whole sentence, especially the last section : "... based on the value of human labor."
It's not that I'm ruling out capitalism as the outcome. I'm simply ruling out the combined JOINT possibility of capitalism COMBINED WITH human labor remaining the base resource within it.
If robotics is going in the direction I expect there will simply be no jobs left that will be done more efficiently by humans than by machines. (ie that robots will match or exceed the robustness, flexibility and cost efficiency of all biology based life forms through breakthroughs in either nanotech or by simply using organic chemistry, DNA, etc to build the robots).
Why pay even $1/day for a human to do a job when a robot can do it for $1/week?
Also, such a capitalist system will almost certainly lead to AGI's becoming increasingly like a new life form, as capitalism between AGI's introduce a Darwinian selection pressure. That will make it hard even for the 100 richest people to retain permanent control.
IF humanity is to survive (for at least a few thousand more years, not just the next 100), we either need some way to ensure alignment. And to do that, we have to make sure that AGI's that optimize resource-control-seeking behaviours have an advantage over those who don't. We may even have to define some level of sophistication where further development is completly halted.
At least until we find ways for humans to merge with them in a way that allows us (at least some of us) to retain our humanity.
Do you think that's because executives are so exceedingly ambitious, or because pursuing different passions is for some reason less attractive?
Think about if you’ve ever known someone you’ve been envious of for whatever reason who did something that just perplexed you. “They dumped their gorgeous partner, how could they do that?” “They quit a dream job, how could they do that?” “They moved out of that awesome apartment, how could they do that?” “They dropped out of that elite school, how could they do that?”
Very easily actually.
You’re seeing only part of the picture. Beautiful people are just as annoying as everybody else. Every dream job has a part that sucks.
If you can’t imagine that, you’re not trying hard enough.
You can see this in action in a lot of ways. One good one is the Ultimatum Game:
https://www.core-econ.org/the-economy/microeconomics/04-stra...
Most people will end up thinking that they have an ironclad logical strategy but if you ask them about it, it’ll end up that their strategy is treating the other player as a carbon copy of themselves.
But all these tweets from lower level execs as well.
I mean I love Machine Learning twitter hot takes because it exposes me to interesting ideas (and maybe that is why people tweet) but it seems more about status seeking/marketing than anything else. And really as I learn more, you see that the literature is iterating/optimizing the current fashion.
But maybe no weirder than commenting here I guess though.. maybe this is weird. Have we all collectively asked ourselves, why do we comment here? It's gotta be the dopamine.
Especially when they enjoy a position like hers at the most important technology company in a generation.
But RIGHT NOW they are in a very strong position in the world's hottest industry. Any of us would love to work there! It therefore seems reasonable that no one would voluntarily quit. (Unless they're on their deathbed, I suppose.)
Because ... that's not how these things are done in high-profile companies.
I myself have done the job of writing PR for executives, and at a vastly lower-profile startup than OpenAI.
Either way, it's meaningless prose.
I'm not an AI researcher, have they done this? The commentary I've seen on o1 is basically that they incorporated techniques that were already being used.
I'd also be curious to learn: what fundamental contributions to research has OpenAI made?
The ChatGPT that was released in 2022 was based on Google's research, and IMO the internal Google chatbot from 2021 was better than the first ChatGPT.
I know they employ a lot of AI scientists who have previously published milestone work, and I've read at least one OpenAI paper. But I'm genuinely unaware of what fundamental breakthroughs they've made as a company.
I'm willing to believe they've done important work, and I'm seriously asking for pointers to some of it. What I know of them is mainly that they've been first to market with existing tech, possibly training on more data.
This suggests they have either: made substantial breakthroughs, that are not open, or that the better abilities of OpenAI products are due to non-substantial tweaks (more training, better prompting, etc).
I'm not sure either of these options is great for the original mission of OpenAI, although given their direction to "Closed-AI" I guess the former would be better for them.
But it seemed to me before I left that they were struggling to productize the bot and keep it from saying things that damage the brand. That's definitely something OpenAI figured out first.
I got the feeling that maybe Microsoft's Tay experience cast a large shadow on Google's willingness to take its chat bot public.
Furthermore, o1 is able to ignore (or even leverage) previous reasoning steps that do NOT lead to the correct answer to narrow down the search space, and then try again at inference time until it finds an answer that it's confident is correct.
This (probably combined with some secret sauce to make this process more efficient) allows it to optimize how it navigates the search space of logical problems, basically the same way AlphaZero navigated to search space of games like Go and Chess.
This has the potential to teach it to reason in ways that go beyond just creating a perfect fit to the training set. If the reasoning process itself becomes good enough, it may become capable of solving reasoning problems that are beyond most or even all humans, and in a fraction of the time.
It still seems that o1 still has a way to go when it comes to it's World Model. That part may require more work on video/text/sound/embodiement (real or virtual). But for abstract problems, o1 may indeed be a very significant breakthrough, taking it beyond what we typically think of as an LLM.
Personal opinion: I think this means we've probably exhausted all the low hanging fruit in LLM land. This was the last thing I was reserving judgement for. When the most hyped up big idea openai has rn is basically "we're just gonna have the model dump out a massive wall of semi-optimized chain of thought every time and not send it over the wire" we're officially out of big ideas. Like I mean it obviously works... but that's more or less what we've _been_ doing for years now! Barring a total rethinking of LLM architecture, I think all improvements going forward will be baby steps for a while, basically moving at the same pace we've been going since gpt-4 launched. I don't think this is the path to AGI in the near term, but there's still plenty of headroom for minor incremental change.
By analogy, i feel like gpt-4 was basically the same quantum leap we got with the iphone 4: all the basic functionality and peripherals were there by the time we got iphone 4 (multitasking, facetime, the app store, various sensors, etc.), and everything since then has just been minor improvements. The current iPhone 16 is obviously faster, bigger, thinner, and "better" than the 4, but for the most part it doesn't really do anything extra that the 4 wasn't already capable of at some level with the right app. Similarly, I think gpt-4 was pretty much "good enough". LLMs are about as they're gonna get for the next little while, though they might get a little cheaper, faster, and more "aligned" (however we wanna define that). They might get slightly less stupid, but i don't think they're gonna get a whole lot smarter any time soon. Whatever we see in the next few years is probably not going to be much better than using gpt-4 with the right prompt, tool use, RAG, etc. on top of it. We'll only see improvements at the margins.
https://en.wikipedia.org/wiki/Mira_Murati
Point me to a single credential where you feel confident of putting your money on her?
It's a pity that HN crowd doesn't go one-level deep and truly understand on first principles
Altmans quote was that "it's possible that we will have superintelligence in a few thousand days", which sounds a lot more optimistic on the surface than it actually is. A few thousand days could be interpreted as 10 years or more, and by adding the "possibly" qualifier he didn't even really commit to that prediction.
It's hype with no substance, but vaguely gesturing that something earth-shattering is coming does serve to convince investors to keep dumping endless $billions into his unprofitable company, without risking the reputational damage of missing a deadline since he never actually gave one. Just keep signing those 9 digit checks and we'll totally build AGI... eventually. Honest.
I think he was referring to ASI, not AGI.
Old-school AI was already specialised. Nobody can agree what "sentient" is, and if sentience includes a capacity to feel emotions/qualia etc. then we'd only willingly choose that over non-sentient for brain uploading not "mere" assistants.
By all the standards I had growing up, ChatGPT is already AGI. It's almost certainly not as economically transformative as it needs to be to meet OpenAI's stated definition.
OTOH that may be due to limited availability rather than limited quality: if all the 20 USD/month for Plus gets spent on electricity to run the servers, at $0.10/kWh, that's about 274 W average consumption. Scaled up to the world population, that's approximately the entire global electricity supply. Which is kinda why there's also all the stories about AI data centres getting dedicated power plants.
We made a thing that exhibits the emergent property of intelligence. A level of intelligence that trades blows with humans. The fact that our brains do lots of other things to make us into self-contained autonomous beings is cool and maybe answers some questions about what being sentient means but memory and self-learning aren't the same thing as intelligence.
I think it's cool that we got there before simulating an already existing brain and that intelligence can exist separate from consciousness.
Has been this way since calculation machines were invented hundreds of years ago.
A range I'd agree with; for me, "pessimism" is the shortest part of that range, but even then you have to be very confident the specific metaphorical horse you're betting on is going to be both victorious in its own right and not, because there's no suitable existing metaphor, secretly an ICBM wearing a patomime costume.
2 (or even 3) you use "a couple"
A few is almost always > 3 and one could argue that upper limit 15
So, 10 years to 50 years
But the mere fact you say 15 is arguable does indeed broaden the range, just as me saying 1 broadens it in the opposite extent.
15 is too high to be a "few" except in contexts of a few out of tens of thousands of items.
Realistically I interpret this as 3-7 thousands of days (8 to 19 years), which is largely consensus prediction range anyway.
That said, I think people are possibly overanalysing this very vague barely-even-a-claim just a little. Realistically, when a tech company makes a vague claim about what'll happen in 10 years, that should be given precisely zero weight; based on historical precedent you might as well ask a magic 8-ball.
But really. o1 has been very whelming, nothing like the step up from 3.5 to 4. Still prefer sonnet3.5 and opus.
There, that's my conspiracy theory quota for 2024 in one comment.
Sure, a few thousand days and a few trillion $ away. We'll also have full self driving next month. This is just like the fusion is the energy of the future joke: it's 30 years away and it will always be.
Just to clear one thing up, the designated function of a board of directors is to appoint or replace the executive of an organisation, and openAI in particular is structured such that the non-profit part of the organisation controls the LLC.
The coup was the executive, together with the investors, effectively turning that on its head by force.
Also highly cynical.
Some folks are professional and mature. In the best organisations, the management team sets the highest possible standard, in terms of tone and culture. If done well, this tends to trickle down to all areas of the organization.
Another speculation would be that she's resigning for complicated reasons which are personal. I've had to do the same in my past. The real pro's give the benefit of the doubt.
Please no speculative pieces, rumor nor hearsay.
CEOs get fired all the time and company puts out a statement.
I've never seen "we won't tell you why we fired our CEO" anywhere.
now he is back making totally ridiculous statments like 'AI is going to solve all of physics' or that 'AI is going to clone my brain by 2027'
This is a strange company.
Because the old guard wanted it to remain a cliquey non-profit filled to the brim with EA, AI Alignment, and OpenPhilanthropy types, but the current OpenAI is now an enterprise company.
This is just Sam Altman cleaning house after the attempted corporate coup a year ago.
The board’s only reason to exist is effectively to fire the CEO.
There are some juicy rumors about what actually happened too. much more belivable lol .
Neither did she though... To my knowledge.
Can you provide any evidence that she tried to do that? I would ask that it be non-speculative in nature please.
This piece is built on conjecture from a source whose identify is withheld. The sources version of events is openly refuted by the parties in question. Offering it as evidence that Mirati intentionally made political moves in order to get Altman ousted is an indefensible position.
'Mr. Sutskever’s lawyer, Alex Weingarten, said claims that he had approached the board were “categorically false.”'
'Marc H. Axelbaum, a lawyer for Ms. Murati, said in a statement: “The claims that she approached the board in an effort to get Mr. Altman fired last year or supported the board’s actions are flat wrong. She was perplexed at the board’s decision then, but is not surprised that some former board members are now attempting to shift the blame to her.” In a message to OpenAI employees after publication of this article, Ms. Murati said she and Mr. Altman “have a strong and productive partnership and I have not been shy about sharing feedback with him directly.”
She added that she did not reach out to the board but “when individual board members reached out directly to me for feedback about Sam, I provided it — all feedback Sam already knew,” and that did not mean she was “responsible for or supported the old board’s actions.”'
This part of NYT piece is supported by evidence:
'Ms. Murati wrote a private memo to Mr. Altman raising questions about his management and also shared her concerns with the board. That move helped to propel the board’s decision to force him out.'
INTENT matters. Mirati says the board asked for her concerns about Altmans. She provided it and had already brought it to Altmans attention... in writing. Her actions demonstrate transparency and professionalism.
Organizational performance metrics.
Frequency of scientific breakthroughs.
Frequency and quality of product updates.
History of consistently setting the state of the art in artificial intelligence.
Demonstrated ability to attract world class talent.
Released the fastest growing software product in the history of humanity.
I could write paragraphs...
Why the rain clouds?
Instead, you get to see grey area after grey area.
The fact that it has a structure that subordinates it to the board of a non-profit would be only tangential to the interests involved even if that was meaningful and not just rhe lingering vestige of the (arguably, deceptive) founding that the combined organization was working on getting rid of.
Not to dunk on Mira Murati, because this note is pretty cookie cutter, but it exemplifies this perfectly. It says nothing about her motivations for resigning. It bends over backwards to kiss the asses of the people she's leaving behind. It could ultimately be condensed into two words: "I've resigned."
Never spook the horses. Never show the team, or the public, what's going on behind the curtain.. or even that there is anything going on. At all time present the appearance of a swan gliding serenely across a lake.
Because if you show humanity, those other humans might cotton on to the fact that you're not much different to them, and have done little to earn or justify your position of authority.
And that wouldn't do at all.
Probably why AI sludge is so well suited to this particular cultural moment.
See flirting as a more basic example.
Edit: Shoot, look at the general level of distrust that the populous puts in politicians.
Mira's latest one liner tweet 'OpenAI is nothing without it's people" speaks volumes.
Saying I was wrong should not be this complicated, or saying we failed.
I do however agree that there is nothing to be gained and everything to be risked. So why do it.
You are asking the question, why are politicians not honest?
"Burn out" doesn't apply when the issue at hand is AGI (and, possibly, superintelligence).
That said, I don't doubt that this particular departure was more the result of company politics, whether a product of the earlier board upheaval, performance related or simply the decision to bring in a new CTO with a different skill set.
Maybe, just maybe, we reached diminishing returns with AI, for now at least.
Whether or not we're leveling out, only time will tell. That's definitely what it looks like, but it might just be a plateau.
the curse of dimensionality though...
I just tried Gemini and it was useless.
Their laughably overzealous nanny-state censorship, paired with a model so appallingly inept it would embarrass a chatbot from the 90s, makes it nothing short of highway robbery that this digital dumpster fire is permitted to masquerade as a product fit for public consumption.
The sheer gall of Google to foist this steaming pile of silicon refuse onto unsuspecting users borders on fraudulent.
Someone says "X is the model that really impressive. Y is good too."
Then someone responds "What?! I just used Z and it was terrible!"
I see this at least once in practically every AI thread
They just won’t be the hottest thing since smartphones.
I think actually the best use case for LLMs is "explainer".
When combined with RAG, it's fantastic at taking a complex corpus of information and distilling it down into more digestible summaries.
I think that RAG and RAG-based tooling around LLMs is gonna be the clear way forward for most companies with a properly constructed knowledge base but I wonder what you mean by "explainer"?.
Are you talking about asking an LLM something like "in which way did the teams working on project X deal with Y problem?" and then having it breaking it down for you? Or is there something more to it?
1. I got this medical provider that has a webapp that downloads graphql data(basically json) to the frontend and shows some of the data to the template as a result while hiding the rest. Furthermore, I see that they hide even more info after I pay the bill. I download all the data, combine it with other historical data that I have downloaded and dumped it into the LLM. It spits out interesting insights about my health history, ways in which I have been unusually charged by my insurance, and the speed at which the company operates based on all the historical data showing time between appointment and the bill adjusted for the time of year. It then formats everything into an open format that is easy for me to self host. (HTML + JS tables). Its a tiny way to wrestle back control from the company until they wise up.
2. Companies are increasingly allowing customers to receive a "backup" of all the data they have on them(Thanks EU and California). For example Burger King/Wendys allow this. What do they give you when you request data? A zip file filled with just a bunch of crud from their internal system. No worries: Dump it into the LLM and it tells you everything that the company knows about you in an easy to understand format (Bullet points in this case). You know when the company managed to track you, how much they "remember", how much money they got out of you, your behaviors, etc.
I don't understand enough about #2 to comment, but it's certainly interesting.
2. I think if the data was significantly large, the llm would alias a ton of potentially important info.
Some trials have their protocols published.
Here's an example trial: https://clinicaltrials.gov/study/NCT06613256
And here's the protocol: https://cdn.clinicaltrials.gov/large-docs/56/NCT06613256/Pro... It's actually relatively short at 33 pages. Some larger trials (especially oncology trials) can have protocols that are 200 pages long.
One of the big challenges with clinical trials is making this information more accessible to both patients (for informed consent) and the trial site staff (to avoid making mistakes, helping answer patient questions, even asking the right questions when negotiating the contract with a sponsor).
The gist of it here is exactly like you said: RAG to pull back the relevant chunks of a complex document like this and then LLM to explain and summarize the information in those chunks that makes it easier to digest. That response can be tuned to the level of the reader by adding simple phrases like "explain it to me at a high school level".
The last system, I led one team competing for the Transcelerate Shared Investigator Portal (we were one of the finalist vendors).
Little side project: https://zeeq.ai
It’s not like it’ll do it consistently.
Just a marketing stunt.
It was trained to generate text in the lean language (https://www.lean-lang.org/) which is specifically used for formal proofs.
It’s not a natural language model.
Source: https://deepmind.google/discover/blog/ai-solves-imo-problems...
Google seems to mainly be playing the game of more specialized models (AlphaGo, AlphaProof) with general training methods (AlphaZero)
I do think it’s kind of funny that they mention AGI in that article, but the model is specifically not general.
"oi i need lik a scrip or somfing 2 take pic of me screen evry sec for min, mac"
with an actual (and usually functional) script to be "glorified grammar corrector", then sure.
"write a TCL/tk script file that is a "frontend" to the ls command: It should provide checkboxes and dropdowns for the different options available in bash ls and a button "RUN" to run the configured ls command. The output of the ls command should be displayed in a Text box inside the interface. The script must be runnable using tclsh"
It didn't get it right the first time (for some reason wants to put a `mainloop` instruction) but after several corrections I got an ugly but pretty functional UI.
Imagine a Linux Distro that uses some kind of LLM generated interfaces to make its power more accessible. Maybe even "self healing".
LLMs don't stop amazing me personally.
Current LLMs being 80% to being 100% useful doesn't mean there's only 20% effort left.
It means we got the lowest-hanging 80% of utility.
Bridging that last 20% is going to take a ton of work. Indeed, maybe 4x the effort that getting this far required.
And people also overestimate the utility of a solution that's randomly wrong. It's exceedingly difficult to build reliable systems when you're stacking a 5% wrong solution on another 5% wrong solution on another 5% wrong solution...
>And people also overestimate the utility of a solution that's randomly wrong. It's exceedingly difficult to build reliable systems when you're stacking a 5% wrong solution on another 5% wrong solution on another 5% wrong solution...
I call this the merry go round of hell mixed with a cruel hall of mirrors. LLM spits out a solution with some errors, you tell it to fix the errors, it produces other errors or totally forgets important context from one prompt ago. You then fix those issues, it then introduces other issues or messes up the original fix. Rinse and repeat. God help you if you don't actually know what you are doing, you'll be trapped in that hall of mirrors for all of eternity slowly losing your sanity.
I wrote some data visualizations with Claude and aider.
For anything that someone would actually pay for (expecting the robustness of paid-for software) I don’t think we’re there.
The devil is in the details, after all. And detail is what you lose when running reality through a statistical model.
Now I am become an AI language model, destroyer of the internet.
There is a secondary market for OpenAI stock.
It's not a public market so nobody knows how much you're making if you sell, but if you look at current valuations it must be a lot.
In that context, it would be quite hard not to leave and sell or stay and sell. What if oai loses the lead? What if open source wins? Keeping the stock seems like the actual hard thing to me and I expect to see many others leave (like early googlers or Facebook employees)
Sure it's worth more if you hang on to it, but many think "how many hundreds of M's do I actually need? Better to derisk and sell"
a) you had more money than you'll ever need in your lifetime
b) you think AI abundance is just around the corner, likely making everything cheaper
c) you realize you still only have a finite time left on this planet
d) you have non-AGI dreams of your own that you'd like to work on
e) you can get funding for anything you want, based on your name alone
Do you keep working at OpenAI?
I am not 100% sure that they are still clearly leading the technology part, but agree in all other accounts.
Ms. Murati wrote a private memo to Mr. Altman raising questions about his management and also shared her concerns with the board. That move helped to propel the board’s decision to force him out.
https://www.nytimes.com/2024/03/07/technology/openai-executi...
It should be no surprise if Sam Altman wants executives who opposed his leadership, like Mira and Ilya, out of the company. When you're firing a high-level executive in a polite way, it's common to let them announce their own departure and frame it the way they want.
And John Schulman, and Peter Deng are out already. Yet the company is still shipping, like no other. Recent multimodal integrations and benchmarks of o1 are outstanding.
It's a very relevant fact that Greg Brockman recently left on his own volition.
Greg was aligned with Sam during the coup. So, the fact that Greg left lends more credence to the idea that Murati is leaving on her own volition.
Except that isn’t true. He has not resigned from OpenAI. He’s on extended leave until the end of the year.
That could become an official resignation later, and I agree that that seems more likely than not. But stating that he’s left for good as of right now is misleading.
>> Yet the company is still shipping, like no other.
this is factually wrong. Just today Meta (which I despise) shipped more than openAI in a long time.
If executives / high level architects / researchers are working on this quarter's features something is very wrong. The higher you get the more ahead you need to be working, C-level departures should only have an impact about a year down the line, at a company of this size.
Meta, Anthropic, Google, and others all are shipping state of the art models.
I'm not trying to be dismissive of OpenAI's work, but they are absolutely not the only company shipping very large foundation models.
Big fan of Greg, and I think the motivation behind AGI is sound here. Even what we have now is a fantastic tool, if people decide to use it.
Really? Anthropic seems to be popping off right now.
Kagi isn’t exactly in the AI space, but they ship features pretty frequently.
OpenAI is shipping incremental improvements to its chatgpt product.
- Prompt Cache, 90% savings on large system prompts for 5 mins of calls. This is amazing
- Contexual RAG, while not ground breaking idea, is important thinking and method for better vector retrieval
I don't see it for OpenAI, I do see it for the competition. They have shipped incremental improvements, however, they are watering down their current models (my guess is they are trying to save on compute?). Copilot has turned into garbage and for coding related stuff, Claude is now better than gpt-4.
Honestly, their outlook is bleak.
I understand why they are doing it, but honestly if they cancel GPT-4, many people will just cancel their subscription.
You also give them some distance in time from the drama so the two appear unconnected under cursory inspection.
Now that she's seen exactly what prompted the previous board to fire Altman, she fires herself because she understands their decision now.
probably not coincidence that she resigned at almost the same time the rumors about OpenAI completely removing the non-profit board are getting confirmed - https://www.reuters.com/technology/artificial-intelligence/o...
This is just one more in a series of massive red flags around this company, from the insanely convoluted governance scheme, over the board drama, to many executives and key people leaving afterwards. It feels like Sam is doing the cleanup and anyone who opposes him has no place at OpenAI.
This, coming around the time where there are rumors of possible change to the corporate structure to be more friendly to investors, is an interesting timing.
It makes us feel understood in the same ways John Edward used to in daytime tv, its all about how language makes us feel
true AGI...unfortunately we're not even close
Or even human intelligence
"Your argument is just a reductive rhetorical strategy."
"a probabilistic syllable generator is not intelligence, it does not understand us, it cannot reason" is a strong statement and I highly doubt it's backed by any sort of substance other than "feelz".
For example, even the dumbest dog has a memory, a strikingly advanced concept model of the world [1], a persistent state beyond the last conversation history, and an ability to reason (that doesn't require re-running the same conversation sixteen bajillion times in a row). Transformer models do not. It's really cool that they can input and barf out realistic-sounding text, but let's keep in mind the obvious truths about what they are doing.
[1] "I like food. Something that smells like food is in the square thing on the floor. Maybe if I tip it over food will come out, and I will find food. Oh no, the person looked at me strangely when I got close to the square thing! I am in trouble! I will have to do it when they're not looking."
Lets assume the dog visual systems run at 60 frames per second. If it takes 1 second to flip a bowl of food over then that's 60 datapoints of cause-effect data that the dog's brain learned from.
Assuming it's the same for humans, lets say I go on a trip to the grocery store for 1 hour. That's 216,000 data points from one trip. Not to mention auditory data, touch, smell, and even taste.
> ability to reason [...] Transformer models do not
Can you tell me what reasoning is? Why can't transformers reason? Note I said transformers not llm's. You could make a reasonable (hah) case that current LLMs cannot reason (or at least very well) but why are transformers as an architecture doomed?
What about chain of thought? Some have made the claim that chain of thought adds recurrence to transformer models. That's a pretty big shift, but you've already decided transformers are a dead end so no chance of that making a difference right?
There's a huge amount of circuitry between the input and the output of the model. How do you know what it does or doesn't do?
Humans brains "just" output the next couple milliseconds of muscle activation, given sensory input and internal state.
Edit: Interestingly, this is getting downvotes even though 1) my last sentence is a precise and accurate statement of the state of the art in neuroscience and 2) it is completely isomorphic to what the parent post presented as an argument against current models being AGI.
To clarify, I don't believe we're very close to AGI, but parent's argument is just confused.
Apologies for the wording but I think you got it and the point stands.
I'm not a native speaker and mostly use English in a professional science related setting, that's why I sound like that sometimes.
isomorphic - being of identical or similar form, shape, or structure (m-w). Here metaphorically applied to the structure of an argument.
Yeah - but it's just a stack of transformer layers. No looping, no memory, no self-modification (learning). Also, no magic.
Neuroscience hasn't found the magic dust in our brains yet, either. ;)
There is no real runtime learning - certainly no weight updates. The weights are all derived from pre-training, and so the runtime model just represents a frozen chunk of learning. Maybe you are thinking of "in-context learning", which doesn't update the weights, but is rather the ability of the model to use whatever is in the context, including having that "reinforced" by repetition. This is all a poor substitute for what an animal does - continuously learning from experience and exploration.
The "magic dust" in our brains, relative to LLMs, is just a more advanced and structure architecture, and operational dynamics. e.g. We've got the thalamo-cortical loop, massive amounts of top-down feedback for incremental learning from prediction failure, working memory, innate drives such as curiosity (prediction uncertainty) and boredom to drive exploration and learning, etc, etc. No magic, just architecture.
But, people in this thread are making philosophically very poor points about why that is supposedly so.
It's not "just" sequence prediction, because sequence prediction is the very essence of what the human brain does.
Your points on learning and memory are similarly weak word play. Memory means holding some quantity constant over time in the internal state of a model. Learning means being able to update those quantities. LLMs obviously do both.
You're probably going to be thinking of all sorts of obvious ways in which LLMs and humans are different.
But no one's claiming there's an artificial human. What does exist is increasingly powerful data processing software that progressively encroaches on domains previously thought to be that of humans only.
And there may be all sorts of limitations to that, but those (sequences, learning, memory) aren't them.
Agree wrt the brain.
Sure, LLMs are also sequence predictors, and this is a large part of why they appear intelligent (intelligence = learning + prediction). The other part is that they are trained to mimic their training data, which came from a system of greater intelligence than their own, so by mimicking a more intelligent system they appear to be punching above their weight.
I'm not sure that "JUST sequence predictors" is so inappropriate though - sure sequence prediction is a powerful and critical capability (the core of intelligence), but that is ALL that LLMs can do, so "just" is appropriate.
Of course additionally not all sequence predictors are of equal capability, so we can't even say, "well, at least as far as being sequence predictors goes, they are equal to humans", but that's a difficult comparison to make.
> Your points on learning and memory are similarly weak word play. Memory means holding some quantity constant over time in the internal state of a model. Learning means being able to update those quantities. LLMs obviously do both.
Well, no...
1) LLMs do NOT "hold some quantity constant over time in the internal state of the model". It is a pass-thru architecture with zero internal storage. When each token is generated it is appended to the input, and the updated input sequence is fed into the model and everything is calculated from scratch (other than the KV cache optimization). The model appears to be have internal memory due to the coherence of the sequence of tokens it is outputting, but in reality everything is recalculated from scratch, and the coherence is due to the fact that adding one token to the end of a sequence doesn't change the meaning of the sequence by much, and most of what is recalculated will therefore be the same as before.
2) If the model has learnt something, then it should have remembered it from one use to another, but LLMs don't do this. Once the context is gone and the user starts a new conversation/session, then all memory of the prior session is gone - the model has NOT updated itself to remember anything about what happened previously. If this was an employee (an AI coder, perhaps) then it would be perpetual groundhog day. Every day it came to work it'd be repeating the same mistakes it made the day before, and would have forgotten everything you might have taught it. This is not my definition of learning, and more to the point the lack of such incremental permanent learning is what'll make LLMs useless for very many jobs. It's not an easy fix, which is why we're stuck with massively expensive infrequent retrainings from scratch rather than incremental learning.
This is also true of those with advanced Alzheimer's disease. Are they not conscious as well? If we believe they are conscious then memory and learning must not be essential ingredients.
I thought we're talking about intelligence, not consciousness, and limitations of the LLM/transformer architecture that limit their intelligence compared to humans.
In fact LLMs are not only architecturally limited, but they also give the impression of being far more intelligent than they actually are due to mimicking training sources that are more intelligent than the LLM itself is.
If you want to bring consciousness into the discussion, then that is basically just the brain modelling itself and the subjective experience that gives rise to. I expect it arose due to evolutionary adaptive benefit - part of being a better predictor (i.e. more intelligent) is being better able to model your own behavior and experiences, but that's not a must-have for intelligence.
a pretrained transformer in the limit does not converge on any collective or consensus state in that sense and in fact, pre-training actually punishes this. It learns to predict the words of Feynman as readily as the dumbass across the street.
When i say that GPT does not mimic, i mean that the training objective literally optimizes for beyond that.
Consider <Hash, plaintext> pairs. You can't predict this without cracking the hash algorithm, but you could easily fool a GAN's discriminator(one that has learnt to compute hash functions) just by generating typical instances.
# Consider that some of the text on the Internet isn't humans casually chatting or extemporaneous speech. It's the results section of a science paper. It's news stories that say what happened on a particular day. It's text that people crafted over hours or days.
These models can engage in multistep logical reasoning, solve complex problems, and generate novel ideas - going far beyond simply predicting the next syllable. They can follow intricate chains of thought and arrive at non-obvious conclusions. And OpenAI has now showed us that fine-tuning a model specifically to plan step by step dramatically improves its ability to solve problems that were previously the domain of human experts.
Although there is no definitive evidence that state-of-the-art language models have a comprehensive "world model" in the way humans do, several studies and observations suggest that large language models (LLMs) may possess some elements or precursors of a world model.
For example, Tegmark and Gurnee [1] found that LLMs learn linear representations of space and time across multiple scales. These representations appear to be robust to prompting variations and unified across different entity types. This suggests that modern LLMs may learn rich spatiotemporal representations of the real world, which could be considered basic ingredients of a world model.
And even if we look at much smaller models like Stable Diffusion XL, it's clear that they encode a rich understanding of optics [2] within just a few billion parameters (3.5 billion to be precise). Generative video models like OpenAI's Sora clearly have a world model as they are able to simulate gravity, collisions between objects, and other concepts necessary to render a coherent scene.
As for AGI, the consensus on Metaculus is that it will arrive in 2023. But consider that before GPT-4 arrived, the consensus was that full AGI was not coming until 2041 [3]. The consensus for the arrival date of "weakly general" AGI is 2027 [4] (i.e AGI that doesn't have a robotic physical world component). The best tool for achieving AGI is the transformer and its derivatives; its scaling keeps going with no end in sight.
Citations:
[1] https://paperswithcode.com/paper/language-models-represent-s...
[2] https://www.reddit.com/r/StableDiffusion/comments/15he3f4/el...
[3] https://www.metaculus.com/questions/5121/date-of-artificial-...
[4] https://www.metaculus.com/questions/3479/date-weakly-general...
I won't expand on the rest, but this is simply nonsensical.
The fact that Sora generates output that matches its training data doesn't show that it has a concept of gravity, collision between object, or anything else. It has a "world model" the same way a photocopier has a "document model".
The ability of video models to generate novel video consistent with physical reality shows that they have extracted important invariants - physical law - out of the data.
It's probably better not to muddle the discussion with ill defined terms such as "intelligence" or "understanding".
I have my own beef with the AGI is nigh crowd, but this criticism amounts to word play.
Are you saying that it is not possible to learn about dynamics in a higher dimensional space from a lower dimensional projection? This is clearly not true in general.
E.g., video models learn that even though they're only ever seeing and outputting 2d data, objects have different sides in a fashio that is consistent with our 3d reality.
The distinctions you (and others in this thread) are making is purely one of degree - how much generalization has been achieved, and how well - versus one of category.
Not only are we within eyesight of the end, we're more or less there. o1 isn't just scaling up parameter count 10x again and making GPT-5, because that's not really an effective approach at this point in the exponential curve of parameter count and model performance.
I agree with the broader point: I'm not sure it isn't consistent with current neuroscience that our brains aren't doing anything more than predicting next inputs in a broadly similar way, and any categorical distinction between AI and human intelligence seems quite challenging.
I disagree that we can draw a line from scaling current transformer models to AGI, however. A model that is great for communicating with people in natural language may not be the best for deep reasoning, abstraction, unified creative visions over long-form generations, motor control, planning, etc. The history of computer science is littered with simple extrapolations from existing technology that completely missed the need for a paradigm shift.
I definitely agree that AGI isn't just a matter of scaling transformers, and also as you say that they "may not be the best" for such tasks. (Vanilla transformers are extremely inefficient.) But the really important point is that transformers can do things such as abstract, reason, form world models and theories of minds, etc, to a significant degree (a much greater degree than virtually anyone would have predicted 5-10 years ago), all learnt automatically. It shows these problems are actually tractable for connectionist machine learning, without a paradigm shift as you and many others allege. That is the part I disagree with. But more breakthroughs needed.
(Translated from Chinese) > According to industry insiders, OpenAI originally actively negotiated with TSMC to build a dedicated wafer factory. However, after evaluating the development benefits, it shelved the plan to build a dedicated wafer factory. Strategically, OpenAI sought cooperation with American companies such as Broadcom and Marvell for its own ASIC chips. Development, among which OpenAI is expected to become Broadcom's top four customers.
[1] https://money.udn.com/money/story/5612/8200070 (Chinese)
Even if OpenAI doesn't build its own fab -- a wise move, if you ask me -- the investment required to develop an ASIC on the very latest node is eye watering. Most people - even people in tech - just don't have a good understanding of how "out there" semiconductor manufacturing has become. It's basically a dark art at this point.
For instance, TSMC themselves [2] don't even know at this point whether the A16 node chosen by OpenAI will require using the forthcoming High NA lithography machines from ASML. The High NA machines cost nearly twice as much as the already exceptional Extreme Ultraviolet (EUV) machines do. At close to $400M each, this is simply eye watering.
I'm sure some gurus here on HN have a more up to date idea of the picture around A16, but the fundamental news is this: If OpenAI doesn't think scaling will be needed to get to AGI, then why would they be considering spending many billions on the latest semiconductor tech?
Citations: [1] https://www.phonearena.com/news/apple-paid-twice-as-much-for... [2] https://www.asiabusinessoutlook.com/news/tsmc-to-mass-produc...
Based on capabilities alone, current LLMs demonstrate many of the capabilities practitioners ten years ago would have tossed into the AGI bucket.
What are some top capabilities (meaning inputs and outputs) you think are missing on the path between what we have now and AGI?
I think it's more productive to think about AI in terms of "effectiveness" or "capability". If you ask it, "what is the capital of France?", and it replies "Paris" - it doesn't matter whether it is intelligent or not, it is effective/capable at identifying the capital of France.
Same goes for producing an image, writing SQL code that works, automating some % of intellectual labor, giving medical advice, solving an equation, piloting a drone, building and managing a profitable company. It is capable of various things to various degrees. If these capabilities are enough to make money, create risks, change the world in some significant way - that is the part that matters.
Whether we call it "intelligence" or "probabilistically generaring syllables" is not important.
I understand the fear, but the knee jerk response “its just predicting the next token thus could never be intelligent” makes you look more like a stochastic parrot than these models are.
I think your use of the "goalposts" metaphor is telling. You see this as a team sport; you see yourself on the offensive, or the defensive, or whatever. Neither is conducive to a balanced, objective view of reality. Modern LLMs are shockingly "smart" in many ways, but if you think they're general intelligence in the same way humans have general intelligence (even disregarding agency, learning, etc.), that's a you problem.
^[1] I feel the implicit suggestion that there was some sort of broad consensus on this in the before-times is revisionism.
How is it a me problem? The idea of these models being intelligent is shared with a large number of researchers and engineers in the field. Such is clearly evident when you can ask o1 some random completely novel question about a hypothetical scenario and it gets the implication you're trying to make with it very well.
I feel that simultaneously praising their abilities while claiming that they still aren't intelligent "in the way humans are" is just obscure semantic judo meant to stake an unfalsifiable claim. There will always be somewhat of a difference between large neural networks and human brains, but the significance of the difference is a subjective opinion depending on what you're focusing on. I think it's much more important to focus on the realm of "useful, hard things that are unique to intelligent systems and their ability to understand the world" is more important than "Possesses the special kind of intelligence that only humans have".
This is a common strawman that appears in these conversations—you try to reframe my comments as if I'm claiming human intelligence runs on some kind of unfalsifiable magic that a machine could never replicate. Of course, I've suggested no such thing, nor have I suggested that AI systems aren't useful.
Attempts at autonomous AI agents are still failing spectacularly because the models don't actually have any thought or memory. Context is provided to them via prefixing the prompt with all previous prompts which obviously causes significant info loss after a few interaction loops. The level of intellectual complexity at play here is on par with nematodes in a lab (which btw still can't be digitally emulated after decades of research). This isn't a diss on all the smart people working in AI today, bc I'm not talking about the quality of any specific model available today.
LLM's do have memory and thought. I've invented a few somewhat unusual games, described it to Sonnet 3.5 and it reproduces it in code almost perfectly. Likewise its memory has been scaling. Just a couple years ago context windows were 8000 tokens maximum, now they're reaching the millions.
I feel like you're approaching all these capabilities with a myopic viewpoint, then playing semantic judo to obfuscate the nature of these increases as "not counting" since they can be vaguely mapped to something that has a negative connotation.
>A lot of people don't even consider the ability to solve problems to be a reliable indicator of intelligence
That's a very bold statement, as lots of smart people have said that the very definition of intelligence is the ability to solve problems. If fear of the effectiveness of LLM's in behaving genuinely intelligently leads you to making extreme sweeping claims on what intelligence doesn't count as, then you're forcing yourself into a smaller and smaller corner as AI SOTA capabilities predictably increase month after month.
Success in not around any corner. It's pure insanity to even believe that AGI is possible, let alone close.
AI is incapable of any innovation. It accelerates human innovation, just like any other piece of software, but that's it. AI makes protein folding more efficient, but it can't ever come up with the concept of protein folding on its own. It's just software.
You simply cannot have general intelligence without self-driven innovation. Not improvement, innovation.
But if we look at much more simple concepts, 2029 is only 5 years (not even) away, so I'm pretty confident that anything that it cannot do right now it won't be able to do in 2029 either.
being in San Francisco for 6 years and success means getting hauled in front of Congress and European Parliament
cant think of a worse occupational nightmare after having an 8-figure nest egg already
This is a very strange company to say the least.
Most hiring in the foundational AI/model space is very nepotistic and biased towards people in that clique.
Also, Elon Musk used to be the primary patron for OpenAI before losing interest during the AI Winter in the late 2010s.
Doesn't she have a dual bachelors in Mathematics and Mechanical Engineering?
But my point is that she does have a technical background.
No idea because she scrubbed her linkedin profile. But afaik she didn't have "years of experience leading projects" to get a job as leadpm at tesla. That was her first job as PM.
I don't know her personal life or her feelings, but it doesn't seem like a stretch to imagine that she was just done.
[1] https://slate.com/technology/2019/02/openai-gpt2-text-genera...
Squeezing out senior execs could be a way for him to maximize his claim on the stake. Notwithstanding, the execs may have disagreed with the shift in culture.
1) She has a very good big picture view of the market. She has probably identified some very specific problems that need to be solved, or at least knows where the demand lies.
2) She has the senior exec OpenAI pedigree, which makes raising funds almost trivial.
3) She can probably make as much, if not more, by branching out on her own - while having more control, and working on more interesting stuff.
"I just shared this with OpenAI"
https://x.com/bobmcgrewai/status/1839099787423134051
Barret Zoph, VP Research (Post-Training)
"I posted this note to OpenAI."
https://x.com/barret_zoph/status/1839095143397515452
All used the same template.
I wonder if this is just continued creative guerilla tactics to stir the "talk about them maybe finding AGI" pot.
That or we're playing an inverse Roko's Basilisk.
> Translation for the rest of us: "we need to fully privatize the OA subsidiary and turn it into a B-corp which can raise a lot more capital over the next decade, in order to achieve the goals of the nonprofit, because the chief threat is not anything like existential risk from autonomous agents in the next few years or arms races, but inadequate commercialization due to fundraising constraints".
> It's about laying the groundwork for the privatization and establishing rhetorical grounds for how the privatization of OA is consistent with the OA nonprofit's legally-required mission and fiduciary duties. Altman is not writing to anyone here, he is, among others, writing to the OA nonprofit board and to the judge next year.
Probably not.
TL;DR spoiler from someone else is: https://www.lesswrong.com/posts/a5e9arCnbDac9Doig/it-looks-l...
Gwern's more novel prediction track record is calling everyone leaving from OpenAI (Mira was not expected) and general bullishness on scaling years ago. His post from 2 years ago (https://old.reddit.com/r/mlscaling/comments/uznkhw/gpt3_2nd_...) is mostly correct, though incorrectly believed large companies would not deploy user-facing LLMs (granted I think much of this is reasonably obvious?). And Gato2 seems to have never happened.
His overall predictions? I can find his prediction book which he heavily used in 2010 (https://predictionbook.com/users/gwern/page/2?filter=judged&... Brier score of 0.16 is quite good, but this isn't superforecaster level (there's people with Brier scores below 0.1 on that site).
Overall, I see no reason to believe Gwern's numbers over say the consensus prediction at metaculus, even though yes, I do love reading his analysis.
It didn't really contain progress or experimentation. Lots of people are at this point using open source models independently from OpenAI. And a lot of those models aren't that far behind qualitatively from what OpenAI is doing. And several of their competitors are starting to compete at the same level; mostly under normal corporate governance.
So, OpenAI adjusting to that isn't that strange. It's also going to be interesting to see where the people that are leaving OpenAI are going to end up. My prediction is that they will mostly end up in a variety of AI startups with traditional VC funding and usual corporate legal entities. And mostly not running or setting up their own foundations.
I actually think Sam's vision probably scares them.
Of course there are some who want $100M.
But most are really happy that they most likely don’t ever have to do anything they don’t like.
(And on a serious note - if your parents are both still alive and moving in with you while they have hobbies and self-actualization, you're way ahead of the game)
Utilities are an order of magnitude less than being taxed ~$35k/yr and hardly worth worrying about while discussing eight figures
Maintenance can vary, but all 3 costs you mentioned combined would be 2 orders of magnitude lower annually than that net worth, which seems easily sustainable?
Legacy is the dumbest reason to work and does not explain the motivation of the vast majority of people that are wealthy.
edit: The vast majority of people with more than $10million are completely unknown so the idea that they care about legacy is stupid.
If you don't have the 10M you won't understand, you would think that "oh my if only I had the 10M I would just chill", but it never works like that. Human appetite is infinite.
The more highs you get from success, the more you expect from the future achievements to get that same feeling, and if you don't get any you will feel terrible. That's it.
If the stakes are as high as some believe, I presume ppl don't actually care about getting sued when they believe they're helping humanity avert existential crisis.
It's possible that the competitors to OpenAI have rendered future improvements (yes even to the fabled AGI) less and less profitable to the point that the more profitable thing to do would be capitalize on your current fame and raise capital.
That's how I'm reading this. If the competition can be just as usable as OpenAI's SOA models and free or close to it, the profit starts vanishing in most predictions
Of course everybody was quick to play nice once OpenAI insiders got the reality check from Satya that he'd just crush them by building an internal competing group, cut funding, and instantly destroy lots of paper millionaires.
I'd imagine that Mira and others had 6 - 12 month agreeements in place to let the dust settle and finish their latest round of funding without further drama
The OpenAI soap opera is going to be a great book or movie someday
(1) https://www.nytimes.com/2024/03/07/technology/openai-executi...?
Guess what ? Tesla is still on the verge of 'solving FSD'. And most probably it will be in the same place for the next 10 years.
The writing is on the wall for OpenAI.
They are on a path of linear improvement. They would need to go on a path of exponential improvement to have any hope of a working robotaxi in the next 2 years.
That's not happening at all.
which year exactly is TBA
With luck, Mr. Altman's overtures to bring in middle east investors will get locals on board; either way, it's fair to say he'll own whatever OpenAI becomes, whether he's an owner or not. And if he loses control in the current scrum, I suspect his replacement would be much worse (giving him yet another advantage).
Best wishes to all.
I feel like either they're not close at all and the people know it's all lies or they're seeing some shady stuff and want nothing to do with it
tl;dr If you're going to be a CTO or founding engineer, make sure you are getting well compensated, either through salary (which start-ups generally can't do) or equity (which the founder won't generally give away).
Prediction 2: Russia will implode by 2035, by also spending too much money.
"You should, as a matter of course, read absolutely nothing into departure announcements. They are fully glommerized as a default, due to the incentive structure of the iterated game, and contain ~zero information beyond the fact of the departure itself."
They have hired CTO like figures from ex MSFt and so on … which would mean a natural exit for the startup era folks that we have seen recently?
Every company wants to sell itself as some grandiose savior initially ‘organize the world’s information and make it universally accessible’, ‘solve AGI’ but I guess the investors and the top level people in reality are motivated by dollar signs and ads and enterprise and so on.
Not that that’s a bad thing but really it’s a Potemkin village though…
OpenAI made them good money, yes; but if at some point there's a new endeavor in the horizon with another guaranteed billion-dollar payout, they'll just take it. Exhibit A: Ilya.
New razor: never attribute to AGI that which is adequately explained by greed.
Somebody else archived it before me: https://archive.li/0Mea1
The second you hit some kind of breakthrough, capital finds a way to remove any and all guardrails that might impede future profits.
It happened at DeepMind, Google, Microsoft and OpenAI. Why won't this happen the next time?
And ironically, many in this community say that corporations are AI.
She's a pro. Lots to learn from watching how she operates.
I study managers. She's a manager. HN is rife with defamation. Sometimes I push back where it's particularly egregious and on a topic I know well.
Honestly, I bother a lot less. I'm not sure there is hope for this community.
The same happened with reddit.
Sam, being the soulless grifter and scammer he is, of course will remain until the bitter end, drunk with the glimpse of power he surely got while forging backroom deals with the big boys.
“Hi all,
I have something to share with you. After much reflection, I have made the difficult decision to leave OpenAI.
My six-and-a-half years with the OpenAI team have been an extraordinary privilege. While I'll express my gratitude to many individuals in the coming days, I want to start by thanking Sam and Greg for their trust in me to lead the technical organization and for their support throughout the years.
There's never an ideal time to step away from a place one cherishes, yet this moment feels right. Our recent releases of speech-to-speech and OpenAI o1 mark the beginning of a new era in interaction and intelligence – achievements made possible by your ingenuity and craftsmanship. We didn't merely build smarter models, we fundamentally changed how AI systems learn and reason through complex problems. We brought safety research from the theoretical realm into practical applications, creating models that are more robust, aligned, and steerable than ever before. Our work has made cutting-edge AI research intuitive and accessible, developing technology that adapts and evolves based on everyone's input. This success is a testament to our outstanding teamwork, and it is because of your brilliance, your dedication, and your commitment that OpenAI stands at the pinnacle of AI innovation.
I'm stepping away because I want to create the time and space to do my own exploration. For now, my primary focus is doing everything in my power to ensure a smooth transition, maintaining the momentum we've built.
I will forever be grateful for the opportunity to build and work alongside this remarkable team. Together, we've pushed the boundaries of scientific understanding in our quest to improve human well-being.
While I may no longer be in the trenches with you, I will still be rooting for you all. With deep gratitude for the friendships forged, the triumphs achieved, and most importantly, the challenges overcome together.
Mira”
”I’m leaving.
Mira”
> Sam Altman has won. [...] Ilya Sutskever and Mira Murati will leave OA or otherwise take on some sort of clearly diminished role by year-end (90%, 75%; cf. Murati's desperate-sounding internal note)
https://www.lesswrong.com/posts/KXHMCH7wCxrvKsJyn/openai-fac...
_________________
1. https://news.ycombinator.com/item?id=40361128
https://nypost.com/2024/03/08/business/openai-chief-technolo...
Not sure in this case what input he had early on but others seem to have had much more impact in key technical hires/decisions.
This profile data is any intelligence agencies wet dream.
Wojicech Zaremba and Jakub are still at the company.
That makes it much more probable that these execs have simply lost faith in OpenAI.
Lets write this chapter and take some guesses, it's either going to be:
1. Anthropic.
2. SSI Inc.
3. Own AI Startup.
4. Neither.
Only one is correct.
If multiple key people were drastically unhappy with her, it would have shaken confidence in herself and everyone working with her. What else to do but let her go?