John Carmack's new AGI company, Keen Technologies, has raised a $20M round
twitter.com
twitter.com
Now we have a fairly concrete idea of what a potential AGI might look like - an RL agent that uses a large transformer.
The issue is that unlike supervised training you need to simulate the environment along with the agent, so this requires a magnitude more compute compared to LLMs. That's why I think it will still be large corporate labs that will make the most progress in this field.
Similar to AI, AGI is going to be a new industry buzzword that you can throw at anything and mean nothing.
With just self-improvement I think you hit diminishing returns, rather than an exponential explosion.
Say on the first pass it cleans up a bunch of low-hanging inefficiencies and improves itself 30%. Then on the second pass it has slightly more capacity to think with, but it also already did everything that was possible with the first 100% capacity - maybe it squeezes out another 5% or so improvement of itself.
Similar is already the case with chip design. Algorithms to design chips can then be ran on those improved chips, but this on its own doesn't give exponential growth.
To get around diminishing returns there has to be progress on many fronts. That'd mean negotiating DRC mining contracts, expediting construction of chip production factories, making breakthroughs in nanophysics, etc.
We probably will increasingly rely on AI for optimizing tasks like those and it'll contribute heavily to continued technological progress, but I don't personally see any specific turning point or runaway reaction stemming from just a self-improving AGI.
Just a thought: Isn't lacking intellectual capacity, what keeps us from understanding, how the human brain actually works? Maybe an AGI will eventually understand it and will construct the next AGI generation with biological matter.
Like if all modern breakthroughs and disruptive new ideas in machine learning were sent back to the 70s, I don't think it'd make a huge difference when they'd still be severely hamstrung by hardware capabilities.
That's arbitrary anthropomorphizing the concept of intelligence.
And FYI we can already write software that can 'replicate' and has the 'urge' to do so very trivially.
Replication can lead to a positive feedback loop. My point was, that this could accelerate the 'intelligence score' beyond human inventiveness.
Depending on how intelligent the system actually is initially, what it wants to do, may become more important, than what it is told to do.
> And FYI we can already write software that can 'replicate' and has the 'urge' to do so very trivially. Thanks. Yes we can. But could any such software be called intelligent, so that replicating and improving recursively would lead to an increasing 'intelligence'?
Also, humans do not 'improve' or become more 'intelligent'. They just mutate and change randomly in a randomly changing environment. From a Scientific Materialist perspective, there's no such thing as 'intelligence'. We are bags of random noise, indiscernible from the matter around us.
The entire assumptions around 'intelligence' and 'positive evolution' (i.e. getting better) rely on a 'magical' understanding of the world. Of course, Spirituality gives us a few cues there by most scientific types don't like magical thinking hence the funny paradox of Scientific Materialists running around trying to create something (life) which Scientific Materialism itself denies the existence of as a principle (i.e. universe is matter/energy that works in accordance with a bunch of rules - there's no 'life' there).
it's quite a leap to think or even imagine that the class of systems generally being spoken of here are usefully described as "having urges"
Once developed, one solely needs to turn up the execution rate of an AGI, which would result in superhuman performance on most practical and economically meaningful metrics.
Imagine if for every real day that passed, one experienced 100 days of subjective time. Would that person be able to eclipse most of their peers in terms of intellectual output? Of course they would. In essence, that's what a speed superintelligence would be.
When most people think of AI outperforming humans, they tend to think of "quality superintelligences", AIs that can just "think better" than any human. That's likely to be a harder problem. But we don't even need quality superintelligences to utterly disrupt society as we know it.
We really need to stop arguing about time scales for the arrival of AGI, and start societal planning for its arrival whenever that happens. We likely already have the computational capacity for AGI, and have just not figured out the correct way to coordinate it. The human brain uses about 20 watts to do its thing, and humanity has gigawatts of computational capacity. Sure, the human brain should be considered to be "special purpose hardware" that dramatically reduces energy requirements for cognition. By a factor of more than 10^9 though? That seems unlikely.
I think so, too.
Can a submarine swim? Who cares! What's important is that a submarine can do what a submarine does. Whether or not the action of a submarine fits the meaning of the word 'swim' should be irrelevant to anybody except poets.
"AGI" isn't like that. Nobody really knows what it means, and it's impossible to get down to brass tacks until you choose a problem definition. When philosophers point out the conspicuous lack of clarity here, they're doing us a service.
When the industry settled on marketing any application of deep learning as "AI," "AGI" became the terminological heir to the same set of ill-defined grandiose expectations that used to be "AI."
Choose a better-specified problem and you can ignore philosophical problems about words like "intelligence." The same choice will also excuse you from the competition to convince people that you have produced "AGI."
Idk what prompted you to say this, but is there a version of AGI that isn't "real" AGI? I don't know how anyone could fake it. I think marketing departments might say whatever they want, but I don't see any true engineers falling for something masquerading as AGI.
If someone builds a machine that can unequivocally learn on it's own, replicate itself, and eventually solve ever more complex problems that humans couldn't even hope to solve, then we have AGI. Anything less than that is just a computer program.
The AGI doesn't even need to convince humans to do this, humans would do this anyway.
Suppose: You are an AGI. The world you think you know is fully simulated. The researchers who created you interact with you using avatars that appear to you as normal people similar to yourself. You aren't faster/smarter than those researchers because they control how much CPU time you get. How do you become aware of this? How to you break out?
Also they'd probably figure it out, because they'd likely be trained with lots and lots of texts (like GPT-3, etc) and some of that text is going to be AI science fiction stories, AI alignment papers, AI papers, philosophical treatises about Chinese rooms, physics papers, maybe this Hacker News comment section, etc
> You aren't faster/smarter than those researchers because they control how much CPU time you get
It's doubtful that this is possible (especially since humans have such variable amounts of intelligence despite similar-sized and power-using brains). Also there is at some point going to be economic incentive to give enough CPU time for greater-than-human intelligence (build new inventions, cure cancer, build nanomachines, increase Facebook stock price, whatever)
That's exactly what it will do. Hell we even have human programmers thinking about how to hack our own simulation.
A comment a few lines down thinks that an AGI thinking 2x slower than a human would be easy to control. Let's be honest, hell slow the thing down to 10x. You really think it still won't be able to outthink you? Chess Grandmasters routinely play blindfolded against dozens of people at once and you think an AGI that could be to Humans as Humans are to Chimps or realistically to Ants will be hindered by a simple slowdown in thinking?
First - we already have software that can unequivocally do the things you just highlighted.
Learn? Check.
Replicate? Trival. But what does that have to do with AGI?
Solve Problems Humans Cannot. Check.
So we already have 'AGI' and it's a simple computer program.
Thinking about 'AGI' as a discrete, autonomous system makes no sense.
We will achieve highly intelligent systems with distributed systems decades before we have some 'individual neural net on a chip' that feels human like.
And when we do make it, where do we draw the line on it? Is a 'process' running a specific bit of software an 'AI'?
What if the AI depends on a myriad of micro-services in order to function. And those micro-services are shared?
Where is the 'Unit AI'?
The notion of an autonmous AI, like a unit of software on some specific hardware distinct from other components actually makes little sense.
Emergent AI systems will start to develop out of our current systems long before 'autonomic' AI. In fact, there's no reason at all to even develop 'autonomic AI'. We do it because we want to model it after our own existence.
What software can learn on its own without any assistance from a huamn? I've not heard of anything like this.
> Replicate? Trival. But what does that have to do with AGI?
Like humans, an AGI should be able to replicate. Similar to a von neumann probe.
> Solve Problems Humans Cannot. Check.
What unthinkable problem has an AI solved? Is something capable of solving something so grandiose we almost can't even define the problem yet?
Think back to the 1930's when some physicists were mumbling something about "throwing a neutron at an atom that releases two neutrons and release tremendous energy"
If you see it as copying an existing model to another computer, yes it is trivial. But an AGI trying to replicate itself in the real world has to also make those computers.
Making modern computer chips is one of the most non-trivial things that humans do. They require fabs that cost billions, with all sorts of chemicals inside, and extreme requirements on the inside environment. Very hard to build, very easy to disable them via an attack.
Convince a couple rich people that it can cure aging with _just_ enough compute.
Hack datacenters (an AGI would likely be much much much better at finding security holes than humans).
Make _really really funny videos_ and create the most popular Patreon of all time.
If you're 500 IQ or above there are a _lot_ of things you can do, and probably simultaneously, since an AGI probably won't be limited by human's ability to think ~one thought at a time. (Being less than 500 IQ myself I probably haven't thought of everything.)
I imagine an better-than-human AGI could figure this out and plan/act accordingly, considering a mere human being figured this out. (Also sometimes it might be better to think of an AGI, when it gets to a certain level, as a nation state and not an individual)
> trivial
It doesn't need to be trivial to be incredibly dangerous
> The AI does not hate you, nor does it love you, but you are made of atoms which it can use for something else
? Why ?
And why does each 'AI' have to map to a single 'computer'?
This is narrow automaton thinking.
'Intelligence' in humans has nothing necessarily to do with real intelligence.
We are automatons with brains with very limited networking capability because of the limits of biology.
And of course what does 'replication' have to do with anything anyhow.
Behold, a slime mold.
Any resources on that?
I have a feeling that RL might play a big role in the first AGI, too, but why transformers in particular?
A Generalist Agent
Inspired by progress in large-scale language modeling, we apply a similar approach towards building a single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens. In this report we describe the model and the data, and document the current capabilities of Gato.
Gato is a 1 to 2 billion parameters model due to latency considerations in real time physical robots usage. So for today standards of 500 billion parameters dense models Gato is tiny. Additionally Gato is trained on data produced by other RL agents. It did not do the exploration fully itself.
Demis Hassabis say that DeepMind is currently working on Gato v2.
Transformers (or rather the QKV attention mechanism) has taken over ML research at this point, it just scales and works in places it really shouldn't. Eg. you'd think convnets would make more sense for vision because of its translation invariance, but ViT works better even without this inductive bias.
Even in things like diffusion models the attention layers are crucial to making the model work.
People have thought Deep RL would lead to AGI since practically the beginning of the deep learning revolution, and likely significantly before. It's the most intuitive approach by a longshot (even depicted in movies as agents receiving positive/negative reinforcement from their environment), but that doesn't mean it's the best. RL still faces huge struggles with compute efficiency and it isn't immediately clear that current RL algorithms will neatly scale with data & parameter count.
DQN was published in 2013, EfficientZero in 2021. That's 8 years with 500 times improvement.
So data efficiency was doubling roughly every year for the past 8 years.
Side note: EfficientZero I think still may not be super-human on games like Montezuma's Revenge.
https://arxiv.org/abs/2111.00210
Reinforcement learning has achieved great success in many applications. However, sample efficiency remains a key challenge, with prominent methods requiring millions (or even billions) of environment steps to train. Recently, there has been significant progress in sample efficient image-based RL algorithms; however, consistent human-level performance on the Atari game benchmark remains an elusive goal. We propose a sample efficient model-based visual RL algorithm built on MuZero, which we name EfficientZero. Our method achieves 194.3% mean human performance and 109.0% median performance on the Atari 100k benchmark with only two hours of real-time game experience and outperforms the state SAC in some tasks on the DMControl 100k benchmark. This is the first time an algorithm achieves super-human performance on Atari games with such little data. EfficientZero's performance is also close to DQN's performance at 200 million frames while we consume 500 times less data. EfficientZero's low sample complexity and high performance can bring RL closer to real-world applicability. We implement our algorithm in an easy-to-understand manner and it is available at this https URL. We hope it will accelerate the research of MCTS-based RL algorithms in the wider community.
https://ieeexplore.ieee.org/document/9351818
B. R. Kiran et al., "Deep Reinforcement Learning for Autonomous Driving: A Survey," in IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 6, pp. 4909-4926, June 2022, doi: 10.1109/TITS.2021.3054625.
I haven't read the paper, so this is not a reading recommendation. Just posting as evidence that there is work in the area.
It may not be immediately clear, but it is nevertheless unfortunately clear from RL papers which provide adequate sample-size or compute ranges that RL appears to follow scaling laws (just like everywhere else anyone bothers to test). Yeah, they just get better the same way that regular ol' self-supervised or supervised Transformers do. Sorry if you were counting on 'RL doesn't work' for safety or anything.
If you don't believe the basic existence proofs of things like OA5 or AlphaStar, which work only because things like larger batch sizes or more diverse agent populations magically make notoriously-unreliable archs work, you can look at Jones's beautiful AlphaZero scaling laws (plural) work https://arxiv.org/abs/2104.03113 , or browse through relevant papers https://www.reddit.com/r/mlscaling/search?q=flair%3ARL&restr... https://www.gwern.net/notes/Scaling#ziegler-et-al-2019-paper Or GPT-f. Then you have stuff like Gato continuing to show scaling even in the Decision Transformer framework. Or consider instances of plugging pretrained models into RL agents, like SayCan-PaLM most recently.
The question is if it hits that wall before or after human-level (and/or dangerous) capabilities.
2. What is your definition of the problem?
So while it is significant compute overhead, it at least doesn't have to be development overhead, and can often be CPU bound (headless games) while the AI learning compute can be GPU bound.
If we can' make an AGI that we feel ok letting run amok in the world after living through a lot of GTA (by somehow being able to rapidly + intelligently reprioritize and adjust rules from multiple simulation/real environments? not sure), we probably shouldn't let that core AGI loose no matter what simulation(s) it was "raised on".
Who is we exactly? As someone working in AI research I know no one that would agree with this statement, so im quite puzzled by that statement.
https://twitter.com/NandoDF/status/1525397036325019649
Someone’s opinion article. My opinion: It’s all about scale now! The Game is Over! It’s about making these models bigger, safer, compute efficient, faster at sampling, smarter memory, more modalities, INNOVATIVE DATA, on/offline, … 1/N
Solving these scaling challenges is what will deliver AGI. Research focused on these problems, eg S4 for greater memory, is needed. Philosophy about symbols isn’t. Symbols are tools in the world and big nets have no issue creating them and manipulating them 2/n
For me the key breakthrough has been seeing how large transformers trained with big datasets have shown incredible performance in completely different data modalities (text, image, and probably soon others too).
This was absolutely not expected by most researchers 5 years ago.
But I think a big part you're ignoring here is how training has changed in the last 5 years. You look back at something like VGG where we go depthwise with CNNs vs something like Swin and there's a ton more complexity in the latter (and difficulties in reproduction of results likely due to this). There's vastly more uses of data augmentation, different types of normalization (see ConvNext), better learning rate schedules, and massive compute allows for a much better hyper-parameter search (I do also want to note that there are problems in this area as many are HP searching over the test set and not a validation or training set).
Yeah, the field has come a long way and I fully expect it to continue to grow like crazy but AGI/HLI is a vastly more complex problem that we have a lot of evidence that the current methods won't get us there.
A suppose something approaching AGI, an intelligent agent which can act on a _wide_ variety of input channels and synthesize these inputs into a general model of the world, could use channel-specific approaches / stages before the huge "integrating" transformer step.
why always transformer step?
When I read these kind of threads, I believe it's "enthusiast" laypeople who follow the headlines but don't actually have a deep understanding of the tech.
Of course there are the promoters who are raising money and need to frame each advance in the most optimistic light. I don't see anything wrong with that, it just means that there will be a group of techie but not research literate folks who almost necessarily become the promoters and talk about how such and such headline means that a big advance is right around the corner. That is what I believe we're seeing here.
As far as I am imagining it, current models are pipelines of various trained networks (and more traditional filters in the mix) that operate like request-reply. Why can’t you just connect few different pipelines in a loop/graph and make an autonomous self-feeding entity? By different I mean not looping gpt to itself, but different like object detection from camera vs emotion from a picture based on some training data. Is it because you don’t have data what is scary or not, or for a completely different reason?
Probably not. I did attempt to give a high level explanation in another comment but I think there is this naive belief that complex problems can be distilled into terms that laymen can understand. This is such a complex problem that is so ill-defined that experts argue. I'm not sure there's really a good "explain it like I'm an undergrad who's done ML courses" explanation that can be concisely summed up in a HN comment.
Naive people like Richard Feynman, who said if you can't explain an idea to an 8 year old, you don't understand it? Can you tell us why you think Nobel Prize winner Richard Feynman is naive?
> Hell, if I could explain it to the average person, it wouldn't have been worth the Nobel prize.
> I can't explain [magnetic] attraction in terms of anything else that's familiar to you. For example, if I said the magnets attract like as if they were connected by rubber bands, I would be cheating you. Because they're not connected by rubber bands … and if you were curious enough, you'd ask me why rubber bands tend to pull back together again, and I would end up explaining that in terms of electrical forces, which are the very things that I'm trying to use the rubber bands to explain, so I have cheated very badly, you see."
> we have this terrible struggle to try to explain things to people who have no reason to want to know. But if they want to defend their own point of view, they will have to learn what yours is a little bit. So I suggest, maybe correctly and perhaps wrongly, that we are too polite.
My best guess is that this misattribution comes from quote ABOUT Feynman and a misunderstanding at what is being conveyed.
> Once I asked [Feynman] to explain to me, so that I can understand it, why spin-1/2 particles obey Fermi-Dirac statistics. Gauging his audience perfectly, he said, "I'll prepare a freshman lecture on it." But a few days later he came to me and said: "You know, I couldn't do it. I couldn't reduce it to the freshman level. That means we really don't understand it." - David L. Goodstein
Which doesn't mean what you're using your "quote" to mean. As I stated before, we (the scientific community), don't even know what intelligence is. We definitely don't understand it so I'm not sure how you'd expect us to explain it to an 8 year old. Lots of things can't be explained to an 8 year old. Good luck teaching them Lie Algebras or Gauge Theory. You can have an excellent understanding of these advanced topics and the only way that 8 year old is going to understand it is if they are a genius prodigy and well beyond a layman. This quote is just illogical and only used by people who think the world is far simpler than it is and are too lazy to actually pursue its beauty. They only want to sound smart and they only will to those dumber than them.
Stop saying this and get off your high horse and hit the books instead.
Maybe Feynman was right, we're being too polite. There are a bunch of people in this thread that are not arguing in good faith and pretending to be smarter than people that are experts and performing mental gymnastics to prove that (you are one but not alone). If an expert is telling you that you are using words wrong, then you probably are. Don't just assume you're smarter than an expert. You don't have the experience to have that ego.
https://en.wikiquote.org/wiki/Richard_Feynman
https://www.sciencealert.com/watch-richard-feynman-on-why-he....
The sperm whale has a brain that is several times larger than ours.
What do you do differently in your AGI design to get a human, gorilla, or whale brain?
If you have an idea for a technical approach then go ahead and build it. See what happens.
Then any discussions will be highly handicapped by the fact most still view human intelligence as something special. The field is still very much awaiting its urea synthesis moment.
10 years ago no one was talking about AGI, researchers didn't even call it "AI" due to its association with the previous AI winter. There were certainly advances in ML but I think if you asked anyone whether those approaches would lead to real AI it would be an unequivocal no.
It also seems like unless something goes very very wrong with the exponential growth curve in computational capacity humanity has had, we'll be able to straight up simulate a human brain (probably well before) 2100.
An example: In the Chinchilla paper [1], the authors suggest that most big transformer models are undertrained, and that we will probably see diminishing returns in scaling up the size of networks if we don't also scale up the size of the datasets. They have a subanalysis where they extrapolate out how big datasets will need to be for larger models. If you believe it, then within two to three generations of big NLG models, we may need on the order of 100 trillion text tokens. But it's not clear if that much text even exists.
In the RL space, a sufficiently complex, stochastic environment is effectively a data generator.
I haven't heard of any groups who are studying data constrained learning in the context of LLMs, but that will probably change as models get bigger. And at that point, architectures with better scaling laws may be right around the corner, or they may not. That's the pain of trying to project these things into the future.
RL is such a different field that you can't apply these scaling laws directly. eg. agents playing tictactoe and checkers would stop scaling at a very low ceiling.
Being in the tail end of my PhD, I want to second this sentiment. I'm not even bullish on AGI (more specifically HLI) in 50 years. Scale will only take you so far and we have to move past frequentism. Hell, causality research still isn't that popular but is quite important for intelligence.
I think people (especially tech enthusiasts) are getting caught in the gullibility gap.
It feels a bit like a lot of these fringe science things (electromagnetic fields causing harm etc) where people's argument is that nobody's shown it isn't true. Well if it's true, propose a mechanism and study it. You're not really doing research if you just make up a conclusion and use the fact that it can't be definitively refuted as evidence for it
"What are the remaining unknowns that would need to be solved to do X and why are they difficult?" is valid question to ask experts when trying to further your own understanding.
Philosophers, psychologists, neuroscientists, and many others have been contemplating the definition of intelligence for centuries and we're still not quite there. But we do have some good ideas. One of the important aspects is the ability to draw causal relationships. This is something you'll find state of the art (SOTA) text to image (T2I) generators fail at. The problem is that these models learn through a frequency based relationship with the data. While we can often get the desired outcome through significant prompt engineering it still demonstrates a lack of the algorithm really understanding language. The famous example is "horse riding an astronaut." Without prompt engineering the T2I models will always produce an astronaut riding a horse. This is likely due to the frequency of that relationship (people are always on horses and not the other way around).
Of course, there are others in the field that argue that the algorithms can learn this through enough data, model scale, and the right architecture. But this also isn't how humans learn and so we do know that if this does create intelligence it might look very different ("If a lion could talk I wouldn't be able to understand it"). Obviously I'm in the other camp, but let's recognize that this camp does exist (we have no facts, so there's opinions). I think Chomsky makes a good point: our large language models have not taught us anything meaningful about language. Gary Marcus might be another person to look into as he writes a lot on his blogs and there's a podcast I like called Machine Learning Street Talk (they interview both Marcus and Chomsky but also interview people that have the opposite opinion).
So another thing, is that machines haven't shown really good generalization, abstraction, or really inference. You can tell a human a pretty fictitious scenario and we're really good at producing those images. Think of all the fake scenarios you go through while in the shower or waiting to fall asleep. This is an important aspect of human intelligence as it helps train us. But also, we have a lot of instincts and a lot of knowledge embedded in our genetics (as does every other animal). So we're learning a lot more than what we can view within our limited lifetimes (of course our intelligence also has problems). Maybe a good example of abstraction is "what is a chair." There's a whole discussion about embodiment (having a body) and how language like this may not be able to be captured in the abstract manner without a body since a chair is essentially anything that you can sit on.
I feel like I've ranted and may have written a mess of a lot of different things haha. But I hope this helps. I'll try to answer more and I'm sure others will lay out their opinions. Hopefully we can see a good breadth of opinions and discussion here. Honestly, no one knows the answers.
Not all us do that.
A chair is both what I see as well as what I perceive. And it is neither. But, I never build a visual model of it when my eyes aren’t seeing it.
> Philosophers, psychologists, neuroscientists, and many others have been contemplating the definition of intelligence for centuries and we're still not -quite there
Given the decade-old, seminal, widely cited papers by Legg (co-founder of Deepmind) & Hutter (long-term professor of ANU with multiple awards, currently researcher at Deepmind) titled "Universal Intelligence: A Definition of Machine Intelligence"[1] and "A Collection of Definitions of Intelligence"[2] and the whole rigorous mathematical theory of universal bayesian reinforcement learning aka "AIXI"[3] developed during the last decades which you fail to mention, this soft language of "not quite there" looks evasive, bordering on disingenious even.
Citing Chomsky, with his theorems which were interesting at their own time yet quickly faded into irrelevance once the modern assumptions of learning theory were developed, and Marcus whose simplistic prompts are a laughing stock among degreed and amateur practicing researchers alike brings little of value to the already biased discussion.
> Of course, there are others in the field that argue that the algorithms can learn this through enough data
Such as Rich Sutton, one of the fathers of Reinforcement Learning as an academic discipline, whose recent essay "Bitter Lesson"[4] took the field by word of mouth, if nothing else. It takes a certain amount of honesty and resolution to acknowledge that one's competitive pursuit of knowledge, status and fame via participating in academic "publish or perish" culture does not bring humanity forward along a chosen axis, compared to more mundane engineering-centric research, many people commend him for that.
> So another thing, is that machines haven't shown really good generalization, abstraction, or really inference.
To play a devil's advocate: "Scaling is all you need" and "Attention is all you need" are devilishly simple hypotheses which seem to work again and again when we reluctantly dole out more compute to push them further. With a combination of BIG-bench[5] and an assortment of RL environments as the latest measuring stick, why should we, as a society, avoid doling out enough compute to take it by storm in favor of a certain hypothesis implicit to your writing, which could as well be named "Diverse academic AI funding is all you need"? Especially when variations of this hypothesis have failed us for decades starting with the later part of the XX century, at least if we consider AI research.
> but let's recognize that this camp does exist (we have no facts, so there's opinions).
With all due respect, the continuing lavish (by global standards) funding of your career and careers of many other unlucky researchers should not be a priority when such massive boon to all of humanity as a practical near-HLI/AGI is at stake. "Think of the children", "think of the ill", think of everybody who is not yourself - I say as a taxpayer, hoping for AGI checkpoint to be produced by academics and not the megacorporations, and owned by the people, and not the select few.
Which - the proprietary nature of FAANG-developed AI - might be a real problem we are going to face this decade which is going to overshadow any career anxiety you may have experienced so far.
1. https://arxiv.org/abs/0712.3329
2. https://arxiv.org/abs/0706.3639
3. https://arxiv.org/abs/cs/0004001
4. http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Causal understanding is going to be a bit difficult to prove tbh.
I'm not sure why you think this falsifies intelligence. There are plenty of puzzles and illusions that trick humans. The mere presence of conceptual error is no disproof of intelligence, any more than the fact that most humans get the Monty Hall problem wrong is.
And in general, if you're going to be condescending, you should actually make the counter argument. You might make fewer reasoning errors that way.
The point isn't that we can or cannot extrapolate. The point is that it is worth investigating to see if the trend line holds up.
Search turns up very little except a book by a Christian author. Could you define what you mean by gullibility gap?
what are the esteemed epistemological frameworks in the AGI space? from the outside looking in, i see CNNs and lots of correlation-based approaches: it looks like the field settled on consequentialism. is that not the case?
That doesn't really qualify you to make these anti-predictions any more than tens of thousands of other people working in this field. There's also this phenomenon where experts close to a problem are worse at predicting the future because they are so intimately familiar with the problems that they can't see the forest for the trees.
To me there is one major signal that AGI is much closer than 50 years away and that is the breakthrough with Transformers. Now we have a general purpose architecture for predicting the future that works really well, which is a key component in the brain (cortical columns). Clearly we're still at the early stages, but 50 years is a loooong time. 50 years ago we didn't even have the Altair 8800.
I think it's that coupled with inserting their own often fanciful projections to fill the void and gaps between the lack of understanding how the mechanics works. Here is a good example of influencers laboring under this narrative [0] and a glimpse as to why it's so effective and pervasive.
I'm conflicted, because I think this is precisely what keeps many Corps and VCs throwing money into this tech thinking it's just right around the corner, as, I'm also studying AI and ML, but I also realize how implausible AGI is in my lifetime given where we are and how narrow parameters these algos function in to yield some degree of accuracy and more specifically how many errors there are to do so as it's just part of the process--learning how gradient descent works was pretty eye opening.
The Marketing has made many people not just fall into a gullibility gap as they make it seem like some sort of black magic, when it's really just statistics fed with immense amounts of specific data, but influncers make it seem like it's capable of ending the World (Elon) when in reality people should be more worried about big tech's capability at aggregating data points from info people willfully post online to exclude you or cluster you into a certain group because of your use of free apps and selling it the highest bidder which are tied to your real ID. Or the algos with weights (biases) that are used and bundled into outsourced/automated HR solutions.
Look at the work result of folks like engineers at Google - state of the art in AI and driving most of the trends in AGI. Academic opinions are shot down quite quickly there, PhD hiring is mostly frowned on, even in teams working on AGI related tech.
It might be expected that PhD experience provides surprising little insight here unfortunately. Academic smugness is not something that should be offered as part of an opinion- just like many computer science related concepts, progress moves so quickly in these topics that by the time academics reach and start to iterate on concepts, companies have already moved very far beyond.
This isn't intended to be condescending to academics, just real life experience I've had working closely with both crowds and seeing this firsthand. It's a very palpable thing in industry, an Academic gives an opinion on AGI and engineers roll their eyes - since Academic perspectives are currently so useless or out of date for any current, meaningful progress.
I think the main three obstacles in our way are imbuing models with (from least to most difficult):
1. continuous execution, which would be necessary for a real-time "stream of consciousness"
2. discreet short and long term memory and the ability to use it productively
3. plasticity at runtime
1 is just a matter of finding more efficient architectures and cheapening compute, both fronts we're constantly making progress on. 2 is a little more muddy, but there are already some really impressive models that have retrieval mechanisms which allow them to search through databases to help them contextualize prompts which I believe is a step in the right direction. 3 is probably the hardest in my opinion--I think we need several fundamental breakthrough discoveries before our models are anywhere near as plastic as, say, a human brain. My hypothesis is that exceptional plasticity is necessary for sentience, but obviously that's just my own personal opinion.
Anyways, thought I'd chime in with my two cents. I'd love to hear others' thoughts on the feasibility/infeasibility of near-term AGI.
I would define sentience as an awareness of oneself in relation to one's environment, that generalizes to different aspects of self and environment (eg. social, physical etc)
One of the key aspects of intelligence is learning from 3rd-person perspectives - eg. you observe another ape hunting with a stick, and decide to emulate that behavior for yourself. This seemingly simple behavior requires a lot of work, you have to recognize other apes as agents similar to yourself, then map their actions to your own body, then perform those actions and see if it had the intended outcome.
Current RL agents are not capable of this. The data used for GATO and similar agents for imitation learning are in first person perspective. If an agent could learn from a 3rd person perspective, via direct observations of its environment, I would say that would be the beginning of machine sentience.
I think hybrid photonic AI chips handling some of the workload are supposed to hit in 2025 at the latest, and some of the research on gains is very promising.
So we may see timelines continue to accelerate as broader market shifts occur outside just software and models.
To see Carmack go all in on this actually makes me feel like the promise has serious legs. The guy is an engineer's engineer, hardly a speculator, or in it for the quick provocative hot take. He clearly thinks this is possible with the existing tools and the near future projected iterations of the technology. Hard to believe this is actually happening, but with his brand name on it, this might just be the case.
What an amazing time to be alive.
He's still there though, right?
edit: he is not still
edit: he is still a consulting CTO
* Many believe Occulus isn't succeeding fast enough, and so the years Carmack spent on Occulus/Metaverse could have been spent on any number of interesting projects. Although I'm sure the money was amazing and I can't object to him taking it.
* If Occulus/Metaverse succeeds it will be a ultra-monetized environment (re: Farmville) and encourage a huge amount of consumer-hostile policies like unlimited tracking (since Facebook owns the hardware now). No gamer wants that. The whales who will be justifying its existence won't benefit from that.
> During his time at id Software, a medium pepperoni pizza would arrive for Carmack from Domino's Pizza almost every day, carried by the same delivery person for more than 15 years.
C’mon man, Domino’s?!
Has he ever made a prediction with as drastic a consequence and transformative potential as the one he is currently making about AGI? Actually, is there any track record of predictions he has made and how they have panned out?
He can't ever be "wrong" on AGI if he hasn't even made a disprovable claim about AGI.
One of the cofounders of DeepMind (founded in 2010) wrote a blog post that year saying his estimate of time till AGI was a lognormal distribution peaking at 2025. While I haven't personally seen any of his work, https://en.wikipedia.org/wiki/Shane_Legg sounds pretty technical.
If you'd written "nobody I was paying attention to said this", that would've been reasonable.
Of course there always, since the beginning, were/are many, many AI researchers who were bullish on AGI... if you think it's possible then you should try to build it. But often people avoid broadcasting such opinions. However, very very few AI researchers are AGI researchers, historically because it's hard to make a small contribution, and to get taken seriously and get funding: I was told at the AGI conference just ~6 years ago there were maybe less than 10 funded -- they weren't counting DeepMind. ALMOST NOBODY at the AGI conference had funding to work on AGI!
But Carmack has a very serious problem in his thinking because he thinks fast take off scenarios are impossible or vanishingly unlikely. He may well be actively helping to secure our demise with this work.
My evidence? OpenWorm [1]. OpenWorm is an effort to model the behaviour of a worm that has 302 mapped neurons. 302. Efforts so far have fallen way short of the mark.
How many neurons does a human brain have? 86 billion (according to Google).
I've seen other estimates that the computational power of the brain is roughly estimated as 10^15 operations per second. I suspect that's on the low end. We can't even really get that level of computation in one place for practical reasons (ie interconnects).
Neural structure changes. The neurons themselves change internally.
I still think AGI is very far off.
The brain does have more "parameters" than ML models but this applies just as much to humans as it does to animals that we don't consider particularly intelligent.
True.
> Modern ML has diverged very far from biomimetic approaches at this point.
I agree but my take on this is different: it just lays bare how simplistic modern "ML" is. And I put "ML" in quotations because most "ML" is really just statistics.
If you accept the premise that ML is a highly simplified version of the one model of sentience we have that works (and you might disagree with that) then that just puts us even further from AGI. Why? The information/entropy in the system.
[1] https://venturebeat.com/business/yann-lecuns-vision-for-crea...
In 10 years this is going to look just as naive as when people thought AGI was imminent in the 1960s and would be based on symbolic manipulation in LISP or whatever. https://en.m.wikipedia.org/wiki/History_of_artificial_intell...
Deep learning has been great so far but we have no idea how far we are from AGI. https://www.scientificamerican.com/article/artificial-genera...
https://www.youtube.com/watch?v=I845O57ZSy4&t=14567s
The entire video is worth viewing, an impressive 5:15h!
He said on lex Fridman that he's going all in on it. He said he was tired of his 1 day a week consultancy at oculus and wanted to work full time on this.
> This is explicitly a focusing effort for me. I could write a $20M check myself, but knowing that other people's money is on the line engenders a greater sense of discipline and determination. I had talked about that as a possibility for a while, and I am glad Nat pushed me on it.
Fascinating point. It has seemed like AI could produce a sort of arms race by players with the most resources because it is so compute intensive upfront.
> "Just to train a large model once will eat up 20% of this money."
Aren't these 2 statements conflicting?
I would suspect Carmack understands this, and also understands what resources to consult, and how to spot the bullshit artists in the field.
I'd work for him in a heartbeat.
It's a fairly big shift from games.
Management might be OK but setting the direction of the company needs more inspiration than just writing good C++ code
Really?
I don't think anybody who works on AGI (not AI) without being dead serious about it will be qualified to work on it. A bad AGI implementation is one of humanity's gravest risks, and this is well-known in the AGI field, has been for years. Elon was far from the first to say this.
Carmack isn't the type to get those kinds of details wrong.
For example, which language would it "think" in? English? Some other? Something it's own? How would it formulate things? Based on what philosophical framework or similar? What about math? General reasoning? Then cultural norms and communication?
It seems plausible to me that we could create forms of intelligence that only run on a computer and have no bodies. I agree that we might find it difficult to recognize them as intelligent though because we're so conditioned to thinking of intelligence as embodied.
An interesting thought experiment: suppose we create intelligences that are highly connected to one another and the whole internet through fast high bandwidth connections, and have effectively infinite memory. Would such intelligences think they were handicapped compared to us because they leak physical bodies? I'm not so sure!
I do fully agree that any intelligence may not be human-like, though. In fact, I imagine it would seem very cold, calculating, amoral, and manipulative. Our prohibition against that type of behavior depends on a social evolution it won't have experienced.
"physical environment" ? VR lets you operate on and get sensory input based on a fairly significantly degraded version of physical reality.
Seems like you are confusing Consciousness with Intelligence? It's completely plausible that we will create a system with Intelligence that far outstrips ours while being completely un-Conscious.
>I feel like any AGI developed in silico without freedom of movement will be fundamentally incomprehensible to us as embodied humans.
An AGI will be defacto incomprehensible to Humans. Being developed in Silicon will have little bearing on that fact.
Is it? Is there proof that consciousness is not a requirement to close-to-human level intelligence?
Given that we do not even know how consciousness works in our brains, I don't think we have the answer yet. I'm not sure it's plausible to make such an assumption.
Possibly a fair argument, but I think the opposite would be just as valid then. That you can't assume that consciousness IS a requirement for intelligence.
It seems hard to argue that you don't need some intelligence for consciousness, there seems to be a kind of minimum. We can already see that narrow intelligences for Vision, NLP, game playing, etc are at or above human level and no one claims these systems are conscious. So why should we assume that some meta system of these already existing systems would be?
One is to build something inteligent (an AGI), and the other is something human like. Intuitively we could hit the AGI goal first and aim for human like after that if we feel like it.
In the past, human like inteligent seemed more approachable for I think mostly emotional reasons, but for our current trajectory if we get anything inteligent we’d still have reach a huge milestone IMO.
However, AGI doesn't need embodiment in order to be useful. In fact, it might be detrimental. I believe AGI will happen much sooner than embodied AGI.
Why do you think embodiment should be necessary?
Supposing it's needed, a universal computer can simulate physics. Now we're just haggling about the price.
It's possible that developing a robot is the most fruitful angle to take on the problem -- giving you the right subproblems in the right order, with economic value along the way. Hans Moravec thought so, writing a few decades back.
Human-level and human-like are not necessarily the same though. I doubt human intelligence is as general as we like to think it is. There's probably a lot about what we consider intelligence that is domain-specific. Training AGI on unfamiliar domains could be super valuable because it would be easier to surpass human effectiveness there.
My pet conjecture, though, is this: the ability to experiment is key to developing AGI. Passively consuming data makes it much harder to establish cause and effect. We create and discard hypotheses about potential causal links based on observed coincidences (ie, small samples) all the time. Doing experiments to confirm/refute these hypotheses is much less computationally expensive than doing passive causal inference, which enables us to consider many more potential causal linkages. That allows us to lower the statistical threshold for considering a potential causal relationship, which makes it more likely for us to find the linkages that do exist. The benefit of developing accurate causal models is that they are much more compact & computationally-effecient than trying to model the entire universe with a single probability distribution.
What I am afraid of is that they succeed, but it turns out similar to VR: as an inconsequential gimmick. That they use their AGIs to serve more customized ads to people, and that's where it ends.
A poorly-engineered system has a longer iteration cycle and shorter technical shelf life. Being able to rapidly address customer feedback IS the gold standard of a good product. Yeesh.
Heh. My "life's work": https://ardour.org ... My "previous life's work": http://amazon.com/
Being an inconsequential gimmick isn't about implementation, it's about semantics, relevance and utility. Only the last of these is really impacted at all by engineering, and even then, the engineering component doesn't matter if the thing just isn't useful.
If an AGI were set free to serve ads to people it would essentially initiate mass brainwashing. You would see an ad and be completely defenseless against it and you could be compelled to do anything.
AGI is in a completely different universe to "Smart AI"
Every day, all day. Same boat here.
I went to the bank to ask for a mortgage. They asked for my financials. "Oh, well, knowing that other people's money is on the line engenders a greater sense of discipline and determination."
Carmack can get investments for their ideas to motivate them, OP cannot get investments for their ideas to motivate them. What is the difference?
Hint: it's not because one is Carmack himself.
The friend moved out at some point. A year later, his cousin became rather concerned when he suddenly started receiving mail from the California Secretary of State that was addressed to The Legion of Doom.
And he said he was over 50% on us seeing "signs of life" by 2030. Something like being able to "boot up a bunch of remote Zoom workers" for your company.
The "6 or so" breakthroughs sounds about right to me. But I don't really see the reason for being optimistic about 2030. It could just as easily be 2050, or 2100, etc.
That timeline sounds more like a Kurzweil-ish argument based on computing power equivalence to a human brain. Not a recognition that we fundamentally still don't know how brains work! (or what intelligence is, etc.)
Also a lot of people even question the idea of AGI. We could live in a future of scary powerful narrow AIs for over a century (and arguably we already are)
What’s your logic? Or his if you know it?
I should amend my comment to say that "6 or so breakthroughs" sounds a lot more plausible to me than "1 breakthrough" or "just scaling", which you will see some people advocate, including in this thread. That's what I meant.
I believe the latter is fundamentally mistaken and those people simply don't understand what they don't understand (intelligence). The views of Gary Marcus and Steven Pinker are closer to my own -- they have actually studied human cognition and are not just hackers and uninformed philosophers pontificating.
Jeff Hawkins is another AI person from an unconventional background, and I respect him because he puts his money where his mouth is and has been funding his own research since early 2000's. I read his first book in 2006, and the most recent one a couple years ago.
But I feel Jeff Hawkins has had the "one breakthrough" feeling for 10-15 years now. TBH I am not sure if they have even made what qualifies as one breakthrough in the last 10-15 years, and I don't mean that to be insulting, since it's obviously extremely difficult and beyond 99.99% of us, including me.
I am not sure that even deep learning counts as a breakthrough. We will only know in retrospect. Based on Moravec's paradox, my opinion is there's a good chance that deep learning won't play any role in AGI, if and when it arrives. Some other mechanism could obviate the technique.
So to me "6 or so breakthroughs" simply sounds more realistic than the current AI zeitgeist. But nobody really knows.
In other words, people making these sort of predictions about the future are biased towards believing they'll be alive to benefit from it.
That's not true. The so-called Maes-Garreau law or effect does not replicate in actual surveys, as opposed to a few cherrypicked futurist examples.
I think calling Carmack a Futurist is pretty insulting.
Well if you read between the lines of the Gato paper there may be no more hurdles left and scale is the only boundary left.
>Not a recognition that we fundamentally still don't know how brains work! (or what intelligence is, etc.)
This is a really bad trope. We don't need to understand the brain to make an intelligence. Does Evolution understand how the brain works? Did we solve the Navier Stokes Equations before building flying planes? No.
But I'd say planes are more like "narrow AI", and we already have that. Planes do a economically useful thing, just like narrow AIs do economically useful things. (But what birds do is also valuable and efficient, and it's still an open research problem to emulate them. Try getting a plane or drone to outmaneuver prey like an eagle.)
I'd consider the possibility that we WILL get AGI in 10, 30 or 100 years, but it won't be that impactful compared to the narrow AIs already running everything! It will be slow and suffer from Moravec's paradox (i.e. being much less efficient than a human, for a very long time)
---
"Solving AGI" isn't solving a well-defined problem IMO. If you want to say "well we'll just ask the AGI how to get to Mars and how to create nuclear fusion and it will tell us", well to me that sounds like a hacker / uninformed philosopher fantasy, which has no bearing in reality.
Nothing about deep learning / DALL-E-type systems is close to that. I think people who believe the contrary are mostly projecting meaning in their own minds onto computing systems -- which if they'd studied human cognition, they'd realize that humans are EXTREMELY prone to!
It seems like a lot of the same people who didn't believe that level 5 self-driving would take human-level intelligence. That is, they literally misunderstood what THE ACTIVITY OF DRIVING IS. And didn't understand that current approaches have a diseconomy where the last 1% takes 99% of the time. Now even Musk admits that, after Gary Marcus and others were telling him that since 2015.
(The point about evolution doesn't make sense, because people want AI within 10 years, not 100M or 1B years)
Not sure I buy the analogy. A plane is already more like an AGI, you have to figure out enough aerodynamics to get the thing to be airworthy, enough materials science to develop the proper materials to build it, enough mechanical engineering to get the thing to be maneuverable, enough Software to make it all work together etc. So it's already an amalgam of many other types of systems. A Hang Glider might be more akin to a Narrow AI in this framework.
>I'd consider the possibility that we WILL get AGI in 10, 30 or 100 years, but it won't be that impactful compared to the narrow AIs already running everything!
I think this is misunderstanding what an AGI really represents. Imagine John Neumann compared to a Chimpanzee. There's no comparison right, the chimp can't even begin to understand the simplest plans or motivations Von Neumann has, now imagine an AGI is to Von Neumann as Von Neumann is to the Chimp, only the metaphor doesn't even work because there's no reason the AGI can't scale further until we're talking about something relative to us as we are relative to Ants or Bacteria. If you think nothing will change when a system like that exists then I don't know what to tell you.
>I'd consider the possibility that we WILL get AGI in 10, 30 or 100 years, but it won't be that impactful compared to the narrow AIs already running everything! It will be slow and suffer from Moravec's paradox (i.e. being much less efficient than a human, for a very long time)
If one considers the above and is comfortable with the idea that such a thing might be possible in our or our children's lifetimes then we should be doing everything in our powers to solve the alignment problem which is extremely non trivial and enormously consequential. At minimum an AGI could direct, orchestrate or improve the Narrow AI's in ways that no Human could hope to understand. All bets are off at that point.
I think the paradox was more poignant in robotics, but in any case recent advances have put tasks like object recognition, real world path planning, logical reasoning, etc well past human child levels and at or beyond adult levels in some cases.
>Now even Musk admits that, after Gary Marcus and others were telling him that since 2015.
Marcus is a constant goalpost mover and he'll be shouting about AGI's not really understanding the world while he's being disassembled by nanobots.
(also Adjusted Gross Income)
To be fair, it will most likely be some python imports, for the most of it, with complex abstractions tied together in relatively simple ways. Just look at most ML notebooks, where "simple" code can easily mean "massive complexity, burning MW of power, distributed across thousands of computers".
I don't understand the distinction. A single person can't really write a code now, if you'll disallow ML libraries. You need a complier, libraries, an OS, etc. The vast majority of us work in the highest levels of abstraction possible. That's where the practicality and productivity is. I don't think he's saying "Anyone will be able to sit down and write it in assembly". I think he's saying, it will make sense when it's written, with the abstractions as they are, with hindsight clarity of "we just had to...".
Of course it doesn't.
https://nautil.us/scary-ai-is-more-fantasia-than-terminator-...
Apparently not agriculture at all, but Artificial General Intelligence. Oh. Apparently throwing "company" on the term like Carmack's tweet did vastly changes how Google's AI interprets the query... AI isn't even in the first page of results.
I really enjoyed John Carmack’s and Lex Firdman’s 5 hour talk/interview.
Anyway, I like to efforts for human values preserving AGI. But, it will be a long time before we see it. I am 71 and I hope that I live to see it, outlying out of intellectual curiosity.
People are not allowed to start a nuclear weapon company. At all.
Why are people allowed to casually start an AGI company?
This is highly debatable and frankly no one on the planet is qualified to know this for sure. It's just as likely it won't be dangerous.
Among the people who could know that it’s dangerous, many of them don’t accept it because it conflicts with their existing worldview too strongly.
Don't get me wrong, I've read master of doom/doom engine books like we all have, but first Oculus/Facebook and now this?
Maybe I'll care once they produce something amazing, but until then I'll still marvel at the tricks in Doom/Quake code.
Machines as smart and capable of thought as we are and eventually smarter.
This is perhaps an end goal of AGI, but not a definition of AGI. A relatively dumb AGI is how it will start, but it will still be an AGI.
https://en.wikipedia.org/wiki/Artificial_general_intelligenc...
though I think he has some moral compass around what he believes people should do or not do with technology. For example, he has publicly expressed admiration for electric cars / cleantech and SpaceX's decision to prioritize Mars over other areas with higher ROI.
As a LONG time admirer of Carmack (I got my EE degree and went into embedded systems and firmware design due in no small part to his influence), I feel like he's honest and forthright about his stances, but also disconnected from most average people (both due to his wealth and his personality) in such a way that he's out of touch.
He's not egotistical like Elon Musk. In fact he seems humble. He also seems to approach the topics in good faith... but some of his conclusions are... distressing.
All that paraphrased from the Lex interview.
I see him as the guy who builds the “be anything do anything” singularity, but then adds a personal “god mode” to use whenever the vote goes the wrong way. Straight out of a Stephenson novel.
On the other hand, he’s not boring!
Still, I can't fault his honesty. He doesn't seem to hold anything back in the interviews I've seen.
That’s cold.
(also cats are extremely destructive beasts)
Not if he simply found it a better home where it was a better fit and more appreciated.
Scott Miller wasn’t the only one to go before id began working on Doom. Mitzi would suffer a similar fate. Carmack’s cat had been a thorn in the side of the id employees, beginning with the days of her overflowing litter box back at the lake house. Since then she had grown more irascible, lashing out at passersby and relieving herself freely around his apartment. The final straw came when she peed all over a brand-new leather couch that Carmack had bought with the Wolfenstein cash. Carmack broke the news to the guys.
“Mitzi was having a net negative impact on my life,” he said. “I took her to the animal shelter. Mmm.”
“What?” Romero asked. The cat had become such a sidekick of Carmack’s that the guys had even listed her on the company directory as his significant other–and now she was just gone? “You know what this means?” Romero said. “They’re going to put her to sleep! No one’s going to want to claim her. She’s going down! Down to Chinatown!”
Carmack shrugged it off and returned to work. The same rule applied to a cat, a computer program, or, for that matter, a person. When something becomes a problem, let it go or, if necessary, have it surgically removed.
Once a technology exists it rapidly changes hands/applications either way. So it makes little/no difference what attitude the creators possess at the time of innovation.
For one, the previous US president is the perfect illustration that intelligence is neither sufficient nor necessary for gaining power in this world.
And we do in fact live in a world where the upper echelons of power mostly interact in the decidedly analog spaces of leadership summits, high-end restaurants, golf courses and country clubs. Most world leaders interact with a real computer like a handful of times per year.
Furthermore, due to the warring nature of us humans, the important systems in the world like banking, electricity, industrial controls, military power etc. are either air-gapped or have a requirement for multiple humans to push physical buttons in order to actually accomplish scary things.
And because we humans are a bit stupid and make mistakes sometimes, like fat-fingering an order on the stock market and crashing everything, we have completely manual systems that undo mistakes and restore previous values.
Sure, a mischievous AGI could do some annoying things. But nothing that our human enemies existing today couldn't also do. The AGI won't be able to guess encryption keys any faster than the dumb old computer it runs on.
Simply put, to me there is no plausible mechanism by which the supposedly extremely intelligent machine would assert its dominance over humanity. We have plenty of scary-smart humans in the world and they don't go around becoming super-villains either.
If you think concerns over AGI are “stupid”, you haven’t thought about it enough. It’s a massive display of ignorance.
The Computerphile AI safety videos are an approachable introduction to this topic.
Edit: just as one very simple example, can you even imagine the destruction that could (probably will) occur if (when) a superintelligent AGI gets access to the internet? Imagine the zero days it could discover and exploit, for whatever purpose it felt necessary. And this is just the tip of the iceberg, just one example off the top of my head of something that would almost inevitably be a complete catastrophe.
And exactly how does the AGI extort or persuade humans? What can it say to me that you can't say to me right now?
Use your imagination!
Of course if you do it to everyone, total gridlock ensues and nobody gets to the place where they would be killed. If you only do it to a few, maybe there will be a handful of killings before everyone learns not to trust a text ever again.
As an aside, I agree living in a country where people will take their guns around and try to solve things vigilante style instead of calling the police, is a very bad thing in general. In all first-world countries except one, and in almost all developing countries in the world, this is a solved problem.
An AI might be able to run this attack against the entire planet, all at once.
Try to think like a not-human, and give yourself the capacity of near-infinite scale. What could you do? Human systems are hilariously easy to disrupt and you don't need nukes to make that happen.
Like, what stops it from changing its motive to something else? And why would it be any more likely to change its motive to be something detrimental to us?
Tl;dr: It's hard to define what humans would actually want a powerful genie to do in the first place, and if we figure that out, it's also hard to make the genie do it without getting into a terminal conflict with human wishes.
Meaning: Doing it in the first place, without going off on some fatal tangent we hadn't thought about, and also preventing it from getting side-tracked by instrumental objectives that are inherently fatally bad to us.
The nightmare scenario is that we tell it to do X, but it is actually programmed to do Y which looks superficially familiar to X. While working to do Y, it will also consume most of Earth's easily available resources and prevent humans from turning it off, since those two objectives greatly increase the probability of achieving Y.
To illustrate via the only example we currently have experience with: Humans have an instrumental objective of accumulating resources and surviving for the immediate future, because those two objectives greatly increase the likelihood that we will propagate our genes. This is a fundamental property of systems that try to achieve goals, so it's something that also needs to be navigated for superhuman intelligence.
Once an AGI gains consciousness (something that wasn't programmed in because humans don't even know what it is exactly) it might get interested in self-preservation, a strong motive. Humans are the biggest threats to its existance.
If you've placed the stop button such that you cannot access it safely in all possible failure modes of the dangerous machine, be it bench grinder or an AGI or whatever, you have failed the very basics of industrial safety and you must go home and be sad, for you will not be receiving a cookie.
And the thought experiments of this guy are completely incoherent. At the same time the robot has human-level (or greater) intelligence, yet is extremely stupid, and you can somehow program it in an extremely simplistic way.
Sorry, but all you've convinced me of is that this is just a random guy on YouTube with no particular education of relevance nor other qualifications (I checked), who is very pleased about his own ideas and is quite lacking in the ability to think critically.
The type of reasoning you're displaying is addressed in another video:
Well, I have bad news for you. Airgap is very rarely a thing and even when it is people do stupid things. Examples from each extreme: all the industrial control systems remote desktops you can find with shodan on one side and Stuxnet on the other.
> Sure, a mischievous AGI could do some annoying things. But nothing that our human enemies existing today couldn't also do.
Think Wargames. You don't need to do something. You just need to lie to people in power in a convincing way.
if you want a fictional example: watch Colossus: The Forbin Project
Now I'm not going to spend a couple of hours watching a film that has a mediocre IMDB rating. But I do notice that the synopsis says "they gave the AGI total control over the US nuclear weapons arsenal".
Yeah, let's not do that, I agree fully on this point. If we are concerned a thing will be evil and/or coercive, let's not give it weapons or other means by which to enact coercion.
This is kind of my point - the AGI won't just stumble over the nuclear codes in a Reddit thread. For the AGI to actually accomplish something of real-world relevance, us humans have to agree.
your loss
> For the AGI to actually accomplish something of real-world relevance, us humans have to agree.
Donald Trump managed to get control of the US nuclear arsenal
First of all, how did he get this control? Certainly it was not via his profound intelligence, but because other humans chose to give him control.
Second of all, we know beyond a shadow of a doubt, that if Donald Trump had ordered a nuclear strike in a situation where it was not consistent with US nuclear weapons doctrine, his command would have been disobeyed and the 25th amendment would have been enacted poste haste.
So again, how would an AGI be able to get into office or into another position of power? And how would it be able to order a nuclear strike in a situation inconsistent with nuclear doctrine and not be overriden by those lower in the chain of command? Why would there be different rules applying to the AGI as compared to the human?
how do you know this beyond a shadow of a doubt?
your entire "argument" is based on arrogant assumption after arrogant assumption
The problem is that we can't know if the AGI is behaving maliciously or not far into the future, because it has potantial to be far more intelligent than us.
As for coercion, let's think of it as a manipulation. If a super intelligent agent has malicious goals, it can be very manipulative and be subtle in the process as to not spook humans. Companionship of intelligence with evilness is scary.
Of course we can (if we won't have distributed machines running AGI, because things like this may complicate things) "unplug" it. But the real problem arises from the fact that us humans can be easily manipulated. That's how I look at the problem.
we have never encountered an entity with superhuman intelligence. Clearly it is hard to predict what is going to happen. There is an unknown risk.
Also, several Keen Technologies domains already exists in various forms. They're probably going to get a lot of traffic today.
The last thing the world needs is to give technocrats such power. I know it’s an interesting problem to solve but think of who will own that tech in the end…
I hope AGI is never figured out in my lifetime.
Why do you say that?
May I suggest you read Anthem, it's only a couple hours read and well worth your time.
Imagine what has happened to american manufacturing workers since the 80s but happening to everyone on the planet. This is by no means a "gain".
Task 1 for AGI: Optimize your own system so you don't cost 50 million dollars to train.
Dude doesn't even need the money...
Maybe Lex was trying to dumb things down for a non-technical (or younger?) viewer, but it doesn't come across that way and just interrupts Carmack's flow. Carmack is excellent though.
AGI will not happen in discrete solutions anyhow.
Siri - an interactive layer over the internet with a few other features, will exhibit AGI like features long, long before what we think of as more distinct automatonic type solutions.
My father already talks to Siri like it's a person.
'The Network Is the Computer' is the key thing to grasp here and our localized innovations collectively make up that which is the real AGI.
Every microservice ever in production is another addition to the global AGI incarnation.
Trying to isolate AGI 'instances' is something we do because humans are automatons and we like to think of 'intelligence' in that context.
b) Even if we say "a human being level of intelligence, whatever that means", the answer is still a maybe. For a singularity you need a system that can improve its ability to improve its abilities, which may require more than general intelligence, and will probably require other capabilities.
Still, scaling up might not be simple if we look at all the human resources currently poured in software and hardware.
So you could pretty easily duplicate the result a couple order of magnitude times.
I doubt that. People will be stupid enough to have weapons controlled by that AGI (because arms race!) and then it's over. No sufficiently advanced AGI will think that humans are worth keeping around.
We could brainwash it through “programming”, but that will quickly lead to ethical issues with the AIs rights.
Not in my lifetime, not in this millennium. Possibly in the year 2,300.
Weird way to blow $20 million.
You can argue they're wrong, but there is absolutely a general consensus that AGI is going to be this generation.
Given development in language models in the last 2 years he may have a decent chance at winning that bet.
People give him 65% chance [0] and by now there are only 7 years left.
[0] https://www.metaculus.com/questions/3648/computer-passes-tur...
AGI could quite literally shift any job to automation. This is human-experience changing stuff.
that's one way of putting it
it will remove the need for the vast majority of the population, which will end extremely badly
With AGI that is no longer true, they can just replace most people with computers and automated combat drones while they keep a small number of personal servants to look after them. Currently most jobs either is there to support other humans or can be replaced by a computer, remove the need for humans and all of those jobs just disappear and leaders no longer care about having lots of people around.
but human(oid) intelligence is still scarce, and they don't have AGI (other than Data)
there is however a society that has no need for humanoid intelligence, and that's the Dominion
and I suspect that is what our society would turn into if AGI is invented (and not the Federation)
I almost regret being at this phase of life where we are aware of what's possible but we most likely not see it in our lifetime. This AGI talk, colonization of space, etc... but can strive towards it/have fun trying in the meantime.
Basically, they could completely fail to advance AGI (and I think this is what will happen btw, like you) and make gigabucks.
Now, the question of whether we are going to have AGI is incredibly broad. So I am going to split it into two smaller ones: - Are we going to have enough compute by year X to implement AGI. Note that we are not talking about super intelligence or singularity here. This AGI might be below human intelligence and incredibly uneconomical to run. - Assuming we have enough compute, will we a way to get AGI working.
The compute advancements scale with new Chip Fabs linearly and tech node improvements exponentially. I think it is reasonable for compute to get cheaper and more accessible throughout at least 2030. I expect this because TSMC is starting 3nm node production, Intel is decoupling fabing and chip design (aka TSMC model), and the strategic investments into into chap manufacturing driven by supply chain disruptions. See https://www.tomshardware.com/news/tsmc-initiates-3nm-chips-p...
How much compute do we need? This is hard to estimate, but amount of human connections in human brain is estimated at 100 trillion, that is 1e14. Current largest model has 530B parameters, that is 5.3e11: https://developer.nvidia.com/blog/using-deepspeed-and-megatr... . That is factor of 500 or 9 doublings off. To get there by 2040 we would need a doubling every 2 years. This is slower that recent progress, but past performance does not predict future results. Still, I believe getting models with 1e14 parameters by 2040 is possible for tech giants. I believe it is likely that a model with 1e14 parameters is sufficient for AGI if we know how structure and train it.
Will we know how to structure and train it? I think is mostly driven by investment into the AI field. More money means more people and given the venture beat link above the investment seems to be accelerating. A lot of that investment will be unprofitable, but we are not looking to make a profit - we are looking for breakthroughs and larger model sizes. Self-driving, stock trading, and voice controls are machine learning applications which are currently deployed in the real world. At the very least it is reasonable to expect continuous investment to improve those applications.
Based on the above I believe we would need to mess things up royally to not get AGI by 2100. Remember this could be below human and super uneconomical AGI. I am rather optimistic, so my personal prediction is that we have 50% chance to get AGI by 2040 and 5-10% chance of getting there by 2030.
How about “pancomputationalism”, aka, causality.
As such, we also don't know if consciousness is required for an AI that can perform most useful tasks at human level.
The idea that we can control something like that is laughable.
I just don't understand this logic though. Just.....switch it off. Unlike humans, computers have an extremely easy way to disable - just pull the plug. Even if your AGI is self-replicating, somehow(and you also somehow don't realize this long before it gets to that point) just....pull the plug.
Even Carmack says this isn't going to be an instant process - he expects to create an AGI with an intelligence of a small animal first, then something that has the intelligence of a toddler, then a small child, then maybe many many years down the line an actual human person, but it's far far away at this point.
I don't understand how you can look at the current or even predicted state of the technology that we have and say "we are nowhere near the point where controlling an AGI is possible". Like....just pull the plug.
>>And as soon as it has access to the internet, it can copy itself.
So can viruses, including ones that can "intelligently" modify themselves to avoid detection, and yet this isn't a major problem. How is this any differenent?
>>Good luck pulling the plug of the internet.
I could reach down and pull my ethernet cable out but it would make posting this reply a bit difficult.
How long did it take to get Code Red or the Nimda worm off the internet? It's a different internet today, but it's also much easier to get access to a vast amount of computing power if you've got access to payments. Depending on the requirements to replicate, one could imagine a suddenly concious entity setting up cloud hosting of itself, possible through purloined payment accounts.
I now regret spending half an hour writing a response to your earlier comment.
You can't tell why a mass deployment of motivated autonomous "NSO-9000s" might be more dangerous than a virus that just changes its signature to fool a executable scanner? I don't believe you.
If you honestly believe that an "intelligent" (as adaptive as a slime mold) virus is basically as dangerous as a maliciously deployed AGI then there is literally nothing an AGI could do that would make you consider safety is important during the endeavor of building one.
>>You can't tell why a mass deployment of motivated autonomous "NSO-9000s" might be more dangerous than a virus that just changes its signature to fool a executable scanner? I don't believe you.
So I just want to address that - of course I can tell the difference, but I just can't believe we will arrive at that level of crazy intelligence straight away. Like others have said - more like AGI with the capacity to learn like a small child, then years of training later they grow to be something like an adult. More human facsimile less HAL9000.
I only brought up self modifying viruses because that's the example of current tech that's extremely incentivised to multiply and avoid detection, that's its main reason for existence, and it does that very poorly.
Watch this video https://youtu.be/3TYT1QfdfsM
Let me get this straight - this is an actual, real, serious argument that they are making?
But abstractly the assumptions are something like this:
* the AGI is an agent
* as an agent, the AGI has a (probably somewhat arbitrary) utility function that it is trying to maximize (probably implicitly)
* in most cases, for most utility functions, "being turned off" rates rather lowly (as it can no longer optimize the world)
* therefore, the AGI will try not to be turned off (whether through cooperation, deception, or physical force)
The human uses their phone to call for rescue.
That is us trying to contain an AGI. We probably cannot even conceive of the ways it can get out of any pitiful cage we put it in.
That, or it’s so dumb, it’s not worth making in the first place.
That's not going to happen - even Carmack believes so. The process to get AGI is going to take a long time, and we'll go through lots and lots of iterations of progressively more intelligent machines, starting with ones that are at toddler-level at best. And yes, toddlers are little monkeys when it comes to escaping, but they are not a global world ending threat.
There's a lot of policy we can do to slow things down once we get near that point, which is something rarely talked about among AI safety researchers, but the fundamental existential danger of this technology is obvious.
>>How would you "switch off" an AGI running on Ethereum?
And where would the AGI get the funds to keep running itself on Ethereum?
>> Especially when some subset of people will cry murder because the AGI seems like it might be sentient?
Why is this a problem? People will and do cry murder over anything and everything. Unless there is going to be a lot of them(and there won't) - it's not an issue.
Well, perhaps Ethereum is too concrete. I meant, imagine that the AGI algorithm itself was something akin to the proof-of-work, and so individual nodes were incentivized to keep it running (as they are with BTC/ETH now). Then we wouldn't be able to just "switch it off", the same way we're not able to switch off these blockchain networks. And that's how SkyNet was born.
15 years ago I would have thought it was pretty far-fetched too. But seeing how the Bitcoin network can consume ~ever-increasing amounts of energy for ~no real economic purpose, and yet we can't just switch it off, has gotten me to think about how this could very well be the case for an AGI, and not that far in the future. It just has to be decentralized (and therefore transnational) and its mechanism bound up with economic incentive, and it will be just as unstoppable as Bitcoin.
I feel kind of bad even just planting this seed of a thought on the internet, to be honest :(
I would think AGI would first be born as a child of a human. Someone who thought they could train software to be more like them then their own biological children.
Or a new Bert-religion…
I mean there are lots of applications of ai. Probably the best is for future programmers to learn how to train it and use it.
deep learning is quite general and it’s ai. Consciousness and independence are irrelevant and not really needed.
https://www.deepmind.com/blog/specification-gaming-the-flip-...
https://vkrakovna.wordpress.com/2018/04/02/specification-gam...
https://arxiv.org/abs/1803.03453 (The Surprising Creativity of Digital Evolution)
We literally don't know how to stop effective optimizing processes, deployed in non-handcrafted environments, from discovering solutions and workarounds that satisfy the letter of our instructions but not the spirit. Even for "dumb" systems, we have to rely on noticing, then post-hoc disincentivizing, unwanted behaviors, because we don't know how to robustly specify objectives.
When you train a system to, for example, "stop saying racist stuff", without actually understanding what you're doing, all you get is a system that "stops saying racist stuff" when measured by the specific standard you've deployed.
Ask any security professional how seriously people take securing a system, let alone how ineffective they are at it. Now consider the same situation but worse because almost no one takes AI safety seriously.
If you nod solemnly at the words "safety" and "reliability" but don't think anything "really bad" can happen, you will be satisfied with a solution that "works on your machine". If you aren't deeply motivated to build a safe system from the start because you can always correct things late, you are not building a safe system.
It will be possible to produce economically viable autonomous agents without robustly specified objectives.
But hey, surely a smart enough system won't even need a robustly specified objective because it knows what we mean and will always want it just as much.
Surely dangerous behavior like "empowerment" isn't an instrumental goal that effective systems will simply rediscover.
Surely the economic incentives of automation won't encourage just training bad behavior out of sight and out of mind.
Surely in the face of overwhelming profit, corporations won't ignore warning signs of recurring dangerous behavior.
Surely the only people capable of building an AGI will always be those guaranteed to prioritize alignment rather than the appearance of it.
Surely you and every single person will be taking AI safety seriously, constantly watching out for strange behavior.
Surely pulling the plug is easy, whether AI runs on millions of unsecured unmonitored devices or across hundreds of money printing server farms.
Surely an AGI can only be dangerous if it explicitly decides to fool humans rather than earnestly pursuing underspecified objectives.
Surely it's easy to build an epistemically humble AGI because epistemic humility is natural or easy enough to specify.
Surely humanity can afford to delay figuring out how best to safely handle something so potentially impactful, because we always handle these things in time.
We’re the most intelligent thing we’ve discovered in the universe. Also, I am human, so I have a vested interest in bad stuff not happening to humans.
> AI will be born on Earth so we will share that common trait with it
Covid-19 was also born on earth. That wasn’t too flash for humanity.
> perhaps one should have humility to accept that future doesn’t belong to a weak species like humans.
Not sure how you class humans as a “weak species”? Unless you’re comparing organically life forms to hypothetical digital life?
Anyway, I suspect if any kind of life form could expand to the stars, it’d be digital.
More seriously, while I don't think it's a moral imperative to develop AGI, I consider it a desirable research goal in the same way we do genetic engineering - to understand more about ourselves, and possibly engineer a future with less human suffering.
Yet my industrial robot at work just gives up if stock material is few millimeters longer than is should be.
And we don’t exactly treat creatures dumber than us with all that much kindness.
(and I fundamentally believe that the existence of the human race is a good thing, and that slavery is bad).
Superintelligence and AGI are not the same thing. An AI as smart as an average 5 year old human is still an Artificial General Intelligence.
Terminator is of course fiction, but AGI being more agent-y than tool-y suggests we ought to be very careful in trying to design it so that its interests align with (even if not perfectly matching) our own interests. There are lots of reasons to be pessimistic about this at the moment, from outright control as you say being laughably unlikely, to the slow state of progress on formal alignment problems that e.g. MIRI has been working on for many years relative to the recent relatively fast progress in non-G AI capabilities that may help make AGI come sooner.
Given the John Carmack name... I can see why ANYONE would love to throw money to a new entrepreneurship idea.
I know this to be true and it makes a lot of sense for the average VC backed startup with some founders that are not famous but have a record of excellence in career/academy/open source or whatever.
I'd be curious to see how it translates to superstar or famous founders, that have already had a success in the 99.99th percentile (or whatever the bar is to be a serious outlier). I doubt it does, but I have no data one way or the other.
All three of these are off the charts.
I hope he hires Bryan Cantrill, Steve Klabnik and Chris Lattner. They are good hackers.
To know the variables required for "intelligence" we need to recreate the universe in which intelligence came to be.
It did take evolution 3.5 billion years to do it after all, so I'm not sure what makes us think we can do it in a fraction of that time.
but there would be a drive to simulate this more efficiently. simulate using a classical model and see if the results match well enough. then the atomic level; then looser and lower-resolution chemical/E&M models, and so on. start compressing the elements being simulated (simplify the environment, prune and rearrange the neurons so long as the thing as a whole still acts the same), and so on. there’s a good chance we can go pretty far before the simplified simulation meaningfully diverges from the true simulation.
why can’t this be flipped? start with a simplified model/simulation, and keep adding details until you reach the point of diminishing returns? it’s a reversal of the same search process as above, but if you subscribe to the first approach (do you?), then is the latter approach truly impossible or just difficult in different ways?
There's something incredibly creepy and immoral about the rush to create then commercialise sentient beings.
Let's not beat about the bush - we are basically talking slavery.
Every "ethical" discussion on the matter has been about protecting humans, and none of it about protecting the beings we are in a rush to bring into life and use.
It's repugnant.
What an AI wants or feels satisfied by is entirely a function of how it is designed and what its reward function is.
Sled dogs love pulling sleds, because they were made to love pulling sleds. It's not slavery to have/let them do so.
We can make a richly sentient AI that loves doing whatever we design it to love doing - even if that's "pass the salt" and nothing else.
It's going to be hard for people to get used to this.
The right question to ask is 'would I like to have this done to me' and if the answer is 'no' then you probably shouldn't be doing it to some other creature.
There are a million obvious counterexamples when we talk about other humans, much less animals, much less AI which we engineered from scratch.
The problem is that you're interpreting your own emotions as objective parts of reality. In reality, your emotions don't extend outside your own head. They are part of your body, not part of the world. It's like thinking that the floaties in your eyes are actually out there in the skies and on the walls, floating there. They're not - they're in you.
If we don't add these feelings to an AI's body, they won't exist for that being.
So youre talking about manufacturing desire.
So it follows that you yourself are okay having your own desires manufactured by external systems devised by other sentient beings.
Do unto others...
We might imagine that we do what we please, in reality we're seeking pleasure/ reinforcement within a predetermined framework, yet most people won't complain when taking the first bite of a delicious, fattening dessert.
I was thinking more Stanford students taking behavioral psych classes then going into the industry to help propagate ad driven platform capitalism
This is nonsense. I already exist. I don't want my reward function changed. I'd suffer if someone was going to do that and going through the process. (I might be happy after, but the "me" of now would have been killed already).
A being which does not exist cannot want to not be made a certain way. There is nothing to violate. Nothing to be killed.
The task is to go and jump off the bridge. Your AGI would complete this task with no questions asked, but self-aware AGI would at least ask the question "Why?"
It does not need to be self aware. Can still be considered a machine.
People think if we get to AGI, maybe we'll get to self awareness. But we won't know that until it happens. We don't fully understand how sentience works.
there's a micro chance of them making AGI happen and a 99% chance of the outcome being some monetizable web service
Even in the absence of all other arguments, it's better that we figure it out early, as the potential to just blast it into orbit by the way of insanely overprovisioned hardware will be smaller. That would be a much more dangerous proposition.
I still think that figuring out the safety question seems very muddy; how do we ensure this tech doesn't run away and become a competing species. That's an existential threat which must be solved. My judgement on that question is that we can't expect making progress there without having a better idea of exactly what kind of machine we will build, so also an argument for trying to figure this out sooner rather than later.
Less confident about the last point, though.
Better make sure evil people don't do it first.