John Carmack and Rich Sutton partner to accelerate development of AGI
globenewswire.com
globenewswire.com
Here’s an updated link: https://youtube.com/watch?v=uTMtGT1RjlY
People want to eat.
Safety here being say; a user asking for instructions on how to make a dirty bomb and the model responds with "Sorry, I can't do that ethically".
If not outright blocked, instructions do seem to be weighted down.
In that sense it is mathematically equivalent to (say) applying an L2 regularization penalty to reduce the occurrance of higher-order terms when fitting a polynomial. Strictly it will produce a worse fit on your training data, however it is done because out of sample performance is important.
The pilots need oxygen masks as a backup while they quickly descend to a safer altitude. Unless you're flying over the Himalayas and can't drop that low, the passengers should be fine.
https://en.wikipedia.org/wiki/Emergency_oxygen_system#Incide...
Given the self-evidence of no existential risk, there is then an objection to “dumbing down” where model performance suffers due to RLHF (the “alignment tax” is a real thing), and often but not always this includes an objection to wokeness or perceived unwillingness to speak “truths” or left-wing bias being imposed on models.
Imagine something that thinks 1000x faster than you, is thousands of times smarter than you, never sleeps, never eats, can self-improve, can self-replicate and it didn't like you, or someone deployed it against you and your ethnicity, family, whatever?
It would be devastating. You'd be fucked. We'd all be fucked.
Personally I don't think such a creation will ever be practical or work the way we think it will, but if we'd like to keep enjoying the world and value our autonomy, I think we should be very cautious when implementing it.
Yeah that would be bad. We're also way way off from that point. A real threat that we actually face now is someone like Sam Altman getting congress to make it illegal for individuals to run models locally because then he can't charge us subscriptions for them or harvest information about our lives, thoughts, and feelings. Imagine an AI therapist (I've already done this with llama 65B a bit) with telemetry that lets commercial interests know all your specific deepest desires insecurities and fears so they can sell you stuff.
Skynet isn't the pressing issue, a Corpo-Techno-Stasi is.
We will need AGI to ensure continued existence of the human race. Without it, our destruction is assured.
What makes you think that?
He worked at one of the most user-hostile companies on the planet for a decade, and only left because he wasn't happy with its operating efficiency.
Someone with that mentality might make technological progress on their own, but I'm not confident that they value morals and human safety over technology. His dismissive comments about AGI safety are certainly a concern.
He, like many engineers, think intelligence is a kind of engineering; so you just need to build something that works.
Intelligence is all about coping under failure. It is a lot more like, 'acting safely when you dont have the answers you need' than 'answering questions correctly'.
That latter impulse resigns the whole field to cheap mimicry. These guys arent even starting from the right point, nor could they, being engineers.
Intelligence is what intelligence is. You don't know what it is even if you dearly believe you do. Nobody really does. We construct ways to probe it, at least to probe the aspects of it we find useful.
>That latter impulse resigns the whole field to cheap mimicry.
"Mimicry" is an assertion that is essentially meaningless. You mean to tell me the plane is fake flying ? Lol Okay. The bee might just as well accuse a bird of mimicry. This is a very rubbish line of argument an astonishing number of people seem to have trapped themselves in. The idea that internal processes matter anything more than a means to an end.
Right, but you know enough -- of course! -- to say that I do not know.
I've written the first draft of a book on animal intelligence. I doubt neither you nor Carmack have even thought that a study of actually intelligent systems (animals) should even enter the conversation.
No no, of course, it's just going to be a modulo trick.
I simply know that sincerely believing something to be so has no actual bearing on truth. I know that results are what matter because that is what we employ with each other. And because that's the only thing Evolution "cared" about.
>I've written the first draft of a book on animal intelligence.
That's good for you.
>I doubt neither you nor Carmack have even thought that a study of actually intelligent systems (animals) should even enter the conversation.
I do know that this study is not the be all end all. Why would I be concerned with following one example of biology to a T when even Nature itself does not (Flight and the bee and the bird amongst others).
>No no, of course, it's just going to be a modulo trick.
There is no trick. There is only what flies and what does not. Trick flying is not a distinction that exists. Reality does not play tricks.
This is the first time I see this definition of intelligence.
My first impression is that this is a relevant aspect of intelligence, but very far from "all about". (other aspects being things like "abstract thought", world knowledge, agency, ...)
> That latter impulse resigns the whole field to cheap mimicry
AIs are inherently non-human intelligence. (I like the shoggoth-comparison [1]). AI companies go to great length to make them behave more human-like. If you look at earlier AIs (e.g. when they play games against humans), you will find that human experts (e.g. the people they play against) they will often note, that the AI behaves very unhuman and un-understandable in certain aspects (e.g. they make a move that seems utter non-sense at first, but will be very important 40 moves latter in the game). (I should note, that the opposite is true as well: sometimes the AI surprises the expert by exhibiting very human-like behavior)
The reason intelligence has evolved in animals is because we do not have the answers. All of modern AI is focused on how to extract behaviour from answers -- rather than how to cope when they arent present.
Intelligence is the solution to the problem that evolution cannot adapt life fast enough. Evolution is the kind of 'intelligence' AI has: delete the failures; know the answers.
The intelligence of interest, in animals, is live adaption to one's environment. It's ecological rationality brought about by sensory-motor adaption.
This is why we find the NN-type solutions to problems deeply unsatisfying. We arent missing something: there is no "there" there. It's just a compression of the answer space.
This is really disappointing: the intelligent solution is the interesting one! Modern AI finds ways of solving problems without intelligence.
Emphasis on the word 'artifical'
It is quite plainly possible to produce a system at least as intelligent as a human literally through random chance, based solely on what happens to work. That is, in fact, the sole example we have for how to make anything like AGI.
Mutations are random - they provide the raw material for evolutionary selection. Evolution itself is absolutely not random chance.
Unless you were referring to something else?
There are a lot of people out there with an opinion on AI right now. Carmack at least has done his homework and he's identified some solvable technical problems. And he's solving those. That's how technology moves forward. He's a bit on the spectrum, so yes, he doesn't have a lot of patience for people spouting nonsense in a hysterical hand wavy way.
Part of AI safety is the likes of China and a few other countries not getting there first and using it against us. They aren't going to slow down because some activists with ethical concerns in the Bay area. Unlike some of the irrational fears, that one is very real because they already have a lot of people working on this and they own a lot of chip factories.
So our choices are to wait for history to happen or make it happen.
The thing is, however, that regardless of how much nonsense might be out there, well though-out arguments for AI risks do exist. Dismissing them together with the nonsense has to be either a conscious choice or utter ignorance. Either of which would be very concerning for someone aspiring to develop such a high-impact technology (and quite possibly being capable of doing it).
We(us humans) will not use A.I. to compete against each other. A.I. will be a competing top species.
In addition to your concerns.
It's quite terrifying how, as we've chosen an apparently very easy path to bake our preferences and quirks into intelligent systems, people became very "responsible" and concerned for survival of human race, parroting alarmist rhetoric that precedes not only LLMs, but even RL successes of early Deepmind and just cites vague shower thoughts of Bostrom and such non-technical ilk. Say what you want about LLMs but there's zero credible reason to perceive them as a more risky approach!
They're not these rigid, logic/rule bound systems that struggle with human emotions. By all accounts, GPT-4 is as emotionally competent as it is on anything else.
I suppose there's something unsettling about building Super Intelligence in humanity's image.
Sure, harmless now, but imagine you let one of these LLMs actually use your browser, logged in as you, and it does something that you didn't intend it to do. We are getting to that point fast where you can ask an LLM to check your emails and answer them for you, order food from uber eats or use some internal company or government system that can control a huge number of different variables.
Funnily enough it seems human error (through bad or misguided LLMs prompts) is going to become much more common as LLMs do more things for us. You would kind expect the opposite from automated computer systems, yet here we are.
It's the same reason people like Jeff Bezos spent 250 mill a decade ago on the Washington Post.
And Musk spent 44 billion on Twitter.
Words matter - words can change the world.
Better copy over here: https://news.ycombinator.com/item?id=37654684
None of the AI features I’ve encountered being added to applications are opt-in.
You could argue something similar happened with cars, which continue to shape American culture and development.
I think AI tools have the chance to help make the world a much better place, but it all depends on how it's handled and implemented.
Personally, I don't trust any single group or person with the full power of AI, so I'm hoping open source models regulate any potential power imbalances.