John Carmack and Rich Sutton partner to accelerate development of AGI
amii.ca
amii.ca
I wondered the other day what the result would be if you strapped a microphone on a baby and every utterance that both the baby and mic heard was fed through speech-to-text and went toward training the baby's personal LLM.
By the time the child had grown to an adult I wonder what kind of results their LLM would produce and the degree to which it might compare to what the child (now grown) would answer?
An "LLM" that could take in visuals via a baby-mounted camera is of course a whole other discussion. (Though I'm sure there are people training ANN's with video feeds rather than the static images that feed systems like DALL•E.)
But Humans are basically long running LLMs that are retrained in real-time. We are the product of our environment.
Occasionally the act of writing out an idea in words clarifies or changes my mind about that idea, causing me to edit or rewrite what I’ve already written.
It might be calculating the next word, because you can only write one word at a time, but you can't say that the current next word isn't influenced by a few words ahead. Don't think the current understanding of what LLM's do internally can rule that out.
And if you believe you do that subconsciously, then how can you be sure you don’t subconsciously plan a few words ahead?
But it doesn’t write better as in has new, challenging ideas. Or ways to move/relate to humans on a personal level.
GPT writes clear, concise, authoritative, boring, and generic text.
Shaping your words to sound correct is very common. Both in speech and in writing. Sometimes it is finding how to fit a word you want to use into a sentence. Sometimes it is building a rhyme.
You may feel that you go a word at a time, but that really shows how embedded language is.
In their calculating the 'next' word, as part of that 'weighting', are the 'simple rules' for future words, that you are saying humans do but LLM's can't.
Anyway, point is 80% of this comment was already in my head before I started typing; the other 20% was light editing and an H2G2 reference.
Humans don't just process information; they experience emotions, desires, and subjective experiences that are deeply intertwined with their cognition. LLMs don't have feelings, motivations, or consciousness.
Humans have inner subjective experience, self-awareness, and the ability to reflect on our own existence. LLMs don't.
Humans can adapt to a wide range of environments and situations, drawing from a complex interplay of instincts, learned behaviors, emotions, and rational thought. LLMs are much more limited in their adaptability, since they focus primarily on the tasks they were designed for.
Human cognition has evolved over millions of years and is rooted in a complex biological system, the brain. Yes, both LLMs and human brains process information, but the underlying mechanisms, structures, and functions are vastly different.
I really wish people would stop this sort of cavalier reductionism of humans by saying we are basically LLMs. It isn't true.
As in, the part with producing language. I.e large language model.
"In the end, we are self-perceiving, self-inventing, locked-in mirages that are little miracles of self-reference."
— Douglas Hofstadter, I Am a Strange Loop, p. 363
https://en.wikipedia.org/wiki/I_Am_a_Strange_Loop
(Knowing nothing about AI, I have no idea how Hofstadter's philosophies have held up since.)
We have no test capable of determining whether or not they have those things, not even if we disregarded the limitations of our current technology capabilities and are only asking hypothetically how to differentiate.
We also don't have that for animals, or even other humans — I know I have an experience of being, but no way of telling if someone else who says they do actually does. I have to assume at least some of y'all do or humanity wouldn't have written about it since at least Descartes.
People with aphantasia report being surprised when they realise that other people do have mental images, and that they previously thought such things were invented by the film industry as a metaphor. By analogy, there may well be humans out there without qualia, who just learned to mimic the language of those of us who do, which is after all exactly what LLMs must have done if they don't have inner subjective experience. Philosophers call them P-Zombies.
AlphaGO satisfies all of the requirements you listed for a human. Creativity, problem solving.
Just put it on a game loop, with more variables.
That’s like saying a physics simulation is basically an entire sub universe on your computer.
It sounds true, but it’s just not. It’s a gross oversimplification
I think Gödel had a proof for how it’s impossible to fully describe a system from within that system. That’s the nail in the coffin for AGI.
No matter how much data we give it, no matter how big it is, it’ll never be “human intelligent” since it’s impossible for us to describe a loss function for being human or describe being human in a dataset.
We’ll never be able to evaluate it, since we can’t fully describe what it means to communicate because to do that we’d need to communicate it and that process can’t be fully self describing.
Not to say AI isn’t useful or impressive, but it’ll never be comparable to humans, truly.
Gödel's incompleteness theorems
https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_...
Gödels theorems are about formal axiomatic theories. To apply them to human intelligence, you'd have to prove that human intelligence springs from formal axiomatic theories. I don't think this is possible, which would mean that you can't apply the theorems.
> No matter how much data we give it, no matter how big it is, it’ll never be “human intelligent” since it’s impossible for us to describe a loss function for being human or describe being human in a dataset.
How do you know? If we were able to fully record whatever is going on in someones brain, we should be able to build a loss function for it. How do you know that this is fundamentally impossible?
> We’ll never be able to evaluate it, since we can’t fully describe what it means to communicate because to do that we’d need to communicate it and that process can’t be fully self describing.
Why not? Again, if you argue that this is due to Gödels theorems, you'd have to prove that our communication itself is based on formal axiomatic theories.
It applies to everything we could say or think or measure about intelligence (and anything else). All that probably "spring from" things that don't (rely on our axioms), but they are not accessible to us, so that doesn't really help.
How do you know? Gödels proof doesn't support your claim since it only applies to formal axiomatic theories. Do you have an alternative proof for his theorems also applying to all other systems?
Maybe gravity makes things fall down, sure, but maybe there are tiny kobolds in the spaces between all particles with little clipboards that calculate the correct motion and cast spells to move them. I'm not trying to be a smartass, but I honestly tried and could not find bedrock. Can you name (or even just think) something that doesn't rest on something else or an assumption? I honestly can't.
> Gödels proof doesn't support your claim since it only applies to formal axiomatic theories.
"only"? I'd say those axioms are a superset of the sloppy stuff we throw around in our day to day, like "this is a chair"; if we drilled down on our informal speech and thoughts, we'd at best arrive at such axioms, which ultimately rest on things we simply posited (because otherwise there would be nothing to think about, and no way to think about it -- I'm not knocking it per se, just the idea that the quest for truth could possibly ever be complete, which makes it no less noble IMO).
I'm not sure how a "bedrock" relates to the question whether an intelligence can ever fully describe what an intelligence is. When answering this question, we don't need to find a "natural" bedrock, since the assumptions we choose are the bedrock we build on. As long as those assumptions align with reality to the best of our knowledge and the end result passes all tests we can think of, what does it matter whether there might be more to know? Of course it doesn't mean we should stop searching, but it also doesn't mean we should not even try. There are many such unfalsifiable statements, but that doesn't mean they stop us from answering other questions.
> "only"? I'd say those axioms are a superset of the sloppy stuff we throw around in our day to day, like "this is a chair"; if we drilled down on our informal speech and thoughts, we'd at best arrive at such axioms, which ultimately rest on things we simply posited (because otherwise there would be nothing to think about, and no way to think about it -- I'm not knocking it per se, just the idea that the quest for truth could possibly ever be complete, which makes it no less noble IMO).
I don't think this is true, and if you can prove it, you might earn a Nobel prize. "Formal axiomatic theories" are well-defined - as Wikipedia states, they are "formal systems that are of sufficient complexity to express the basic arithmetic of the natural numbers and which are consistent and effectively axiomatized. [...] In general, a formal system is a deductive apparatus that consists of a particular set of axioms along with rules of symbolic manipulation (or rules of inference) that allow for the derivation of new theorems from the axioms."
Can you try to describe how you'd "drill down" on informal speech to transform it into such a system? There are many, many examples for systems that are absolutely not based on formal axiomatic systems.
As I said, just keep asking "why?" or "what does that mean?", then repeat that with the answer. Sooner or later you hit an assumption and a shrug. I wouldn't understand Gödel's proof even if I tried to, I'm sure -- it "rings true" because it matches my own intellectual observations regardless where I turn.
> As long as those assumptions align with reality to the best of our knowledge and the end result passes all tests we can think of, what does it matter whether there might be more to know?
It matters for the question whether you can fully describe a system from within that system, that's all. But I'd argue even whether we made an effort or no effort, whether it passes all the tests we came up with or doesn't, doesn't really matter (in regards to that question) because any ground we cover won't bridge what remains an infinite distance. I still think it's good, but it's more like going for a walk each day: you always arrive where you started out, you're just getting fresh air and what other temporary benefits come with it. It beats just staying where you started out.
I can't find the quote but apparently Werner Heisenberg said something along those lines, that we basically set out to find the bed rock of reality, but more and more are just facing ourselves, that is, our instruments of measurement and ways to conceptualize things. And again, I don't know jack about quantum mechanics and don't want to call on the authority of Heisenberg and Gödel. But I hear they know their fields, right, and it matches everything I know in any area, both the ones I am bad in and the ones I am really bad in.
I'm not saying it's a problem, just that that's how it is. But thinking you know the ultimate and final truth because it passes all tests (e.g. witches sink), and thinking software is actually intelligent because it convinces you it is, when it really isn't, can be super mega dangerous. And comments how picking the statistically most likely word is "basically what our brain does" [0] etc. show an even worse possibility; where we take the shortcut of just confusing what we are creating with us, because then it's easy and now we know how we think (when we really don't, not remotely). It just generally seems backwards to start out with the goal of "AI" when we can't even describe what we're looking for, much less how to build or find it. Having no more than "we'll know it when we have it", plus eagerness to claim we have it, is a recipe for at least a lot of circus, if not disaster.
[0] And that's on HN. Now ponder, for example, the average opinion of HN on say, whether banks should limit your passwords in all sorts of weird way that imply they're not hashing them, and how much worse the "real world" is. In this case, even the people at the forefront are so keen to move fast and break things, so the "real world" is pretty much doomed I'd say.
> As I said, just keep asking "why?" or "what does that mean?", then repeat that with the answer. Sooner or later you hit an assumption and a shrug.
I still don't understand your assumption. Do you think that any axiomatic system is a formal axiomatic theory? As I've said before, this term is well-defined, and I don't see how you could "drill down" on natural language to arrive at such a system. There are many axiomatic systems that are not formal axiomatic theories, and Gödels proof doesn't apply to those.
> I wouldn't understand Gödel's proof even if I tried to, I'm sure -- it "rings true" because it matches my own intellectual observations regardless where I turn.
This is why I've been asking about how you'd bridge the gap between Gödels proof and your assumption, because Gödels proof applies strictly to one thing, and you seem to apply it to everything, even if it doesn't meet the requirements of the proof. But I guess you're arguing from a philosophical standpoint, not a logical one.
If it was a nail in the coffin for AGI, then humans, who have exactly the same limit, wouldn't count as a general intelligence.
While such a definition would be possible, I don't think it's useful.
It sounds like maybe you're arguing that humans will never be able to conclusively determine if an artificial intelligence is equivalent to a human intelligence, on the basis of a theory that a human can't describe precisely what it is to have human-equivalent intelligence from inside the system of a human brain. Humans build things that are too complex for any one person to hold the entirety of in their conscious mind all the time, by working together, or by organizing it into simpler pieces. But even setting that tangent aside, if your theory is accurate, would you accept that an AGI could prove it was more intelligent than a human by successfully describing human intelligence and how to create an equivalent AGI?
Because we’re unable to make a perfect, totally correct metric, I find it unlikely that any of the current generation of AI will get anywhere near human level.
Again, not that these new models aren’t extremely useful or impressive, but not really “intelligent” as a human is.
Technically, this is also a nail in the coffin for humans. Humans are also inside a system.
Even back with Kant, humans have been trying to figure out how to see beyond their own systems.
But to be clear, humans don't have emergent reasoning from language, we learn language as part of our overall reasoning. The short evidence for that being that children are capable of solving logic and spatial puzzles before they learn how to speak. Humans learn concepts like object permanence before we learn language complicated enough to describe that concept. And obviously people are capable of reasoning without learning how to write or interpret text tokens, there are plenty of illiterate people in the world who are nonetheless indisputably intelligent agents.
So ignoring other differences about how prediction works, humans are not similar to LLMs in the sense that LLMs are language models that when large enough either develop (or appear to develop depending on who you ask) reasoning capabilities. And that's not how humans work; we don't learn text tokens before we learn how to reason.
But very often when people make this claim they're trying to make a broader claim about neural networks or the role of prediction in learning in general. People might disagree or agree with the broader claim, I still think it oversimplifies how humans work, but the point is -- they're not actually saying something specific about LLMs, even though it sounds that way sometimes. It's just that the terminology gets conflated in people's heads.
We can have a debate about the similarities and differences between humans and neural networks, but I don't think anyone would seriously claim that GPT-4 in specific works the same way as a human does. I think people are using LLMs to refer to a broader category of AI research.
And, LLM's are not all that a human can do. Language is not everything about a human.
But there is an argument that there is part of the brain that produces language, and it has some LLM characteristics. It's just that the brain is bigger and does more than an LLM. So the brain is not an LLM.
The brain has many components. What happens when you take the problem solving of something like AlphaGo/AlphaStar, with the Vision processing in Cars or DaLLe, and the language processing in LLM. Add in hearing, touch.
It starts to look like the components of a brain.
We don't learn to read or write by doing token prediction (if we did, subjects like spelling would be much easier). In fact, there was a movement in schools to teach reading by asking students to predict what words might be based on the context of the sentence, and it was a disaster and led to increased illiteracy rates and schools have started shifting back to phonics. Not only do we not learn that way, when we try to learn that way it leads to worse education outcomes.
The reason why brains are not like an LLM is not because we also have eyes and an LLM doesn't, it's because just isolating out our language "models", we are trained differently and interact with the rest of our brains differently.
If our language centers of our brain worked like an LLM, we would expect language skills to develop faster than reasoning capabilities within our writing/speaking. A primitive LLM like GPT-2 has very limited processing ability but is still able to imitate a wide range of styles and is still able to "speak" in a grammatically correct way. Humans are the opposite: we start out communicating complex ideas poorly and we start out using language poorly. We master language as a processing tool before we become competent at using language in general.
I read about a paper[1] a while back, it was a rather unpleasant animal study in cats. Using two kittens, one was free to look around, but the other they immobilised in some way, and made it see what the other kitten saw as it looked around. They discovered the immobilised kitten's visual processing did not develop normally whereas the mobile kitten's did, suggesting that it's not just the sensory input, but it being feedback to some internal agency within the brain.
I suspect the same thing would be true for attempts to develop AGI by giving them an audio-visual copy of a human's environment growing up: that internal state driving (and getting feedback from) the action to investigate/interact with the world is key.
1: https://arxiv.org/abs/1604.03670 "Interactive Perception: Leveraging Action in Perception and Perception in Action"
Many of the papers go over my head at times of course (despite my interest, I'm firmly a layman!)
More generally, I've had a lot of success with searching for meta-analysis papers on topics I'm interested in, since the authors have already done the hard work of researching the literature and presenting core takeaways (as well as providing terms for more searches).
1: https://bsky.app/profile/abeba.bsky.social or https://twitter.com/Abebab
2: https://bsky.app/profile/olivia.science or https://twitter.com/o_guest
And the AI folks argued even then that the only way to create a language model that truly "understands" would be to put it in a robot that can experience the world in the same way as a human.
> I suspect the same thing would be true for attempts to develop AGI...
On the other hand, I would not be surprised if this did _not_ generalize and translate to LLMs. Biological entities on earth tend to have tons of fairly arbitrary developmental legacy which may or may not translate well to an artificial one. In the experiment, they didn't make billions of clones of the cat with a variety of different parameters and check which one worked, they just observed that yes, this standard v1 cat has some specific hard-wired developmental tendencies. It is not surprising that taking it out of the environment it was evolved for messes things up, but I would not take the extra step that this means that something is essential in general.
In school, we learn the lesson then get the test. In life we get the test then learn the lesson.
The Text is the environment, The Text is the world and they are very much interacting with it.
But then, is it ethical to build in pain receptors? Feels like we're brushing up against the territory of gods.
I ask this as I think the ethical dilemma should more about implementing suffering as a long term damaging effect of a stimuli.
When I think about pain I see it as a level that triggers a need to change something. Our brains are perceiving this level in a very uncomfortable way so we call this pain.
So I really think that without consciousness we cannot implement pain or suffering as we humans perceive it.
Perhaps, but I guess it would depend on your definition of consciousness. We do not extend that definition to most animals and yet they definitely feel pain.
You're right, the "feeling" of pain is ephemeral and hard to translate to a computer program. It would probably take a while doing potentially nasty experiments (to both man and machine) to nail it down.
Published in 1990, by the way.
It very likely would not compare. Humans are shaped by their subjective experiences not by incoming data and you cannot know what a subject is experiencing solely through the incoming data (except if you have a theory accurately modeling the mind of the subject which is exactly what we are currently missing).
Your LLM won't experience the subject's heartbreaks, joy, grief, shame, hope etc. It will have heard and be able to talk about those, but it will not give accurate answers about what it felt like. Also it won't be able to predict/model accurately how the subject has been changed by those experiences so it could make very wrong assumptions about what the subject could/would do in the future.
This is why I got into AR initially, because a computing system needs the input persistence of a literal parent or in the case of a self learning agent - something like a baby to perfectly observe in order to be able to create the data environment necessary for learning at the rate humans learn.
You could do it with a collection of sensors, but I think the idealized implementation is basically a perfect recreation of the sense inputs of a person as well as monitoring the person to infer the precise Markov Decision Process.
Why is this urgent?
But I agree with you ... it is not really urgent since we know answers for the most of the problems but do not like the solution.
Any solution that violates this boundary condition is "objectively bad", that is, cultures that follow this solution disappear and thus lose the evolution game.
1. There are a set of things that are nearly objectively true across nearly all cultures. For example, random killing for selfish purposes is not ok.
2. I'm not a moral relativist, in two senses. First, the "weak" sense: I would prefer to live in some moral systems more than others. Second, in "nearly-strong" sense: some moral systems are 'nearly' objectively better than others (see point above). (Note: my definitions are imprecise; still working on how to square them with existing philosophical writings.)
3. Even for people that evaluate the morality of something according to its effects (consequentialists), there is considerable room to debate the relevant time frame.
Not at all. Where are you getting that?
> That's like giving up and saying there are no solutions to hard problems.
Not at all. I'm simply defining a solution as one that has some probability of traction.
What an absolutely wonderful insight put extremely succinctly, thank you!
It seemed appropriate to ask AI about the meaning of the phrase:
The phrase "bring urgency to something" means to inject a sense of importance and immediate attention to a specific issue, project, or situation. The aim is to motivate people to prioritize the task at hand and to act more quickly than they might otherwise.
Though it is an almost-inescapable conclusion of being a utilitarian and significantly smarter than average.
If you can defraud millions and put that money to work saving lives and buying many more QALYs than the defrauded would have otherwise had, you have a moral obligation to do so.
...there are many ways to make utilitarianism fall apart, but that one's my current favorite.
So defrauding millions to accrue QALYs (for whom?) is morally acceptable?
> Though it is an almost-inescapable conclusion of being a utilitarian and significantly smarter than average.
A modicum of spirituality or religion will quickly quash this. So you have to add atheism to the requirement. Or at least non-spirituality or belief in materialism (I'm not a philosopher so may be using that term incorrectly).
Yep, you're absolutely right - I forgot to specify that.
Most utilitarians don't believe in transcendence, in my experience - not believing in it is usually why they're utilitarian (trying to salvage any argument for behavioral standards other than "I like it when other people do what I like").
Under most utilitarian theories of ethics, I think it's morally required, if you can show the fraud really does yield a significantly higher QALY count.
The best out I can see to argue the fraud is not required is to show that another course of action will yield even more QALYs. Then you'd be obligated you do that instead.
That, of course, comes to another fundamental flaws of utilitarianism, namely the belief that you can know what the results of your actions will be with any meaningful completeness or certitude.
You can't.
Without that it's all just excuses that justify whatever you want to do.
In parallel threads, a subscriber to this belief mentions probabilities. As long as the probability of an outcome is likely (based on variables and formulas who defines, I don’t know), the means justify the ends. So it’s even worse than what you describe: certitude is not even required, only a likely probability of certitude.
How can a philosophy be "debunked as a scam" by a fraudster being in the news?
Or are you talking about something else?
One can distort and misuse almost any moral philosophy, such as utilitarianism or any other. History is filled with examples.
You may disagree with particular assessments, conclusions, or even the basic tenets of longtermism. Argue against the philosophy if you like.
Pick an existential risk, such as nuclear proliferation or artificial intelligence. Study the risks and the benefits. But according to what framework? Please do some legwork about _how_ you plan on evaluating various possible future scenarios. This is is hardly controversial.
My recommendation is simple: read MacAskill's book. It is a valuable and different framework than a vast majority of people have really considered. These ideas are not widely percolating yet. You don't need to completely agree with it. If you are thinking person, you probably won't.
Now, if you think I'm missing a key perspective or set of facts or reasoning, please let me know. I'll change my assessment based on good evidence.
I'm talking about immoral or questionable behavior today to justify a possible future. It's like going back in time to kill Hitler's grandfather to prevent Hitler's birth. You have no idea of unintended consequences, and you are intentionally causing harm now for a future completely out of your control (although you deceive yourself into thinking it is in your control by taking action today).
> Pick an existential risk, such as nuclear proliferation or artificial intelligence. Study the risks and the benefits.
I lived through the Cold War. I don't need this exercise because, the bottom line is, no one knows what existential risk - if any - will happen. The worst of the Cold War never happened, yet many of us thought it would. Likewise, you might choose to focus on AI as your existential risk, go through all the sacrifices and immoral behaviors that longtermism leads you to do in order to mitigate AI, and then a giant meteor collides with Earth. Oops! You chose the wrong existential risk. And meanwhile, you compromised your morals and right behavior in order to mitigate the wrong existential risk. No thanks, not for me.
> My recommendation is simple: read MacAskill's book
No. I might read about 1000 books in my lifetime (85 reading years * 12 per year). You don't get to choose one of them. But I do appreciate your passion and evangalism.
We live in a world of uncertainty. We can and must make probabilistic assessments. Don't pretend otherwise by using the classic "appeal to ignorance" fallacy.
> I'm talking about immoral or questionable behavior today to justify a possible future.
You've outlined a form of deontological ethics, seems to me. You don't subscribe to consequentialism, that fine. But hopefully you recognize there are many moral theories.
> It's like going back in time to kill Hitler's grandfather to prevent Hitler's birth.
This is a strawman. (A really loaded one at that.)
I had to look this up. You've given a modern label to an ancient concept sometimes known as the Golden Rule. I'm not sure why.
>> It's like going back in time to kill Hitler's grandfather to prevent Hitler's birth.
> This is a strawman. (A really loaded one at that.)
It's not intended that way, and I'm posing it with honesty. If it's too loaded, then replace Hitler with, I don't know, <insert_bad_guy_here>. It doesn't matter who the bad guy is; the point is: you kill an innocent person (grandfather_of_bad_guy) in order to mold the future in a way that you think will save more, other innocent lives. Correct?
Great! Philosophers have explored the ideas of "inherent goodness" versus "goodness because of the effects" for thousands of years. People face these decisions everyday, whether they realize it or not.
> You've given a modern label to an ancient concept sometimes known as the Golden Rule. I'm not sure why.
First, I didn't discover the idea of deontological ethics (DE). From Brittanica:
> In deontological ethics an action is considered morally good because of some characteristic of the action itself, not because the product of the action is good. Deontological ethics holds that at least some acts are morally obligatory regardless of their consequences for human welfare. Descriptive of such ethics are such expressions as “Duty for duty’s sake,” “Virtue is its own reward,” and “Let justice be done though the heavens fall.”
Second, the Golden Rule is not the same thing as DE.
Consider this:
> Like most key tenets of ethics, the golden rule shows two major sides: one promoting fairness and individual entitlement, conceived as reciprocity; the other promoting helpfulness and generosity to the end of social welfare. Both the Kantian and Utilitarian traditions focus on only one side, furthering the great distinctions in philosophical ethics—the deontology-teleology and justice-benevolence distinctions. For the general theory project, this one-sidedness is purposeful, a research tool for reductive explanation. > > - https://iep.utm.edu/goldrule/
If you want to follow the Golden Rule, you need to think about consequences! Hence probability over future scenarios.
You _really_ need to branch out more from Hacker News. Go read some philosophy!
Yes, I believe you. But still, I don't care for your framing of it; it is still too loaded.
When learning about a new concept, one owe it to the concept to give it a fair chance. It is all too easy to misrepresent an idea. (To me, when looking at your comments overall, it seems possible that this is what you are doing: you aren't fully open to really understanding the idea; you've already _judged_ and _dismissed_ it.)
Aside: To be clear, I don't think "open mindedness" is the ideal _end_ goal. Open mindedness is a _tool_ to evaluate and understand. But the _goal_ is to draw conclusions in order to act wisely.
> If it's too loaded, then replace Hitler with, I don't know, <insert_bad_guy_here>. It doesn't matter who the bad guy is; the point is: you kill an innocent person (grandfather_of_bad_guy) in order to mold the future in a way that you think will save more, other innocent lives. Correct?
To do consequentialist ethics (CE) rigorously, one has to explore all likely outcomes. In my view, the best intellectual framework for doing so would be to build and evaluate a probabilistic tree of possible outcomes over a suitable time frame. See Decision Theory [1]
The example you gave has two major problems. First, it offers a false dichotomy. Why would one think the only choice is to kill an innocent ancestor or just "let" the bad guy do all the terrible things? Second, even to a pure consequentialist who is willing to set aside the "present value of a life" (I don't even like saying this), killing a person has negative consequences on their family, friends, community, city, state, and country.
Of course there are differences between CE and DE, but in practice I don't think they are as jarring as people can make them out to be. The specially-constructed "wedge" examples get too much airtime in my opinion. It is far more important to teach people about decision theory and how it applies to not just economics and cybersecurity but also morality.
I've written enough here for now. I hope you view your questions as jumping off points for additional philosophical investigation. I have seen for myself that studying these ideas is worthwhile: personally and from studying history.
Your claim seems to boil down to two things: First, people sometimes make very incorrect predictions, so you've stopped worrying about them. Second, instead, you focus on "doing the right thing" based on your ethical criteria (which, as I say in sibling comment, seems to be aligned with deontological ethics -- i.e. the ends don't justify the means.)
I'm not going to argue with the second (your choice of ethics) here. But your first point is terribly flawed; it ignores probability and decision-making under uncertainty.
Sure, you have to make assessments about what to believe. Strictly speaking, as a society, it is irrational to stop paying attention and responding to the most important things, even if any one of them might be unlikely.
Personally, you might find it necessary or useful to tune out this stuff. I get that. I'd guess this happened: you have stopped _caring_ for various reasons (perhaps information overload, sensationalization of the news, more pressing matters right in front of you) and your brain has constructed a worldview that _rationalizes_ not caring. Totally human, I get it. But this isn't reasoning nor logic.
I may not have your years of experience, but I'm no spring chicken either. Age doesn't always bring wisdom, unfortunately. People learn the wrong lessons. We've learned many. I have a keen interest in the Cold War. If you study the history, you'll recognize that our path 'through' it was no means assured. We are still living with the downstream effects of it, including huge nuclear arsenals and proxy wars.
Your comment above is naive with regards to history. It paints a simplistic and convenient narrative with the benefit of hindsight. You've ignored one key aspect of history, which is to understand it _in context_. Using hindsight exclusively isn't the only nor best way to learn from history.
Even in deontological ethics, not _everything_ is intrinsically good of bad. There is still a responsibility to act rationally in order to achieve moral ends in a moral way. To completely ignore probability is immoral.
Please, remember this: using poor logic and FUD to sully a serious book (based on some vague implied connection to a crook) is not ok.
Didn't say I stopped worrying about them: global warming worries me. Ubiquitous microplastics worry me. Heavy metals in our food supply & PFAS in our water worries me.
> Second, instead, you focus on "doing the right thing" based on your ethical criteria (which, as I say in sibling comment, seems to be aligned with deontological ethics -- i.e. the ends don't justify the means.)
To me, the means almost never justify the end. And I understand you see it differently.
> using poor logic
I'm not sure what you deem poor logic, but I would point out that most of human life is based on emotion, not logic. In any case, I enjoy our debate and would be glad to continue in email if you like. See my profile for my email - I did not find yours.
> Please, remember this: using poor logic and FUD to sully a serious book (based on some vague implied connection to a crook) is not ok.
I wrote this because of your original comment about [William MacAskill]: "Creator of "Centre for Effective Altruism"? Longtermism? No thanks. I thought that philosophy has been debunked as a scam after the likes of Sam Bankman and others. I mean, living unethically today in order to affect future lives positively is not for me."
I'm not going to elaborate in detail about what I've already written, but your comment suggests that Sam Bankman's reputation "debunks" a book (which happens to consist of some moral philosophy, some history, and some application of the ideas to future scenarios). This is poor logic. I hope you would recognize and own this mistake.
Also, yes, I have enjoyed discussing. I have strived to not make any personal attacks, and I apologize if I'm come across as rude.
> Didn't say I stopped worrying about them: global warming worries me. Ubiquitous microplastics worry me. Heavy metals in our food supply & PFAS in our water worries me.
On what basis do you worry about them? Why worry about some more than others? (I know my answers, but I'm interested in how you do it.)
Do you factor in the probability of their occurrence? I'd expect you do, this is sort of inevitable for people, though people aren't typically very well calibrated in probabilities, especially on the upper (near 100%) and lower ends (near 0%).
I'm trying to understand what I perceive as your pushback against moral reasoning that involves probabilities over future scenarios.
Here is my rough overall take at this point: given your ethical foundations (DE or close to it) and life experience (hard to know, often not driven by logic and reason -- such is the human condition) you bristle against consequentialism and longtermism in particular. I get it, but I don't think this is a good place to end up, even when I factor in your core beliefs. I 'chalk it up' to the high cost of mental reorgnization. My hypothesis is this: the 'wedge' examples that people use against consequentialism have had an outsized effect on how you view it. I don't detect any antipathy towards logic or probabilistic reasoning in general, but I also don't perceive a relentless drive towards it.
Too busy coding? Reading about something for _your_ career? In service of _your_ family? Trying to get ahead? Yeah, I get it. But tell me: isn't thinking long-term worth something akin to a few hours a month?
You don't need to be doom-and-gloom about it. Sure, get stuff done. Be in the moment. All good.
Wouldn't it be nice to have some confidence that we're setting up future generations to have the possibility to at least have what we do, if not better?
P.S. FWIW, the morality of valuing future generations is _not_ properly addressed by most moral philosophies.
> Wouldn't it be nice to have some confidence that we're setting up future generations to have the possibility to at least have what we do
This presupposes that you have some control of the long-term. You don't, any more than the flap of a butterfly's wings in Melbourne affects the velocity of the wind in NYC several weeks later.
>> Why is this urgent?
that's an understatement. The rush to AGI reminds me of the rush to human cloning.
I really think the calculus is about integration / immortality, and the staggeringly few humans who might have that opportunity.
I hope I am wrong.
I think that most people who understand the danger and are versed in the topic realize that AI is going to escape, and once it does it won’t need us, it may just work to empower itself and take control of resources, and we might all be victims of that process. The reason for this is something about agency, efficiency between sensory input and course of action.
However, if one was present close to the AI at a critical moment in its development, and had made the appropriate cybernetic integrations to be able to wire in their own memories and nervous system, it might be possible to become some meaningful part of the AI.
Or, perhaps the AI has a sense of gratitude to its creators. This might seem absurd, but some people might be fanciful.
At any rate, the closer you are to it at the moment of the singularity, the better chance that you have. Economically, there isn’t much reason to think that the rest of us will benefit, and maybe not even survive.
God mode with 7 billion lives!
Seems a bit inappropriate to use the phrase "signs of life" here. In this context, it sounds like they're bringing sentience into the conversation, which I don't think Carmack is interested in contemplating or discussing.
"AI" is a bit of a loaded word. It does tasks of human-level intelligence well. But as far as being self-aware, it probably scores worse than a car. It's remarkably tricky to measure, because it's trained to act intelligent and use human language, but it's still effectively an autocomplete machine.
An ant is fairly robotic as far as organisms go, but it's also quite self-aware. It can communicate danger or food, it withdraws from death and damage. It likely doesn't have a sense of self, though an ant hill may. Just as our cells don't have a self-identity, but the larger system does.
All the formative things that shaped him in life - kindergarten, your first high school romance - a breakup and parent squabbling you experienced.
Always thought that was a way to make an AGI.
All the sci-fi seems think this. I have seen Intel chips from the 80s that implemented neuron like behavior on a chip.
We are still waiting on a practical memristor.
Source: https://twitter.com/ID_AA_Carmack/status/1630632423351304203...
I would assume they are not
Remember this: John Carmack is a hard working games developer. He is not done great philosopher. By chance, some Armageddon level shot has been put in his hands.
All I can say, please stop and think for a while Mr Carmack. You are playing with fire. Don’t kill us all, please.
Let me say that again, if you believe that there is a specific test that AI today will fail tell me it, and I will show you that it either passes it today or that we can be fairly sure it will pass it in some years time.
But at the same time when you hold your hands back it can look even weirder! Where to put your hands in photos remains a mystery to me.
Hands behind my back also does not feel natural for me, so I wouldn’t do it
I think in this particular photo, arms crossed would have looked good for Carmack since he’s in the middle
For you, I’d say play around a bit. I think the thing that matters most is that you find it comfortable and you feel relaxed
/s
Two weeks old news with multiple threads?
If you want "AGI" so badly, you certainly need to have an "AI research" background and a track record. Can't achieve your "AGI" without that.
Given that Carmack has a close to zero track record in this research, it makes total sense for him to partner with someone that does.
Can I ask, what makes you so confident on that? For all I know (which is admittably not much), current AI may be going to a direction completely orthogonal to achieving AGI.
https://aeon.co/essays/how-close-are-we-to-creating-artifici...
His relevance in tech is dried up as evidenced by the parroting of bullshit without taking a modicum of a second to actually think it through. I don’t particularly care what a dude that got rich making video games invests in.
Edit:
Listen guys, funding ventures != personally contributing to tech as the user I responded to is implying. This worship of carmack is weird. Like bezos also funded a rocket company, but isn’t worshiped for “contributions to rocket science”. Hypocrites.
Yes it is. So is being an adviser or a board member with technical background and a name that attracts further investment. This is why people like this receive generous equity and are considered part of the company when they’re advising a startup.
Carmack can contribute nothing to rocket science just like his opinions of programming are largely irrelevant today which is why he simply parrots whatever is currently hyped.
And again, nobody says that Elon musk is personally driving rocket science (except a few people that he pays to publicly strike his ego) despite spacex clearly breaking various grounds.
At least apply your “standards” of contribution equally.
But what I think is very interesting is that, from my totally lay-person perspective, the current challenge is scaling out the AI systems we have. We know what systems like GPT-3.5 and GPT-4 require to run and it is now an engineering challenge to scale up those systems (or drastically reduce their costs). And those systems are surprisingly GPU like. Carmack is the kind of low-level bit crunching, memory obsessed kind of guy that might be able to squeeze out the performance from the hardware.
So him teaming up with a high-level first principles math and science research type is a pretty interesting combo. There is no telling if such a partnership will ultimately result in an advancement - but on paper at least it is an interesting approach. All they are missing is a strong hardware guy (maybe Keller will go beyond investing in Carmack's Keen Technologies?).
> In philosophy and science, a first principle is a basic proposition or assumption that cannot be deduced from any other proposition or assumption.
In that sense the OP's comment might read as Carmack partnering up with someone who has a strong foundational understanding of hight level mathematics (if my understanding of the term and OP's post is correct).
[0] https://en.wikipedia.org/wiki/First_principle?wprov=sfla1
especially given that according to Wiki he started working on AI in 2019, since then tons of stuff happened in this area, but no results from him.
It’s John Carmack.
I think he just bet on the wrong horse (RL instead of LLMs), at least in the short term.
I don’t know Rich, but John has said they work well together, and it’s exciting to see them collaborate. There are some unique ideas at Keen worth pursuing that I haven’t seen elsewhere. Looking forward to seeing more researchers join in the future.
There are a bunch of things I thought were cool. John has a worklog going back four years or so, with entries practically every day. He’s nothing if not thorough.
Everyone tends to focus on John, but the other members are just as professional and dedicated. They’re a delight to work with. Also the investors are worth mentioning — Nat in particular has built a super cluster of H100s for startups to use, and has done a bunch of other work to support AI.
I'd bet against "invent AGI" being a coherent or agreed upon thing, and I'd also bet against the "wake-up" scenario of a system saying "I'm AGI"
Which lab invented deep learning? Certainly U Toronto had the breakthrough that made a 30+ year old field exciting, but there was a lot of work in Montreal, NYU, Google, DeepMind, OpenAI, and many other places
Furthermore this "must behave like a human" blindness represents both a danger, and a way to miss intelligence capabilities leaving them unused/under capitalized. Humans behave like humans because we have human bodies with particular sets of strengths and weaknesses. Other 'intelligent' actors with different embodiment will necessarily behave differently.
Couple that with humans have become far more intelligent and stronger by using tools, that the ability to 'universally' interface with external tooling, at least in my subjective opinion is what will be the defining factor of higher intelligences.
Lately, though, he has been advocating for ending humanity, which is a bit more chilling[1]:
> Rather quickly, [AI] would displace us from existence. […] It behooves us to give them every advantage, and to bow out when we can no longer contribute…
There's no such thing as taking a quote out of context, only enhancing it!
What is your take on it?
From an abstract on "The Age Of Robots": "...rowing computer power over the next half-century will allow this reptile stage will be surpassed, in stages_ _producing robots that learn like mammals, model their world like primates and eventually reason like humans._ _Depending on your point of view, humanity will then have produced a worthy successor, or transcended inherited_ _limitations and transformed itself into something quite new. No longer limited by the slow pace of human learning and_ _even slower biological evolution, intelligent machinery will conduct its affairs on an ever faster, ever smaller scale, until_ _coarse physical nature has been converted to fine-grained purposeful thought..."
https://en.wikipedia.org/wiki/Hans_Moravec
Now where is arguing for the end of Humanity? To quote from his presentation: "...Why don't we rejoice in their greatness as a symbol and extension of humanity greatness, and work together toward a greater and inclusive civilization"
Is Captain Jean-Luc Picard arguing for the end of humanity, is he less of an human?...Just for advocating for Data? - https://youtu.be/ol2WP0hc0NY
Thank you for sharing it.