John Carmack: I’m going to work on artificial general intelligence

What we really need is a scalable, distributed, physics pipeline so we can scale sims to 1000x realtime with billions of colliding objects. My guess is that Google/Stadia or Unity/UnityML are better places to do that work than Facebook, but if Carmack decides to learn physics engines* and make a dent I'm sure he will.

Until our environments are rich and diverse our agents will remain limited.

*More, I'm sure his knowledge already exceeds most people's.

But that kind of realism is not needed for all AGI research.

I also spent some years on using evolutionary algorithms to evolve control networks for simple robots. The computational resources available at the time were rather limited though. Should be more promising these days now that your commodity gaming pc can spew out in 30 minutes what back then took all the labs networked machines running each night for a few weeks.

I don't think so. Game NPCs don't need AI, which would be way overkill; they just need to provide the illusion of agency. I think for general AI you need a field where any other option else would be suboptimal or inadequate, but in videogames general AI is the suboptimal option... more cost effective is to just fake it!

In some sense you can think of interfacing w/ the online world + trying to win attention to yourself as the kind of general game that is being played.

https://en.wikipedia.org/wiki/The_Talos_Principle

[0] https://en.wikipedia.org/wiki/Simulation_hypothesis

I mean: maybe it's more efficient to have it read all of wikipedia really well before adding all the other noisy senses.

It is nowhere near good enough to avoid running into Moravec’s Paradox like a brick wall as soon as you try and apply it outside the simulator.

Now Alphago and it's implementation framework are much more sophisticated than Deep Blue. It's actually a framework for making single-task solvers, but that's all. The fact it can make more than one single-task solver doesn't making it general in the sense we mean it in the term AGI. AlphaGo didn't learn the rules of Go. It has no idea what those rules are, it's just been trained through trial and error not to break them. That's not the same thing. It's not approaching chess or Go as an intelligent thinking being, learning the rules and working out their consequences. It's like an image classifier that can identify an apple, but has no idea what an apple is, or even what things are.

To build an AGI we need a way to genuinely model and manipulate objects, concepts and decisions. What's happened in the last few decades is we've skipped past all that hard work, to land on quick solutions to specific problems. That's achieved impressive, valuable results but I don't think it's a path to AGI. We need to go back to the hard problems of “computer models of the fundamental mechanisms of thought.”[0]

[0]https://www.theatlantic.com/magazine/archive/2013/11/the-man...

Sounds like the start of a truly horrifying Black Mirror episode

Rodents? Try insects [1]. In the late 40s and early 50s, when neural networks were first explored with great enthusiasm, some of the leading minds of that generation believed (were convinced, in fact) that artificial intelligence (or AGI in today's terms) is five/ten years away; the skeptics, like Alan Turing, thought it was fifty years away. Seventy years later and we've not achieved insect-level intelligence, we don't know what path would lead us to insect-level intelligence, and we don't know how long it would take to get there.

[1]: To those saying that insects or rodents can't play Go or chess -- they can't sort numbers, either, and even early computers did it better than humans.

I think you'd be surprised how much progress is also being made outside those two factors. It's sort of like saying graphics only improve with more RAM and faster compute. We know there's more to it than that.

In many cases, the cutting edge of a few years ago is easily bested by today's tutorial samples and 30 seconds of training. We're doing better with less data and orders of magnitude less compute.

The key, of course, is redefining life and intelligence as whatever the current state-of-the-art accomplishes. (Cue explanations that the brain is just a giant pattern matcher.) It makes drawing parallels and prophesying advancements so much easier. Of all our sciences, that's perhaps the one thing we've perfected--the science of equivocation. And we perfected it long ago; perhaps even millennia ago.

Rodents can't play Go or a lot of other humanly-meaningful tasks. We don't need to build an artificial cell. A cell is too many components that by blind luck happened to find ways to work together, this is as far from efficient design as can be. The same way we don't build two-legged airplanes, we don't need anything that's close to the wet spiky mess that happens in human brains. It's more likely that we have all the ingredients already in ML, and we need to connect them in an ingenious way and amp up the parallelism.

Actually it's not so obvious that the brain is not differentiable. If you do a cursory search, you'll find quite a lot of research into biologically plausible mechanism for backpropagation. Not saying the brain does backprop, we just don't know and it's not outside of the realm of plausibility

In a sense, everything is affected by quantum effects. However, with neurons, they are generally large enough that quantum effects do not dominate. Voltage gated channels are dozens to hundreds of amino-acids long. Generally, there are hundreds to millions of ion channels in a cell membrane and the quantum tunneling of a few sodium ions in or out of the cell will generally not affect gestalt behavior of the cell, let alone a nervous system's long term state. Suffice to say, ion channels are not dominated by quantum behavior.

Largely, we have the building blocks to replicate neurons (as we currently understand them) in silico. However, as is typical with modeling, you get out what you put in. Meaning that how you set your models up will mostly determine what they do. Setting your net size, the parameters of you PDEs, boundary values, etc. are the most important things.

Now, that gets you a result, and it's likely to take a fair bit of time to run through. To get it up to real time the limiting factor really ends up being heat. Silicon takes a LOT of energy as compared to our heads, ~10^4 more per 'neuron'. If we want to get to real time, we're gonna need to deal with the entropy.

But, if this is a 100% replicated brain, doesn't that mean its suffering is just as real as a real brain's suffering, and therefor just as cruel? And if not, what's the difference?

Gene expression is often tied to the environment the organism is in. Mere possession a gene isn't enough to benefit from it. Some expressions don't take effect immediately, but rather activate in subsequent generations.

Epigenetics is a whole equally large layer on top of this system. A single-focus approach may not be sufficient, and even if it is, it's not likely to cope with environmental entropy very well.

Like what is language, what is intelligence? Some of the smartest linguists and philosophers would proudly declare they have no fucking clue.

Making Alexa turn on the lights or using Google Translate are cool party tricks though.

Idc how many Doom games ya made, but I’m sorry to say a bunch of software engineers aren’t gonna crack this one.

I wouldn't, however, bet against some kind of insanely clever development coming out of his new endeavor. Something like an absurdly efficient new object classifier, that reduces the compute requirements for self-driving cars by a non-trivial factor, would be a very Carmack thing.

I genuinely felt a sense of disappointment when he moved to Facebook (via the Occulus acquisition). So yea, fuck you, Facebook and your manipulative, life values corrupting and PR machinery.

I place John Carmack miles above Zuckerberg.

This may be his biggest impediment. ML has gotten very far with looking at problems as linear algebraic systems, where optimizing a loss function mathematically yields a good solution to a precisely defined (and well circumscribed) classification or regression problem. These techniques are very seductive and very powerful, but the problems they solve have almost nothing in common with AGI.

Put another way, Machine Learning as a field diverged from human learning (and cognitive science) decades ago, and the two are virtually unrecognizable to each other now. Human learning is the best example of AGI we have, and using ML tech as a way to get there may be a seductive dead end.

I'm glad to see he's aiming big with his billions and time. This is what rich people should be doing. Hl3 Gaben!

Yes, he seemed to put a lot of effort to try to get things through FB internal politics, and not always successfully. I really wish his experiments with a scheme-based rapid prototyping environment / VR web browser had been allowed to continue [1]. VR suffers from a lack of content, and VR itself is well-suited to creating VR content, and his VR script would surely facilitate closing that loop among other things. Although now four years later I guess FB has a large team working on a locked-down, limited world building tool (closed platform, no programming ability). Oh well.

I don't think this is the end of this wave of VR, but at this point I wouldn't be at all surprised if say Apple or someone else ends up bringing it to the mainstream instead of Facebook. [2]

[1] https://groups.google.com/forum/#!msg/racket-users/RFlh0o6l3...

[2] https://www.theverge.com/2019/11/11/20959066/apple-augmented...

To be honest anyone who has a very good working knowledge of Linear Algebra can learn much of ML-math in a day. There really isn't anything mathematically super-sophisticated that is in popular use today.

Sigh. I assumed the whole point of hiring John Carmack is that you trust him to identify critical problems - and to find the best way to solve them.

I don't put learning state of the art ML past Carmack, at all. However, does ML tech of today lead to general AI? It's a strong assumption.

Yes and I really wished he hadn't. Before he joined oculus they were working on the rift2, he steered them away from that to focus on mobile efforts.

I do see the appeal of mobile vr but at the end of the day it is basically an android phone in a vr headset.

PCvr is already 2 big steps back in graphical quality from desktop games. Mobile vr is like 10 steps back. 8 more steps than I'm willing to take even if it affords me mobility.

As far as I can tell Carmack is an old engineer whose name gets thrown around for headlines. If there weren't articles about his stealing stuff to take to Oculus I don't think his presence there would be observable.

Now people are talking like Carmack switching topics is going to change the world. It's just going to change his schedule. There are smarter engineers already working on this problem.

You must be joking, right? I'm as much of a Carmack fan as anyone here, but overstating the skills of one personal hero does no good to anyone.

Since you have no formal way to compare environments to each other, you can't prevent this from happening. Therefore you just pushed the subjectively as to which AI is smarter to which environments are chosen by the user to run against.

For what I skimmed from your paper, it looks like the LH agents may be viewed as discrete optimization processes trying to optimize an objective/utility function across an infinite space of possible environments (infinite voters).

If it is the case, and if each environment vote has the same weight, you may be in a case of no free lunch, where the performances of all possible agents (including the random agent) will average to the same across all possible environments.

Or, to restate the above, for each environment in which an agent is doing well, it is possible to construct an "anti-environment" where the agent is performing exactly as bad.

My personal opinion on the topic of AGI is that it is actually a case of NFLT.

https://ti.arc.nasa.gov/m/profile/dhw/papers/78.pdf

I.e. across the space of all possible environments, all agents perform equally well

That's an odd definition of intelligence. By that definition, a bird is more "intelligent" than a human at the task of opening a nut. Seems like "fitness" would be a much more appropriate term.

It seems especially strange to consider this work in the field of general intelligence. Nothing about what you just described is general. By this definition, a chess bot is much more intelligent than the average person. I don't think we'd say a chess bot has general intelligence.

I've long thought that the issue with VR is a conceptual one, not a technical one and maybe that frustration comes from there. "Running forward" is an unsolved problem in room scale VR. For a seated experience, you're basically back to a neat display gimmick + accurate hand tracking.

Any real solutions need, on the one side, real-world physical constructions (think running threadmills) that soon hit holodeck-level limitations and, on the other, software that actually benefits from the real technology VR brings to interactive media: super accurate hand- and head-tracking. The first gets impractical/impossible soon, the second limits development to a few niche genres: Shooting ranges, cockpit sims, dance/party games and some vague "experiences" where the actual tech is pretty much ignored and you just say "but it feels so immersive!" (honestly, it does work for horror games!). It's basically motion controls 2.0.

The only place I could see the technology shine is, oddly enough, AR. It has way less mainstream hype to it but it makes much more sense because you actually benefit from the tracking of your real-world movement: You're still a part of it! The holo-lens demos that pop up on youtube might seem clumsy, but I can totally see a use case for replacing physical monitors with arbitrarily sized and positioned displays you can virtually move in any office space. There's rumors of Apple working with Valve on AR tech. If there's any technology that could follow the smart phone, AR is my bet. I'm honestly surprised Carmack didn't move in that direction rather than deciding to become a general AI guru.

For anybody who hasn't played an untethered VR experience, I highly recommend it. It makes a world of difference with games like Echo combat and Beatsaber. Tons of fun. It reminds me of the first time I played wii bowling.

People with this level of track record should not be underestimated, there aren't many of them out there... They matter.

You can get up to date in the field in under half a year of extensive reading. And many of those scientists are too busy solving more specific goals, that their labs set. I doubt there are more than 1,000 researchers in the world specifically working on AGI.

You are making a common mistake of assuming that just because someone is good at something like programing computers the same skill would translate identically to a completely different domain.

If anything he is lucky to have been born in an era where his skill of programming computers could be put to use - otherwise his talents may have gone to waste, he may have ended up toiling fields his talent untapped and undiscovered, like that of millions before him.

I see Carmack as a very(as in uniquely) talented Engineer. Usually, engineers are not the type who do very well in pure research topics. And AGI is certainly a pure research topic, since we don't have a clear leading us there. So while it's great to see he is interested in it, now sure if we should have any kind of expectation there.

Do I wish him the best of luck and hope he cracks the problem? Of course, all the same I would wish that of an upstart PhD student. Yet, the announcements of a brilliant PhD student attending a university to work on AGI is somehow not on hacker news.

This is cult worship of the personality Carmack has amassed, perhaps completely accidentally. When Carmack actually achieves something interesting let us discuss it then, not the mere announcement that he will try, as if that means anything. Read: it doesn't.

Do you all realize you're arguing about nothing?"

Absolutely. I should mention why. AGI is an open field. AGI is opennest of open fields. Advances in deep learning tell quite little about what AGI will look like. We don't know if AGI will be a hundred incremental innovations from deep learning, ten deep advances from deep learning or five incredible advances with only a slight relation to deep learning. We don't know if it will just appear when 100 super-computers are hooked together or if a genius at home on their laptop could cobble it together. Sure, you could extrapolate and say compute has mattered more than theory, so far. But you could also say impressive things have been done but they haven't approached robust generality and there's something we're missing. Pick the approach but then you'll have to see if it's possible.

Etc.

The human brain really can't handle the crippling adversity of a fellow human announcing an aspiration. Hopefully our AGI replacement can.

But yea, I agree with your general point. I'd just note that having that ability to be insanely productive in working on things people haven't done before means to me that if it's possible for someone like him to really get good at this field, he's probably gunna do it.

Who knows how far you can get with just "putting stuff together." That's what Edison did.

> When I think back over everything I have done across games, aerospace, and VR, I have always felt that I had at least a vague “line of sight” to the solutions, even if they were unconventional or unproven. I have sometimes wondered how I would fare with a problem where the solution really isn’t in sight. I decided that I should give it a try before I get too old.

This demonstrates how hard the problem is. When you're tackling a really hard problem "mostly not a success" is a success. Most people faced with the same problem would return "no successes".

AGI is a very active research position that arguably lacks the engineering/real-world arm that I believe Carmack could provide.

His early work in 3D graphics and math are supportive arguments for that. Research ideas turned into viable real-world systems.

Is it too early? I think us armchair hnews users can go back and forth all day. But in the end, we'll only find out the answer after the fact.

I hope all the best for him in this. I think this is a perfect space for him to fit int.

Scientific pursuits have an extremely steep risk profile and are systematically underfunded because nobody knows how to capture the value that comes out of them. If someone wants to chase one on their own dime, we should celebrate that contribution to society rather than dwell on the fact that the median (even 99th percentile) outcome is that the project goes nowhere and the person continues on to retirement proper. The mean outcome could be very different and the benefit of the doubt costs us nothing, so why not give it?

He's openly talked about his work ethic in a bunch of places. He's the type of guy who after a life time of coding calculated he's 100% efficient up until 13 hour work days and then he drops off[0]. Although he did mention working those long hours is often best working on multiple things instead of 1 topic but maybe with AGI there's a bunch of different avenues to explore.

[0]: https://www.youtube.com/watch?v=udlMSe5-zP8&t=4773

I wouldn't be surprised if the opposite was true, at least with the theory part. AI didn't really go anywhere for decades, because people focused too much on theory.

Otherwise, there's a lot of data and hardware at your disposal, even from the comfort of your home.

> Wouldn't it make more sense for him to join a cutting-edge team, like DeepMind or OpenAI?

You mean they guys that are training with videogames that people like John developed?

Substitute math for anything you want.

> I am going to be going about it “Victorian Gentleman Scientist” style, pursuing my inquiries from home, and drafting my son into the work.

Which to me reads like part time work on Oculus, part time work on this AGI project. If it is with Facebook it isn't at all clear from the post (plus I'd assume it would be accompanied by marketing copy in that situation).

As for what I am going to be doing with the rest of my time: [...] For the time being at least, I am going to be going about it “Victorian Gentleman Scientist” style, pursuing my inquiries from home, and drafting my son into the work.

But even so, history will have to judge whether the author's statements were true regarding the risk of super-human AI. Or whether a lot of _quite_ smart people weren't smart _enough_ to realize that there was a real likelihood of this being possible to achieve faster most thought.

Also, using ridicule as a rhetorical technique isn't the most sound type of reasoning, regarding the author of your link ;)

Runner up for next project was cost effective nuclear fission reactors, which wouldn’t have been as suitable for that style of work.

[1] - https://twitter.com/id_aa_carmack/status/352192259418103809

Reminds me of when I was new to Quake 3 and found an amazing server: it was always full of players and full of non-stop action. I played with these people all the time after school. Nobody said anything, they were 100% business which was cool. I would often congratulate them on nice kills or commentate on my victories. Everyone was about the same skill level.

Eventually I realized I was playing on a server that simply filled empty slots with bots. I was the only human player.

Nuclear fission shocks you but not AGI?

He's been a major shareholder in 2 companies that have been acquired (Id and Occulus)

BUT if anyone can clear a hurdle or two...

Pretty sure you don't get to AGI with some hacks.

I wouldn't bet against him when he sets his mind to something.

> That's exhilarating but also terrifying. Our still-barbarian level human systems are still nowhere near ready to deal with the socioeconomic problems that may arise with AGI.

I think you are reading way too much into his statement. It's extremly unlikely that he just magically figured out a way to tackle the problem (just knowing where to get started would be massive).

I personally don't believe modern machine learning is remotely close to AI, except perhaps the very lowest rung of the ladder of self-serving AI definitions. I base that belief on what seem to be the unknowns, reinforced by predictable failures[1]. But I have very little reason to believe it's impossible. Not even the possible necessity of quantum effects would seem to preclude it. Heck, we've already begun harnessing quantum effects in materials science, computing, biology, and other areas.

Unless you mean that whatever we could eventually come up with would be more biological than machine or that only a human could think like a human, but that seems more like word play, the kind of game AI believers play. (That said, that poses an interesting question: which is more likely to be achieved first--a designed-from-scratch, DNA-based cellular intelligence, or something not based on DNA or otherwise mimicking existing organic life? If at all, of course. Also presuming such a distinction isn't in fact hopelessly quaint and naive.)

[1] I'm not a naysayer. While I never believed self-driving cars were around the corner (not even 5 or 10 years out; you can Google my HN comments from years ago), I have no doubt the science has been useful and can and will and is put to great, largely unseen use, as is typical of most science.

Because we can simulate physics.

I like the guy as much as most, but so far it seems like he has been wandering around. He’s had much success in the early days of 3D video games and that’s about it. A guy of his calibre would probably make a good impact if he joined one of the expert teams like DeepMind. No matter how smart he is, AI today is a completely different ballgame then what he was part of so far. I hope I’m wrong, but I don’t see him making any sort of breakthrough on his own, with his son. Maybe he wants to spend more time with his family, which is great, or he drank koolaid about his own legend. Odds are against him, heavily so. Good luck, in any case.

My brain bit on that remark; would you care to elaborate?

/ok, maybe his project falls under "total bullshit"...

That is a ridiculous exaggeration. Carmack was clever enough to gain ~1 year advantage in performance over his competitors for the Doom engine, using Binary Space Partitioning, which was first applied to 3D graphics in 1969, before he was born. The Quake engine got a significant performance boost from Michael Abrash, who is a specialist in code optimization.

No, he didn't, and that is not a claim that he would ever make himself.

We don't have the laws of AGI like we had the laws of optics (Asimov notwithstanding.) Tons of research effort was poured into the wrong avenues in vision (hand-tuned HoG, transforms, optical flow analysis) and ML (support vector machines, computational learning theory) until a chain of breakthroughs hit on the right mathematical approach for vision and supervised learning more generally.

We have some mathematical approaches to try with AGI (e.g. policy optimization/max-Q in reinforcement learning), but they equations are plagued with fundamental issues (e.g. reward sparcity, easily-gamed artificial objectives.)

Carmack optimized some very difficult equations when he worked on graphics, but in AGI we still don't have the right equations to optimize.