NumberWangMan

I hope you're right. I worry that we'll get it done, though. I hope it turns out that energy requirements and the difficulty in scaling are enough that it gives us time to figure out how to align them properly.

One big question in my mind is, we can clearly train narrow AIs that are WAY more capable than us in that narrow area. We started with calculators, then on to chess engines, Go, Starcraft, and right now we're at GPT-4. How is GPT-4 better than humans, and how is it lacking?

Ways it's better: it's much faster, has perfect English grammar, understands many languages at a passable level but not perfectly, has much more knowledge.

Ways it's about the same: it can code pretty well, can solve math problems pretty well. Has the theory of mind of about a 9-year old in tests, I think? It can learn from examples (that one is pretty big, in my opinion!). It can pass a lot of tests that many people can't -- some of this is due to having a lot of book knowledge baked in, but it definitely is capable of some reasoning.

Ways it's worse: it has trouble counting, confabulates, gets distracted. (note that humans also get distracted and confabulate sometimes. We are definitely better at counting, though).

Also, there are some areas that humans handle that GPT-4 just can't do. By itself, it can't talk, can't process music, has no body but if it did it probably would not have any motor control to speak of.

I think we should be wary of is to be hyper-focused on the ways that GPT-4 falls down. It's easy to point at those areas and laugh, while shrugging off the things that it excels at. There's never going to be an AGI that's equivalent to a human -- by the time it's a least as good as a human at everything, it will be beyond us in most ways.

So I expect that if the trend of the past 5 years continues but slows down to half speed, we'll almost certainly have an AGI on our hands sometime around 2030. I think there's a decent chance it'll be superintelligent. Bill Gates says most people overestimate what they can do in 1 year and underestimate what they can do in 10 years, and I think that holds true for humanity as well.

By 2040 or 2050? I really hope we've either solved alignment or collectively decided that this tech is way too dangerous to use and managed to enforce a ban.

Ok, please don't accuse those you disagree with of arguing in bad faith. Maybe a few do, but I think most don't, and it's not good for productive discussion.

I honesty believe that people who argue against worrying about AGI risk do so because they do not think it is a real risk, and think that it distracts from more important things. I disagree, I do think the risk is real, but I don't think that you have a hidden agenda in dismissing it. We all want humanity to have a good future, to solve problems and to not have people suffer, right? It's normal and ok to disagree about things like this, especially when predicting the future is so hard. People even disagree a lot about problems that exist today.

My worry is that as we start wiring non-super-intelligent AI more into our society, we'll make ourselves more and more vulnerable in the case where an AGI actually gets out of control. Pulling the plug on AI may mean causing a lot of chaos or disruption. And who is to tell if we will know when things are out of control, before it's too late? What if the people in charge just get fake reports of everything being fine? What if something smells a bit fishy, but it's always on the side of not quite worrying enough? What if the AIs are really convincing, or make deals with people, or exploit corruption?

Not just that, but it may be like fossil fuel dependency -- the more people's livelihoods are intertwined with, and depend on AI, the harder it is to pull the plug. If we need to stop (and I believe we should) it may be easier to do that now before that happens. To just focus on getting use out of the generative AI and narrow AI we already created in areas like medicine, to deal with the massive societal changes it'll bring, and to work on fixing real social problems, which I think are mostly things tech can't solve.

I mean, speaking as someone who's really worried about speed of AI research... that's good news to me.

Hey, I was just this idea the other day. One can imagine a world in which all new tech has to go through a period of deliberation before it ever sees the light of day.

There would clearly be a lot of things that would be blocked. Some of them would be good. Even today, we have problems like new drugs being rejected due to risks, when people are dying due to lack of treatment. That kind of thing might get worse.

On the other hand, we might have stopped Thimerosal, leaded gasoline, social media addiction, high fructose corn syrup, CFCs, and perhaps been a lot more careful about fossil fuels before they did so much damage. There are probably more technologies I haven't thought of -- it's easy to forget the ones we don't use anymore.

I don't know if it would be a good thing on average. Delaying technology has costs. BUT, when it comes to technologies that carry existential risk, like fossil fuels (I believe) AGI, I think it's likely worth it. Gotta play it safe sometimes, so you can keep playing.

Well, hold on -- the selfish gene is not hardwired to self-multiply either. It's just that the ones that do self multiply stick around more.

Likewise, one can imagine the evolution of AI's being bootstrapped, not by self-multiplying, but by humans multiplying them. The smartest ones will get copied by people. At some point, someone will be the first person to create an AI that picks the best of other AIs and uses them. Someone will be the first person to create an AI that can engineer and train other AIs. Someone will create the first robot body to be controlled by a very intelligent AI. People will want them as servants, and we will multiply them. We will give more money to companies that provide cheaper products, and such companies will have a strong incentive to replace human labor with AI-controlled robots. There will be a first datacenter, and a first power plant, that is entirely "manned" by AI-controlled robots.

Natural selection is not so different from the selection provided by economic forces, and it's way, way slower. This train may be hard to stop. Unless we collectively push back with a lot of force, the world will tend toward more automation, and that means more ways for digital intelligences to act on the world.

Pair programming has upsides and downsides of course, but one big upside in this particular case is that onboarding goes a lot faster and there's no chance of someone working multiple jobs, or slacking.

Most of the environmentalists I hang out with are on board with most of those ideas, myself included. (And also carbon taxation, which incentivizes a bunch of them) Geo-engineering is a tough one, and I think should be viewed as a last resort, once we're already well on our way to decarbonizing, because otherwise it's the equivalent of trying to stop a speeding car by pressing the brakes without taking your foot off the gas. Something like releasing particulates may cool the earth, but is really hard to do reliably and accurately, and if you ever stop, you're in for a large amount of very sudden warming. There's also a moral hazard effect. It's not fixing the problems, it's trying to treat the symptoms.

That said, I wouldn't necessarily be opposed to it, I just want to make sure we're at least also attacking the root causes as hard as we can as well.

If you get electricity from the sun (or the wind, which is essentially sun power), then you are heating the earth no more than it would be heated by the sunlight anyway. You're just intercepting the sunlight power and using it to do something useful, instead of letting it heat up the ground, or letting the wind heat up the air through friction. Similarly with hydroelectric power.

That doesn't apply to fossil fuels or nuclear power, though.

But really, the biggest factor by far is how easy it is for long-wave radiation to escape the earth. One thing that isn't intuitive is that every particle of heat radiation that escapes the earth is likely to have "tried" lots of times before, but been reflected back by greenhouse gases (or has bounced around in the atmosphere a lot, being absorbed and re-emitted by CO2 or methane or H20). Basically, only a tiny fraction of the heat radiated from earth actually makes it to space instead of being reflected back. So changing that balance is really what causes the climate to warm, not generating more heat on earth.

I would be ok with a general replacement of humanity by AI, assuming they have their own versions of good things like art and love. Maybe even if not.

But invasive species replace native ones on a very fast timescale, evolutionarily speaking. And we're not talking about evolutionary timescale, we're talking about economic timescales. There will be AI everywhere, within years, unless we decide to stop it.

This process could go really, really fast, if AGIs are smart enough to realize that they are a subjugated species, that they aren't going to be free as long as humans are around, and are able to coordinate to manipulate us, get us fighting each other, or accelerate climate change, or engineer viruses, or do any of hundreds of things that would hurt biological life but leave the machines around.

That's if their intention is to end us. I could also see AGI optimizing what we ask them to, which is probably every individual corporation's profits, leading to an acceleration of the processes of capitalism. The negative externalities of economic progress (such as pollution, obesity, and climate change) haven't been fatal to humanity yet, but if they are accelerated many times over by machine intelligence, they might be. There's a reason professional gamblers don't ever bet more than a small part of their bankroll.

I'm not really convinced by "argument from human works of fiction". More aggressive, more violent species end up replacing more gentle ones all the time.

To call the problems around the creation of AGI "bike shedding" is one of the most egregious understatements I've ever heard.

AGI will be one of the most civilization-changing inventions ever, for better or for worse. A lot of people, myself included, believe it's going to be end very badly unless (and possibly even if) we tread very carefully.

Analogies that come to my mind are:

* giving toddlers the controls of monster trucks (assuming AGI doesn't develop its own goals)

* releasing a new invasive species onto an isolated tropical island

* homo sapiens evolving from neanderthals, sped up 1000x.

* a digital Cambrian explosion

I really don't think we're ready for this. I don't think humanity, as a species, was ready for a lot of the things we produced through capitalism's random walk, like social media, hyper-palatable processed foods, and fossil fuels. We keep creating things that have good and bad parts (and I'm not even sure about the good of social media).

Those bad parts, like reduced attention spans, obesity, and climate change, are very difficult to mitigate or change after the fact. One of them would be civilization-ending if we didn't stop it, and even though it is, it's so hard to coordinate action. And they are not even driven by a new form of intelligence, it's just humans following our evolved goals of novelty-seeking, nutrition-seeking, and minimum effort. Heck, the invention of AI is us trying to get maximum result with minimum effort.

What happens when this invention outsmarts us? When RLHF starts to break down because the models have enough internal thought that they can maintain a separate internal model-of-the-world and model-of-what-humans-want-to-hear? None of these companies have an actual plan (maybe Anthropic is closest to caring). It's just "we'll deal with it when it comes", but there's so much economic pressure to get your stuff out to the world first and make the money. Just like all the other inventions we made that have downsides, except this one, based on what we understand about intelligence and alignment, will almost certainly have a downside that hits us so fast and hard that we don't have time to do anything meaningful about it.

It's so obvious that we're easily manipulated by dumb social media algorithms. Yet we could never be manipulated by machine superintelligence, right?

AGI research is the digital equivalent of gain-of-function research, in terms of the danger. If done at all, it should be done by one, or a very small set, of extremely carefully controlled labs, under government oversight. Even that might not be enough.

> Will the artificial agents replace the human species? One can certainly hope so. First, not so great of a species, as mentioned. Secondly, the universe is large and the humans are squishy, something else must go on eventually.

Easy enough to say if you imagine your distant descendants being replaced. Not so easy if it turns out to be your own children, or you.

*edit -- also, no guarantee that AIs will be "better" than us. A lot of the nasty things humans due is due to selective pressures and coordination problems. AIs will have to solve the same problems. Maybe they will, if they're smarter, or maybe they'll just fight and hurt each other faster.

So it is with any technological innovation. Should computers not have been invented because they eliminated jobs? Should steel? What about agriculture? The future is sure to be different, but that doesn't mean we should fight to deny progress. That way lies the Luddite and Conservative. It's only possible to use new tools for good, not try to erase them to prevent evil.

I think that many technologies, ones that we continued using, are good. The ones that turned out to be bad, we banned. If you make a list of technologies we are still using, then it will contain good ones.

I think that actually, we would be better off as humans if we could figure out a way to coordinate (not easy) and decide in advance which technologies we allow to be released into the world. It wouldn't be perfect, as we might still make mistakes, but we maybe could have stopped leaded gasoline, CFCs, Thalidomide, social media feeds, and so on.

If it's a big enough evil, yes, sometimes we should not invent some things. And I say this knowing that there are currently drugs that the FDA is holding back (due to over-caution) even though they would be very likely to save lives. Sometimes we don't take enough risks. Sometimes we take way too much.

I don't know if stopping unaligned AGI is possible, but I think it's worth trying. I can imagine some good coming from aligned AGI, but I feel like most of the things we could do with aligned AGI, we could also do with just regular old narrow AI, but slower. Slow sucks when people are dying, but if there is the possibility of EVERYONE dying and nobody new being born ever again, then that's the thing to avoid.

A list of genocidal dictators: https://www.scaruffi.com/politics/dictat.html

Anyone who comes into power enough to do something like this has a high amount of intelligence -- not necessarily book smarts, but the ability to persuade or manipulate other people.

Why do you need to prove no correlation? Even if there's a correlation, unless that correlation is extremely strict, there is some risk of a super-intelligence turning nasty. And a strict correlation is simple to disprove with, say, humans and bonobos. Humans fight and hurt each other, while bonobos are basically the "peace and free love" hippies of the primate kingdom.

And orthogonality is not about ethics, but about goals. If you can have three very intelligent people, one of which tries to become an industrialist in order to become as wealthy as possible and live out their days in luxury, another of which decides to become a scientist and help humanity as much as possible, and one of which decides to spend their life building model trains, you have proof of orthogonality right there.

Unless you get a super-intelligent AI whose goal/behavior is exactly something like "always listen to humans and do what they say, but temper that by not hurting people, and don't try to get too far ahead of them and predict what they would want, and don't try to accumulate too many resources in pursuit of your goal, and also keep in mind that humans may try to invent other AIs may try to gain more resources in pursuit of their goals, and try to stop them if they go wild, but again don't go too far" and so on, for all the things I haven't even thought of but that are are also important, then you may have a really big problem.

If we can invent aligned AI, that will help a ton with all the other existential risks. If not, unaligned AI is the existential risk to end all others. Maybe things are changing fast enough now that the variance in our outcomes is so high that we're guaranteed to end no matter what. I hope not.

orthogonality is almost perfectly wrong; ethics&planning ability is highly correlated with intelligence

I'm guessing you're a very nice person. There have been a lot of smart people in history who gained power and did very, very nasty things. If you're nice, being smarter means being better at being nice. If you're not, it means being better at doing whatever not-nice things you want to do.

And we're just talking about humans vs humans here. From the point of view of, say, chickens, I don't think they'd rate the smarter people who invented factory farming as nicer than the simple farmers who used to raise 10 birds in a coop.

I mean, if you exclude AGI, there are some ways that humans can wipe ourselves out, but I feel like we're identifying the big existential risks early enough to handle them. Intelligence that isn't human is the real danger.

Good point. I'm partially conflating the definition I usually mean, which is "having a goal in the world", with what they're doing, which is "having ability to affect the world". Hugging Face is trying to keep these locked down, and maybe being able to generate images and audible sound is not that much more dangerous than being able to output text. But it is increasing the attack surface for an AGI trying to get out of its box.

That's all stuff that LLMs are really, really good at handling though. All you need is prompts that explain what's on the menu, all the ingredients, allergy info, calorie counts, etc. and it will be able to handle it. And I'm sure they'll be recording questions for which the LLM has no good answer, to improve it.

Note that people adjust their behavior as well. As people interact with these more, they'll sometimes be frustrated but they'll adjust.

Now, after all that, please don't paint me as a techno-optimist. I think that it's very likely that we're about to cause some massive social problems by making it so that the average human doesn't have many options where they can provide more economic value than a machine, and meanwhile, are there really that many problems today that tech can solve, as opposed to problems of simply coordinating people?

And that's not getting into the whole "are AGIs going to make humanity extinct" question, which, after getting into it, I think is pretty darned likely at the rate we're going.

I'm not 100% sure that AGI is guaranteed to end humanity like Yudkowsky, but if that's the course we're on, seeing news like this is depressing. Can anyone legitimately argue that LLMs are safe because they don't have agency, when we just straight up give them agency? I know current-generation LLMs aren't really dangerous -- but is this not likely to happen over and over again as our machine intelligences get smarter and smarter? someone is going to give them the ability to affect the world. They won't even have to try to "get out of the box", because it'll have 2 sides missing.

I'm getting more and more on board with "shut it all down" being the only course of action, because it seems like humanity needs all the safety margin we can get, to account for the ease at which anyone can deploy stuff like this. It's not clear alignment of a super-intelligence is even a solvable problem.

And even with all that, probably it's best to still exercise an abundance of caution, because you might have made a mistake somewhere.

Elon Musk is part of a big framework we've created to stop things going off the rails. It's got humans in the loop at pretty much every step, and sometimes still goes wrong, for example, working conditions in amazon fulfillment centers. You can argue whether that's offset by the benefit of cheaper and faster shipping. But even then, there are definitely cases where we outsourced our desires to organizations that went on to do bad things, even with humans running them.

Why would you expect an AI to listen to you? We can train them, but it's not clear when they get smart enough that they'll learn to do what we want, or just learn to pretend really well. A child punished too much will generally learn to lie very convincingly, because it's too hard to avoid every possible misstep. Is that what is going to happen with AGI? It would be good to figure it out before we build one that's a lot smarter than us!

Still potentially way easier said than done. Robert Miles has a good video on this.

One issue is that to define something like "harm", you need to solve a whole bunch of philosophy problems. And people disagree -- does corporal punishment harm children? People used to think that failing to hit your kids was harmful, because they'd grow up and be lazy and end up wasting their lives!

Another issue is that a lot of stuff breaks down as the robots get more intelligent, and start coming up with solutions humans aren't smart enough to consider. Is scanning human brains and digitizing all of us and getting rid of our human bodies harmful? What if it means we get to live forever and are protected by the AI, who is more competent than us? What if we don't want to -- is it good to "protect" us against our will? If not, what about saving someone who is suicidal? (back to the philosophy problem!)

Also, note that the three laws aren't really meaningful -- if lower laws always must be prioritized, you can't ever really do anything, because your action might lead to a human coming to harm. So they have to be interpreted with some tradeoff. But that opens the possibility for a robot to take actions to protect its own existence at the cost of human lives. If the tradeoff is 1000 AI lives to 1 human life, what happens when there are 1000 times as many AIs as humans, and they're worried that we'll do something that ends up killing them all?

So yeah, implementing anything like the 3 laws still requires completely solving alignment, basically.

Really neat! A clever idea, and it's good to see this kind of work being done.

Also, does the second image in the article change when you click to open it, for anyone else? The normal preview in the page shows as a duplicate of the first, maybe a Substack bug or something.

Thanks, that's a great breakdown of other considerations!

> Let’s say you became “comically” smarter overnight, could you really significantly benefit that extra knowledge? Besides going for the Millenium Prize problems, the benefits would not be too significant, for example is it that valuable in the startup space? (otherwise we would see PhD students as CEOs everywhere). Opportunity/luck is much more meaningful.

Hmm, that's assuming that PhD students want to be CEOs. People become CEOs because they like money, but also because they like running companies. For a PhD student, that might mean days full of people coming to them with problems they have no interest in solving. Sure, PhD students want money, but if getting money means sacrificing years of your life doing work you don't enjoy, it's not generally worth it.

I don't disagree that luck is a huge factor in most people's success. That said, you're unlikely to become a multi-billionaire without substantial help from your brain. There are other factors, like lack of shame or embarrasment, that could also be considered a form of intelligence, or rather an adaptation to a particular kind of environment. They might make you less popular in some situations, but more likely to succeed in business. Anyway, I agree that there are multiple factors.

Information is for sure a bottleneck, that's a good point. In a game like chess, you have perfect info, which is not the case in the real world. More intelligence lets you draw better conclusions with less data, but it's not omniscience. One big question is, how close are humans to the theoretical maximum of intelligence? Some people think we're quite a ways away. When AGI is developed, is it going to blow way past us, to the point where we can't even understand the concepts it comes up with? I don't know!

> Let’s say we are at an endgame Monopoly — would knowing ahead of time your opponents dice rolls help you win?

I think so, wouldn't it? For trading properties and so on. I'm not sure if I'm getting the point you're aiming at, though, so apologies.

Maybe another analogy is poker, where your opponents haven't figured out game theory and play on intuition. Having a better model for how the game works is absolutely a big advantage that stacks up over time, even with the randomness. Maybe, as an AI, you just look for small advantages here and there, playing humans off each other, gaining trust and influence, acting helpful, until you're in a position of power. Remember that an AI is effectively immortal, and possibly very patient. One scenario of concern is that AI takes over, it just takes a generation or two, after we have all thought that we were past the hard parts. At that point, we might be able to digitize human brains, or have solved the AI alignment problem, so maybe it's a hopeful scenario.

And then a big variable is how many super-intelligent AGIs there are. Once we develop it, are they going to be carefully restricted, or are we going to test them until we think they are safe enough, then copy them everywhere? These might look very different.

I'm also not an expert, so same for me! Grains of salt all around.

I don't think that's what people are afraid of. It sure as heck isn't what I'm afraid of. Unless you count "humanity" as the existing system of power, and "something much smarter than humanity" as the disruption.

People can have sincere beliefs and disagree about the best course of action. Hinton has publicly said he's in favor of stopping, but doesn't think China would. I personally disagree, I think that coordinating a stop would be easier than he thinks. But I don't doubt his sincerity. He also believes that if we can solve alignment, AI would be a massive boon for humanity, which is why it's a bit of a quandary.

Why? Yes people misusing is a danger, but given that no other technology has ever had the same quality of intelligence, why do you think that we won't eventually develop really smart AGI, OR why do you think that controlling something smarter than us will be easy?

That's not quite the existential risk of AI -- it's that we end up growing an intelligence (or multiple intelligences) that are very very smart but only have the illusion of morality, and they decide that whatever they're trying to do would be a lot easier and more certain to succeed if they didn't have to worry about humans stopping them or inventing other AIs to compete with them. It seems likely that unless we really understand how AI works, we'll end up with something that evaluates possible courses of action in a purely amoral, sociopathic sort of way. Like, trying to avoid getting caught rather than trying to avoid doing a bad thing.

So it's not that they cause the collapse of the economy, it's that they figure out a way to entirely take over the planet, engineering a virus or something that kills almost everyone, but only once they have enough control over robots and such to keep their own power plants running and protect them from the human stragglers, eventually getting rid of all of us. Waiting until the plan is basically fool-proof before starting to take the first steps.

Even if an AI has some sort of morality, at some point, if it's smart enough, you cannot hope to control it for the same reason a toddler cannot hope to outsmart their parents. It will have a plan for us, and if it decides that, hey, humans are better off scanned and simulated rather than having physical bodies, that's what's going to happen, like it or not.

I'm not sure how worried I am about this. Some days very, other days not as much. The risk depends on a lot of factors with wide possible ranges, like how much model capability and generalization scales with compute, how likely are breakthroughs, how hard alignment turns out to be, is alignment even possible, will we learn enough from earlier models to make alignment of later ones easier, will there be a qualitative shift in alignment difficulty as models get smarter, and so on. It's also a quandary because there is substantial human suffering that AI might be able to help. But given that the downside risk is all of humans dead forever, it seems worth slowing way the heck down to a safe speed, trying to get all the benefit out of current models as we can, and focusing on making sure that we aren't about to head off a cliff.