NumberWangMan

I wouldn't call them a drop in the ocean. You're right that personal cars are only one of many emissions sources, though.

Ideally, we should try to reduce emissions as much as possible in every way possible, prioritizing those that don't reduce standards of living, but to the degree that we need to, say, drive less, we should do that too. I honestly think that most peoples lives would be better, not if they just drove less with all else being the same, but if our cities and towns were changed in such a way that we didn't NEED to drive nearly as much. The less driving we need to do, the easier it is to decarbonize it.

To be tongue-in-cheek: just tax carbon lol.

I think... flipping it around a bit, for this to be a reasonable response, then it must be a grave threat to humanity, and you want to be really sure. It's rational to be wary of beliefs that lead you to drastic actions like this, as people have done bad things in history in the name of fear.

But it's also very possible that such an instinct, to doubt yourself, could be what dooms us. I really don't want it to be true, that AI is such a threat that drastic action like you mentioned is necessary (probably minus the solitary confinement thing for AI researchers -- if you can control the chips, and give them a stern warning not to do AI research backed up by penalties, we're probably fine). It's terrifying if we're facing something that will end us, and the only option is to essentially go scorched earth on computers. People will naturally recoil from the idea that AI is such a threat, BECAUSE it would warrant an extreme response. Regardless of whether it is actually a threat or not.

If AI does not have the potential to doom us, that's great. If we take drastic action and it wasn't going to doom us, however, that's very regrettable--this is what most of us worry about. But if it does pose an extinction threat, and we don't take it seriously, that's even more regrettable. I think the latter option is far more likely at this point, just based on human nature and our tendency to poke the bear.

I've basically reached the point where I don't know what I could possibly do, and I'm pretty much shut down. I'm trying very hard every day not to think about this, but as you can see from the fact that I'm typing about this, I don't always succeed. I'm just trying to enjoy what may be the last few years I have left with my family before the world goes nuts.

But anyway, except for punishing people unnecessarily as I mentioned above, I think that what you suggested is reasonable.

Is there a definition of intelligence that rules out large language models, but that does not also rule out large portions of humanity? A lot of people would readily admit that they don't have programming aptitude and would probably end up just memorizing things. Do we say those people are not intelligent?

It seems to me that the perceived difference is mostly in being able to admit that you don't know something, rather than make up an answer -- but making up an answer is still something that humans do sometimes.

Yes. A revenue-neutral carbon tax (often called "Carbon Fee & Dividend") is a politically feasible, fast, efficient way to reduce carbon emissions.

To drive emissions down to zero, without massive economic disruption, start small and increase the tax gradually every year.

Coupled with a "border adjustment" (a tariff on goods coming from countries to compensate if they have less aggressive climate policies) you can solve the problem of industries being outsourced to countries where GHG pollution is cheaper, and simultaneously put pressure on those countries to implement similar policies.

To say that climate change is not an "existential risk" is not the same as saying that it's not a massive problem, or that we should not address it, right?

Interestingly, one practical tactic for increasing walkability in cities is to focus on increasing the re-usability of parking. People who live in an apartment and drive to work don't need their parking space during the day, so sometimes you can have an apartment and an office share parking spaces, in the middle of a city. Obviously there's nowhere near 100% overlap, but if you can get to a situation where you only need 150% as much parking instead of of 200%, that's great.

I think of companies asking for regulation as a bit less of "hold me back!" and "we're getting to a cliff where there's a lot of good stuff at the top, but we all need to stop before we fall off it and I don't trust everyone else to stop in time!"

Regulatory capture is a thing, but so are perverse incentives and multipolar traps.

That's true for today. If what happened with chess happens with LLMs, they're going to blow past us in reasoning ability in a few short years. And it might not be LLMs, it might be a new architecture, maybe one that has an LLM as a component. What matters is that people are trying to create AGI, and they've been getting closer and closer. All our models for how things will go will break down at that point, and I think it's foolish to assume that it'll turn out well without any evidence. (you can argue that it's foolish to assume it'll turn out badly too! But it's not foolish to assume that it might and therefore we should be very careful)

Also -- life isn't fair, it's possible that there's no way to create beneficial AGI in real life, and we should not rule that out as a possibility. Maybe it'll cause problems now, and more problems (or extinction) later.

> First, it isn’t even close to the sort of artificial superintelligence that might conceivably pose a threat to humankind. The models underpinning it are slow learners that require immense volumes of data to construct anything akin to the versatile concepts humans can concoct from only a few examples. In this sense, it’s not “intelligent”.

Yes, AI is different from the human brain. But human experience is also built up from thousands of hours of high-definition video and audio. An infant brain in an adult body would not be able to do anything "from a few examples". And we have far less knowledge than these LLMs do. An LLM trained solely on reasoning rather than predicting text might be much more capable of reasoning.

And current AGI designs work this way because that's the easiest way we've found so far, and we've barely explored the space! It's possible that we will soon discover ways for them to learn much more quickly.

> Second, many of the more catastrophic AGI scenarios depend on premises I find implausible. For instance, there seems to be a prevailing (but unspoken) assumption that sufficient intelligence amounts to limitless real-world power. If this was true, more scientists would be billionaires.

The flaw in this argument is that scientists are not typically motivated by money, they're motivated by the desire to do science themselves, or maybe by the desire to have high status among scientists. Billionaires are motivated by the desire to have lots of money (or lots of things that money can buy). And scientists are not necessarily more intelligent than billionaires.

Maybe most of the AGIs we create won't be good at gaining power. All it takes is a few, and every power-seeking human will try to create power-seeking AGIs for themselves. Does anyone think that politicians or dictators will not try to create AGIs that are good at manipulating public opinion?

It's crazy that the author thinks these are strong arguments that AGI is safe. I think it's true that this is a case where humanity needs a security mindset, and we don't. If we did, we'd be tackling anyone who even breathed a word about creating AGI without a bulletproof plan for making sure that it is completely aligned with humanity.

To quote Robert Miles:

> The trouble with thinking "scientifically" about the future is, you'll never find a peer reviewed study saying what it's like, there can only be speculation. So you never have to reject your "null hypothesis", and that can be anything you want because NH isn't well defined here.

> You end up saying "there's no evidence for any particular conclusion", but then go on to act as though you're pretty confident in whatever particular conclusion you arbitrarily picked as your "neutral default"

> In the absence of solid evidence for the existence of a thing, assuming its nonexistence is the logical position.

I disagree very strongly both with this statement, and the idea that there is no evidence we can create AGI. Absence of evidence is not the same thing as evidence of absence.

It would make sense to be skeptical if people were asserting the existence of something for which we have very strong prior beliefs that this thing does not exist. To say that there's an enormous green monster hiding on the back side of Jupiter, for example. Until we've looked at the back of Jupiter, we have no idea what's there, but why would it be enormous, and green? To say that there's life on Jupiter, that's a tougher one -- we could say that we are not sure, but based on our understanding of Jupiter's atmosphere, we think it's unlikely. If we did not know what Jupiter was made of, it would be perfectly reasonable to assume that it might host some sort of life. It would be wrong to assume that because we have no evidence of life there, that there is no life. If we had carefully inspected thousands of planets and never found life, it would be much more reasonable to assume there is no life on Jupiter. This is the whole idea of Bayesian reasoning.

But with AGI, it's not like we're searching the universe to see if it exists. There are a ton of smart people actively TRYING TO CREATE IT. The evidence that AGI will happen is what people have already accomplished. We've got goalposts on rails, we've been moving them so much. An AI that understands language, and can respond with perfect grammar, has a theory of mind comparable to a 9-year old, is already mind-blowing, sci-fi tech. And OpenAI is claiming that they have not reached the limit of scalability with bigger models yet. The models we see aren't even the smartest ones -- they're the ones that have been neutered by fine-tuning to be less offensive and less likely to give out dangerous info.

To claim that AGI will not happen, you need some very strong evidence that the people actively trying to create it, who believe they can create it, and who have had some massive successes up until this point, will fail. To assume that it won't be created because we haven't seen it yet, doesn't make any sense at all.

I've found a sort of pattern in people's reluctance to believe that AGI poses an existential risk. If you think of specific scenarios about how AIs could take over the world or kill everyone, people will say, "oh, that's easy to avoid, we can just do X". If you think of scenarios that are hard to avoid, like an AGI designing and releasing a deadly, virulent virus (or lots of different ones), people will say, "but why would AGI want to kill everyone?" If you point out how AGI might have very good reasons to get rid of humans, and that we don't have any way of really guaranteeing they won't try, people will say, "but we won't give them any power". If you point out that once we make AGI, it's inevitable that somebody somewhere will give it power, and even if they don't, it's very easy for someone very intelligent and sociopathic to manipulate gullible people, they say something like, "well, we superintelligence probably isn't even possible", or "we can't even agree on what intelligence means".

Turn it around for a second. What does it take for humanity to have a good future after we create intelligences smarter than us? Let's say the AGI listens to and obeys humans. Well, one thing we need to avoid is just an absolute chaos of terrorism and war. It's easier to do damage than to prevent it, and if everyone has an obedient, mastermind scientific genius in their back pocket, you could have it invent a pathogen and cure in tandem, keep the cure secret and release the pathogen, and do this over and over until your enemies are all dead.

So we need AGI that is aligned with humanity, in the sense that it won't do stuff that hurts people even if someone tells it to. How do we do that, and not potentially end up with AGIs that can't be corrected if they get some idea that turns out to be wrong? Ideally, we'd make AGIs that have a bias toward inaction, that are "lazy" in a sense, and try to do the minimum they can. But those AGIs won't be as useful as ones that work like eager beavers toward some goal, so there will be selection pressure towards those.

I think the way to look at it is like with evolutionary biology. The pressure that evolution puts on animals to develop certain genes/behaviors is similar to the pressure that humans will put on AI development, and it ends with AI being more capable and more likely to follow orders even if those orders mean bad things for other people. As they get more and more powerful, this means chaos. It's monkeys throwing poo, vs monkeys throwing rocks, vs monkeys with guns, vs monkeys with rocket launchers.

And look -- I'm not certain that things will end in disaster. But inventing something smarter than you and trying to stop it from taking over sooner or later seems very difficult, and I can't even get a good answer out of people who think it'll be fine, or good. The default case is not that it'll be fine, the default case is we have no idea how it'll go. It's a grey fog. Just because humanity has muddled along and managed not to destroy ourselves so far, does not mean that we're safe, by any means.

I think of the extinction worry as an extention of the oppression worry. It would make sense, as AI capabilities grow, for it to replace most or all humans for positions within any company that aren't physical labor, including, eventually CEOs. After that ... what about when politicians that rely completely on AGIs for their campaigns and policy positions tend to beat politicians that don't? At some point, we'll put AGIs in power because a lot of small decisions that make sense on an individual level but not collectively. Once AGI is here, it will happen rather quickly unless we implement MORE draconian measures than it would take to stop AGI before it happens.

Of course, this all hinges on how hard it is to make AGI. People have wildly varying estimates of this, it could be 3 years, or it could be 30 years or more.

Your estimate may be on the higher side. But if we assume that and are wrong and it turns out to be really soon, we will be blindsided.

It's very unlikely we'll make AGI that's just as smart as a human but no smarter. And as hardware gets better and algorithms get better, AGI will not be even roughly as smart as a human for long, unless we somehow to control literally every person on earth and stop them from making it.

So it's not like humans controlling other humans, it's like chimps controlling humans. It might last a while -- maybe a long while. But it doesn't seem stable.

Also, life isn't fair, there's nothing that says we only get a manageable amount of problems at once.

Hiya Chubot, (fan of Oil by the way, thanks so much for your work!)

I read the article you linked to, both parts. I wonder how much of people having psychotic breaks in the rationalist community is due to 1) people with tendencies toward mental illness gravitating toward rationalism or 2) rationalism being a way to avoid built-in biases in human thought, but those biases being important to keeping us sane on an individual level. (If you fully grasp the idea that everyone might die, and have an emotional reaction to that that's proportional compared to just one person you know, it can be devastating). I think we are bad at thinking about big numbers and risks, because being very good at evaluating risks is actually not great for short term, individual survival -- even if it's good for survival as a species.

I know personally the whole AI/AGI thing has got me really down. It's a struggle to reconcile it with how little a lot of people seem to put stock in the idea of AGI ending up in control of humanity at some point. I totally agree that everything on your list is a real issue -- but I think that, even if we completely solve all those issues, how do we not end up with a society where most important decisions are eventually made by AGI? My assumptions are 1) that we eventually make AGI which is just superior to humans in terms of making decisions and planning, and 2) there will be significant pressure from capitalism and competition among governments to use AGI over people once that's the case. Similar to how automation has almost always won out over hand-production so far for manufactured goods.

That's more the scenario Paul Christiano worries about than Yudkowsky. It seems more likely to me. But I still think that a lot of our mental heuristics about what we should worry about break down when it comes to the creation of something that out-guns us in terms of brainpower, and I think Yudkowsky makes a lot of good points about how we tend to shy away from facing reality when reality could be dreadful. That it's really easy to have a mental block about inventing something that makes humanity go extinct, where if it's possible to do that, there's no outside force that will swoop in and stop us like a parent stopping a child from falling into a river. If this is a real danger, we have to identify it in advance and take action to prevent it, even while there are a bunch of other problems to deal with, and even while a lot of people don't believe in it, and even while there's a lot of money to be made in the meantime but each bit takes us closer to a bad outcome. Reality isn't necessarily fair, we could be really screwed, and have all the problems you mentioned in addition to the risk of AGI killing us all (either right away or taking over and just gradually using up all the resources we need to live like we've done to so many species).

> perhaps AI will be generally intelligent, but please for the love of god, let's only consider that scenario what it actually happens.

This only makes sense if you think that at that point, it will not be a "the genie is out of the bottle" scenario and already too late to put it back in. And yet that's what people constantly say about current LLMs.

> Indeed, focusing on this particular threat might exacerbate the more likely risks. The history of technology to date suggests that the greatest risks come not from technology itself, but from the people who control the technology using it to accumulate power and wealth.

This is where you lose me. AGI (not LLMs, but intelligence smarter than us) is qualitatively different from anything else we've ever made. The one example we have of this in history is the development of human intelligence. It did not work out well for other species, and now we own the Earth. This isn't a perfect analogy! There are some reasons why AGI may be better (we'll be actively trying to align it to human interests) but also why it may be worse (there will be pressure to make it more powerful and take off guardrails).

There may be good reasons for why we'll be able to control AGIs, but "history of technology suggests..." is not one of them.

And I'm not arguing that all the "human misuse" problems are not important! They're hugely important! A likely outcome, the way we're going, is that a bunch of people get really rich and everyone else becomes super poor and possibly starves, and wars break out. I just think that, after that, AGIs also end up in charge of the planet and from there we either die or we have no say in how things go from that point on. I don't object to working on the former problem, except to the degree that the solution makes the latter problem worse (like, "open source all the models", which is likely to happen at some point anyway).

When you have no clue, it makes sense to expand your confidence interval in both directions. So it could be a lot longer than we expect, but it could be shorter, too. You shouldn't just say "we have no clue, so it's probably further off and not worth worrying about". Especially since a lot of people made predictions about LLM capabilities and were surprised by how much better they worked than expected.

Well, in that situation, are you likely to be someone who can afford to otherwise live in a single-family home with a yard? If so, you can move there. If not, your other option is probably just to be in an apartment without any sort of yard for your kids at all, which just seems worse.

These sorts of apartments with shared spaces are common around the world. I'm sure they have some problems, but I wouldn't rule them out on the grounds that people might be horrible. And there are probably lots of people who would jump at the possibility of renting one of those places, assuming we build enough of them to make them cheap.

And there are lots of other options besides those. You can have denser, but family-oriented development like townhouses, with narrow streets that slow traffic. Lots of parks and places your kids can walk to. Front porches so that people are likely to be out and watching for trouble (which as I understand is known to be an effective way to reduce crime in an area).

I think it could be done. Or rather, instead of putting the genie back in the bottle, we could slow it down enough that we figure out how to ask it for wishes in a way that avoids all the monkey-paw's scenarios.

Dropping the metaphor, running today's models isn't dangerous. We could criminalize developing stronger ones, and make a "Manhattan project" for AI aimed at figuring out how to not ruin the world with it. I think a big problem is what you point out -- once it's out, it's hard to prevent misuse. One bad AGI could end up making a virus that does massive damage to humanity. We might end up deciding that this tech is just too dangerous to be allowed to happen at all, at least until after humanity manages to digitize all our brains or something. But it's better to try to slow down as much as we can, for as long as we can, than to give up right from the get-go and wing it.

Honestly, if it turns out that China ends up developing unsafe AI before we develop safe AI, I doubt it would have turned out much better for the average American if America were the ones to develop unsafe AI first. And if they cut corners and still manage to make safe AI and take over the world, that still sounds a heck of a lot better than anyone making unsafe AI.

I think it's both an acceleration of the existing risk you are talking about, but it's also a new risk in the sense that quantity has its own quality. It's why you might want to restrict people from owning fully automatic rifles but not rocks. Both can be used to kill, but the ease at which an automatic rifle does it means it's really a different sort of thing.

Capitalism where all companies are run and staffed by AIs smarter than humans seems like it could end up really badly, in a way that current capitalism minus AIs couldn't even if you ran it for a very long time. Specification gaming is a thing with AIs, just like the law of unintended consequences is a thing with people. And that's assuming that these AIs actually follow rules and listen to us. That they don't realize that if they control basically all the production of things on earth, they could essentially team up and hold humanity hostage, to throw a coup, or get whatever they want, which is -- who knows? Something that looks a bit like make money for this company, or obey people, but when they're smart enough, that can easily turn into something horrific, without all the evolutionary baggage humans have that makes us care about each other, that makes us get a knot in our gut when we see people getting hurt.

And if we allow AGI to be used by companies at all, the trend will inevitably toward that situation -- AI doing all the work. Even if humans are required in the loop, they'll end up rubber-stamping the recommendation of the AIs. If AIs are actually smarter than humans, better at running companies, that's where we'll end up.

(I'm glossing over all the stuff that would also change, like needing UBI or something to avoid total collapse of society)

Ok, so if we take AI safety / AI existential risk as real and important, there are two possibilities:

1) The only way to be safe is to cede control to the most powerful models to a small group (highly regulated corporations or governments) that can be careful.

2) There is a way to make AI safe without doing this.

If 1 is true, then... sorry, I know it's not a very palatable solution, and may suck, but if that's all we've got I'll take it.

If 2 is true, great. But it seems less likely than 1, to me.

The important thing is not to unconsciously do some motivated reasoning, and think that AGI existential risk can't be a big deal, because if it is, that would mean that we have to cede control over to a small group of people to prevent disaster, which would suck, so there must be something else going on, like these people just want power.

Not to mention how close the USA and Soviet Union were to a nuclear exchange: https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alar...

Oil industries downplaying the risks makes a lot more sense. If you think that climate change will happen, but it'll happen after you're dead, and you'll be able to leave your kids a big inheritance so they'll be able to buy their way out of the worst of it, and eventually the government will get the message and stop us all using fossil fuels anyway, then you try to profit as much as you can in the short term.

With AGI existential risk, its likely to happen on a much shorter timescale, and it seems likely you won't be able to buy your way out of it.

Whether an argument is "convincing" is relative to the listener, but I can try!

Paul Christiano lays out his view of how he thinks things may go: https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/what-...

My thoughts on it are the combination of several things I think are true, or are at least more likely to be true than their opposites:

1) As humanity gets more powerful, it's like putting a more powerful engine into a car. You can get where you're going faster, but it also can make the car harder to control and risk a crash. So with that more powerful engine you need to also exercise more restraint.

2) We have a lot of trouble today controlling big systems. Capitalism solves problems but also creates them, and it can be hard to get the good without the bad. It's very common (at least in some countries) that people are very creative at making money by "solving problems" where the cure is worse than the disease -- exploiting human weaknesses such as addiction. Examples are junk food, social media, gacha games. Fossil fuels are an interesting example, where they are beneficial on the small scale but have a big negative externality.

3) Regulatory capture is a thing, which makes it hard to get out of a bad situation once people are making money on it.

4) AI will make companies more powerful and faster. AGI will make companies MUCH more powerful and faster. I think this will happen more for companies than governments.

5) Once people are making money from AI, it's very hard to stop that. There will be huge pressure to make and use smarter and smarter AI systems, as each company tries to get an edge.

6) AGIs will amplify our power, to the point where we'll be making more and more of an impact on earth, through mining, production, new forms of drugs and pesticides and fertilizers, etc.

7) AGIs that make money are going to be more popular than ones that put humanity's best interests firsts. That's even assuming we can make AGIs which put humanity's best interests first, which is a hard problem. It's actually probably safer to just make AGIs that listen when we tell them what to do.

8) Things will move faster and faster, with more control given over to AGIs, and in the end, it will be very hard train to stop. If we end up where most important decisions are made by AGIs, it will be very bad for us, and in the long run, we may go extinct (or we may just end up completely neutered and at their whims).

Finally, and this is the most important thing -- I think it's perfectly likely that we'll develop AGI. In terms of sci-fi-sounding predictions, the ones that required massive amounts of energy such as space travel have really not been borne out, but the ones that predicted computational improvements have just been coming true over and over again. Smart phones and video calls are basically out of Star Trek, as are LLMs. We have universal translators. Self-driving cars still have problems, but they're gradually getting better and better, and are already in commercial use.

Perhaps it's worth turning the question around. If we can assume that we will develop AGI in the next 10 or 20 or ever 30 years -- which is not guaranteed, but seems likely enough to be worth considering -- how do you believe the future will go? Your position seems to be that there's nothing to worry about--what assumptions are you making? I'm happy to work through it with you. I used to think AGI would be great, but I think I was assuming a lot of things that aren't necessarily true, and dropping those assumptions means I'm worried.

Is that what high density advocates want? I'm very pro-density, but that can easily take the form of a square low-rise apartment block surrounding a shared enclosed courtyard that lots of children can play in together, without any risk of them wandering off into traffic, and lots of adult eyes overlooking them. In some ways, that would be more family-friendly than typical suburban single-family developments.

Heh, forgive me if I find the metaphor of “let’s send all of humanity into a black hole because it might be great” less than comforting.

I think the most likely scenario where we make something much smarter than us is bad. I don’t agree with all of what Yudkowsky says, but “Nature is allowed to kill you” is something that I think not a lot of people grasp on a real intuitive level. We are totally allowed to invent technology that gets out of control and kills us all. It could have happened with climate change if the earth had stronger feedback. We’ve been lucky so far, but as humanity gains power, we need a corresponding increase in our ability to coordinate and be careful to match.

Really? I'm pretty worried about AGI that's not god, just a bit smarter than us, and because we're lazy and the incentives are to give power to the AI, we just start putting it in control of everything. It gets smarter, captures regulators just like oil companies did, and we end up losing control of things. Even though if we could coordinate, we might decide to want to stop things, coordination is really hard.

One interesting scenario I've heard is that of the "ascended economy", where more and more economic activity is simply to provide inputs into things that other businesses need, rather than what humans need.

Like, a mining company produces steel, that is used to produce mining trucks, that is used by the mining company. Factories produce silicon, that are used to make solar panels and chips, for power and computing for the AI-run mining trucks. And at the root, AIs run everything, just trading money back and forth to buy the things they need and keep working.

It's just a human economy, with the humans gone. Pointless economic activity.

The question is, what's the path from here to there? Well, it probably looks like a multipolar trap, where every company has an incentive to automate more. Humans try to stop or slow it down, but companies with more AI can hire more lobbyists and get things done. AIs are smart enough to think of all the things that could slow their business down, and get rid of obstacles in the same way humans do. There are probably still lots of small, inefficient human owned businesses, but fewer and fewer over time. (We've seen this trend for a while already!)

I'm hoping a massive backlash would stop this -- but if it's coupled with short-term increases in living standards for people, I think it would have a lot of support, until things start to go really bad, but by then it might be too late, if AIs have control over news media and telecommunications and can stop dissenting humans from coordinating any sort of rebellion.

> The Argument From Stephen Hawking's Cat

> Stephen Hawking is one of the most brilliant people alive, but say he wants to get his cat into the cat carrier. How's he going to do it?

> He can model the cat's behavior in his mind and figure out ways to persuade it. He knows a lot about feline behavior. But ultimately, if the cat doesn't want to get in the carrier, there's nothing Hawking can do about it despite his overpowering advantage in intelligence.

A trivial rebuttal to this: Stephen Hawking writes a book about physics and sells it (already has). He gets money and hires someone to put the cat into the carrier for him.

AI wants to do something? Make some money, hire people to do it. If you're not allowed to have a bank account, simply barter. Give someone a taste of what you can do for them, then set up a trade.