NumberWangMan

Some people can handle complexity, but I think they're in the minority. I'm not sure that tacking on "oh, but also climate change will be good in a few ways" is helpful in getting people to take action. So there are sure to be some misinformed climate activists out there who don't know about greening, but that may be a preferable situation to everyone being very informed about the nuance and therefore feeling like this is a smaller problem than it really is.

> It’s all very clever, yes, but at bottom it’s just a brute force approach.

This is the bit that I think you should focus on a bit more. I don't think it's the case you need complicated, clever algorithms and architectures in order to get complicated, clever behavior.

If you start with particle physics, then work your way up to chemistry, and then biology, you can see how we start with very, very simple rules, but at each level there is more and more complexity. The universe "running" physics is the epitome of a brute-force approach. It would be a mistake to say that because the rules of particle physics are so simple, that nothing made of those particles could ever think.

Likewise, even though these models are just big arrays of numbers that we stir in the right way to make them spit out something closer to what we want over and over again, I think it's a mistake to say that out of that, can never arise something much more capable than humans.

> You can’t just train an LLM with more and more parameters and more and more tokens and expect it to be smarter than the data it was trained on. And such models don’t bring us any real understanding of what it would take to make super-intelligent machines.

Others have addressed the first point here, but for the second -- yes, that's true. As for the second, I think we'll have a bit of warning before we get to true superintelligence, but even now, I it seems to me like we have half of an AGI in large LLM models. They don't seem to be conscious, can't really evaluate its their thoughts except by printing them out and reading them in again, and are only superintelligent in terms of knowing lots of facts about lots of things. But I think we are probably going to figure out how to create the other parts and we'll be there.

I am worried that humanity is on a bit of very-high-inertia train of "more and more progress" without enough safeguards. It was ok in the past, but as our world gets more and more connected and new inventions get spread far and wide in less time than ever before, it's possible for damage to be done on a very wide scale before we can figure out how to counteract it. It also means that good things can spread in the same way -- but the problem is that it's not just the average that matters, it's the variance. It doesn't matter if you create and disseminate 9 out of 10 new technologies that are massively beneficial if the other 1 ends up with humanity gone or completely disempowered, and you can't take advantage of the good stuff.

Individuals will always be able to do this, but it might be enough if it's illegal and a bit of extra effort. You don't need to eliminate 100% of AI-generated spam, just make it risky and unprofitable enough that it's not a common practice, and have people call out those who seem to be doing it. I'm not sure it's possible, but it seems worth trying.

And if someone is operating a troll farm in another country, this won't help.

Nitpick: I think we can comprehend an intelligence greater than our own, up to some point, but that's different from being able to predict its actions.

And we could contain an intelligence greater than our own, up to a point. But if there are a lot of incentives not to, because letting that intelligence act on the world gains the "handler" money/power, then once there's one, there will likely be many, many more.

There's a game-theory solution to this, which is to impose a tax on imports from countries in order to make up for their lack of aggressiveness in addressing climate change, aka, a border-adjustment mechanism. It's not perfect, but it provides a way to encourage every country to work together by punishing ones that don't. It also discourages industries from outsourcing in order to skirt environmental regulations.

I would add the caveat that Chess is a game where the nature of the rules mostly eliminate any advantage of numbers, which is not necessarily the case for other sorts of competition.

I'm still quite concerned about the risk of rogue AI, as it's not even necessarily true that humans will have the advantage of numbers, or be able to bring our numbers to bear. And a smart AI will try to divide us and conquer, if it comes down to a conflict, including recruiting humans to its side.

On the other hand, the case for reverse causation is particularly strong. Sleeping on the street is very likely to cause anxiety and depression. Being around lots of people using drugs 24/7 makes it a lot easier to pick up a drug habit.

This feels related to the idea in "Billionaires, Surplus, and Replaceability" (1)

Basically, if, say, Jeff Bezos never existed, it seems likely that someone else would have created Amazon, perhaps a few years later, and maybe not quite as good. "Big online retailer" is sort of a natural niche, with a bit of a natural monopoly. So while the classic argument for letting people keep most of their wealth gained in the free market is that they provided a lot of value, maybe that doesn't make as much sense when you consider that if they hadn't done it, somebody else probably would have soon after.

It's obviously impossible to gauge exactly how much excess value someone provided compared to the world in which they never existed, but to round it up to 90% or doesn't seem very accurate.

So it seems like in our current world, there's a winner-take-all effect that a lot of venture-backed startups are trying to exploit. If we were a lot more aggressive in taxing companies like Amazon, my intuition is that it would go a long way toward reducing this effect.

(1) https://astralcodexten.substack.com/p/billionaires-surplus-a...

Well, let’s tax carbon and start collecting it!

Vecr gives good analogies, but for AGI, I think investing a lot into alignment research, trying to set up incentive structures to actively avoid "arms races", establishing global oversight and agreements, having a "fire alarm" or process in place that a company can trigger if they develop an AI that they think is close to becoming dangerous, or even individuals. Strong whistleblower protections may help. Tracking GPUs so we know where compute is being gathered together. Starting talks with other countries like China, right now, to build a common understanding of how to deal with a potential problem.

I think it's true that it may be very difficult to solve alignment theoretically, and solving it when we have real-world systems to look at may make it easier (but still probably not easy). But we need to be able to buy companies time so they can slow down as we approach what seems to be the "edge" of dangerous capability / consciousness / goal directedness, without feeling like they will lose out. And to be very strict about the conditions the models are trained under and what testing they undergo before release.

A lot of this is stuff that people are already suggesting, but not everyone is taking the risk seriously.

Haha, it was me, Moloch! https://slatestarcodex.com/2014/07/30/meditations-on-moloch/

In other words, these are situations where the incentives for individuals are not aligned with incentives for the group.

And Altman revealed that he does not own stock in OpenAI, and declined to take a position at the proposed new agency. He said he could recommend other people. If those all turn out to be OpenAI employees, then I will be disappointed and concede the point.

Yes, him being the head of an AI company means you should be skeptical. But jumping right to "let's do the exact opposite" is not sensible.

Also, sitting right next to him was Gary Marcus, who doesn't have a dog in the fight as far as I know, and he agreed with a most of the recommendations.

Existential threat is a really hard thing to grapple with. You can't wait for real-world evidence of danger to deal with it.

If there's a possibility of an asteroid coming that will wipe out life on earth, waiting until our telescopes can see it to start taking action may not give us enough time to react. We have to plan for the theoretical possibility.

Maybe we'll never be able to create artificial superintelligence. Maybe artificial superintelligence will be much easier to align than we thing. Maybe it'll be hard, but manageable, possibly because we have more time to work on the problem than we think.

All of those are possibilities, but the fact that if you ask 100 people about it, you'll get 100 different opinions, means that we have very little idea how things will go. Given that, I don't like the approach of wing it / hope for the best. If we end up stifling AI innovation unnecessarily, that may mean that our techno-utopia is delayed for a few years, but we get there in the end. On the other hand, if we under-react and things turn out really bad, we may not get another try. Scenarios where AGI kills a billion people are really bad, but scenarios where we create it and end up permanently disempowered or dead are worse.

I wonder sometimes if people are just really dissatisfied with their lives, that they react with such vitriol to the idea of AI progress being delayed. I know we have a lot of problems, but how many of them are technological vs. having poor human institutions and incentive structures? AI probably won't help much with that.

Here as a post on his blog from 8 years ago: https://blog.samaltman.com/the-software-revolution

"...development of an AI that could end human life may only require a few hundred people in an office building anywhere in the world, with no equipment other than laptops.

The new existential threats won’t require the resources of nations to produce."

and

"The fact that we don’t have serious efforts underway to combat threats from synthetic biology and AI development is astonishing."

So (2) could be a factor, for sure. But (1) is consistent with what seems to have been his viewpoint for a long time. It's possible he's updated his viewpoint and is faking concern about safety, but given how recent advances have seemed to make a lot of people more concerned, I don't think that's likely.

His point of view, that he's stated on the Lex Friedman podcast, is that AGI is potentially dangerous and inevitable, but safer to develop as soon as possible and try to make it safe by studying real-world models empirically, rather than waiting until computing power is much greater and risking a "fast takeoff".

I don't know that I agree with this strategy, but I can respect that it's a consistent viewpoint.

So what I hear from him is "Yes, this is dangerous stuff, and we're trying our hardest to make it safe, but we can't control everyone, so we need regulation ASAP because market forces are going to push everyone to race and release unsafe AIs".

Yep, and dismissing doomsday claims on this basis is a heuristic that almost always works.

I mean, if it’s reasoning, we may be very close to AGI, and if we are, that’s a big problem. We’re on a trajectory for these models to become a lot smarter than us, and we’re not ready to deal with that.

That's easy to say in the abstract, but when it comes down to the people you love actually getting hurt, it's a lot harder.

> There is nothing dangerous in current ai models or ai itself other than the people controlling it.

Totally agree! but...

> If it were intelligent then yeah maybe but we are not there yet and unless we adapt the meaning of agi to fit a marketing narrative we wont be there anytime soon.

That's the bit where I don't agree. I don't think we can say with certainty how long it will be, and it may be just years. I never imagined it would be so soon that we have AI that can imitate a human almost perfectly, and actually "understand" questions from college level examinations to write answers that pass them.

Ok, so just to confirm out of your 4 scenarios, you don't include:

5) There are advanced AGIs, and they will compete with each other and trample us in the process.

6) There are advanced AGIs, and they will cooperate with each other and we are at their mercy.

It seems like you are putting a lot of weight on advanced AGI being either impossible or far enough off that it's not worth thinking about. If that's the case, then yes we should calm down. But if you're wrong...

I don't think that the fact that no one knows anything is comforting. I think it's a sign that we need to be thinking really hard about what's coming up and try to avert the bad scenarios. To do otherwise is to fall prey to the "Safe uncertainty" fallacy.

I need to make a montage of the thousands of hacker news commenters typing "The REAL danger of AI is ..." followed by some mundane issue.

I'm sorry to pick on you, but do people not get that the non-human intelligence has the potential to be such a powerful and dangerous thing that, yes, it is the real danger? If you think it's not going to be powerful, or not dangerous, please say why! Not that current models are not dangerous, but why the trend is toward something other than machine intelligence that can reason about the world better than humans can. Why is this trend of machines getting smarter and smarter going to suddenly stop?

Or if you agree that these machines are going to get smarter than us, how are we going to control them?

It won't, which is why people are also calling for international regulation. It's a really hard problem. If you think AGI is going to be very dangerous, this is a depressing situation to be in.

I'm more interested in hearing how someone who expects that AGI is not going to go badly thinks.

I think it would be nice if humanity continued, is all. And I don't want to have my family suffer through a catastrophic event if it turns out that this is going to go south fast.

Having a bunker is also consistent with expecting that there's a good chance of apocalypse but working to stop it.

Patrickas responded with some examples: https://news.ycombinator.com/item?id=35966307

We've already ceded all authority to an algorithm that no one can turn off. Our political and economic structures are running on their own, and no single human or even group of humans can really stop them if they go off the rails. If it's in humanity's best interest for companies not to dump waste anywhere they want, but individual companies benefit from cheap waste disposal, and they lobby regulators to allow it, that sort of lose-lose situation can go on for a very long time. It might be better if everyone could coordinate so that all companies had to play by the same rules, and we all got a cleaner environment. But it's very hard to break out.

Do I think capitalism has the potential to be as bad as a runaway AI? No. I think that it's useful for illustrating how we could end up in a situation where AI takes over because every single person has incentives to keep it on, even when the outcome of all people keeping it running turns out to be really bad. A multi-polar trap, or "Moloch" problem. It seems likely to end up with individual actors all having incentives to deploy stronger and smarter AI, faster and faster, and not to turn them off even as they start to either do bad things to other people or just the sheer amount of resources dedicated to AI starts to take its toll on earth.

That's assuming we've solved alignment, but that neither we or AGI has solved the coordination problem. If we haven't solved alignment, and AGIs aren't even guaranteed to act in the interest of the human that tries to control them, then we're in worse shape.

Altman used the term "cambrian explosion" referring to startups, but I think it also applies to the new form of life we're inventing. It's not self-replicating yet, but we are surely on-track on making something that will be smart enough to replicate itself.

As a thought experiment, you could imagine a primitive AGI, if given completely free reign, might be able to get to the point where it could bootstrap self-sufficiency -- first hire some humans to build it robots, buy some solar panels, build some factories that can plug into our economy to build factories and more solar panels and GPUs, and get to a point where it is able to survive and grow and reproduce without human help. It would be hard, it would need either a lot of time, or a lot of AI minds working together.

But that's like a human trying to make a sandwich by farming or raising every single ingredient, wheat, pigs, tomatoes, etc, though. A much more effective way is to just make some money and trade for what you need. That depends on AIs being able to own things, or just a human turning over their bank account to an AI, which has already happened and probably will keep happening.

My mind goes to a scenario where AGI starts out doing things for humans, and gradually transitions to just doing things, and at some point we realize "oops", but there was never a point along the way where it was clear that we really had to stop. Which is why I'm so adamant that we should stop now. If we decide that we've figured out the issues and can start again later, we can do that.

I honestly don't question Altman's motivations that much. I think he's blinded a bit by optimism. I also think he's very worried about existential risks, which is a big reason why he's asking for regulation. He's specifically come out and said in his podcast with Lex Friedman that he thinks it's safer to invent AGI now, when we have less computing power, than to wait until we have more computing power and the risk of a fast takeoff is greater, and that's why he's working so hard on AI.

I'm what you'd call a doomer. Ok, so if it is possible for machines to host general intelligence, my question is, what scenario are you imagining where that ends well for people?

Or are you predicting that machines will just never be able to think, or that it'll happen so far off that we'll all be dead anyway?

I mean, just like you can create 1-line python script that claims "I am an AGI" and have that fact be false, you can have ChatGPT tell you it has no explicit world model, while exhibiting behaviors that can only really be explained by it having some sort of model of the world inside it.

Fine-tuning is like a PR agent teaching someone what sorts of things not to mention on TV even though they may be true.

You sound skeptical of regulation. I think being skeptical of regulation may be a heuristic that almost always works. (1)

I won't argue that the incentives for regulators aren't really messed up.

But I argue that AI is different from just about any tech we've ever created because of:

* its potential to destroy vast number of jobs faster than any previous kind of tech. * its ability to destroy those jobs permanently and thoroughly. To not leave any space left where humans can provide more value than machines. * its ability to vastly accelerate the development of many other kinds of technology, good and bad, and destabilize our already unstable society further. * its potential, not that far off, to eventually think, make decisions for itself, and have its own goals which may conflict with humanity. All the existential risk stuff.

(1) https://astralcodexten.substack.com/p/heuristics-that-almost...