Hacker News Reader

Comment by theptip | Hacker News Reader

A tap on the brakes might make sense right now. The risk with that strategy is that we want to make sure that we don't over-regulate, then get overtaken by another actor that doesn't have safety concerns.

For example, I'm sure China's central planners would love to get an AGI first, and might be willing to take a 10% risk of annihilation for the prize of full spectrum dominance over the US.

I also think that the safety/x-risk cause might not get much public acceptance until actual harm has been observed; if we have an AI Chernobyl, that would bring attention -- though again, perhaps over-reaction. (Indeed perhaps a nuclear panic is the best-case; objectively not many people were harmed in Chernobyl, but the threat was terrifying. So it optimizes the "impact per unit harm".)

Anyway, concretely speaking the project to attach a LLM to actions on the public internet seems like a Very Bad Idea, or perhaps just a Likely To Cause AI Chernobyl idea.

View on HN

I very much doubt LLMs are the path to AGI. We just have more and more advanced "Chinese Rooms." [1]

There are two gigantic risks here. One: that we assume these LLMs can make reasonable decisions because they have the surface appearance of competence. Two: Their wide-spread use so spectacularly amplifies the noise (in the signal-to-noise, true fact to false fact ratio sense) that our societies cease to function correctly, because nobody "knows" anything anymore.

[1] https://en.wikipedia.org/wiki/Chinese_room

The difference between AGI and a more advanced Chinese Room may not be relevant if enough people see the latter as the former. The goalposts have been moved so often now that what is and isn't intelligent behavior is no longer a bright and sharp divide. It is more like a very wide gray area and we're somewhere well into the gray by some definitions with tech people with an AI background claiming that we are still far away from it. This in contrast to similar claims by those very same people several years ago where what we take for granted today would have definitely been classified as proof of AGI.

Personally I think the definition isn't all that relevant, what matters is perception of the current crop of applications by non technical people and the use that those are put to. If enough people perceive it as such and start using it as such then it may technically not be AGI but we're going to have to deal with the consequences as though it is. And those consequences may well be much worse than for an actual AGI!

Well, I think a dividing line might be that if you put a Chinese Room in charge of a justice system, a corporation, or a regulatory agency, it's gonna do a pretty cruddy job of running it.

I don't think that is what will happen. What I do think will happen is that a lot of people in lower level functions will start to rely on these tools to help them in their every day jobs and the lack of oversight will lead to rot from within because the output of these tools will end up embedded in lots of places where it shouldn't be. And because people are not going to own up to using these tools it will be pretty hard to know which bits of 'human' output you can trust and which bits you can not. This is already happening.

Right, that's the point I was making about increased noise in the system.

You'd be able to convince an AGI that is the year it is. The chat bot will be stuck on the story and statistical nearness of related responses.

The chat bot can't verify, because it doesn't Know anything.

> For example, I'm sure China's central planners would love to get an AGI first, and might be willing to take a 10% risk of annihilation for the prize of full spectrum dominance over the US.

This is the main problem - no matter what constraints the US (or EU) puts on itself, authoritarian regimes like Russia and China will definitely not adhere to those constraints. The CCP will attempt to build AGI, and they will use the data of their 1.4 billion citizens in their attempt. The question is not whether they will - it's what we can do about it.

Saying we shouldn't "tap the brakes" on AI out of safety concerns because Russia/China won't is a little like saying we shouldn't build containment buildings around our nuclear reactors, because the Soviet Union doesn't. It's a valid concern, but the solution to existential danger is not more danger.

I think it's more like we shouldn't put a upper limit on the number of nuclear weapons we hold because the Soviet Union/Russia may not adhere to it.

We were able to (my understanding is fairly effectively) negotiate nuclear arms control limits with Russia. The problem with AGI is that there isn't a way to monitor/detect development or utilization.

"The problem with AGI is that there isn't a way to monitor/detect development or utilization."

This is not completely true, although it is definitely much more trivial to "hide" an AI, by e.g. keeping it offline and on-disk only. To some extent you could detect disk programs with virus scanners, encryption or obfuscation make it somewhat easy to bypass. Otherwise, these models do at least currently take a fair amount of hardware to run, anything "thin" is unlikely to be an issue, any large amount of hardware could be monitored (data centers, for example) in real time.

Its obviously not fool-proof and you would need some of the most invasive controls ever created to apply at a national level (installing spyware into all countries e.g.), but you could assume that threats would have these capabilities, and perhaps produce some process more or less demonstrated to be "AI free" for the majority of commercial hardware.

So I would agree it is very, very difficult, and unlikely, but not impossible.

Yes but you’d never be sure. Not sure enough to disarm.

Not so sure your analogy works here. Aren't containment buildings meant to protect the area where the reactors are? I think the closer analogy would be saying the US needed to tap the breaks on the Manhattan Project because nuclear weapons are dangerous even though Nazi Germany and Soviet Russia are going full steam ahead during WW2 or the cold war with their nuclear weapons programs. The world would probably be very different it we had chosen the 'safer' path.

> Saying we shouldn't "tap the brakes" on AI out of safety concerns

I didn't say that we shouldn't tap the brakes, nor is that the only strategy. Other ones include, in rough order of viability: global economic sanctions on hostile actors attempting to develop AGI; espionage/sabotage of other AGI effort (see the Iran centrifuges); developing technologies and policies meant to diminish the impact of a hostile actor having AGI; and military force/invasion of hostile actors to prevent the development of AGI.

I'm sure you can think of others - regardless, there are far more options than just "more AI research" and "less AI research".

Reply on news.ycombinator.com