Hacker News Reader

Comment by glimshe | Hacker News Reader

The problem is that even if the majority of labs decide to slow down, there will be one lab, somewhere, with dark money moving forward. The military wouldn't let that chance pass by.

View on HN

I think they will all conveniently "decide" to slow down because models are becoming crazy expensive, both per token and how much tokens they need to do anything meaningful. The difference between Sol and Astra is 150% pricing increase per token. The next one could be, I don't know, Orion, at $150 and it won't do anything meaningfully different than Astra. It's quite safe to say that they have ingested all human knowledge and because models are inherently just statistical generators, they are not expanding it, just rehashing what we already know anyway.

And what framework would there be for international cooperation? Without that element, this would just be self-destruction in a global race.

Maybe we're about to test the scifi trope that a common enemy - like a superior invading alien life form - is the only thing that could unite all of humanity.

> there will be one lab, somewhere, with dark money moving forward

If this is really an existential threat, that's better than fifty of them.

No it's not. It's worse. Do you have any idea how much power that one lab would have?

How is worse?

If they manage to leap ahead they can effectively kill all competition, present and future and you'll be at their mercy.

So the solution is to make the company that can’t avoid sub-AGI models from going rouge to try RSI?

Also, AI performance has proven to be a function of power and compute, and right now most of the compute lies between two companies rushing towards RSI not because they are afraid someone is going to do this before but because they are the ones who wants to build it.

If the position is that one of these models will go out of control and destroy humanity then why would having two such models be safer than one?

Given that the arrival of this stuff has been compared to nuclear weapons already the question is whether or not the same mechanisms that kept us from annihilating the planet would hold there too.

I am not sure that it will.

But a balance duopoly tends to be better than a monopoly and if there are multiple poles then so much the better.

I'm past the point where I believe that any of these a-holes is in it 'for the good of humanity', their behavior so far can be extrapolated safely to what they will be like when they are in the control of what effectively would be the largest private arsenal ever created.

For a while I thought the AI doomers were off their meds and too paranoid but I'm coming around to the fact that they may have a point: we are rapidly approaching a point of no return, if we are not already past it and the people in charge are exactly the wrong people to be in charge at such a point in time.

How specifically would a balanced duopoly be safer?

> If the position is that one of these models will go out of control and destroy humanity then why would having two such models be safer than one?

Models don't go out of control, they generate totally harmless tokens. The people directing the models go out of control. The owners of the single powerful model are seriously tempted to go rouge and they're going to do it, sooner or later.

Two or more competing groups force the development of defensive capabilities which require strong models too. In addition, offensive capabilities in possession of different competing groups introduce the option of MAD (Mutually Assured Destruction) which is a form of defense that doesn't require defensive capabilities. Worked with nukes.

They will if it would be against international treaties to do otherwise, mutually enforced by every other world power.

We don’t even need something that powerful, you can control the hardware manufacturers and distributors. If you control GPUs, HBM, and the hardware needed to cool all that down and provide the power, that’s enough to make it very hard to scale up the risk.

You cannot really hide an AI datacenter consuming GW of power for too long.

Except no one country controls all that. And if you put limits on it, other countries will speed up doing it themselves.

If you stop going "nobody will do anything actually effective", and start going "how do we do something actually successful", any number of successful ideas start readily arising. It is possible to stop. Not easy, but possible.

Like a war or something? I can't think of a successful non-bad way for everyone to stop besides a treaty.

Outside of the current nonsense in Iran, we've never had to go to war to prevent nuclear arms. We've never had to go to war to ban weapons in space. These things are possible. Hard but possible

That's why treaties sound reasonable to me. Though I'd imagine they would be more like Climate Change ones. Before Iran there were bombings to stop nuclear arms. Imagine a country getting bombed because it's detected their building a data center.

> we've never had to go to war to prevent nuclear arms. We've never had to go to war to ban weapons in space

We've mostly been blindsided by covert nuclear programmes. As for weapons in space, they don't have any purpose. The moment they make sense we'll see weapons in space.

US + South Korea + Taiwan gives a control over pretty much all the hardware used for AI training and inference. Of course having a psychopath president like Trump doesn’t really help at anything that requires international collaboration

> international treaties to do otherwise, mutually enforced by every other world power

This does not exist.

Look at how we regulate nuclear weaponry. Consider how much more widespread it would be if we hadn't.

Nuclear weaponry is effectively regulated because the nuclear powers have agreed that nobody else should be allowed to have the capability.

So this is basically suggesting that USA and China should work together to ensure that they are the only ones allowed to develop and maintain capable AI. And between the two of them they can carve up and techno-colonize the rest of the world.

We've also agreed, for instance, not to do certain types of testing anymore. Just as we could agree to stop training even larger frontier models.

Except...

(From WP)

India

In 1998, India said it would only sign the treaty if the United States presented a schedule for eliminating its nuclear stockpile, a condition the United States rejected.[18]

Israel

In 2016, Israeli Prime Minister Benjamin Netanyahu said that its ratification was dependent upon "the regional context and the appropriate timing".[19]

United States

The United States has signed the CTBT, but not ratified it; there is ongoing debate whether to ratify the CTBT.

https://en.wikipedia.org/wiki/List_of_parties_to_the_Compreh...

> Nuclear weaponry is effectively regulated because the nuclear powers have agreed that nobody else should be allowed to have the capability

Nuclear powers also gave security guarantees that e.g. convinced Ukraine to give up its Soviet stockpile. We're almost certainly entering a new regime of proliferation due to the breakdown of that system.

They made a movie about that.

Between:

   "Iron Sky"
   "Whoops Apocalypse" 
   "Wag The Dog"

and

   "Idiocracy"

You'd think we were amply warned.

Reply on news.ycombinator.com