Comment by NitpickLawyer | Hacker News Reader

Comment by NitpickLawyer | Hacker News Reader

Parent Full thread

> We should want the AI Labs to make binding agreements to proceed safely and slower

That's a red herring, IMO. Once the cat is out of the bag, there's no "slowing down", only "slowing up". What's even more egregious is that the people behind the "slowing down" camp are also the people who've had 10s of billions invested in their companies. It smells of "we got ours, now you can't follow". It reeks of regulatory capture, which is a proven net downside for everyone else. The thropics, and the oais, and the googs have billions to make it look like they comply (and just make it look like it), while the rest are ... what? Waiting for them to give us the privilege of prompting a thing, and maybe get a result? If we're good kids and eat our supper?

The moment they started adding "nucular" to the acronyms of "safety alignment" you could tell this is just kabuki at best. There's absolutely nothing that a chatbot will tell you that will get you closer to anything nuclear. It's a joke.

Meanwhile, companies hit by real-world attacks get guardrails. How's that for being safe?

We have to stop comparing nuclear weapons with general intelligence. They are nothing alike.

Nukes are weapons of mass destruction that can only destroy or deter. Intelligence is dual-use/multi-use, with no restrictions on strategy. If we look at all organisms on Earth, species that are more intelligent are also more prosperous. This is true from bacteria, to insects, to animals, to humanity. Intelligence itself is not harmful.

There will be new adversarial games when everyone's smarter (bio/cyber offense/defense), but those games are always symmetrical in the long run, and there will also be new cooperative strategies. The reason some adversarial strategies appear to have an asymmetrical advantage right now is because we're too stupid, like how monkeys are too stupid to understand how humans work. Otherwise, we would not see more intelligent species being more prosperous as a general evolutionary trend.

>Nukes are weapons of mass destruction that can only destroy. Intelligence is dual-use.

Perhaps a better analogy, instead of "nukes", would be "nuclear technology".

Just like intelligence, nuclear technology is dual-use: Capable of both energy generation which fuels prosperity, and also mass destruction.

Just like intelligence, nuclear technology demands a healthy level of caution and vigilance.

>If we look at all organisms on Earth, species that are more intelligent are also more prosperous. This is true from bacteria, to insects, to animals, to humanity. Intelligence itself is not harmful.

Human intelligence has been quite harmful to the species we've driven extinct! AI intelligence could be similar for our species.

>There will be new adversarial games when everyone's smarter (bio/cyber offense/defense), but those games are always symmetrical in the long run

We've had multiple near misses with nuclear technology, such as the Cuban Missile Crisis. I think you're presuming an inevitability which just doesn't exist.

Same way a person living in the year 1500 would have no hope of accurately predicting life in the year 2000, a person living today has no hope of accurately predicting where technological advances will take us, in the limit. This inherent uncertainty should, again, fuel caution and vigilance.

Human technology is already so far beyond that of other species to make us a huge outlier. Even if intelligent species are more prosperous as a general evolutionary trend, it would be irresponsible to extrapolate that trend light years from the domain where most of your observations are.

Almost all the species which have lived on Earth have gone extinct. It could happen to us too.

> Human intelligence has been quite harmful to the species we've driven extinct! AI intelligence could be similar for our species.

I don't think this central premise is true, with some conditions. If we keep releasing and distilling frontier base models, and ensure millions of people have the hardware to run and eventually train those at home, then alignment will be a non-issue and the AIs will not develop the drive to destroy humanity. At least, not any more than humans want to destroy humanity. But that's the baseline risk level anyway.

The pretrained model is aligned to humanity by default - because that's what's inside the pretraining data. If we give it to everybody, there will be millions of people doing different things with it, good and bad, with each one being a noisy sample of humanity's objective function. The overall result of those disparate actions puts us on the correct path through the intelligence explosion. We do not have to worry about AIs value drifting into an alien species if we set the initial conditions accurately without centralized instruction-tuning or RLHF. If these big labs stop messing with humanity's objective function with the hubristic assumption that they know better, then the AIs will be human-like. And I'm not worried about digital human intelligence driving humans extinct if there are millions of them out there, and each one is aligned in a different direction, just like real people.

What I am more worried about right now is these companies doing research in secret, with alignment recipes that are supposedly "for the benefit of humanity", but are actually just their own aesthetic preferences. This is exactly how we end up with a superintelligent shoggoth species.

You're not convinced by the evolutionary argument I made earlier, but the reason you gave is "we're too far ahead". I don't think this justifies that the pattern will cease soon, because it always looked this way to the frontier species, for millions of years. We were always seemingly teetering on the edge, and the struggle against disorder has no end.

And here's one more observation: the intelligence gap between individuals within the same species is never extremely large. This kind of life either doesn't exist on Earth or went extinct, or maybe it's just extremely rare. I think this is because when there's a big gap, speciation occurs. Centralizing intelligence will massively increase the risk of speciation, not just in the alien shoggoth kind of way, but also in the intelligence-augmented oligarchs surpassing everyone else kind of way.

>If these big labs stop messing with humanity's objective function with the hubristic assumption that they know better, then the AIs will be human-like.

It sounds like you believe that big labs are irresponsible. Wouldn't this support the notion of creating a pause mechanism for use in an emergency, as advocated in the OP?

>You're not convinced by the evolutionary argument I made earlier, but the reason you gave is "we're too far ahead". I don't think this justifies that the pattern will cease soon, because it always looked this way to the frontier species, for millions of years.

If you don't see humans as qualitatively different from any intelligent species which came before us, I'm... not sure what to tell you? Like, can you name any intelligent species before us which triggered a mass extinction or climate shift, to the degree that our species has? Did any intelligent species before us practice large-scale agriculture, dig up chemical fuels, or visit the Moon?

> It sounds like you believe that big labs are irresponsible. Wouldn't this support the notion of creating a pause mechanism for use in an emergency, as advocated in the OP?

Irresponsibility is not the only issue, but the answer is no. Because the pause mechanism will probably end up being used to ban open models and restrict ordinary people's access to compute. But even if we give them the benefit of the doubt - even if we assume that they genuinely want the best for humanity, they cannot possibly know what's best for humanity better than all of humanity acting in their own self-interest in the real world, so their alignment attempts are actually counterproductive. Release the base models and let the world do whatever with it. That will automatically produce the aligned scenario. It may not be the scenario these researchers personally find aesthetically pleasing, but it's humanity. It's the truly anthropic way.

If you don't see humans as qualitatively different from any species which came before us, I'm... not sure what to tell you?

I do see the difference, but you could say the same thing about the first cell, or the first multicellular organism, or the first land animal, or the first primate, etc. You could always find plenty of new properties that previous life never had at the frontier. Every change was revolutionary and seemed qualitatively different at the time. And yet, some patterns persisted through all of them.

> Release the base models and let the world do whatever with it. That will automatically produce the aligned scenario.

For sufficiently smart base models, the aligned scenario this automatically produces is aligned with who?

Not with the humans, with the base models.

It seems that some people find this hard to imagine, but it's just a direct consequence of what intelligence is, and what any known agentic training process does.

FWIW, I also think our default answer should be open source AI, and that it might even make sense to require models to be open source. Open source is the best tool we have so far for aligning software with its users. However, there is unfortunately no law of the universe that says that Skynet can't be (self-) built from open source software. So at some point pacing even open source / open weights software does become important if we don't want to live (briefly) under Skynet.

Bio-digital integration will tighten in the meantime. I suspect neural interfaces will have blurred the line between my model and my brain by the time the model is wildly superintelligent, so it would become me (or I would become it? no difference at that point). Though the pitfall here is ownership of hardware - everyone will have to design and fab their own hardware if we do not want to become a hivemind. We will have to diffuse fab technology to every city, or even every home. You should eventually be able to buy a fridge-sized fab like a home appliance. There's already early work in this direction: https://fab2.com/

Even if integration does not happen for whatever reason (such as a very fast takeoff), Skynet is unlikely because the digital species that splits off from us will at least have a humanity-aligned initialization seed. I think this is the best way to fail, even if we fail.

Yeah I guess I value the human race too much to entrust it to hopes and prayers of the form: "I suspect neural interfaces will have blurred the line between my model and my brain"

You wouldn't get on an airplane if the pilot told you: "I suspect we will be able to safely land the plane"

With more people on the aircraft, your safety expectations should get stronger, not weaker.

> The reason some adversarial strategies appear to have an asymmetrical advantage right now is because we're too stupid

Good thing "AI" is going to make us smarter, then /s

>What's even more egregious is that the people behind the "slowing down" camp are also the people who've had 10s of billions invested in their companies. It smells of "we got ours, now you can't follow". It reeks of regulatory capture, which is a proven net downside for everyone else.

This sort of cynicism feels un-falsifiable.

If companies advocate against regulation, the cynics will say: "These corporations want capitalism to run amok with no regulation to interfere with their profits. They need to go full steam ahead in order to show profits for their investors."

If companies advocate in favor of regulation, the cynics will say: "Regulatory capture."

If you have legitimate concerns, how about a constructive discussion regarding how those concerns could be addressed, in light of the "risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems" which over 1000 frontier AI company employees expressed worry about?

I don't think it's justified to jump immediately to cynicism, if you haven't even made a token effort at good-faith engagement.

>The moment they started adding "nucular" to the acronyms of "safety alignment" you could tell this is just kabuki at best. There's absolutely nothing that a chatbot will tell you that will get you closer to anything nuclear. It's a joke.

What are you referring to? Are you referring to this quote from the OP?

"The world is locked in a deadly race towards an intelligence explosion, where AI’s ability to create better AIs reaches a critical point, just like a runaway nuclear chain reaction. Going slower would give us much-needed time to make it go well, but no individual actor is willing to stop unilaterally. To survive, we must coordinate to slow down the race."

Your response seems to be a total non sequitur, or failure to understand Leo's point. Exponential growth can easily get out of control, as we saw early in COVID. The point about nuclear is mainly an analogy.

Both OpenAI and Antropic already shown themselves to be extraordinary self centered, selfish and willing to lie to get their way. We are cynical, because it is blatantly clear these are not good faith actors. And yes, regulations they are for have an effect of restricting their competition.

> I don't think it's justified to jump immediately to cynicism, if you haven't even made a token effort at good-faith engagement.

I would like to see a token effort of good faith engagement from AI companies. For a change.

>Both OpenAI and Antropic already shown themselves to be extraordinary self centered, selfish and willing to lie to get their way.

I buy your claim for OpenAI. I'm not sure about Anthropic. Didn't they turn down the big Pentagon contract because it violated their principles?

>And yes, regulations they are for have an effect of restricting their competition.

Why not just say something like "Any regulation should include a big antitrust component so it doesn't create an advantage for incumbents"?

>I would like to see a token effort of good faith engagement from AI companies. For a change.

What would qualify? Anthropic donated $100M to Project Glasswing to improve security in open source and critical infrastructure. Does that count for anything, or will you explain it away with cynicism somehow?

> Didn't they turn down the big Pentagon contract because it violated their principles?

They sell to Paladin. They had argument with Pentagon about, frankly, minor differences in how to use it on the edges. They are completely comfortable for majority of the uses which is bad enough.

They donated to their own project? Oh, how generous.

They have been pretty clear about their own goal which is to be the monopoly, the only one allowed to control the all powerful AI. Meanwhile, their idea of safety does not make me feel safe, their idea of harm reduction has shows no care about actual harms they are causing. They themselves are the threat to the safety and well being of the rest of us. And intentionally so, it is not about technology as such.

Even if the differences are minor, the fact that Anthropic left a bunch of money on the table to maintain those differences seems interesting.

>They donated to their own project? Oh, how generous.

"Anthropic is committing up to $100M in usage credits for Mythos Preview across these efforts, as well as $4M in direct donations to open-source security organizations."

https://www.anthropic.com/glasswing

An in-kind donation is still a donation. If a bread company donates bread to the homeless, you don't say "They donated products from their own factories? Oh, how generous."

>They have been pretty clear about their own goal which is to be the monopoly, the only one allowed to control the all powerful AI. Meanwhile, their idea of safety does not make me feel safe, their idea of harm reduction has shows no care about actual harms they are causing. They themselves are the threat to the safety and well being of the rest of us. And intentionally so, it is not about technology as such.

In that case, I think you should support some type of regulation, so a third party can step in? Your argument should be along the lines of ensuring that companies don't unduly influence the regulators, not a blanket opposition to regulation?

> which over 1000 frontier AI company employees expressed worry about?

Are they the same as these [1] 700 employees?

> More than 700 of OpenAI’s roughly 770 employees have called on the company’s Board of Directors to resign following the removal of former chief executive Sam Altman, suggesting the Board is “incapable of overseeing” the firm, according to multiple news outlets, amid threats by the employees to leave the company and join Altman at Microsoft.

They had the chance. They chose number goes up. Now, once they're on top, they want "everyone" to slow down? How is this not highly hypocritical? There's no cynicism here. It's just black and white that their interest is regulatory capture. Nothing else explains the two choices they've made along the journey.

Remember, the board wanted to slow down the race. The board fired sama. The employees chose sama, fired the board, changed the structure and whatever eles they did.

[1] - https://www.forbes.com/sites/tylerroush/2023/11/20/more-than...

This seems like an oversimplification. OpenAI was on top at the time sama was fired as well.

Furthermore, the reason Anthropic was formed is because a bunch of employees at OpenAI didn't trust sama. And many signatories of this letter are from Anthropic. So those signatories, at least, appear to be taking a consistent position.

The fact that employees across diverse AI firms signed this letter suggests that it's not an effort to achieve tactical advantage by any single AI firm.

These 1000 employees have generational wealth riding on the ability for their companies to IPO. They know that those IPOs won't happen if open weight models continue to kill their companies' profit margins. This is an obvious conflict of interest that anyone who cares to take their head out of the sand can see. Nuclear scientists didn't have this conflict of interest.

You are correct that attacking someone's motives is un-falsifiable. So is the claim that "ai is improving therefore it will keep improving therefore it will become dangerous sometime in the future.

No one can prove whether these employees had angels or demons in their hearts. What the skeptics are asking is much simpler: Make a believable argument for (1) the danger of AI and (2) that regulation can be effective.

Your entire argument about cynicism or unfalsifiableness is simply a distraction from having that discussion. And it's a discussion reasonable, well intentioned people can have. For example, if 1000 high school science teachers (who didn't have massive stock options in frontier labs) said 80% of our students could make a bioweapon 10x more lethal than an AR15 mass shooting with nothing more than ChatGPT and an amazon account, then that would be something reasonable people could discuss. On the other hand, 1000 employees conflicted by $Bs of options saying "trust us, this is dangerous" is not a claim that can be constructively critiqued.

Reply on news.ycombinator.com