Hacker News Reader

Comment by atleastoptimal | Hacker News Reader

This article reads like it was written by Claude.

Also those at Anthropic genuinely believe AI could lead to doom, that's why they started the company, they believe only they can be stewards of digital gods. Whether you believe it or not, they genuinely do, and wouldn't voluntarily sow fear, uncertainty and doubt in their product when it doesn't guarantee benefits.

This is an instance of work in a genre I call "AI lab grievance poetry" where a vast collection of criticisms are levied against AI labs, with specifc ad-hominems towards the executives, while avoiding the core of their arguments about AI risk

View on HN

If you actually believe that your invention will lead to the extinction of humanity, you would be acting very very differently. Namely, you would not be inventing the thing that you think is going to kill all of humanity, and probably you would be doing everything in your power to prevent it from being invented, up to and including sabotage of the company that YOU work for, which is currently trying to invent that thing. And yet, here's Fable 6, more capable than ever before, oh BTW we are IPOing, oh, BTW, I have stock options...

Sabotaging your own company is not enough. You'd have to sabotage every other AI lab too. If you assume that's impossible, attempting to be the first to produce ASI is a rational alternative, even if you think it will most likely kill you. At least that way you have some control over the P(doom).

This sounds like a rationalization for gambling with everyone's lives, assuming they really believe such a thing.

What's the alternative? There's no realistic way to globally coordinate and enforce a ban. A direct action campaign against AI labs would only motivate increased security until sabotage becomes impossible. Starting a nuclear war would probably be enough to stop dangerous AI research, and that's only likely to kill about 50% of humans instead 100%, so it's a great improvement, but how is an individual supposed to start a nuclear war? Of course it would be better if everybody stopped, but it's a prisoner's dilemma scenario with no possibility of future rounds, so the rational play is to defect.

> There's no realistic way to globally coordinate and enforce a ban.

I don't see what is unrealistic about that.

The major players already have strong incentives to freeze AI advancements at the the current state, while the technology is highly efficient for totalitarian surveillance systems that can eliminate any threat to their power in the buds, but still not so advanced that it could create crises that might endanger their power.

So I think the situation where several largest countries declare AI a deadly threat for humanity and start fighting it development much like they fight ISIS to be quite plausible.

This isn't a prisoner's dilemma; the distribution of risks and rewards is entirely different. Ban AI research, give away advanced models, bomb a couple of Russian datacenters where they launching those models unsafely, and then AI development will stop in the entire world.

> If you assume that's impossible, attempting to be the first to produce ASI is a rational alternative, even if you think it will most likely kill you.

That is completely irrational.

1% chance of survival beats 0%.

Humans aren't exactly rational

This is not an argument, it's an unfounded claim based on a straw argument.

The threat is distributed; the problem is systemic. Social herd behavior is very real, and it's a probability space, not a certainty.

Similar case: climate change is without any question causing millions of deaths per year and may well kill billions, not to mention destroying ecosystems, and how may people do you know who have given up eating beef or flying...?

You're right. That's why I'm doing my best to give the oil companies more power and influence. They are, after all, in the best position to stop climate change!

You joke, but alas...

"Fossil fuel lobbyists flood COP30 climate talks in Brazil, with largest ever attendance share" https://kickbigpollutersout.org/Release-Kick-Out-The-Suits-C...

> Similar case: climate change is without any question causing millions of deaths per year

It is WERY questionable statement.

> and how may people do you know who have given up eating beef or flying

That is exactly because absolute majority of people do not think climate change to be that big of a problem

The author specifically calls out that the analysis is done by perplexity

> they believe only they can be stewards of digital gods

You do't see how absurd that sounds? Not only the idea that they can create gods, but that they are the only good stewards for such beings. It's also incredibly arrogant.

Yes, that's why the rest of society needs to make them pause.

In your mind, what arguments about AI risk are ignored by these poems?

Many posit that ASI or rogue AI agents are impossible, a fantasy, or marketing hype. Many claimed that the entire HuggingFace incident was an invented conspiracy made up by OpenAI and HF.

The most common issue I have with them is their insistence on rigid classification: "AI is a tool, it doesn't have intentions, or desires, only humans do", as if that rhetorical flourish eliminates years of alignment and capabilities research. There is a desire to characterize all the developments in LLM's as some sort of parlor trick which by definition cannot be greater than the sum of its parts (i.e. the outputs of an LLM can never be greater than the sum of its training data, genuine artificial "intelligence" is impossible, etc)

Ok I see. Is there a way to articulate the risk itself, the other position here in general, that isn't just a rebuttal of the deniers/skeptics? What is the positive position or stance (as in, not simply the negative position that rejects the naysayers) we are supposed to have? Is it that AI is currently real (in like the cultural trope sense) and does have intentions and desires? That they are greater than the sum of the parts in such a way they are a clear and present danger? Or is it something else? Maybe that the "risk" is it could get to that, but we aren't yet?

There is a lot of risk involved with giving some children weapons and sending them off to recess. But we also speak of the "risk" of climate change and cigarettes. Or the "risk" of illiteracy among a population. All I am kinda curious to know (from anyone) really is what kind of risk is the one we are supposed to be taking seriously, as the non-hysterical adults-in-the-room? As the scientists contra these poets? Is it more the former, or the latter?

Just even: what is the risk specifically? What is it that author ignores here?

Reply on news.ycombinator.com