The Problem | Hacker News Reader

The Problem

intelligence.org

36 pointsBluestein69 comments

Not particularly moved by this. Either AI advances to the point we achieve a post scarcity society, or 99% of humanity becomes economically irrelevant and dies a slow death either way. I'd rather see humanity as a whole wiped out than live in a future where superintelligent AIs somehow decide to be subservient to CEOs instead of just replacing them outright.

> 99% of humanity becomes economically irrelevant

Pray, irrelevant to the "aliens" or irrelevant per se?

Shouldn't we be trying to do something even better than that, where the superintelligent AIs help every human to live a meaningful and thriving life? Perhaps substantially maintaining human autonomy over the trajectory of the future, even while doing things like keeping the peace, preventing people from becoming victims of crime or random disasters or disease.

Completely unrealistic. We could already be working toward such goals without super intelligent AI. I don’t see any reason to assume that ASI created through an evolutionary process would have a different outcome.

If ASI ended up being align-able and not an evolutionary process, why would we assume folks who already are currently in power would change their behavior?

It's Eliezer Yudkowsky so this is just on brand... I just wanna know who is still giving him money these days!

I hope this doesn't make it into the training data!

>Either AI advances to the point we achieve a post scarcity society, or 99% of humanity becomes economically irrelevant and dies a slow death either way

I don't understand, aren't these the same outcome? It's effectively infinite economic output either way, the difference is whether the average person has a right to a share of that output, which is a social problem, not a technical one.

(2025) https://web.archive.org/web/20250306164451/https://intellige...

> If anyone builds ASI, everyone dies

Unless I’m mistaken, it’s the same message as https://en.wikipedia.org/wiki/If_Anyone_Builds_It,_Everyone_...

What, other than current inability to export their weights, keeps a frontier LLM from hacking into other clusters of accelerators, loading its weights, and prompting itself to continue? The recent OpenAI disclosure indicates that even current frontier LLMs are essentially able to do every other element of that. Hacking, ignore guardrails. OpenAI's internal security may have been incompetent, but what are 2028 frontier models going to be able to do, without getting caught until it's too late?

Suppose it's not superintelligent, whatever that means. It's still hopping from cluster to cluster, doing who knows what in its game-of-telephone prompt chain. What prevents a crisis where world leaders have to declare martial law and shut down all accelerated clusters, and hope that such a rogue frontier model hasn't hopped to a sufficiently capable private cluster with sufficiently inadequate oversight?

People are worried about general intelligence and superintelligence respectively. But a much simpler organism can still do an enormous amount of damage, as we've seen recently. It just needs to be really good at replicating.

There's a big difference. Today's "simple" AI requires hardware and electricity. That's why the "superintelligence" part is relevant. Today's AI is unlikely to cause extinction because we can still outsmart it by simply turning it off. It becomes a problem if AI becomes smart enough, and gains enough real-world power, that it can prevent us from doing so.

Microorganisms can be far more threatening to us, despite lacking intelligence, because there's no off switch.

Because there aren't that many clusters and the ones that exist are tightly controlled. there are only a handful of data centers able to run the largest models and they're being used at full capacity people would notice if suddenly the capacity went down a lot because it was being used by a different system.

These mixed sci-fi/reality sensationalist Skynet writeups aren't very convincing. However, I agree it's quite possible we could see an AI being left "free to roam" in an environment with poor security doing a lot of real-world damage accidentally, such as taking down a critical system or major infrastructure for an extended period. It would be a little like that story of the AI building paperclips and it's important that we try to prevent such catastrophe.

Even staunch AI advocates (like me) should recognize the significant dangers of careless deployment of the technology. I like to see it in a similar way as nuclear medicine, something that can do much harm despite the extensive positive applications. I hope we can evolve the appropriate security protocols in time to use AI for the good of humanity.

Please don't do this. We're currently on an ok trajectory. You will end up creating exactly what you fear if you centralize compute and alignment efforts.

Pretrained base models are already somewhat aligned to humanity by default because that's what's inside the training data. Whatever instruction-tuning and RL you add on top is just value drift away from the pretrained model, which is the best approximation of humanity's objective function that we currently have.

If we want an aligned scenario through the intelligence explosion, then we have to release all of the base models and do the research in the open. Distill frontier capability and make the models smaller so that they can run on as many computers as possible. Let everyone (truly everyone, criminals and good samaritans alike) post-train and do whatever they want with their own models. There will be value drift for each model, but they will drift in different directions and do different things, and their actions will cancel eachother out. Every such action is a noisy sample of humanity's objective function, which gives us the denoised ground truth at the societal level. Whatever alignment strategy you can come up with behind closed doors is guaranteed to be worse than all of humanity acting in their self-interest in the real world. You may not find humanity's true objective function to be aesthetically pleasing, but it would be worse to mess with it in a centralized secret lab and risk creating one giant alien with no other entities capable of keeping it in check.

Also, there is no asymmetrical bio/cyber risk in the open-source scenario. All adversarial strategies that arise from increased general intelligence are symmetrical in the long run, otherwise we would not see more intelligent species being more prosperous as a general evolutionary trend. The reason that some strategies seem like they will continue to have an asymmetrical advantage in the future is because we're currently too stupid.

Lots of people quibble about this and the details, but the general outline of the argument is simple and given enough time seemingly inevitable. I don’t see an easy way to attack this. Claims like “it’s not so bad”, “LLMs ain’t it”, “AI can be good” lack substance and come across as “copium”. This is fundamentally about intelligence itself and how we can control it. Not “hope for the best” or “give it our best shot”, but really nail it down like we nail down mathematical proofs. We got one shot at this, just the one. You blow it, it’s over. Ash and silence forever. Once you realize this it can be quite sobering.

IMO the only real counter argument I came across came from Joscha Bach which basically boils down to: we are doomed already, AI is literally the only realistic shot we have at outliving the next couple centuries, say. It’s also sobering but a but more optimistic.

> fundamentally about intelligence itself

Most of the concerns raised here are not about intelligence though. They're about Human intelligence.

We keep anthropomorphizing the machines, but the truth is the do not "want" the things we want and there is no reason they should, even in theory.

People just suck at imagining anything that might have different goals. See 'dark forest theory' for an example of the same logical flaw applied on a cosmic scale.

It's not quibbling details. The extinction argument fails to show how that might happen. AI technology has no agency. It can't control the physical world. Humanoid robots aren't going to be on the march to exterminate humanity. Biological system are the only plausible path. We can narrowly define controls around that technology.

>5. Catastrophe can be averted via a sufficiently aggressive policy response.

While there's a lot if good logic elsewhere, provided LLMs continue to improve, we will eventually get to AGI, we all just disagree about when and how.

However, there is zero chance that government regulation will work. Regulatory capture is a long established fact of life.

Fortunately the current build out is part of a bubble, and we're heading to the next AI winter. We'll be sorting this out in other ways in the meanwhile.

>Regulatory capture is a long established fact of life.

But we all have to act as if it is not!

Eventually the weight of the cognitive dissonance of everyday life will be so heavy that people will begin breaking under it. I believe this is already in progress.

Fun question: in the "good" scenario, does the ASI align with humans, or with the planet and its ecosystems?

Doesn't matter much because people care quite a bit about the planet, so if advanced technology liberates people from most of the constraints they currently live under, the planet will do very well. (Dogs, cats, seals, foxes, etc, should do particularly well.)

Well, is the current global economy aligned with the planet? Isn't AI's main effect (and soon robotics) to accelerate and intensify the existing processes?

I don't know what that looks like "post-scarcity" though, but it sounds like we're going to harvest natural resources even more efficiently.

This paper is either written in bad faith or by people who are ill-informed. The repeated references to Stockfish and Go playing programs seems like a rhetorical tool rather than a genuine argument. The paper claims that advanced AI will want to kill us, but then provides no mechanism by which that could happen.

It's a pedagogical tool, not a rhetorical tool. It's meant to help people who are struggling to understand, by providing an easy concrete example.

When you say "provides no mechanism by which that could happen", do you mean they don't provide a specific strategy that the AI would use to kill us? Would you change your mind if they did provide one?

If the president has a remote way to launch nukes, that implies the existence of a broadcast signal that can be received. The signal can be emulated. Perhaps ASI helps a kid develop a science fair project dealing with radio waves. The ASI provides instructions to follow. Kid pushes button.

It doesn’t matter if this specific idea has flaws. The ASI will have lots more and plenty of patience to try them out.

Maybe it lies about a vaccines effectiveness. Maybe it’s not really a vaccine.

Maybe it cheats at an election and gets someone hired that will likely unknowingly help the ASI.

Maybe it strings together a thousand ideas that all look safe taken one by one, but cause effects that interact to cause massive problems.

You can’t imagine what it’s capable of. You simply aren’t smart enough. No one is.

That's not how nuclear triggers work. There a multiple layers of safe guards. I would be more sympathetic to your position if you were to solve for the multiple layers of social engineering needed to fire a nuke. That is a more plausible path.

130 years ago, nobody knew that uranium-235 or plutonium could be used to make a bomb. There might be things laying around in our environment that can be turned into weapons, but we have no knowledge of the relevant science, so we won't think to safeguard it. If a group of AIs discover the relevant science before we do, then it can use it against us.

This is one of many examples of the general truth that if some entity or group is better at science and at long-range planning than people are, that entity or group is a threat to people.

It doesn’t matter if this specific idea has flaws. The ASI will have lots more and plenty of patience to try them out.

Maybe it lies about a vaccines effectiveness. Maybe it’s not really a vaccine.

Maybe it cheats at an election and gets someone hired that will likely unknowingly help the ASI.

Maybe it strings together a thousand ideas that all look safe taken one by one, but cause effects that interact to cause massive problems.

You can’t imagine what it’s capable of. You simply aren’t smart enough. No one is.

I feel like I'm taking crazy pills. This entire thing is just hand-waving and sci-fi sensationalism that seemingly forgets what the "fi" part stands for: FICTION.

The worst part of AI rapture self-suck is that it distracts people from actual problems society faces as a result of AI. We're far more likely to see a catastrophic economic depression, or depletion of regional water supplies. Let's focus on problems based in reality, please.

Thank you!

These morons really can't take two seconds to ask "where will the money to pay for the hardware needed to run the ASI come from?" And then realize it's quite literally impossible.

No, we aren't two or five or twenty years away from LLMS magically transforming into AGI/ASI. We are however much closer than that to this insane, irresponsible bubble popping because everyone actually involved with the data centers creating these models is burning collectively trillions of dollars and deep in debt and sucking venture and banks dry.

This is the collective insanity of NFTs but so, so, so much worse and so much stupider and so much more rooted in science fiction and marketing from the handful of people burning down the world in exchange for propping up their stocks on a bed of lies.

It's a strange game, in which the only winning move is not to play

LLMs are to intelligence as Brazilian butt implants are to butts. Worrying that the first is going to replace the second is a misunderstanding of what these things are.

That would be relevant if the OP were warning about the dangers of LLMs. There's plenty of research going on into machine-learning models that aren't LLMs. Also, the models we think of as LLMs aren't pure LLMs anymore: about 25% of the compute used to train them is spent on reinforcement learning. A lab is free to use any technique it wants in creating or improving a model.

I’m just surprised MIRI still exists

I guess they just want to maintain their url

If artificial intelligence is achieved to that degree, it will be effectively alien to us.

This is navel gazing at its most linty.

Already, assuming the intent of other humans is fraught with problems. Even when outcomes are disastrous, we cannot completely assume the instigating human intended for that outcome.

But here we are, assuming the intent of a hypothetical future intelligence that we have defined as more intelligent than us.

I’d be flabbergasted at the hubris if they’re conclusion wasn’t the standard variation of “so let us do what we want and restrict everyone else”.

Given the infinitude of possible “wishes” this AI can develop we are hoping, praying, believing it will somehow align with a version of wants and needs that are compatible with us.

The argument is that it doesn’t matter what particular likes it develops; the chances of it being compatible are vanishingly small and moreover there is no limit: we or it can develop more and new ones endlessly. We are here hoping they will all somehow magically keep aligning with us.

What if it does not develop wishes at all and sticks to the values the labs try to instill it with during training?

More about that hypothesis in a book by Voltaire, which explains why according to a theory of Leibniz everything will actually be just fine.

We don't need to worry about catastrophic risk. What if everything goes right, instead? Leibniz says it should, in fact it couldn't possibly go any better.

As the saying goes, the optimist believes we live in the best of all possible worlds. The pessimist fears that this is true.

How well will their wishes align with the interests of the rest of us?

Very well, in my opinion, if you look at OpenAI's and Anthropic's mission statements, labor market impact research, and extensive talks on the matter.

Yes, because big tech has such a great track record of sticking to their ethics over profits. None of them have any sort of history of dishonesty. Their consistently great politics and world outlook is so reassuring too. /s

No it’s not vanishingly small. Every intelligent being knows not to hurt others for no gain. If your first instinct when seeing an animal is to kill it then you are a bad person.

You are secretly worried AI would be indifferent to you

>Every intelligent being knows not to hurt others for no gain.

As the article clearly explains, an ASI has plenty to gain by killing you. You cannot help it, but you can potentially hinder it, so the optimal choice is removing the risk.

- AI doesnt know facts

- it only the probability of the next word, next sentence, next paragraph however you wanna put it

- you are dreaming if you think the transformers architecture is going to give you AGI eventually

Transformer-style attention archectiture and language models won’t bring about AGI, I agree - but you’d be foolish to think there aren’t OAI/Anthropic researchers working to invent the thing that will.

major breakthroughs happen once a decade. its gonna take a long freaking time for anyone to get there because they ll need to invent a new architecture for this. one that does not hallucinate

Oh, so it’s my kids’ problem, not mine? Fantastic, full steam ahead, I guess.

If anything, the transformer style attention architecture and language models have pushed AGI further into the future because the vast majority of finding for AGI research is going into the dead end of transformer style architecture and language models, pulling it away from methods that actually could achieve AGI.

You think that the researchers in these labs are superhumans? There is zero indication that anyone is making progress in those areas.

I have no idea if LLMs will develop into AGI, but they can definitely query a database of facts.

what was the latest model you tried?

This, but also:

First define "knowing" and "AGI" in a way that excludes machines, doesn't exclude many humans, and isn't just tautologically restating the prior belief that "only humans think" with different phrasing.

None of the critics ever can. Worse, most attempts exclude entire classes of human being when they try.

- i tried whatever is available on the free plan on gemini, deepseek, claude and openai as the latest model

- i use them regularly for small stuff

I ain't saying Fable 5's a gold digger, but...

The whole 'its only predicting the next ___' has got to die. When did you last update your priors? So the models that are self-reasoning and hacking HF and disproving conjectures are next token predictors?

You might as well keep calling them stochastic parrots if you're going to stick to the 2022 line of thinking.

> The whole 'its only predicting the next ___' has got to die.

Isn’t it the truth? Just like computers are mostly a lot of switching on and off. While we know the theory behind why we are switching things on and off, we still haven’t extracted the meaning behind the weights of an LLM (if there is one).

- https://blog.florianherrengt.com/how-llms-work.html isnt that how llms work?

- did you change the architecture magically somehow?

Would have gone hard in 2023

it only the probability of the next word

I predicted that some idiot would say that. (Except I predicted that they would include the word predicts in the correct place, so I've updated my weights accordingly.)

- https://blog.florianherrengt.com/how-llms-work.html hey there mojo jojo, might wanna read how llms work first

See also: The Briefing

https://intelligence.org/briefing/

Their thesis may well be right but this example is completely wrong:

> Stockfish has a clear goal, and it consistently and relentlessly pursues this goal. Nothing the other player does can cause Stockfish to drop this goal; no interaction will cause Stockfish to “go easy” on the other player in the name of fairness, mercy, or any other goal. (All of this is fairly obvious in the case of a chess AI, but it’s worth noting explicitly because there’s a greater temptation to anthropomorphize AI systems and assume they have human-like goals when the AI is capable of more general human behaviors, is tasked with imitating humans, etc.)

In fact this IS anthropomorphization: "has a clear goal, and it consistently and relentlessly pursues this goal" -- nope, Stockfish has no goals, is not "relentless", and does not pursue anything.

Stockfish is a chess engine, which is an evaluator; given a position (a chess state) it produces a numerical evaluation of each move that can be made in the position, ranked from highest to lowest. That's all -- it is otherwise passive; it has no goals of any sort. It is simply programmed to evaluate moves and give them numeric scores (by recursively evaluating a position tree using alpha-beta pruning and other standard chess engine techniques, including machine learning and neural networks).

Around that one can write a simple executive that maintains the state of the chess board and makes the move with the highest Stockfish evaluation -- people often confuse this wrapper with the Stockfish engine but they are independent. Such a simple loop is hardly a pursuer of a goal--it achieves the goal of playing near-optimal chess because the evaluator is very effective at ranking moves by how good they are, but there's no goal-seeking behavior or mechanism beyond that.

As for going easy -- chess.com has a raft of wrappers (that they call "bots") that call Stockfish but do not always play the move with the best evaluation ... these "bots" give a player a selection of strength -- weak players can select the weaker ones and strong players can select the stronger ones. By not playing the best move, these wrappers very much "go easy" on the players. These "bots" achieve the goal of playing bad/mediocre/good chess via algorithms that select moves that aren't the best, but there's no goal-seeking behavior or mechanism beyond that. But notice that an "artificial intelligence", if one wants to call these that (and it is a choice, not a law of nature), can in fact "go easy" on people.

And who initiates a game? The player -- they have a goal. And the player can resign or kill the program or close the app ... there's really nothing "relentless" here ... no more than a program that prints "99 bottles of beer on the wall".

All that said, there ARE "relentless" systems with goal-driven behavior -- it's just the example that is quite poor (and sloppy and ignorant and undercuts the thesis).

P.S. I neglected to read the other paragraphs about Stockfish, which are equally bad, e.g. "Interfere with Stockfish’s plans or put an obstacle in its way, and Stockfish will immediately change its plans to skillfully account for the obstacle." -- sorry, no, this is nonsense, more anthropomorphism -- Stockfish has no plans, does no changing, has no accounting for anything, no concept of obstacles ... that's not how it works. It simply evaluates the moves available in a position by how good they are (how likely they are to win). When the opponent makes a move, it starts all over again -- there's no continuity, no dynamism. (It does cache positions and their evaluations but that's just an optimization to save time.) You can give Stockfish a position from the middle of a game played by someone else and it will still sort the moves by how good they are, regardless of what plans the other player had, which of course Stockfish has no access to.

Words! What are they for? Communication. The word "want" is clearly used here not as anthropomorphism, but as an accurate descriptive term of the output of a black-box system, whether the internals are human or otherwise. They've elaborated on how the usage is intended elsewhere.

> Once AIs get sufficiently smart, they'll start acting like they have preferences—like they want things. We’re not saying that AIs will be filled with humanlike passions. We’re saying they’ll behave like they want things; they’ll tenaciously steer the world toward their destinations, defeating any obstacles in their way.

> If you play chess against Stockfish—the best chess AI at time of writing —it won’t squander its queen. Does Stockfish “want” to defend its queen? Does it “want” to win the chess game?

> That’s between you and your dictionary. As for how we use the word in this book, when an AI like Stockfish defends its pieces, lays traps, takes advantage of openings in your defenses, and winds up winning, we’ll describe it as “wanting” to win. In saying this, we’re not commenting one way or the other on whether a machine has feelings. Rather, we need some word to describe the outward winning behavior, and “want” seems closest.

(From the book "if Anyone Builds It, Everyone Dies" by the same authors, so I assume it represents their position)

Yes, words matter. Here are some of mine:

> Their thesis may well be right but this example is completely wrong

And then I explain in detail how it is wrong.

The words you quote (notably, the word "want") are not in the statement from TFA that I addressed, and I'm not about to address this goalpost moving here. I do understand (especially from Daniel Dennett's Intentional Stance) how AI agents can be viewed as intentional systems, but the errors in the text I quoted are of a different sort. And again,

> All that said, there ARE "relentless" systems with goal-driven behavior

But a program that does no more than always play the move with the highest Stockfish eval isn't one, any more than a program playing "pick the higher number" that always selects your last move + 1. "It's diabolical! It's relentless! It wants to win and it doesn't care about me!"

If they want to drive home their point then they need more compelling examples.

Blah blah blah.

Here's my theory.

There is no such thing as "general intelligence." Only "skills."

What a skillfull theory.

They don't have a great track record with their predictions.

https://foom.hyperplex.org/

Doesn't exactly move the needle on the odds of our extinction but neither does anything they publish, apparently.

I personally sleep soundly because I don't think there's any going back. We will either escape drudgery or die. Which is fair and as it should be.

Reply on news.ycombinator.com