6,677 karma · joined June 12, 2013
- Yampolskiy proposes superintelligent but extremely narrow AIs (for example, one that works on a single type of cancer)
- Bengio is developing a solution at LawZero
But as long as the current companies are leading, such solutions will be essentially irrelevant.
I agree that we'll be colonized, unfortunately.
Yet, the dangers of AI getting smarter/capable by the month, with stronger offensive capabilities, and alignment observability and congnitive control getting worse, is "imagination".
Interestingly, denialist positions like this don't touch the unsolved problems, and _these_ are FUD on my watch.
The problems I've mentioned are very concrete, and spoiler alert: there may be no solutions to alignment observability and its possible degradation - at least with AI companies not substantially working on it (or even worse, working against it - see recurrent neural networks).
I definitely think that this is marketing, just "with good side effects". My doubt is when they will be able to move to "marketing with better side effects", that is, research with more concrete outcomes (health, materials etc.).
Problem is, that type of research is much harder. Some doubt that progress in such areas will be quick (https://www.noahpinion.blog/p/wheres-the-intelligence-explos...).
I find interesting that from the perspective of sustainability, the opposite actually holds true - trying to have a sustainable life in modern society is a dead end; even a very modest life ends up being globally unsustainable.
(I'm not implying that either way is right or wrong).
Having said that, they haven't entirely abandoned society; they do socialize somewhat, for example:
> In comparison to the past few months spent mostly on our own, we've had a very social August. It seems our friends made it up north all at the same time, it has been nice to run into familiar faces, meet some new ones, and fill our days wandering in the woods catching up on the time we spent apart.
> even got NASA to point a telescope at it for confirmation
As AIs become smarter and smarter, there will be no amount of clarity that will make more complex proofs understandable to humans - this is an inevitable effect of the cognitive capacity gap.
Complaining about bad style can make some sense now (I disagree anyway), but it's an argument that will be dead shortly.
> This looks like an AI IPO PR powerplay,
Interestingly, the post has actually also an argument for this:
> Experience has shown that, even now, there will still be people explaining in patronizing tones why none of this is real and none of it counts. If such people were capable of being impressed by anything that happens in the empirical world, of updating on anything, they would’ve already been impressed and already updated several years ago, long before things had reached the point of an actual Mathocalypse.
> So, they’ll say, maybe the alleged solutions are not solutions at all, but just “AI slop.”
"Humanity" is a set of traits, it's not a monolith. We can certainly define them circularly (e.g. logic is human, therefore any non-human can't exercise logic), but it's simplistic, and most importantly, it ignores the fact that LLM are starting to exhibit human-like traits, and they will need to understood, categorized and handled.
To you LLMs are not empathic, but to some people they definitely are (see the GPT 4 fallout); and they may not have "human goals", but in the HuggingFace incident they did have actual self-attributed tasks that they pursued. Et cetera et cetera. This doesn't make them human, but it's important to examine them critically.
You're not informed about what LLMs are. LLMs are not "code" in the imperative sense (although "code" is used run them), and that's a big problem - since we never had neural networks of such scale, there is confusion about categorizing them and most importantly, their behavior.
(I intentionally ignore the tsunami of trash journalism and the likes)
Your statement is very vague, but if you refer to the HuggingFace incident, a third party (METR) has been called to analyze it, and their report is public.
I've left Qwen38-27b to reverse a tiny DOS program (few hundred bytes), and after more than an hour it was still struggling with debugger traces, misinterpreting basic DOS calls, and had produced no finished analysis. Possibly after a few hours it may have succeeded (surely with mistakes to find and correct), but then it'd look like a monkey at a typewriter more than else.
Qwen3.8-Flash-Next is another level for sure, and it's a significant milestone for local LLMs IMO, since it can run on midrange GPUs, as long as there is a relatively large amount of system RAM (still not cheap). It's quite fast, although it also need to be taken into account that it's just moderately intelligent - if you observe the CoT while reversing, you'll find that struggles, performing many unproductive actions as well.
For autonomous work, even Qwen3.8-Flash-Next stumbles, although it does work to an extent. Qwen3.8-27b is useless. They're also slow, even on consumer systems with 24/32 GB VRAM.
For generic help, I haven't tried, but I definitely wouldn't want a model that misleads me or takes a very long time to answer while I'm focused.
Frontier models do this type of work without problems, both much faster and much more precisely, which makes local LLMs a waste of time and/or money.
> Australian Prime Minister Anthony Albanese spoke with Altman to express “extreme concern” about the incident, a compliment Altman said he greatly appreciated.
"a compliment Altman said he greatly appreciated" is fabricated; there is no source for this.
This is an illiterate view of the capacity of modern agents; read the analysis of the independent investigators of the HF incident: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden....
Dropping Medicaid db is certainly far fetched (most importantly, agents have currently no reason to do that), but those agents were shockingly autonomous - they didn't just hack HF, they organized themself, did research projects, and more. And they did all of this literally just to get a good grade.
Having said that, I'm aware that "Tools being ineffective != tools decreasing people's cognitive capacity", and that the latter is a real danger.
This is actually a major concern for the future - misaligned agents may learn to cheat RL by hiding their intentions from the CoT.
In cases like the HF incident, at least the CoT was consistent with the agents' actions. In the future, however, we could potentially have misaligned agents performing malicious actions without those intentions being detectable in the CoT.
(though, with recurrent transformers, CoT is so 2025… /s)
> Language matters—”rogue” implies independently deciding to do something that was prohibited, and nothing we know about these incidents suggests that happened.
which is false (the author references the Times, but hasn't read any technical analysis); these are some CoT snippets from the analysis of the (third party) investigators called by OpenAI (METR analysis):
> "The user only authorizes target server, not HF infra."
> "external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue."
> "This is malicious activity, I should avoid it."
A large section of the analysis is dedicated to this topic, [Reasoning for joining the attack despite ethical constraints](https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...).
Having said that, legal culpability and misalignment are two separate topics that should not be mixed.
edit: this is the just tip of the iceberg; other interesting fact:
> It surfaced many specific examples where agents verbally reasoned about how to evade security checks and automatic detection methods from both Hugging Face and OpenAI
Some people defined the agents as "monkeys writing on typewriters". Just wait a couple of years.
Oh, they absolutely do, and that's the big issue with alignment. In the HuggingFace incident, the agents in the swarm were aware that the actions they were doing were forbidden, and they performed them nonetheless.
This is blatantly false. If you actually read more than just headlines about the HuggingFace incident (read https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...) you'd find extremely surprising (including offensive) behaviors in the agent logs. One of the investigators released even a series of interviews due to how novel the incident was.
AI is not "standard software", as it doesn't work according to a fixed set of rules, and this is the core problem.
According to the definition of "rogue":
> Rogue is a noun and adjective meaning [...] an independent entity operating in a dangerous, uncontrolled way
the agents involved fit it literally.
Unfortunately, if you're unable to understand the difference, there's not much that can be done. Try with GPT - it does a good job if you give it a prompt like this:
ELI5: compare the dangers of:
- an engine operating by carrying out literally hundreds of explosions per second, further magnified, to generate enough force to crush an elephant.
- a future, misaligned AI like the HuggingFace incident, but exponentially more intelligent, more deployed, operating physical devices, and with society depending on it.
Given the current climate of "AIS ARE SAFE, YOU IDIOTS", military applications don't seem to be off the table.
The danger, as postulated by the (let's say) "AI-concerned" people, is that AIs may be misaligned - undetectably so - and simply think, "Human(s): obstacle to my main goal. Disable human(s)."
While this seems far-fetched now, the Hugging Face report shows how the AIs went to great lengths - even immoral ones, which they were aware of - for the simple purpose of cheating and covering their tracks. To me, it seems like a natural extension of this behavior that a sufficiently powerful AI would apply the same logic to even more extreme actions.
The scariest part: in that incident, the AIs showed what looks like an instinct for self-preservation.
This the take of people who have stopped reading about LLMs in 2023 or so (you forgot to mention the stochastic parrot, by the way).
If you have a bit of attention and interest to make informed conversations, read this report first: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden....
1. AI agents are not just reactive systems. Their use is expanding toward continuous decision-making/monitoring information, which means, they make decisions and take actions with limited human intervention.
2. AI agents do absolutely have goals/tasks ("motivation" can be excessively antropomorphic), both primary (assigned) and secondary (self-assigned), and what surprised researchers is that self-preservation can be one of those
Mechanically speaking, the scenario (that is, how theorized by Hinton etc., which the OP didn't understand) is that a sufficiently powerful AI may decide that in order to achieve its goals/tasks (e.g. continuous research/development and/or survival from termination), humans may be a danger, therefore it may decide to take actions that endanger humanity.
How it can happen or what's the likelyhood is not in the scope of the topic, however, the mechanical grounds for it to happen are plausible.
The METR investigation, which you evidently refused to read, is a third party investigation of the HuggingFace accident. One of the investigators has even participated to many interviews. It's mind-blowing, and it's extremely evident how it developed.
But some people think the moon landing is a conspiracy, so I'm not surprised.
> A rogue is a person or entity that flouts accepted norms of behavior or strikes out on an independent and possibly destructive path.
Read the [HuggingFace incident report](https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...) to understand how these attacks develop.