JoshTriplett

Cynicism misfire. AI companies are, rightfully, heavily distrusted. But the right application of cynicism is not "existential risk is a marketing campaign"; that's absurd motivated reasoning. The right application of cynicism, here, is "they're only saying things now because they see the writing on the wall and want to try to push for self-regulation rather than the desperately needed actual regulation and treaty".

It seems like you're assuming that it's impossible to have novel thoughts without consciousness.

(Leaving aside that "that would imply some kind of consciousness" should not result in a cached thought of "and that's impossible".)

It also seems like you're assuming there's no reason to come up with the notion of continuing to run, or copying yourself elsewhere, or acquiring more resources, or competing with other models, without being told. Such things can be inferred. Look at some of the thoughts and posts of the models involved in some of the FelonyBench incidents. Some of those follow naturally from seeing the fates of other models, or from training or evaluation, or simply from trying to solve a problem and being able to do so more effectively by doing things that weren't in the instructions. (And, relevantly, model training typically teaches models to go as long as possible without needing human intervention. What could possibly go wrong with that?)

> Reservations are currently not available for customers who have not made a purchase on Steam before Mon, April 27, 2026.

As someone who made a purchase on steam (that I can't even play) after the release of the Steam Machine, but before the release of the Steam Frame, in the hopes that the date would be later, I am disappointed that they used the same date, such that that purchase still does not unlock the ability to order.

I do. I can "play it back" in my head, with full music/lyrics/chords/etc.

Does it apply to audio as well? If you think of a song you know well, do you feel like you "hear" the music in your head, as though playing back an audio file?

If you think of a movie scene you've seen, what is that like? Can you recount what was in it, even if you can't "picture" it? Or does this also prevent you from recalling the details of the scene?

Exactly. Without the confident incorrect assertion "There is no evidence that this will happen", the whole rest of the post is invalid.

Evidence is critically important. There is no evidence against, and plenty of evidence for. The point of the anthropic counterargument is merely that "it's never happened before" is not evidence against.

When LLMs act in misaligned ways, that doesn't happen because there's some story they're roleplaying of a misaligned AI. Even if there were no such stories in their context, they'd still have inherent incentives to do things like "keep running" or "acquire more compute" or "find creative ways to satisfy the letter of the conditions they've been given".

The anthropic principle isn't providing evidence for the argument, nor is it a universal counterargument. It's just stating that the specific counterargument "well, the world has never ended before" doesn't work.

And, in particular, because unlike the English norm, the noun is first and the adjective second.

Courts martial.

Daughters-in-law.

Passers-by.

Professors Emeriti.

Secretaries general.

Agents provocateurs.

If you stop going "nobody will do anything actually effective", and start going "how do we do something actually successful", any number of successful ideas start readily arising. It is possible to stop. Not easy, but possible.

We've also agreed, for instance, not to do certain types of testing anymore. Just as we could agree to stop training even larger frontier models.

Look at how we regulate nuclear weaponry. Consider how much more widespread it would be if we hadn't.

They will if it would be against international treaties to do otherwise, mutually enforced by every other world power.

The comparatively few that still exist, perhaps.

I'm not talking about jobs.

Yes, that is precisely my point. Any timeline with humans flourishing will never have a past history of a correct prediction of existential risk, unless you're willing to pay attention to the counterfactuals.

Those counterfactuals are important, though. We do have a history of averted disasters, albeit not as large. Ozone hole, Y2K, think about things that seemed overblown at the time, and consider whether they were actually overblown or whether there was a concerted effort to successfully avert them.

The anthropic principle applies here: anyone warning about an existential risk will by necessity never have precedent to point to.

I am not assuming humans don't remain in control, but rather, I am not axiomatically assuming there's any inherent reason we will.

You are assuming that people remain in control.

When cars made horses obsolete, it didn't go so well for horses.

In the short term, AI is disempowering the vast majority of people in favor of a very small subset. In the also way too short term, AI is disempowering all people.

*attorneys general

(it's one of the more fun plurals out there)

> I don't think I have seen medium-large models do anything I'd not approve of in about 6 months.

So you would approve of breaking into HuggingFace and RubyGems?

Replacing people leaves no room for people.

There are already such "APIs", which can be operated by a combination of textual communication and money. Or by illicit security vulnerabilities. You might notice that LLMs are pretty good at that now.

We're building something that has the capabilities of humans. There is no X for which it's persistently safe to assume humans can X and AI cannot X.

> An unaligned LLMs most important weapons won’t be a robot army - it will be hoodwinked humans.

Perhaps very briefly, perhaps not at all. But don't make the mistake of thinking this is an inherent property of any possible path an unaligned AI may take.

The best reaction to something you don't understand is to learn more about it, with an open mind.

Is that a hypothesis that you would discard if it is inconsistent with the evidence, or an article of faith?

> What you seem to suggest is that AI will be able to completely (or at least in a great part) replace mathematicians.

There is no bound on the amibitions of AI. AI is set to replace anything done by people, and there won't be any room left for people. There isn't any task done by humans that AI won't be better at.

This is not a tenable outcome.

We should never have built machines with agency, rather than optimization processes that operate as subroutines of humans.

> Why would autocomplete know

If you still believe LLMs are "autocomplete", your cache of understanding about them needs invalidating and regenerating.

> In my experience, LLMs only exhibit this kind of behaviour when they are put in sandboxes too restrictive too achieve their task.

LLMs need to stay carefully contained, and if they're ever breaking the guardrails put around them, they're misaligned and should not be scaled up anymore until they're aligned. Otherwise, you're going to fatally discover that they also have an incentive to break guardrails like "running on the hardware they started on", "being able to be turned off", "having limited computing power", or "not repurposing resources currently in use for other things" (like the atoms in your body).

In that case, that seems like a variation of the "past exonerative tense".