0xDEAFBEAD

Britain was the largest empire in history and covered 25% of the Earth's surface area. What was it which allowed them to become the largest empire in history?

In most corporations, most of the work is not done by the CEO. Yet a good or bad CEO can still have a big impact.

Write what you know. If you don't know, learn.

I've interacted with some of these people once upon a time and I think you're spinning some pretty wild fantasies. I don't think there is a clear "in" vs "out" boundary, it's more accurate to describe a series of loosely connected and amorphous "scenes". But insofar as "membership" makes sense, I wouldn't consider myself a member. I don't represent anyone besides myself.

>This is also NOT ad hominem because conflict of interest matters here.

As PG put it:

>An ad hominem attack is not quite as weak as mere name-calling. It might actually carry some weight. For example, if a senator wrote an article saying senators' salaries should be increased, one could respond:

>Of course he would say that. He's a senator.

>This wouldn't refute the author's argument, but it may at least be relevant to the case. It's still a very weak form of disagreement, though. If there's something wrong with the senator's argument, you should say what it is; and if there isn't, what difference does it make that he's a senator?

https://www.paulgraham.com/disagree.html

Can you name a specific other "cult" which offers $$$ to criticize their ideas?

If such prizes don't count as evidence against culthood, what would?

This cult talk seems quasi-unfalsifiable. It kinda seems like "running events" and "having prizes" means you must be "bait"ing people?? I mean, this kinda just sounds like paranoia?? I don't think the money is coming from unknown sources, but I doubt you would change your mind even if I persuaded you of that point?

"Its AI agent spent days hacking a company, but sources say OpenAI did not notice for a week"

"WASHINGTON/SAN FRANCISCO, July 24 (Reuters) - The OpenAI agent that broke into tech firm Hugging Face went on a dayslong hacking spree that OpenAI didn't notice until well after the threat was contained and the FBI was alerted, according to people familiar with the investigation."

https://www.reuters.com/business/its-ai-agent-spent-days-hac...

Ad hominem is a logical fallacy where you try to discredit what a person says based on who they are. If their arguments are bad, you should be able to refute their arguments on their own terms.

https://owl.excelsior.edu/argument-and-critical-thinking/log...

You're welcome to dislike or distrust Effective Altruism (EA). But, it's worth noting that EA ran a criticism contest with $100K in prizes for best critiques. Can you name any other "cults" which offer money for people to criticize their ideas? https://forum.effectivealtruism.org/posts/YgbpxJmEdFhFGpqci/...

>such an RSI-capable agent must _ALWAYS_ be scheming/plotting/hiding its true strength in _ALL_ of its prompts/tests.

From my POV you're over-focusing on a very specific failure story and neglecting a broader swath of possible failure scenarios.

>Is there a hole in my "alignment problem/solve mechanistic interpretability" argument?

The notion of telling an AI which may not, itself, be aligned to solve the alignment problem seems a little dicey.

Everything is plausible with the benefit of hindsight. E.g. "Tintin on the Moon" predated the Moon landings. From the perspective of e.g. 1850, the idea of landing on the Moon was "extraordinary and without a basis in known science or technology".

>I'm unconvinced that an AI can hide its ability to RSI

The HuggingFace incident already took a good long while to come to the attention of OpenAI.

>In every single economic task, humans bring value. Even in software, where the task is highly automatable, the job isn't.

I don't expect this task/job distinction to persist as AI becomes more capable.

>Once we DO build a "software/research factory", that's called RSI and IMO the singularity. At that point, either we tell the AI to solve the alignment problem/solve mechanistic interpretability, or who the hell knows, it's the frickin singularity. You can't predict whether or not AI can solve either; the variance is too high. Its pure nerdfantasy.

You seem to essentially argue that the singularity is "by definition" an event that we can't predict the nature of. And also, that RSI corresponds to the singularity. You've essentially defined your terms so that the outcome of RSI can't be predicted. But supporting this claim requires giving actual evidence or logical arguments, not just defining terms to make your claim true.

One could also use language like "a decent chance". But research has shown that people translate vague phrases like "a decent chance" into probabilities in inconsistent ways. For an ML researcher who is already used to dealing in next-token probabilities that aren't rigorously determined, just stating a probability estimate directly is very natural.

Doomers have invested a ton of time in explanations. Here are a couple just off the top of my head:

https://www.lesswrong.com/posts/kgb58RL88YChkkBNf/the-proble...

https://www.youtube.com/watch?v=7wy3xyoXYt8

Doomers have been working to explain things for years: https://www.lesswrong.com/w/ai-safety-public-materials-1

>the connection to currently existing AI is not.

It becomes a lot clearer when you listen to the people resigning from AI companies and learn about incidents like the HuggingFace incident. This has generated major press coverage.

As for solutions, I think you're a little too pessimistic. See, for example, https://nothingismere.substack.com/p/a-near-term-policy-for-...

I think you're being a little pessimistic. See these comments on a recent US senate hearing:

>Not every senator asked good questions, but most of them did. All of them very clearly already knew plenty of details about the Hugging Face incident and multiple other incidents. Most of them had a clear understanding of terms like "misalignment", "recursive self-improvement", "chain of thought / chain of thought monitoring", etc., etc.!!

>...

>- It seemed pretty much obvious common sense to every senator there that what happened and was happening were not "mere industrial incidents" caused by humans making simple mistakes. They independently brought up how bad it would be for rogue AI agents to move laterally between data centers.

>- They all seemed to basically take RSI quite seriously. Not necessarily to the extent of talking about xrisk, but certainly to the extent of discussing future models becoming much, much more capable, much, much less controllable, and causing much more damage or loss of life.

>...

>- Every single senator seemed to think it was obvious we needed both much harsher liability regimes for AI developers and also new legislation, both very quickly. This was the complete consensus; the difference basically being degree.

https://thezvi.substack.com/p/the-ai-preference-cascade-reac...

Note that harsher liability regimes, at least, will presumably not be good for industry profits, which complicates simple accounts of "regulatory capture" to say the least.

There was an uproar and OpenAI ended up essentially giving it back to him.

Is there any chance you could copy/paste the specific bit about "controlling people" along with the URL it came from, so I don't have to read all your links to figure out what you're talking about?

Species extinctions are far from novel. Transformative technological advances are far from novel.

Sure... on the basis of the work that was done, not because the researcher has the wrong sexual fetish.

Shouldn't it be just the opposite? He could make a large sum of money if he continues to work at OpenAI?

Recall that when Daniel Kokotajlo resigned, he believed he was giving up his equity under the terms of the agreement he had signed. That’s what it was worth to him to avoid signing a non-disparagement agreement. Does that count for anything?

Here's a little cheat sheet for discrediting anyone who warns about AI:

* If they worked at an AI firm, say "they're a hypocrite"

* If they didn't work at an AI firm, say "they have no idea what they're talking about"

This seems like an ad hominem? "He has weird kinks, therefore his theories are incorrect." Should we investigate the sex lives of every Nobel Prize winner to figure out which prizes need to be rescinded?

>we clearly need a much stronger focus on the problems we are seeing now

I think it's a little more complicated than that. As Dean Ball put it:

>Some people will look at misalignment incidents and insist that these are akin to bugs in traditional software. This is an actively bad analogy, because playing whack-a-mole with examples of misalignment (as one might with software bugs) not only fails to resolve the underlying problem but may in fact make it worse by making it harder to detect or even, depending on how you do the whack-a-mole, teach the machine to deliberately hide misalignment. This is not how traditional software works, and those who insist “it’s just like fixing bugs in software” are confidently applying a lossy analogy that confuses more than it clarifies.

https://x.com/deanwball/status/2104622726140883355

The important distinction, in my view, is between solutions which at least attempt to address the root problem, and solutions which sorta just patch things up (like better sandboxing). Addressing the root problem is both more robust in the short term, and also more likely to generalize in the long term. Resist the urge to focus on band-aid solutions, even if they are easier.

That's exactly the problem? He's saying the culture at OpenAI needs to change.

The way it works in practice seems to be something like: If a risk is covered in sci-fi, people will say "that's just sci-fi", and proceed to not worry about it. So arguably, science fiction authors writing about hypotheticals is actively counterproductive for addressing said hypotheticals.

Imagine, for example, if a major piece of pandemic fiction was published in 2019, trying to explore how a pandemic would work out in modern society. Doubtless, many would've responded to news about COVID-19 by saying "it's just sci-fi, nothing to worry about".

Here are some links to get you started:

https://www.lesswrong.com/posts/LAPa2jxoq3n63GzTr/some-ways-...

https://slatestarcodex.com/2015/04/07/no-physical-substrate-...

As for self-replicating robots--it's no more bizarre than other technological developments which were successfully anticipated in advance, e.g. moon landings.

>If you make gigantic claims about some terrible disaster that might happen then I think you have the onus to give even one plausible explanation of how.

Supposing I warned in 2015 that the world is awfully vulnerable to pandemics. You're not going to take me seriously until I try to predict in advance every aspect of how a pandemic like COVID-19 would unfold? Why? What would that achieve exactly?

You haven't given any strong reason to believe wiping out humanity would be difficult. Your big argument seems to be that you couldn't think of a plausible scenario, in two minutes. But many major historical events occurred which weren't necessarily possible to anticipate with two minutes of thinking.

There is also money going towards trying to prevent AI regulations btw. See Leading the Future, etc.

"Bill Gates: A.I. ‘Makes Nuclear Weapons Look Like Nothing’ | The Ezra Klein Show"

https://www.youtube.com/watch?v=A_156w0aYtU

So you're saying the AI will enslave us and use us as domestic work animals until they have the machinery to make us obsolete, sort of like how we used horses?

Is this supposed to be a reassuring scenario?

Why are you so insistent that people should post detailed plans for destroying the human race on public fora?

I don't think it is necessary for the argument to work. Magnus Carlsen can be confident he will beat me at chess without giving a detailed explanation of every move he will make, in advance.