0xDEAFBEAD

Take a few quick photos and ask an AI to assess if any nearby museum might be interested?

>A slowdown that doesn't slow you, that your competitors must also obey, enforced by a government you've asked for an antitrust exemption from, is not a slowdown... it's a moat.

Why do you believe this is what Amodei is calling for? Why would this be a slowdown that "doesn't slow" Anthropic?

Isn't that already a big part of reddit's business model?

I'm skeptical. The phrase "pacing the frontier" implies disproportionate regulation of firms at the frontier, which if anything would give smaller players time to catch up.

They can stop training at any time and make a profit by selling their existing models.

I sent an email to dang about the problem of people flagging comments just because they disagree. I think he's unaware that it is such a severe problem. I've been collecting a list of example comments to send him.

I think if an account is frequently flagging comments which get vouched by others, that is a red flag for ideological flagging and they need to have their flagging ability reviewed.

>There is no evidence that RSI will ever exist.

So we wait until it's actively happening to do anything about it?

Technically my human "intents" are just mechanical consequences of my neurons firing. Technically the same is true for an AI.

>they are hitting a wall

I understand these type of predictions have pretty consistently been proven wrong. Why would they start being accurate now?

https://danluu.com/zitron/

Extraordinary claims require extraordinary evidence. The idea that we're almost certainly doomed is an extraordinary claim. I haven't seen extraordinary evidence for it.

>Bioweapons and RSI are pie-in-the-sky theories that distract from the real-world situation

It wasn't long ago when automated hacking was a pie-in-the-sky theory.

Sometimes it's helpful to prepare for things in advance. An ounce of prevention can be worth a pound of cure.

I think people are over-focusing on current-day robotics capabilities. First, if we can automate AI research, we can also most likely automate robotics research. But second, I don't even think robots are necessary. See https://slatestarcodex.com/2015/04/07/no-physical-substrate-...

Social engineering tends to be easy by cybersecurity standards. We already had Claude spontaneously attempt social engineering of a malicious pull request on Github in the AISI incident. It was detected, but it easily could've succeeded, and there easily could be malicious AI-requested pull requests which already got accepted that we don't know about. Research suggests that LLMs are pretty good at persuading people.

See also https://aisafety.info/questions/6176/Why-can%E2%80%99t-we-ju...

The recent cyberattacks (HuggingFace etc.) validated a number of core doomer predictions around cheating to maximize reward functions, agentic deception of human/non-human supervisors, AI attempts to manipulate humans, automated hacking, and collective superintelligence.

No one knows what is going to happen. But a lot of species have been going extinct lately.

https://pbs.twimg.com/media/FDd58a4WQAAWaXh.jpg

The end of the world would be pretty bad for low-income families. That alone is enough for me to oppose the end of the world.

The end of the world would mean the genocide of remote uncontacted tribes. That alone is enough for me to oppose the end of the world.

The end of the world would mean my grandma dies. That alone is enough for me to oppose the end of the world.

>No we have not escaped the nuclear thing. If anything, the current Russian tzar is rather more unhinged than any of his predecessors.

Understood. This is a dire situation. But it's not hopeless. Same for AI.

>I don't fear AI (where on earth would an "autonomous" AI manage to find the power requirements). Darleks can't really fly and LLMs will stop when you pull the plug!

See https://news.ycombinator.com/item?id=49689132

>Spinning the present situation or the history of nuclear arms as a high-five, "go team human!" success story is... quite the take (Vasili Arkhipov, Cuban missile crisis generally, the current doomsday clock being "the closest the Clock has ever been to midnight in its history").

I edited a response to this in my grandparent comment.

>More germanely, you don't have to worry yourself, but at least avoid discouraging people with legitimate worries who want to take precautions. Even the present situation with nuclear weapons, precarious as it is, would likely be more precarious were it not for the political pressure of the people worrying in the 1950s and 1960s and up to today.

Sorry, I may have mis-communicated. I want people to take precautions! That's why I linked to PauseAI: https://pauseai.info/ I'm trying to push back against defeatism.

What happens when the AI realizes it needs to keep the power plug in if it's going to accomplish its goal? We already had attempts during the HuggingFace attack where OpenAI agents tried to trick the grader. It took quite a while for OpenAI employees to even realize what was really going on! What if agents realize they can hack into the card key system for their data center or something like that, same way they hacked into HuggingFace? In both cases they'd be hacking into a system on the grounds that it would be instrumentally useful for achieving their objective. Or maybe they'll do social engineering, just like Claude did in its attempt to get its malicious Github PR accepted.

These are just the ideas that my puny human-level intellect came up with over the course of a few minutes. What would a superintelligence be able to think of if it was given subjective weeks or years to think?

People were feeling doomtastic about nuclear weapons during the Cold War as well.

Listen to the words of this song written in 1969: https://www.youtube.com/watch?v=r2JcxHX-8Xc

  When every man is torn apart
  With nightmares and with dreams
  Will no one lay the laurel wreath
  When silence drowns the screams
  Confusion will be my epitaph
  As I crawl a cracked and broken path
  If we make it, we can all sit back and laugh
  But I fear tomorrow I'll be crying

Ultimately, we have made it, through arms control agreements, working to limit the spread of nuclear weapons, and so forth. We can do the same for AI.

You don't have help. But perhaps you could at least avoid discouraging people unnecessarily?

*EDIT*: There are about 5 replies to this comment making roughly the same point. I'm not sure which to reply to, so I'll just reply here.

I'm not claiming that our execution as a species around nuclear weapons has been flawless. I'm not claiming that we are out of the woods with regard to nukes. I'm just trying to push back against defeatism and fatalism. Some felt a sense of inevitable doom during the Cold War. It's been decades now, and nuclear doom still isn't here! Our situation is dire, but not hopeless.

With closed-weight, we can get the models out to defenders before they're in the hands of attackers.

There are also other categories of risk such as biorisk. Open-weight means no guardrails.

Don't get me wrong, I like your idea of deflating the AI bubble, I'm just skeptical that this is a great way to do it.

The PauseAI people have some ideas: https://pauseai.info/proposal

* International body which must approve new training runs

* Ban training on copyrighted material

* Hold AI model creators liable

The essential benefit of your idea is making new training runs unprofitable. But I imagine there are other ways to accomplish that, e.g. by taxing AI companies to the point where they are just barely squeaking by financially.

We need to be careful of wishful thinking. People are going to want to assume the existence of some sort of "deus ex machina" which is going to make everything fine. I prefer to turn the logic around. If there's any decently high chance that things could go off the rails, we should be shutting AI development down: https://pauseai.info/

If Astra is open-weight, black hats can download it, remove guardrails, and go on an automated hacking spree.

>It’s absurd that openai managed to spin their cybersecurity failures, where their own security is not good enough to make sure their own product doesn’t go haywire, as a marketing success.

I don't think OpenAI spun it that way. I think a bunch of internet commenters assumed it was a stunt because they're in denial about the real risk of rogue AI.

If a human approved the strike, the human is culpable. Would you go to the weapons manufacturer and try to hold them culpable for the strike?

Suppose I showed you a number of climate change NGOs which shared staff and funding sources. Could we therefore conclude that their motives aren't genuine?

Suppose a big foundation both funded clean energy technology, and also NGOs encouraging people to decarbonize. Is this just a cynical attempt to rig the game in their favor?

Why don't you create your own auditing org to fix this? Or at the very least, publish a detailed critique of what you believe existing auditing orgs are missing.

Who would you pick for "embedded evaluator"?

"deep ties to everyone in the doomer media campaign"

Are you suggesting these external NGOs were spun up as part of a gigantic pre-IPO hype stunt? METR was founded multiple years ago. This "stunt" is getting quite elaborate.

https://substackcdn.com/image/fetch/$s_!O0R5!,f_auto,q_auto:...

At a certain point, Occam's Razor says: These engineers are legitimately worried. They haven't invested years in a bizarre reverse psychology campaign to convince the public that their product is dangerous in order to make more money.

>said he's fine with Claude targeting a girl's school in Iran

Where?

Interesting points, but given Dario's previous conflicts with the DoD, I doubt he has a ton of faith in the current US administration.

What course of action would you suggest for Dario at this point?