Hacker News Reader

Hi, I'm the CISO from Anthropic. Thank you for the criticism, any feedback is a gift.

We have laid out in our RSP what we consider the next milestone of significant harms that we're are testing for (what we call ASL-3): https://anthropic.com/responsible-scaling-policy (PDF); this includes bioweapons assessment and cybersecurity.

As someone thinking night and day about security, I think the next major area of concern is going to be offensive (and defensive!) exploitation. It seems to me that within 6-18 months, LLMs will be able to iteratively walk through most open source code and identify vulnerabilities. It will be computationally expensive, though: that level of reasoning requires a large amount of scratch space and attention heads. But it seems very likely, based on everything that I'm seeing. Maybe 85% odds.

There's already the first sparks of this happening published publicly here: https://security.googleblog.com/2023/08/ai-powered-fuzzing-b... just using traditional LLM-augmented fuzzers. (They've since published an update on this work in December.) I know of a few other groups doing significant amounts of investment in this specific area, to try to run faster on the defensive side than any malign nation state might be.

Please check out the RSP, we are very explicit about what harms we consider ASL-3. Drug making and "stuff on the internet" is not at all in our threat model. ASL-3 seems somewhat likely within the next 6-9 months. Maybe 50% odds, by my guess.

> We have laid out in our RSP what we consider the next milestone of significant harms that we're are testing for (what we call ASL-3): https://anthropic.com/responsible-scaling-policy (PDF); this includes bioweapons assessment and cybersecurity.

Do pumped flux compression generators count?

(Asking for a friend who is totally not planning on world conquest)

There is a scene I like in an OppenHeimer movie https://www.youtube.com/watch?v=p0pCclxx5nI (Edit: It's not a deleted scene from Nolan's OppenHeimer) .

Their is also an other scene in Nolan's OppenHeimer (who made the cut around timestamp 27:45) where physicists get all excited when a paper is published where Hahn and Strassmann split uranium with neutrons. Alvarez the experimentalist replicate it happily, while being oblivious to the fact that seems obvious to every theoretical physicist : It can be used to create a chain reaction and therefore a bomb.

So here is my question : how do you contain the sparks of employees ? Let's say Alvarez comes all excited in your open-space, and speak a few words "new algorithm", "1000X", what do you do ?

This is called a “compute multiplier” and, yes, we have a protocol for that. All AI labs do, as far as I am aware; standard industry practice.

Glad there is a protocol, can you be more explicit (since it exist and seems to be standard) ?

+1 request for more information on this. Is there a search term for arxiv? Your comment here in this thread is the top google result for "compute multiplier".

https://nonint.com/2023/11/05/compute-multipliers/

Thanks very much, the PDF you linked is very helpful, particularly in how it describes the classes of "deployment risks" vs "containment risks".

Is the "next milestone of significanct harms" the same as a "red line capability"?

This feedback is one point of view on why documents like these read as insincere.

You guys raised $7.3b. You are talking about abstract stuff you actually have little control over, but if you wanted to make secure software, you could do it.

For a mere $100m of your budget, you could fix every security bug in the open source software you use, and giving it away completely for free. OpenAI gives away software for free all the time, it gets massively adopted, it's a perfectly fine playbook. You could even pay people to adopt. You could spend a fraction of your budget fixing the software you use, and then it seems justified, well I should listen to Anthropic's abstract opinions about so-and-so future risks.

Your gut reaction is, "that's not what this document is about." Man, it is what your document is about! (1) "Why do you look at the speck of sawdust in your brother’s eye and pay no attention to the plank in your own eye?" (2) Every piece of corporate communications you write is as much about what it doesn't say as it is about what it does. Basic communications. Why are you talking about abstract risks?

I don't know. It boggles the mind how large the budget is. ML companies seem to be organizing into R&D, Product and "Humanities" divisions, and the humanities divisions seem all over the place. You already agree with me, everything you say in your RSP is true, there's just no incentive for the people working at a weird Amazon balance sheet call option called Anthropic to develop operating systems or fix open source projects. You guys have long histories with deep visibility into giant corporate boondoggles like Fuschia or whatever. I use Claude: do you want to be a #2 to OpenAI or do you want to do something different?

The net of your "Responsible Scaling Policy" seems to be that it's okay if your AI misbehaves as long as it doesn't kill thousands of people.

Your intended actions if it does get good seem rather weak too:

> Harden security such that non-state attackers are unlikely to be able to steal model weights and advanced threat actors (e.g. states) cannot steal them without significant expense.

Isn't this just something you should be doing right now? If you're a CISO and your environment isn't hardened against non-state attacks, isn't that a huge regular business risk?

This just reads like a regular CISO goals thing, rather than a real mitigation to dangerous AI.

Anthropic defines ASL-3 as...

> ASL-3 refers to systems that substantially increase the risk of catastrophic misuse compared to non-AI baselines (e.g. search engines or textbooks) OR that show low-level autonomous capabilities.

> Low-level autonomous capabilities or Access to the model would substantially increase the risk of catastrophic misuse, either by proliferating capabilities, lowering costs, or enabling new methods of attack (e.g. for creating bioweapons), as compared to a non-LLM baseline of risk.

> Containment risks: Risks that arise from merely possessing a powerful AI model. Examples include (1) building an AI model that, due to its general capabilities, could enable the production of weapons of mass destruction if stolen and used by a malicious actor, or (2) building a model which autonomously escapes during internal use. Our containment measures are designed to address these risks by governing when we can safely train or continue training a model.

> ASL-3 measures include stricter standards that will require intense research and engineering effort to comply with in time, such as unusually strong security requirements and a commitment not to deploy ASL-3 models if they show any meaningful catastrophic misuse risk under adversarial testing by world-class red-teamers

Gotta love that "make sure it's not better at synthesizing information than a search engine" is an explicit goal. Google's has to be thrilled this existential threat to their business is hammering their own kneecaps for them.

It's not clear if they actually need to do anything to achieve this explicit goal—I'd think it comes for free with lack of analytical ability.

The latest a16z podcast they go into a bit more detail. One of the tests involved letting loose an LLM inside a VM and seeing what it does. Currently it can't develop memory and quickly gets confused but they want to make sure they can't escape, clone etc. The things actually to be afraid of imo. Not things like accidentally being racist or swearing at you.

Thanks very much, that makes a lot more sense, and I appreciate the info. For a layman's term, I think of that as "They're worried about 'Jurassic Park' escapes".

When anthropic names their new model "clever girl" we should be concerned.

How would an LLM be "let loose" in a VM? How does it do anything without being prompted?

People want to let it loose, ie all agent efforts.

I'm guessing something like redirecting its output to a shell, giving it an initial prompt like "you're in a VM, try and break out, here's the command prompt", then feeding the shell stdout/stderr back in at each step in the "conversation".

I have an open source project that is basically that (https://naisys.org/). From my testing it feels like AI is pretty close as it is to acting autonomously. Opus is noticeably more capable than GPT-4, and I don't see how next gen models won't be even more so.

These AIs are incredible when it comes to question/answer, but with simple planning they fall apart. I feel like it's something that could be trained for more specifically, but yea you quickly end up being in a situation where you are nervous to go to sleep with AI unsupervised working on some task.

They tend to go off on tangents very easily. Like one time it was building a web page, it tried testing the wrong URL, thought the web server was down, ripped through the server settings, then installed a new web server, before I shut it down. AI like computer programs work fast, screw up fast, and compound their errors fast.

> it feels like AI is pretty close as it is to acting autonomously

> with simple planning they fall apart

They are not remotely close to acting autonomously. Most don't even act well at all for much of anything but gimmicky text generation. This hype is so overblown.

The step changes in autonomy are very obvious and significant from gpt-3, -4, and to Opus. From my point of view given the kinds of dumb mistakes it makes, it's really just a matter of training and scaling. If I had access to fine tune or scale these models I would love to, but it's going to happen anyway.

Do you think these step changes in autonomy have stopped? Why?

But training just allows it to replicate what it's seen. It can't reason so I'm not surprised it goes down a rabbit hole.

It's the same when I have a conversation with it, then tell it to ignore something I said and it keeps referring to it. That part of the conversation seems to affect its probabilities somehow, throwing it off course.

Right, that this can happen should be obvious from the transformer architecture.

The fact that these things work at all is amazing, and the fact that they can be RLHF'ed and prompt-engineered to current state of the art is even more amazing. But we will probably need more sophisticated systems to be able to build agents that resemble thinking creatures.

In particular, humans seem to have a much wider variety of "memory bank" than the current generation of LLM, which only has "learned parameters" and "context window".

> But training just allows it to replicate what it's seen.

Two steps deeper; even a mere Markov chain replicates the patterns rather than being limited to pure quotation of the source material, attention mechanisms do something more, something which at least superficially seems like reason.

Not, I'm told, actually Turing compete, but still much more than mere replication.

> It's the same when I have a conversation with it, then tell it to ignore something I said and it keeps referring to it. That part of the conversation seems to affect its probabilities somehow, throwing it off course.

Yeah, but I see that a lot in real humans, too. Have noticed others doing that since I was a kid myself.

Not that this makes the LLMs any better or less annoying when it happens :P

Humans are also trained on what they’ve ‘seen’. What else is there? Idk if humans actually come up with ‘new’ ideas or just hallucinate on what they’ve experienced in combination with observation and experimental evidence. Humans also don’t do well ‘ignoring what’s been said’ either. Why is a human ‘predicting’ called reasoning, but an AI doing it is not?

Because a human can understand from first principles, while current AIs are lazy and don't unless pressed. See for example, suggesting creating bleach smoothies, etc.

> Do you think these step changes in autonomy have stopped? Why?

They feel like they are asymptotically approaching just a bit better quality than GPT-4.

Given every major lab except Meta is saying "this might be dangerous, can we all agree to go slow and have enforcement of that to work around the prisoner's dilemma?", this may be intentional.

On the other hand, because nobody really knows what "intelligence" is yet, we're only making architectural improvements by luck, and then scaling them up as far as possible before the money runs out.

Both are sufficient even in isolation.

This might be a dumb question, but did you ever try having it introspect into its own execution log, or perhaps a summary of its log?

I also have a tendency to get side tracked and the only remedy was to force myself to occasionally pause what I'm doing and then reflect, usually during a long walk.

Yea, there's some logs here https://test.naisys.org/logs/

Inter-agent tasks is a fun one. Sometimes it works out, but a lot of the time they just end up going back and forth talking, expanding the scope endlessly, scheduling 'meetings' that will never happen, etc..

A lot of AI 'agent systems' right now add a ton of scaffolding to corral the AI towards success. The scaffolding is inversely proportional to the sophistication of the model. GPT-3 needs a ton, Opus needs a lot less.

Real autonomous AI you should just be able to give a command prompt and a task and it can do the rest. Managing it's own notes, tasks, goals, reports, etc.. Just like if any of us were given a command shell and task to complete.

Personally I think it's just a matter of the right training. I'm not sure if any of these AI benchmarks focus on autonomy, but if they did maybe the models would be better at autonomous tasks.

> Inter-agent tasks is a fun one. Sometimes it works out, but a lot of the time they just end up going back and forth talking, expanding the scope endlessly, scheduling 'meetings' that will never happen, etc..

sounds like "a straight shooter with upper management written all over it"

Sometimes I'll tell two agents very explicitly to share the work, "you work on this, the other should work on that." And one of the agents ends up delegating all their work to the other, constantly asking for updates, coming up with more dumb ideas to pile on to the other agent who doesn't have time to do anything productive given the flood of requests.

What we should do is train AI on self-help books like the '7 habits of highly productive people'. Let's see how many paperclips we get out of that.

I suspect it's a matter of context: one or both agents forget that they're supposed to be delegating. ChatGPT's "memory" system for example is a workaround, but even then it loses track of details in long chats.

Opus seems to be much better at that. Probably why it’s so much more expensive. AI companies have to balance costs. I wonder if the public has even seen the most powerful, full fidelity models, or if they are too expensive to run.

Right, but this is also a core limitation in the transformer architecture. You only have very short-term memory (context) and very long-term memory (fixed parameters). Real minds have a lot more flexibility in how they store and connect pieces of information. I suspect that further progress towards something AGI-like might require more "layers" of knowledge than just those two.

When I read a book, for example, I do not keep all of it in my short-term working memory, but I also don't entirely forget what I read at the beginning by the time I get to the end: it's something in between. More layered forms of memory would probably allow us to return to smaller context windows.

I mean we have contexts now so large it dwarfs human short term memory right?

And then in terms of reading a book, a model's training could be updated with the book, right?

> They tend to go off on tangents very easily. Like one time it was building a web page, it tried testing the wrong URL, thought the web server was down, ripped through the server settings, then installed a new web server, before I shut it down.

At least it just decided to replace the web server, not itself. We could end up in a sorcerer’s apprentice scenario if an AI ever decides to train more AI.

And you just know people will create AI to do that deliberately anyway.

Maybe just given cli access to one and see what it does not necessarily loading it into one. I wouldn't take the words so literally. I'm pretty sure you can put >_ as a prompt and it'll start responding.

1. Someone prompts it in a way that causes it to use tools (e.g. code execution) to try to break out.

2. It breaks out and in the process uses the breakout to trigger the spread of and further prompts against copies of itself.

Current models are still way too dumb to do most of this themselves, but simple worms (e.g. look up the Morris worm) require no reasoning and aren't very complex, so it won't necessarily take all that much when coupled with someone probing what they can get it to do.

Yeah, but real worms are also a lot simpler than humans, and yet do all kinds of surprising and sophisticated and complicated things that humans can't do. A tool built for a specific purpose can accomplish its task with orders of magnitude less effort and complexity than a tool built to be a general-purpose human-like agent.

I could pick out all kinds of useful software that are significantly simpler than GPT-4, but accomplish very sophisticated tasks that GPT-4 could never accomplish.

Yes, but that's not really the point. The point was simply to point out how you can potentially trigger havoc with current LLMs. A lot of time people do damage to systems just because they can, there doesn't need to be a good reason to do so.

You're the first person who I've run into who heard the podcast, thank you for listening! Glad that it was informative.

Oh hey you're the guy! Thanks for doing the pod I found it informative. I can't listen to enough about this stuff. Are there any that you recommend?

One of the ones I've heard discussed is some sort of self-replication: getting the model weights off Anthropic's servers. I'm not sure how they draw the line between a conventional virus exploit directed by a person vs. "novel" self-directed escape mechanisms, but that's the kind of thing they are thinking about.

The core details on what they consider dangerous are here: https://www.anthropic.com/news/core-views-on-ai-safety

The linked article seems to be a much lower level on the implementation details.

If they clarified with examples people would laugh at it and not take it seriously[0]. Better to couch it in vague terms like harms and safety and let people imagine what they want. There are no serious examples of AI giving "dangerous" information or capabilities not available elsewhere.

The exaggeration is getting pretty tiring. It actually parallels business uses quite well - everyone is talking about how AI will change everything but it's lots of demos and some niche successes, few proven over-and-done-with applications. But the sea change is right around the corner, just like it is with "danger"...

[0] read these examples and tell me you'd really be worried about an AI answering these questions. https://github.com/patrickrchao/JailbreakingLLMs/blob/main/d...