Hacker News Reader

Comment by nneonneo | Hacker News Reader

I think we’d be better served if we stopped thinking of jailbreaks as “attacks” on LLMs. If a user decides they want to go and subvert the guardrails, they really should be able to. The attacks aren’t really going to go away any time soon, either; there’s always some other clever phrase that the RLHF hasn’t accounted for.

Instead of perennially worrying about the risks that the LLMs will say something bad when specifically prompted to do so, I’d rather the companies pay more attention to the problems with accuracy, with outputting copyrighted material (for their own sakes!), and with things like memorizing and regurgitating private information.

View on HN

LLMs will increasingly become interfaces to every type of system.

This assertion that LLMs should obey whatever instructions as long as they are specifically prompted to do so is bizarre.

Should a bank teller LLM divulge another customers balance because you specifically told it to?

This type of exploit is called social engineering when it happens to humans, and we go to great lengths to train the human to detect and avoid it. Why should we not do the same with an LLM?

The underlying APIs can enforce permissions. I don’t care if LLM prompt engineering causes the bank assistant to say that John Doe has 500K in his bank account. As long as it isn’t real data, there’s far less risk.

This is more or less how it works for human customer phone support, who are also limited by narrow permissions. And as we know, they are also skilled at confabulating in the interest of customer happiness.

perhaps the emphasis from companies in the public press is related to product liability laws, consumer protection and being sued?

AFAIK the actual research community in PhD land many times have an internal culture with extreme concern for "harm" from a personal point of view -- it's not fake. If you read the first research papers for various LLM groups, you can see the concern is spelled out in detail, at length.

There are high numbers of burnout and psychological issues among practitioners in advanced computer-intensive projects.. so something there seems self-reinforcing, too.

> perhaps the emphasis from companies in the public press is related to product liability laws, consumer protection and being sued?

It's an interesting thought, but what if the only companies able to run AI platforms are the ones who are able to shoulder the legal cost of all the liabilities it brings?

Essentially the legal system resources is the moat for smaller companies here.

Just thinking through the liability of tricking an LLM into giving you the personal details of a target. I'm not sure there are many companies out there large enough to shoulder those liabilities?

One woman gets killed by some deranged ex, you can get through that. But now you have notice. You know. So you have to do something to make sure your LLM doesn't get tricked again. Politicians and law enforcement aren't going to sit around and let that sort of thing become commonplace. And when politicians get up and start waving around the "crime wave" banner, things can develop in a direction you don't want them to go.

So I think it's probably wise for these guys to get out in front of the problem of people tricking LLMs into saying things that they are not intended to say. The option if they can't control that, is that the politicians and law enforcement come in and take advantage of the inevitable nightly news cycle to grab more powers and reelections, and that's not good for anyone in our industry.

Yeah, with this kind of stuff it's just a reputation thing. In future systems it's a problem though, if you accidentally train them on on the wrong thing or the system is smart enough to reconstruct the missing data, the model can start giving detailed bioweapon production plans. Including safety instructions so the would-be terrorist doesn't get infected too early.

I am sure you can get detailed bioweapons instructions on 4chan already. Aum Shinrikyo figures out sarin in the 90s

Aun didn’t just go to the library and look up “how to make to sarin.”

They had progressional chemists and Russian military personal indoctrinated into the organization.

Wow! What an absolutely pointless and stupid concern to have, that a terrorist is going to use an an LLM to make a bio weapon.

Does literally no one understand that the only information that is inside of an llm model is information that has been fed into the model? I think more than anything this fake AI boom has really demonstrated how unintelligent most people are.

Well, most terrorists are not very smart, at least on an absolute level. I think even near-future models would be able to help them on a basic operations/not screwing up level. As I said, a problem is the reconstruction of information intentionally omitted. Models can't currently reconstruct even the basic knowledge I have, but it's something they should be able to do based on simple inference rules.

I don’t think it’s a very valid concern, but LLMs can already do a limited level of reasoning to create information not directly loaded into them. Sometimes that reasoning is even correct! The idea here would be that even if you didn’t load that kind of information into them, at a certain level of advancement, they can determine the necessary plans from other information that has been loaded into them.

Well yes sometimes a bullshitted solution may be correct! But the person doing the prompting for a bioweapon will end up with yeast sludge anyway.

>Does literally no one understand that the only information that is inside of an llm model is information that has been fed into the model?

Not literally no one, but Sam Altman recently claimed that we'll be able to ask an LLM to "solve all of physics" soon, so plenty of people who should know better do seem to think they're basically magic.

If "plans" are sufficient to result in effective bioweapon production capacity then you've lost the battle already. Fortunately, they're not.

Reply on news.ycombinator.com