Hacker News Reader

Comment by phillipcarter | Hacker News Reader

I kind of have two somewhat complementary, perhaps ill-formed thoughts on this:

> The whole point of security attacks is that you have adversarial attackers. You have very smart, motivated people trying to break your systems. And if you’re 99% secure, they’re gonna keep on picking away at it until they find that 1% of attacks that actually gets through to your system.

If you're a high value target then it just seems like LLMs aren't something you should be using, even with various mitigations.

And somewhat related to that, the purpose of the system should be non-destructive/benign if something goes wrong. Like it's embarrassing if someone gets your application to say something horribly racist, but if it leaks sensitive information about users then that's significantly worse.

View on HN

I just published a blog post showing that that is not what is happening. Companies are plugging LLMs into absolutely anything, including defense/threat intelligence/cybersecurity/legal etc. applications: https://kai-greshake.de/posts/in-escalating-order-of-stupidi...

There's a couple of different stages people tend to go through when learning about prompt injection:

A) this would only allow me to break my own stuff, so what's the risk? I just won't break my own stuff.

B) surely that's solveable with prompt engineering.

C) surely that's solveable with reinforcement training, or chaining LLMs, or <insert defense here>.

D) okay, but even so, it's not like people are actually putting LLMs into applications where this matters. Nobody is building anything serious on top of this stuff.

E) okay, but even so, once it's demonstrated that the applications people are deploying are vulnerable, surely then they'd put safeguards in, right? This is a temporary education problem, no one is going to ignore a publicly demonstrated vulnerability in their own product, right?

With a slight modification, this basically applies to just about all security vulns ever :)

Honestly the it seems like they play for wiring up an LLM to something can actually take action is to only give the LLM the same access that the same user querying your API would have.

I’ve been exploring an LLM -> API layer for our app and I’m not worried about prompt Injection because if the user was actually malicious they could just used the interface or the API to do the same thing.

In other words if you treat the LLM like any other frontend then you really should have a problem from a security standpoint. Your would have your iOS application super user access your system, why would you treat an LLM different than any other client.

If you're completely confident that there's no way an attacker might get their text into your user's LLM session then yeah, you have nothing to worry about.

Potential vectors to consider:

- Your app lets users run it against text from other sources - fetched web pages, incoming messages - server logs - which an attacker might be able to influence

- Your users can copy and paste text into your app - and an attacker might be able to trick them into eg copying in a dozen paragraphs of text without first reading it to check for weird hidden prompt instructions

Same as CSRF protections and MacOS random binary from internet running protections.

@charrondev

>I’m not worried about prompt Injection because if the user was actually malicious they could just used the interface or the API to do the same thing.

I think you might have missed that the injected prompt might not come from the end user.

There was an example of someone adding a prompt injection to their LinkedIn profile to override a recruiter's prompt and generate an embarrassing email instead. Not sure if it's fake, but it demonstrates the point either way.

SQL injection enters the chat

I'm a little cautious of comparisons to SQL injection now, because while some of the comparisons are very valid (particularly around the risks), prompt injection isn't really the same category of vulnerability as SQL injection -- so mitigation techniques for SQL injection (escaping input, sanitizing) aren't going to work to stop prompt injection.

But otherwise yeah, it can be helpful to think of prompt injection as if someone is effectively doing XSS on your AI agent (again, keeping in mind that the mitigation techniques are not the same, it's an entirely different method of attack). People tend to think of the jailbreaking examples or getting the agent to swear -- which can be embarassing but also mostly harmless. The reality is that prompt injection is basically arbitrary reprogramming of the agent, and arbitrary insertion of new tasks, and data poisoning/replacement, and data exfiltration, etc...

Yeah, the confusion between jailbreaking and prompt injection is definitely a big problem.

People who are frustrated at the safety measure that jailbreaking aims to defeat often assume prompt injection is equally "harmless" - they fail to understands that the consequences can be a lot more severe to anyone who is trying to build their own software on top of LLMs.

I was referring specifically to the timeline and how there was a sarcastic expectation that they would fix it at a certain stage

Yes, but most companies aren’t allowing unfettered access to promoting, either.

My insider risk — a developer who attempts to extract training data, a LLM being leaked of internal data, or an employee who wants to break the prompt for competitive gain — is a lot different of a threat than allowing all of my customers a tool to query their data using LLM’s.

I mean, people were surprised at Snapchat’s “AI” knowing their location and then gaslighting them. [0]

These experiences are being rushed out the door for FOMO, frenzy, or market pressure without thinking through the way people feel and what they expect and how they model the underlying system. People are being contacted for quotes and papers that were generated by ChatGPT. [1]

This is a communication failure above all else. Even for us, there’s little to no documentation.

[0] https://twitter.com/weirddalle/status/1649908805788893185

[1] https://twitter.com/katecrawford/status/1643323086450700288

I don't think SnapChat's LLM has access to your location. I think a service that it uses has access to your location and it can't get it directly but it can ask for "restaurants nearby".

Here’s the full Snapchat MyAI prompt. The location is inserted into the system message. Look at the top right. [0] [1]

Snapchat asks for the location permission through native APIs or obviously geolocates the user via IP. Either way, it’s fascinating that: people don’t expect it to know their location; don’t expect it to lie; the model goes against its own rules and ”forgets” and “gaslights.”

[0] https://www.reddit.com/r/OpenAI/comments/130tn2t/snapchats_m...

[1] https://twitter.com/somewheresy/status/1631696951413465088

Proven wrong thanks. But there is no reason for it to have access and doing it the way I suggested they already were is superior :)

Yeah, non-destructive undo feels to me like a critically important feature for anything built on top of LLMs. That's the main reason I spent time on this sqlite-history project a few weeks ago: https://simonwillison.net/2023/Apr/15/sqlite-history/

Have you looked at Dolt? It seems similar but I'm not sure how it relates.

Yeah, Dolt is very neat. I'm pretty much all-in on SQLite at the moment though, so I'm going to try and figure this pattern out in that first.

With the sheer amount of affordable storage available to even individuals at retail, it's crazy how much database-integrated software doesn't have sufficient measures to undo changes. Every company I've worked at has had at least one issue where a bug or a (really idiotic) migration has really messed shit up and was a a pain to fix. Databases should almost never actually delete records, all transactions should be recorded, all migrations should be reversible and tested, and all data should be backed up at least nightly. Amazing how companies pulling in millions often won't do more than backup every week or so and say three hail Marys.

And then gdpr fucks that up that nice clean concept completely

GDPR only affects data you shouldn't have or keep in the first place.

No, it really doesn't. E.g. deletion of data about a contract or account that just expired (or expired <mandatory-retention-period months ago>) is data you were totally fine/required to have, but can't be deletion that can be rolled back long-term.

Section 17 (Right to be forgotten) 1a and 1b both refer to situations where there was a legitimate need to process and/or keep a subjects data.

https://gdpr-info.eu/art-17-gdpr/

Implementing this as a rollback-able delete will not be compliant.

If it’s so hard to be a good steward of data, don’t collect it in the first place.

It’s not that GDPR is overly onerous to implement. It’s simply that GDPR is fundamentally incompatible with unlimited undo.

Seems to me that it shouldn't be too difficult to extend an "unlimited undo" system to have specific time-based limits for some data.

Blockchains do not delete records, etc. Also, you have to pay pretty much real money to put your record there. So we can have a good aproximation of what can happen with your proposal even if you need to pay extra for it.

[1] https://ycharts.com/indicators/ethereum_chain_full_sync_data...

Ethereum [1] routinely accumulates around 300G per year and routinely hits over one terabyte of data to sync. Remember, this is the size of the data to sync/transmit, not size of the data that is stored, which we may safely assume to be several times more, because of indices, etc.

Also, your proposal makes two tier database system: one that maintains current consistent view of the state and another for log purposes. The logging system needs high throughput storage with key range read request, which makes it, well, another pretty much fully fledged database (SELECT...GROUP BY...ORDER BY is needed).

The reason nobody does what you described because it is really prohibitive in storage space aspect and really is quite complex - a database on top of another database.

People rightfully see these LLMs as a piece of discrete technology with bugs to fix.

But even if they’re that, they behave a whole lot more like some employee who will spill the beans given the right socially engineered attack. You can train and guard in lots of ways but it’s never “fixed.”

I think the idea is perhaps today you shouldn’t be, but there’s intense interest in the possible capabilities of LLM in all systems high or low value. Hence the desire to figure out how to harden their behaviors.

It also seems that most bigger startups that are calling the OpenAI API will now need to invest tonnes in security.

> If you're a high value target then it just seems like LLMs aren't something you should be using

If you're a high value target then it just seems like ____ aren't something you should be using

I remember when people were deciding if it was worth it to give Internet access to their internal network/users

That’s when people already had their networks and were connecting them to the internet

Eventually, people started building their networks from the Internet

Reply on news.ycombinator.com