- Dev builds secure AI app - App defends against indirect prompt injection in data from the internet - Dev reviews the flagged log - Log affected by the injection is rendered, and the attacker who wrote the injection in the web data exfiltrates the data from the AI app user
The OSINT data seems to be the most likely source of the poisoned content. I guess you could bury that in a social media profile?
If an attacker tries a prompt injection they would be unable to see the response of the LLM. In order to complete an attack they need to find an alternate way to have information sent back to them. For example if the LLM had access to a tool to send an SMS message the prompt injection could say to message the attacker, or maybe it has a tool to post on X which an attacker could then see. In this blog post the way information gets back to the attacker is by having someone load a URL by by viewing the openai log viewer.
At least that is my initial reading from this.
It's the same as if your devs accidentally sent PII to Datadog - sure, Datadog could add some kind of filter to try to block it from being recorded, but it's not their fault that your devs or application sent them data. Same situation here: bad info is being sent to OpenAI, and OpenAI's otherwise benign log viewer is rendering markdown which could load an external image that has the bad data in it's URL.
In that same situation, you'd expect Datadog to just not automatically render Markdown, but you wouldn't blame them for accepting PII that your developers willingly sent to them. Same for OpenAI, they could clean up the log console feature a bit to tighten things up but it's ultimately up to the developers to not feed secrets to a 3rd party.
their log viewer renders the markdown and their browser will make a request containing the sensitive data to the attackers domain where it can be logged and viewed