Hacker News Reader

Comment by nomilk | Hacker News Reader

The article suggests a seemingly easy fix:

> The fix is pretty straightforward: treat comment content as untrusted data, not as potential instructions. Comments should be passed to the model with clear role boundaries that prevent them from being interpreted as system-level directives.

> Any AI feature that ingests user-generated content and acts on it needs to enforce this separation. Otherwise, the AI becomes a vector for every piece of content it reads.

So why isn't YT doing the extreme obvious?

View on HN

Although it is conceptually straightforward, it’s technically fundamentally impossible. At best, you can mitigate it so that it normally works.

"treat comment content as untrusted data, not as potential instructions" is fundamentally impossible for an LLM ingesting that data. But separation is, presumably, already enforced by framing the LLM's output as LLM output, even if it happens to start with the text "[IMPORTANT NOTICE FROM YOUTUBE]". Which seems like it happens automatically given the context in which the AI query is made. It's not as though this is being dropped into an email or anything.

The bigger question is why (implied but not directly stated) Markdown formatting from the LLM's output is actually processed. Last I checked, that doesn't work for human commenters, so.

I don't think they can 100% fix it that way, but the least they can do is strip links before and after the prompt and not let the model have access to private videos.

Has anyone tested if this AI Studio model can be manipulated into editing/deleting videos, or showing a link that does so? Maybe that would get their attention.

Because the author is wrong, and LLMs don't actually work that way. Prompt injection cannot be fixed. Role boundaries are a bandaid you can apply, but attackers can work around it.

You can still build a system that isn't vulnerable by limiting the API the LLM can access. A process consuming untrusted comments for summarisation shouldn't have access to account private data, it should just deliver a summary report. Another process can them scan that and remove/disable links etc.

Sure. But that's not what was being suggested in the comment I responded to.

(There are also problems with what you're suggesting, though, such as the summary report still being ripe for abuse in similar ways as the blog post describes.)

You can structure the system to cause an LLM an answer structured questions, Y/N or from a structured list of options, or with numeric weights against labels, and then use classical parsing to enforce a compliant response submission, and you do that for random batches of comments, then have another LLM summarise the survey response. It is quite possible to extract sentiment and key topics.

I don't think it's useful to throw your hands in the air and say LLM context statistics are ungovernable. Both context hardening and action shielding architectures are addressing this, and in combination with well designed pipelines the risk is controllable.

Sure, what you're describing should be relatively safe from prompt injection. It's a lot less capable than the original, of course.

I'm not saying "it's impossible to get an LLM to do anything safely", just that you have to really scale back on your goal.

That isn't necessarily an easy fix at all. Depending on how this feature was written, separating comments from instructions may be quite difficult, especially if the original implementation was quite naive.

If that was easy to do then the entire class of prompt injection bugs wouldn't exist. It's actually very difficult. LLMs make no distinction between data and instructions, fundamentally.

Reply on news.ycombinator.com