Accidental prompt injection against RAG applications

I've advocated at work for a similar strategy using prompt injections and jailbreaks in the dataset, and to abort when those documents are matched. So far no traction. I think overall it is a mistake to build any such system with only positive examples or documents, but I'm a security person, and still learning machine learning.

Is the best match dot product (or other distance metric used) below some level? What the good level is would have to be examined for your particular use case, maybe make a histogram of the best (or several best) dot products from many good searches and pick some quantile.

Ooh that's an interesting idea.

Another way to do this could be to calculate the "average" of all of the embeddings of documents in the index, then try to figure out if the query embeds to a point that's a significant distance from that average. Maybe figure out that distance through experimentation?

It's a custom system that they built themselves - see Twitter thread: https://twitter.com/deepfates/status/1798578490759078263