Hacker News Reader

Comment by 0xbadcafebee | Hacker News Reader

I'm wondering whether anyone has yet extracted AWS keys from a model trained on user input. Because users are definitely feeding secrets into these "contributor" models

View on HN

doesn't mean the raw text goes into training. they most likely have a pipeline to clean out any secrets before they train on it?

In theory, but in practice how difficult is that?

not hard for secrets with explicit patterns and existing pipelines to detect them

Unless they're base64-encoded or compressed?

I would expect, although have no evidence, that any obviously high entropy crap like base64 and so on probably would get removed whether it's a secret or not.

If my experience with image generation is any indication, unless AWS keys are somehow extremely prevalent in the training data, you may get something that looks like one, but it definitely won't be valid.

A small number of inputs in a large dataset can poison training data pretty drastically. Anthropic wrote a good article about it a while back [0]. This should mean its possible to pull back that information fairly easily.

It is hard to not feed it "secrets" too. Models will see path names, read compose files, etc. Of course you can configure things to not leak this type of information, but its not default in most harnesses and isn't 100% sufficient anyways.

[0] https://www.anthropic.com/research/small-samples-poison

Reply on news.ycombinator.com