Bad characters: imperceptible Natural Language Processing attacks [pdf]

I think they should write their next paper about adversarial attacks on ‘eval()’.

Models aren’t designed to understand Unicode. It’s the tokenizers job to chop that up and feed it to the model properly. What this person found is a step missing from the preprocessing pipelines of these models, or, more likely, something that should be part of the steps that come way before this data gets anywhere near an ML model. So, I think saying this is an adversarial attack on the machine learning model it’s self is a bit disingenuous, because for the input they were given, they did pretty well.

A similar example to this for a computer vision model would be something like inserting some kind of malformed payload in the headers of an image. The data is already bad long before it gets to the model.

I was kind of excited reading the headline, but this seems mostly to be a pretty straight forward application of existing homoglyph attacks that have been common for decades at this point (albeit maybe not common against these types of programs previously).

I thought it was going to be adding imperceptible changes that allow you to choose the interpretation (sort of like the visual adversarial examples) not just substitute a homoglyph and the binary classifier no longer works.

Don't get me wrong, this is still important work, i just got a bit too excited.

It seems like most of these imperceptible changes could be addressed by something like ascii folding (https://www.elastic.co/guide/en/elasticsearch/reference/curr...) but this might not apply for non-english use cases.

If you're interested in adversarial NLP, I also recommend reading this blog post on adversarial attacks on GPT2 with universal triggers (e.g. adding "nobody" as prefix for all inputs causes all entailments to be predicted as contradiction).

Which browser add-ons and bookmarklets exist that help me conveniently obfuscate messages so as to avoid automated surveillance?