Hacker News Reader

Why not sanitize and denormalize (or whatever it's called) the text before feeding it to the spam filter? Or the LLM prompt?

View on HN

Throwing it as-is into a simple bayesian filter works great too - nobody honest writes fun&zwsp;ding.

This is what I always tell people. If something is obviously sketchy, don't bother trying to make it safe. Just discard it. No reason to play the attacker's game. (Obviously you need to know what inputs are sketchy vs just uncommon, like people with a single quote in their last name)

Do you have any evidence they didn't?

My suspicion is that the spam filter programmers didn't do a comprehensive evaluation of every code point on every plane of Unicode because... well that's a massive job. So your "sanitize and denormalize" tasks are actually massive mappings which were likely imperfectly created.

I'm basing my comment on the article. Since they are susceptible to ASCII smuggling, they can't have been doing unicode normalization.

The task isn't that massive. Python's unicodedata (for example) contains all the info already. Including the normalization (not denormalization as I wrote before) function.

Is there something I'm missing?

Reply on news.ycombinator.com