Remind HN: Unicode hacks | Hacker News Reader

Remind HN: Unicode hacks

37 pointsolalonde19 comments

Just a friendly reminder that some Unicode characters[1] look like spaces and should be taken into account when writing filtering/trimming functions. Of course it's not a big deal but something to keep in mind to prevent stuff like usernames who are basically a bunch of spaces.

[1] http://www.cs.tut.fi/~jkorpela/chars/spaces.html

View on HN

This is a classic web security problem; most famously, WinAPI systems have a "flattening" function that would convert things like PRIME U+2032 into ASCII 0x27 (the tick that terminates SQL statements). Database engines can also interpret character sets differently than the rest of the app stack, leading to similar problems. UTF-7 cursed Wordpress for something like a year in which multiple preauth SQL injection flaws were discovered.

The answer to these problems is whitelist filtering and neutralization; if a character isn't known-safe, substitute its HTML entity alternative. If you're writing blacklist filters that need to know what spaces are, you're already playing to lose.

I just want to drop a thank-you for your dedication to good security practice and steady generosity with advice - in what is likely an intimidating topic for many developers (at least it is for me).

Sorry, whitelisting isn't the answer to SQL injection - bind parameters are.

With bind parameters you can pass data out of band, and the DB engine never tries to parse it as SQL.

You're right, of course, that you should never use user-input data as code. SQL bind parameters solve this problem for SQL. I'm sure there exist other things without such an easy solution, in which case you do need to sanitize with a whitelist.

I wasn't trying to be prescriptive about SQL injection. But, it always skeeves me out when people knee-jerk out "parameterized queries" as the answer to SQL injection. Yes, they're better and safer and you should use them wherever you can. But they don't "solve" SQL injection; for instance, there are query fragments that can't be parameterized (which is why you still find SQL injection in sortable table headers and in pagination and in custom query builders).

Be careful.

Hm, how does this work? Wouldn't the WinAPI convert the characters before I do the security parsing (and I agree with the bind parameters comment anyway)? Or is the problem that you run the app on a Linux server and the DB on a Windows server?

In any case: don't use Windows on a server :-)

Seems like Twitter is "vulnerable" to U+00A0 tweets: http://twitter.com/#!/olivierll/status/7852651047817216

For those who are wondering, you can type Unicode codes directly from your keyboard (Ubuntu: Ctrl-Shift-u, other OS: http://en.wikipedia.org/wiki/Unicode_input)

On OS X this particular character is alt+space.

A more innocent trick with unicode and twitter is squeezing extra characters in a tweet by using unicode ligatures:

http://en.wikipedia.org/wiki/List_of_precomposed_Latin_chara...

Unfortunately the amount of ligatures is small but it might come in handy.

http://twitter.com/#!/citricsquid/status/7864128295145472

Same with 0173, although mine seems to produce nothing, whereas yours is a line break (I think?)

Interesting - just tested it in python and everything is removed with str.strip(), except "\ufeff", which also has zero width.

    >>> print("\ufeff#")
    #
    >>> print(len("\ufeff#".strip()))
    2

For more details on the potential visual spoofs: http://unicode.org/reports/tr36/#visual_spoofing

‏There are also other unicode hacks like changing ‏ text direction (U+200F)‏.

It used to have funny effects on websites (browser name in title bar spelled backwards), but it doesn't seem to work now. The above comment contains the unicode character three times.

Seems ALT+0173 works here as a "blank" character. I'm not sure of its exact purpose, but I've never seen it dealt with and often use it as "nothing". The only solution I've seen to properly sanitising Unicode characters is just to disable them entirely and print their name.

If you break my “typographer’s quotes” by overzealously sanitizing when you don’t absolutely need to, you’ll end up 6' under a † if you know what I mean. ;-)

It's called a "Soft Hyphen"

http://www.cs.tut.fi/~jkorpela/shy.html

While it makes sense to make usernames consist of only alphanumeric characters (something people are used to - thanks to email), I don't see why you'd need to "sanitize" Unicode characters.

┌( ಠ_ಠ)┘ How dare you take my unicode from me.

Reply on news.ycombinator.com