Hacker News Reader

Comment by azalemeth | Hacker News Reader

The are quite a lot of "gotchas" with passwords being Unicode though. One of my favourite tricks is to include a null byte in passwords and see what happens. Sudo treats it as the end of the string. Passwd gives "token manipulation error" and nopes out. Many websites seem to fail in interesting ways...

View on HN

I've run into a service where having a password longer than 15 characters _truncates_ it but the password change dialog does not so it's stuck in limbo.

The PayPal forgot password form had this bug just a couple years ago. I changed my password to a generated one. And it was too long. But it didn’t tell me. Just silently truncated.

I used the password reset to change it. This time I used a pretty short password I could type (to rule out a weird copy-paste bug or something). Logged in, went to the change password option and THAT page informed me there was a character limit.

Password character limits are important.

They reveal that the back-end service probably doesn't hash the passwords, which is a good time to GTFO.

I would consider setting a (high) limit for a password a good practice. No one wants an attacker to run a megabyte of password data through your slow password hash. It should be a limit that normal users never reach, but something like 100 chars seems entirely reasonable.

Some web frameworks have such limits built-in. Django for example has a limit of 4096 characters for password input, which should be more than enough for anyone, including people using long random-generated passwords from password managers

The reasonable thing to do is to prehash the user password on the client with a Unicode normalization and a SHA-512 pass.

This way you can have strong input validation server side but also allow almost arbitrary inputs client side.

PS: you likely could also salt the client side hashing and use bcrypt, but bcrypt has a quite short maximum length and I am not sure if it would provide significantly better security here.

The typical password managers allow you to generate up to ~120 characters, I would go for 500 just to be on the safe side.

My bank requires a password of exactly 8 digits. In 2023.

I know of banks which, in 2023, requires exactly 8 digits, the first four being numbers, the other four later letters. No digit can be repeated. Also no consecutive numbers.

It's like they're deliberately trying to reduce the pool of valid inputs.

Not only this, but they also have a sadistic trait against their users. My password on another site (a bank too, IIRC) requires the password not to have my email username and consecutive, or repeated letters. My email username is "u" (as is u@example.com) and I usually generate a 50 or 70 characters long password. There is usually a "u" (they check the case, too). And come characters are repeated in the whole password.

So I reduce the length, significantly, sometimes to 8 characters.

The people who make rules in security in some areas are complete idiots.

A bank I used until about 2018 (no idea if they've fixed this yet - I left) had an exactly 6 character password, and when you used telephone banking it just needed the 6 digits that corresponded to that word. Those 6 numbers also worked online, so at best they were turning all passwords into numbers before hashing them, ensuring there are less than 900000 different possible passwords, which was trivially easy to brute force in 2015, nevermind today.

Bruteforcing should not work as the attempts are either throttled, or lock the account. Provided that they are in place, otherwise the account is wide open.

This i what happens with the 4 digits of a CC PIN and the 3 attempts before the card switches into PUK mode.

One site I went to lowercased my password before storing it unencrypted in the DB, which I only discovered after failing to log in and asking for a password reset mail, so it helpfully emailed me my clear text lowercased password.

I worked at a place that did this, case-insensitive passwords and all. The worst part? They had unencrypted production database backups on all of their dev laptops, the majority of which left the premises after every night. I couldn't get a figure of how many laptops were lost but statistically speaking there had to have been at least 3 lost laptops with unencrypted production database backups per year using industry statistics.

Eventually I convinced leadership to invest in basic security after conservative but still embarrassingly high 6-to-7-figure estimates of annual loss expectancy that only took a measly 5 figures a year to eliminate 75% of the risk, but the company only went around to it a long while after I left the place.

I have former employers who still didn't fix the security bugs I uncovered years after.

I don't know what makes a manager turn off snooze on open PRs for fixing blatant holes.

But if you've got that skill, it can take you far!

I think the lesson is "stop asking for permission and just do want is right". It's rare that a manager needs to approve a PR - would be a culture smell for me in an org.

Incredible. Please tell us who did this, not to shame them, bit to help us!

It was an online store I bought something from quite a few years back. I would hope most stores are using more modern back ends nowadays that doesn't allow this to happen.

I ran into a service where the password entry input enforced a shorter length than the change password.

If I pasted my password it failed, but if I auto-filled from bitwarden it succeeded. Took me weeks to figure out why...

This was, at least a few years ago, the case with Blizzard's Battle.net passwords. I picked a nice secure passphrase, but had to cut it off mid-word because of this truncation. (I think the behavior was for it simply to take the first 16 characters of whatever you typed in the password field, so typing the full passphrase didn't fail, but I wasn't confident that that would remain true in all cases, so I made a habit of only typing the first 16 chars.)

I believe the British Airways site still does something similar to this, or it did until very recently if it's been fixed. Add a password that is too long, it gets truncated and the truncated version saved with no warning or error.

It took me several password reset attempts to realise what was going on, as my literally just set and password manager-saved password didn't work.

I think in BA's case the password input also had a character limit on it, which is ultimately how I realised what was happening, even though there was no info anywhere that such a limit existed.

This is still the case with PeopleSoft (32 chars limitation, and truncation)

This has nothing to do with Unicode (assuming that by "Unicode" you mean "UTF-8"). The null byte has the exact same meaning in UTF-8 as ASCII: https://stackoverflow.com/questions/6907297/can-utf-8-contai...

> The null byte has the exact same meaning in UTF-8 as ASCII: https://stackoverflow.com/questions/6907297/can-utf-8-contai...

And what meaning is that? That link doesn't refer to an authority on the meaning of the nul byte.

In C, strings are nul terminated. Thus when at some point your password is passed along a tool written in C (or any language with the same behaviour) it’ll be truncated at that position.

If by Unicode you mean UCS-2 or UTF-16 though it applies.

A bigger deal happens with ambiguities concerning composed versus decomposed characters. Linux and Windows tend to prefer writing composed characters, whereas Mac OS will prefer writing decomposed characters. The letter é will have two distinct byte sequences to represent the same character (more precisely, the same grapheme).

Shouldn't normalization functions handle that?

Yes, now see if any random service actually uses said functions...

If Unicode weren't broken it wouldn't need any "normalization functions".

And no, I don't have anything better to propose at the moment. Unicode is a mess, but currently without realistic alternatives.

Still I would like to live long enough to see that we finally manage to create some proper text format without all kinds of crazy gotchas and deficiencies.

Care to elaborate?

Are you suggesting that a better version of Unicode does not allow combining characters? [1]

Some languages have an explosive amount of characters if you regard each possible modifier in combination with each base character, some of which aren't practically used. By allowing the composition of modifiers and base letters, all the historically used ones are available, and all the odd ones are technically expressible.

I'm not sure how you encode all the world's languages, live and dead ones, without a few gotchas and deficiencies.

What would an always-normalized Unicode look like, if not either having really, really many characters, or having a non-trivial syntax?

[1]: https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf#G306...

One solution to allow combination characters, but only in a normalised way is to only allow one of the same modifiers, and only allow modifiers in the normal order. This makes parsing Unicode quite complex. If you allow for non-normalised Unicode and write a post-processor that normalises, you get to status quo.

So either you are suggesting to disallow non-normalised Unicode with the current definition, or something different altogether. I can't imagine what that alternative looks like.

(Also, I'm a Unicode fanboy. Sorry for the intensity.)

I think normalization and compound glyphs are orthogonal concepts.

The point would be to have only one canonical binary representation. A representation that is also free of all kinds of "compromises" which are only there for legacy reasons…

Additionally the whole madness should be resolved that Unicode mixes content, representation, and layout (especially as it fails miserably at all of them).

Than there are the problems that Unicode is actually incapable of representing all kinds of scripts. (Just think for example about stuff written top-to-bottom, and not LTR / RTL). Some "unimportant" things like Math can't be represented in Unicode also…

And I won't even talk about the issue that the committee was taken over by some political movement, namely the woke fraction.

I think it's actually a kind of joke that we still didn't mange to invent some sane text format for computers. Unicode doesn't cut it.

But frankly I admit that we won't get anywhere — as even Unicode is full of madness it works "good enough" by now through even more layers of madness on top, so we won't get rid of it ever again. That ship sailed I fear. Still I can't stop banging my head when I hear about things like issues with normalization, something that shouldn't exist in the first place in a sane world.

I wouldn't call that "being Unicode" though, since null byte is in ASCII.

It depends on the encoding: In UTF-16, a lot of characters have 0x00 as part of their encoding. That's the null byte, even if the software isn't technically supposed to see it that way, so it's a valid test. (Easy, too: Any character that's also in ASCII will be represented with a 0x00 byte in UTF-16.) Of course, UTF-8 solves that problem, plus the problem of surrogate pairs.

> That's the null byte, even if the software isn't technically supposed to see it that way

that's a null byte, even if the spec says it's not the string terminator.

i just think it's worth referring to things by using words that more precise semantics if possible

Reply on news.ycombinator.com