Hacker News Reader

Comment by grardb | Hacker News Reader

This is remarkable. I always find it interesting when bugs like this occur.

It reminds me of a hackathon I attended where a food ordering startup (I forget the name, but they were chosen to feed us dinner that night) had a similar bug, which baffled me beyond belief. Without going into crazy detail about my password, it typically follows a certain pattern but is never the same across websites. For some reason, the website kept saying my password was invalid. It met all the password requirements that the website asked for (length, capital letter, etc.).

I forget the exact details, but it ended up being the exact location of a capital letter, the location of a number, or some combination of both. I could never figure out how a bug like that could even be coded up. My best guess is that it was some poorly-formed regex.

> Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.

View on HN

"Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems."

Yes. My favorite was a Coyote Point load balancer bug. If the last character of the HTTP options is "m", the connection will not get past the load balancer.[1] I found this because a web crawler was having trouble with one site. Fortunately, I knew someone with their own Coyote Point load balancer, and was able to establish that the connection went into the load balancer and never came out.

The load balancer has a big file of rules which contain regular expressions. Somewhere, I think there's a "\m" where they meant "\n". Reporting this to the vendor, along with a Python program to demonstrate the problem, was of course futile; they suggested "upgrading the software". I demonstrated that the bug existed on their own load balancer on their own site. I finally added a completely useless field to the HTTP header so that the last character was not "m".

[1] https://www.webmasterworld.com/webmaster_hardware/3312997.ht...

Tell them they've violated their contract and you are migrating from them ASAP.

That doesn't work when it's software on your client's computer. Remember when all the web devs revolted and ditched IE6 back in 2001, because MS was just taking the Mickey? Yeah, me neither.

Right. I didn't have a Coyote Point unit. Many other sites did, and they all appeared to be down to my web crawler until I figured out the problem.

(Current web crawler problem: sites that won't let you read their robots.txt file if they don't like your user-agent string.)

>sites that won't let you read their robots.txt file if they don't like your user-agent string

That's hilarious. So do you use borrow a browser's user-agent or do you ignore the robots.txt?

I'd assume they want you to crawl their website. When they say you are ignoring their robots.txt file, tell them you actively prevented you from seeing it and you could only assume that mean they WANTED to be crawled.

That would get them to fix the issue pretty quickly :-)

No `robots.txt` indeed means weapons free, but crawler gleans useful info from it, e.g. which areas of the site are dynamically generated.

More likely the site is trying to serve custom versions of `robots.txt` to different bots, with good intentions, and the code is buggy.

The strict interpretation is that if "robots.txt" returns 403 Forbidden, it's interpreted as "deny all". That's what the Python library does. We list those sites as "Blocked".

> upgrading the software

they told you this because they probably fixed the bug in a newer release. what response were you expecting, exactly? to dive in with a hex editor?

for what it's worth we run into the same issues with our network devices, it's a pain in the ass, and that's why we're shifting to SDN.

Proper response would be something like

"You've obviously put a lot of time in to this. Thanks. We actually had found that bug 4 months ago in R1892, and it's patch in R1899. Please upgrade, or let us know that you're, in fact, using R1899 and still experiencing the issue (it's issue #847 on our bug tracker at foobugz.xyz)"

Or maybe...

"You've obviously put a lot of time in to this. Thanks. We actually had found that bug 4 months ago in R1892, and we're planning on a maintenance release next quarter. If you'd like to test out our upgrade version, please contact jaz@balanceco.com and let them know you want to be in the testing group - he'll get you the appropriate documentation and code. Thanks for helping fix and test these issues!"

right, so in other words, a software upgrade.

In other words, acknowledge that someone did some work, and actually ask if they're using the latest version, and vitally - acknowledge that they read the correspondence and that upgrading will address the issue.

"Update your software" is often a BS answer that can introduce many other issues or breakages in production systems. Without a clean "downgrade" path (which many companies don't provide) you run a risk of introducing more problems.

i think you're misunderstanding me. i'm agreeing with you - network device vendors are universally terrible across the board, and that's why nobody should count on them unless they absolutely have to.

in other words, what i'm saying is to expect that kind of response is naive, and that you should seek alternatives.

Besides the usual regex aches and pains, the grammar for email addresses is far more complex than most people realize. According to a highly-voted Stack Overflow answer [1], the current RFC-specified grammar for addresses can't even be matched with regex alone. Combining the edge cases of the grammar with (say) Unicode normalization sounds like a recipe for hours of fun.

[1] https://stackoverflow.com/questions/201323/using-a-regular-e...

I find that quite unbelievable. When I had a similar problem last year, the first resource I found was a W3C specification[1] about <input type=email>. The specification clearly states that email addresses should match:

    /^[a-zA-Z0-9.!#$%&’*+/=?^_`{|}~-]+@[a-zA-Z0-9-]+(?:\.[a-zA-Z0-9-]+)*$/

Since this is an official W3C doc, I see no reason why people shouldn't use this.

Edit: There is also a version by WHATWG[2] here:

    /^[a-zA-Z0-9.!#$%&'*+\/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$/

It apparently does a more thorough validation than the W3C one in the domain part, but the difference between the two is not apparent in practice.

[1]: http://www.w3.org/TR/html-markup/input.email.html [2]: https://html.spec.whatwg.org/multipage/forms.html#e-mail-sta...

>Since this is an official W3C doc, I see no reason why people shouldn't use this.

Well, as far as authority/canon goes, it's typically dictated by the IETF and not W3. And, sure, they could cooperate with one another but, really -- if there's an authority on the protocols that describe email (SMTP, POP, IMAP, etc) -- it shouldn't be the World Wide Web Consortium.

That said, the IETF does tend to draft RFCs that reflect actual implementations (at least their intended design), but since they often bias towards interoperability, it's unlikely they'd narrow the scope of the email address grammar.

Practically, those W3C/WHATWG regexes probably are fine, but they aren't correct. For instance they fail to match the valid address c."@".t@gmail.com because they don't handle quoted strings correctly. I'd expect they also match some invalid addresses. My understanding was that only a Perl regex can test an address against the standard: http://www.ex-parrot.com/pdw/Mail-RFC822-Address.html

It may be incorrect to reject that email, but it's good to reject that email.

Why reject a perfectly valid email? Why do you get to thumb your nose at that particular one?

It's only 'valid' because of the particular decisions made by someone writing a random RFC. There's next to no reason to allow quoted strings, and even less reason to allow a different character set inside of them.

Email addresses do not need escape sequences.

Doesn't this sort of RFC typically summarize and codify real-world usage, rather than coming up with new stuff? If a feature is in there, I'd have thought it's because somebody uses it.

At some point you just have to reach the level of "Yeah... don't do that." I would suspect have "@" as part of your email probably approaches that line.

It would be interesting to know the context. If this feature was already in a popular piece of software, then saying "don't do that" in your standard risks fragmenting things rather than codifying them, and that's no good either.

Where do you draw the line? I occasionally have valid GMail addresses with "+" in them rejected because someone wrote a crappy validator.

Allow all the characters that are supposed to be valid outside of quotation marks. Letters, numbers, .!#$%&'*+-/=?^_`{|}~

That way if you don't mind letting through double periods or domain segments having more than 63 characters, you can validate an email with two character classes separated by an @. The basic regex looks like [xx]+@[yy]+

> It's only 'valid' because of the particular decisions made by someone writing a random RFC.

You know that's exactly what "valid" means, when talking about something standardized, right?

In HTML there's no reason to use " outside of an attribute value. Should browsers reject it?

I don't think gmail will let you register that email address.

Okay then, c."@".t@someotherdomain.com

The spec of what HTML will accept for an email in an input form is quite different from what email (which predates HTML by a lot, the major RFC 822 being from 1982) actually accepts.

The email input regexp is only about matching the atext@domain subpart of the syntax. The full email address parsers are about parsing the full allowed syntax you can put in an email to: field that includes address lists and display names.

Also W3C produced HTML specs are hardly gospel about email-related things (but I don't know whether there's anything wrong in this case).

But the question is, if a signup form is asking for your email address, will you type in your name, a pair of angle brackets, and then your email address in between? That's simply not what the web developer wants here.

Will I type that in? No. Will I copy-paste my email from my address book program? Quite possibly.

Replying to myself (can't edit anymore), looks like the W3C spec bungled it in your [1] by referencing "atext" from the RFC. For example, as specified it disallows addresses with periods in them. In the RFC, "atext" is just a building block in the syntax of the local-part before @ sign.

The .NET Framework's email address validation has a pretty ... comprehensive ... regular expression [1].

[1] http://referencesource.microsoft.com/#System.ComponentModel....

Holy line of cartoon swear words Batman!

I would hate to be the programmer who had to debug that regex.

You clearly haven't seen the regex that matches emails according to RFC822:

Behold: http://www.ex-parrot.com/pdw/Mail-RFC822-Address.html

/me falls off chair

The W3C and WHATWG are web standards bodies. Email isn't in their bailiwick. The acceptable format for email addresses is defined in IETF RFCs.

This forces users who have internationalized email addresses (say whatever@maré-design.fr - just a random IDN I know) to lookup the punycode-encoded version of the domain.

Ditto for internationalized TLDs like something@taiwan.台灣).

I'm not sure if this is really that bad, but I don't like it.

It's the job of the validation engine to convert all URLs and email addresses to a canonical form -- either punycode or actual UTF-8, but never both. I prefer standardizing on punycode because it's easier to handle.

Otherwise you're just asking for trouble when the same user who signed up with an email address at maré-design.fr later tries to reset their password with an email address at xn--mar-design-d7a.fr. Sorry, you don't seem to have an account with us.

The same thing happens with URLs. Some browsers send the Host: header in punycode but send the referer in UTF-8. Who knows how they encode CORS headers and all the other newfangled stuff that contains bits and pieces of URLs. You have to consistently convert one to the other before using any of them.

IDNs are a mess.

> It's the job of the validation engine to convert all URLs and email addresses to a canonical form

I don't think <input type="email"> does that though. Maybe it should, but users might be surprised to see their address automatically turn into some ugly unreadable xn--whatver-d7a domain. After form submission then yes of course any sane process will convert them to a canonical form.

I know there were good reasons to use punycode instead of UTF8 for IDNs, but it sure is a mess.

Yeah, I think we'll have to treat email addresses as just another kind of untrustworthy input that needs to be escaped in various ways depending on the context. Punycode for storage and validation, back to UTF-8 for display, back to punycode when it goes in a mailto: link, etc. Ditto for URLs. The days of easy parsing are sadly forever gone.

Saying "it can't even be matched with a regex" is trivial. Here's another, very simple thing that can't be matched with a regex: Matching pairs of nested parentheses.

direct link to image visualizing the complexity of the state machine required: https://i.stack.imgur.com/SrUwP.png

I think many people are thinking about email address validation the wrong way.

RFC 822 describes how messages are encoded when email servers talk to each other. It isn't really about email address validation and is not intended to be used to validate a form field on some registration page.

Unless you're writing an MTA or similar piece of infrastructure there is no reason you should be using the RFC grammar. Even if implementation were easy, it probably isn't what you want. For example, the spec permits inline comments but that's a nonsensical thing to have in the middle of an address you typed into an HTML form. Email addresses entered on a web form should be rejected if they contain comments, IMHO.

I think what most developers really want to know is something like: Can this given email address receive messages? Or: Does this given address actually belong to this user? Well, the only way to test that is to send it a message. At best, regex validation might warn you earlier that a given address couldn't possibly work because it's so obviously malformed. But you can't validate your way into getting people to enter their real email address if they don't want to or if they don't know what it is. If your intent is really just to help catch typos and mistakes, you'd be much better off looking to something like mailcheck [0] which will flag common typos like "foo@hotnail.com" even if they result in valid looking addresses.

[0] https://github.com/mailcheck/mailcheck

I agree with you "100% in spirit", but feel the need to point out that RFC 5322 (822 is ancient...) is for MIME, and thereby isn't what is even used "when e-mail servers talk to each other": it is the format of the e-mail message itself that is used only within a mailbox (when either your client or, if you are using IMAP, your server parses the message).

The actual standard used by e-mail servers talking to each other is 5321 (which you might still know 821 ;P), the standard for SMTP: this protocol actually has a different way to write comments and escape characters, as it is embedded into a different structure (which I think decimates the arguments people tend to make that you should validate comments).

Years ago I was working quite in earnest on an e-mail server suite, and at the time I was extremely deep in the various standards, and wrote a comment that goes into somewhat more depth on the semantics of e-mail verification. The example I was really happy with is the notion that you would never ask your user to HTML escape their username or password ;P.

https://news.ycombinator.com/item?id=4486872

I very much agree because in the end the only validation seems to be "does it contain a @ character somewhere in the middle".

For example: example@localhost is valid but is refused by most validators.

Localhost is obviously not a fully qualified domain name which would be expected in any internet setting.

Top level domains do resolve though, email@bs is completely legal to try to send email to.

> Email addresses entered on a web form should be rejected if they contain comments, IMHO.

Why? I use several addresses that start something like myfirstname.mylastname@... . A comment at the start could make it much easier for me to use browser autofill. (Arguably that's "really" a browser UX issue, but we work with the tools we have)

I really hate browser- or webapp-side email validation. I've rarely seen a web developer get this right. My advice is to treat email addresses like blobs and attempt delivery with a confirmation link. That will catch users who typo their own address, and it will let people use whatever address format they want (the "+" in an email address not being considered valid is my own personal bugaboo).

I dunno. I think people with edge case emails (i.e. those not easily validated with a simpleish regex) should be nudged towards getting more standardized emails.

Our company won't accept weird emails for free trial sign ups. We should be nudging users towards good behavior.

Of course, if your email address is provided at the corp level, then you don't have as many options.

Non-standard emails can be used as tools for phishing is another reason why we should not validate them.

> Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.

Great quote. However, I think regexes got a bad reputation just because of the way people use them. In essence they are a pretty reliable way of parsing because the parsing engine is well tested. But the expression should be kept as simple as possible and developer should avoid using any nonstandard / nonexplicit extensions. I even avoid using \w because, well, what IS a word character? I am sure it is defined somewhere... but I'll always use explicit form (like "[a-zA-Z]" when I want ASCII chars) instead.

Anyway, if you use the form as used in the regex puzzle [0], you'll be fine. As long as you use regex only for what it was meant for, of course [1]...

[0] https://news.ycombinator.com/item?id=10787509

[1] http://blog.codinghorror.com/parsing-html-the-cthulhu-way/

I find a lot of websites will eat passwords that contain special characters. I don't mean that they'll tell you it doesn't match the password policy, I mean that they'll accept the password and then tell you the password is wrong when you come back to sign in. I eventually had to teach my password generator to use only a few usually-properly-handled special characters when generating to avoid the hassle of having to reset the password every time. The same thing is often true of long passwords -- websites will accept the password at the UI level, but it probably gets truncated somewhere in the processing, and you don't know which character you got cut off at, so you have to reset to something shorter.

If it's going to eat a character it should at least be consistent and also eat it at the validation stage.

From what I've seen from "regular people" writing regular expressions, they seem to not have the slightest clue on how to do it

And then putting it into the program without testing it properly

So, sorry, the issue is not regexes, but people just going for it at an "trial and error" fashion (and sometimes just trial)

For people here that may ask themselves just how exactly one could test regular expressions, I recommend a visual tool like [0]. Having the regex structure and meaning drawn in front of you helps tremendously.

That, and for the love of God please comment any non-trivial regexp. Either like this, with 'x' option:

  preg_match('/^
              .*              # Match any number of characters...
              (?=.{6,})       # ... AND match at least 6 characters (lookahead) ...
              (?=.*\d)        # ... AND match one digit after any number of characters (lookahead) ...
              (?=.*[a-zA-Z])  # ... AND match one letter after any number of characters (lookahead) ...
              .*              # ... AND allow any number of characters later.
              $/x', $password);

... or just with normal programming language comments and stitching regexp from multiple strings in multiple lines. Also give some semantic meaning to groups if you use them, e.g. tag them with constants so that your code isn't full of stuff "result.get(3)", which makes you waste time on trying to recall what was that group 3 in the code from last month.

I know it's pretty much software engineering 101. It's the basics of basics. But from my experience, even the brightest of engineers in most serious projects suddenly forget how to write code when they touch regular expressions.

[0] - https://www.debuggex.com

Using tools is good, but also people should test them in their unit test, for what it should/should not be accepted

If you're forbidding items beginning with numbers, just have a test try passing '1a' and failing the match

I agree of course. Unit tests like that are both basic sanity checks and, more importantly, protect you from stupid mistakes when the regexp has to be changed.

It should be common knowledge at this point, but just in case:

If you're doing regex or any other text manipulation on user input when you ask them to set a password, you're doing it wrong.

What's the best way to deal with this problem, and what's the correct way to deal with this problem?

The input the user puts into the password prompt should by taken exactly as is, pushed into bcrypt/scrypt/etc, then stored as the user's password hash.

I'm not entirely opposed to requiring a minimum length, but imposing max lengths / character class rules / etc ends up hurting people who want to pick strong passwords more than it helps people who would pick weak ones (enforcing character classes just gets us lots of password1A! and similar)

I agree that the only real restriction that makes sense is a minimum character count. The others just tend to get in the way. I haven't seen anyone implement it in the wild, but it'd also be cool if there was a wordlist of the 25 most common passwords that the site matched against and refused to accept. I think those two policies, minimum length and no super common passwords, would do a lot to minimize the effectiveness of dictionary attacks.

I personally like the Stanford policy[1]. It basically boils down to: the longer the password, the fewer restrictions. Each password needs to be at least 8 characters, but if you only have 8 characters, you might need uppercase, lowercase, a symbol and a digit. If you have 12 characters, you only need upper/lower/digit. Once you hit 20 characters, you can have whatever you want.

I think that this is a good balance between security for short passwords, while still allowing ridiculously long ones (pass-phrases).

[1] http://arstechnica.com/security/2014/04/stanfords-password-p...

Almost like you're validating on entropy and not specific rules..........

The concept is nice, but their numbers are horribly, unforgivably wrong.

8 random upper/lower/digit/symbol characters are equal to 9 random mixed-case letters. Not 16.

Their cutoffs for different mixes are 8, 12, 16, 20. Realistic cutoffs would look more like 10, 11, 11, 14.

Even worse, they encourage counting the individual letters in words. Never do that. Random words are only as good as two random characters.

    $ dd if=/dev/urandom bs=32 count=1 | xxd -p

That nearly every site on the Internet will refer to that output as "not strong enough" and instead suggest P@ssword1 as a better alternative definitely speaks to the issue.

> I haven't seen anyone implement it in the wild, but it'd also be cool if there was a wordlist of the 25 most common passwords that the site matched against and refused to accept.

I have implemented that on a site. It wouldn't accept the top 10,000 most common passwords. (or was it 1k).

Otherwise ~1% of your users will pick "password" as a password.

> I haven't seen anyone implement it in the wild, but it'd also be cool if there was a wordlist of the 25 most common passwords that the site matched against and refused to accept

It's now in django[0]. And it's a hoot to read...

[0] https://github.com/django/django/blob/master/django/contrib/...

> The input the user puts into the password prompt should by taken exactly as is

I would however suggest trimming trailing spaces.

Also, for a nicer user experience try the password twice: As is, and with case reversed. This lets people login even with capslock on and has little impact on security (i.e. don't be case insensitive! just case reversed.)

I see the reasoning behind trimming trailing spaces, but it could cause at least as many problems as it solves. Plenty of people use automatically generated passwords, and while hopefully most auto-generators don't include spaces, I expect some might. Better might be to warn the user if their password has a trailing space.

Is there any password generator in this reality which does that? Serious question. More than zero?

Probably not a serious one, but a lot of people write their own little scripts and utilities to do this. So definitely more than zero. Perhaps not a significant number, but I'd still be concerned about changing a user's entered password without letting them know. relearn's solution is a good one though.

Could you not trim spaces upon attempted login as well to make this a non-issue?

I do like the idea of warning the user upon creation.

Good idea; I agree that that's better.

I would guess the number of people who accidently enter "password " is far greater than the number of people who have "p " as their password. Plus, for the people who do deliberately input spaces they will just have a weaker password, not issues logging in.

I can see how that could fail horribly when you have arbitrary Unicode data.

For instance, the German sharp s (ß) has an asymmetrical casemapping.

From the Unicode standard[0]:

>The German sharp s character has several complications in case mapping. Not only does its uppercase mapping expand in length, but its default case-pairings are asymmetrical. The default case mapping operations follow standard German orthography, which uses the string “SS” as the regular uppercase mapping for U+00DF ß latin small letter sharp s. In contrast, the alternate, single character uppercase form, U+1E9E latin capital letter sharp s, is intended for typographical representations of signage and uppercase titles, and in other environments where users require the sharp s to be preserved in uppercase. Overall, such usage is uncommon. Thus, when using the default Unicode casing operations, capital sharp s will lowercase to small sharp s, but not vice versa: small sharp s uppercases to “SS”, as shown in Figure 5-16. A tailored casing operation is needed in circumstances requiring small sharp s to uppercase to capital sharp s.

[0] http://www.unicode.org/versions/Unicode7.0.0/ch05.pdf#G21180

Why would that fail horribly?

> For instance, the German sharp s (ß) has an asymmetrical casemapping.

As they say "so don't do that".

This is about capslock, not lettercase in general. Only switch characters that change with capslock on the keyboard.

And even if the mapping is not perfect - so what? The worst that will happen is nothing.

> imposing max lengths .. ends up hurting people

In that case you'll want to pre-hash your BCrypt-encoded passwords, otherwise you'll be imposing a silent 72 character limit.

Remember to encode the hashes, since BCrypt also silently truncates after NULL bytes.

> In that case you'll want to pre-hash your BCrypt-encoded passwords, otherwise you'll be imposing a silent 72 character limit.

Or just use PBKDF2 or scrypt, neither of which imposes an artificial length limitation on passwords.

Not that it really matters, since a 72-character password will have hundreds of bits of entropy as long as the alphabet is more than two characters; it's overkill.

I wouldn't expect a passphrase that long to be particularly entropy-dense - quite the opposite really. Some people do things like pick phrases from Hamlet and stuff some numbers on the end, so truncating them would seriously compromise their expected strength.

Or XKCD style passphrases. Using the Oxford 3000 dictionary gets you about 1 bit of entropy per character. If I have my password manager generate 128 bit passphrases using them for ease of use in the real world, plain bcrypt will quietly reduce their strength by about 17 orders of magnitude. If nothing else that's obnoxious.

And that's ignoring the soft 55 byte limit both the bcrypt spec and the original scrypt paper mention. If we're looking at less than one bit per byte that's starting to get into worryingly weak territory.

Either way, it's all solved by stuffing a (e.g.) base64 SHA384 in there. Bam, every input bit equally effects the output hash regardless of length and position, and the entire thing even becomes NULL byte safe.

The problem is that lots of people will pick "password" as a password. I think it's the responsibility of websites to prevent that from happening.

That's why you download a list of the 10k most common passwords and not let a user choose one from the list. Way better than some arbitrary character class rules that are easily worked around by appending "1A!" to the end of an awful password like GP mentioned.

Use https://github.com/dropbox/zxcvbn.

The correct way is not to use passwords. Use X.509 client certificates, and let the user secure theirs whatever way makes sense to them (whether that's a password, a smartcard, both, or something else). Unfortunately the browser UX for them is terrible.

> Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.

Yeah sure. I think I heard that quote before... just about a million times?

People expect regex to be an easy-to-use tool. Well it's not, and it's a foot gun if you don't take your time to learn it right. But no, people hack up some expressions, hit their feet and blame... the tool of course, not themselves.

Just learn it right, it's a great tool if you know how to make it work for you :)

Regex's in perl5 are what introduced me to test automation. Hard.

Reply on news.ycombinator.com