Hacker News Reader

I tried using that expression for a while, but then a user with a valid email address containing upper unicode characters showed up. I switched to a simpler expression:

    ^[^@\s\x00-\x1f]+@[^@\s\x00-\x1f.]+(:?\.[^@\s\x00-\x1f.]+)*$

It requires exactly one "@", disallows whitespace and control characters, prevents repeated dots in the domain name, and ensures the domain doesn't end with a dot. It catches a few typos and I think it allows every real email address I've heard of.

Domains ending with a dot are valid though, and it's needed sometimes. For example, someone@ai. (ai. is a TLD) is a different email than someone@ai (ai is a local hostname)

Good to know. I guess I'll simplify again. Besides, I wonder whether the expression is vulnerable to catastrophic backtracking.

> it's needed sometimes

When is this needed in the context of a Rails app? You never want to send emails to local hosts, and emails directly on the TLD is possible, but very little will work with them in practice.

> You never want to send emails to local hosts

That's not entirely true, it depends on how your mailing infrastructure is set up.

But in the context of a web app though? That's the key part: you're taking in email addresses from myapp.example.com. For other cases: sure, local email addresses can be useful (not often these days, but sometimes still are). But in this context I'm not really seeing any use case.

At previous $work, I wrote a rails app that managed releases. On the app you could sign up for email notifications when your change was released. These went through the unix local email system. So yes there is very much a use case

It's still a bit silly to tell every SaaS developer that they shouldn't validate that they have a normal-looking domain because some people's use cases call for sending emails to local addresses or to TLDs directly. If a developer needs to handle those cases, they know already, and they're in a vanishingly small minority.

I'm pretty sure the one guy behind ai. doesn't justify that you change your regex for him. Let's not be overly pedantic.

https://en.wikipedia.org/wiki/Top-level_domain#Dotless_domai...

> Domains ending with a dot are valid though,

That doesn't mean you have to support their use. I have never seen a company support an email ending in a period and the world seems to continue along.

I invite you to try my test suite at https://github.com/daurnimator/lpeg_patterns/blob/master/spe...

> containing upper unicode characters

Are e-mail addresses case-sensitive? I would always lowercase&trim a string meant to represent an e-mail address (or a domain name) before doing anything else with it.

The local-part before the @ can be case-sensitive, yes. I've never heard of a mail server that treats it as such, but the RFC says it's case sensitive.

Not all uppercase characters have just one lowercase representation, and not all lowercase characters have an uppercase character. If you try to make the addresses case insensitive, you're bound to run into unnecessary complexities at some point.

Email addresses aren't case insensitive in nature, but many commercial mail services (and misguided developers) assume they are.

To be clear, by "upper" unicode characters I mean non-ASCII international characters, not uppercase characters. For example, 我買@屋企.香港 is now a valid email address. [1]

1. https://en.wikipedia.org/wiki/Email_address#Internationaliza...

I'm pretty sure email addresses are allowed to have multiple @'s. I believe everything after the first is (supposed to be) considered part of the domain.

Domain labels cannot contain @. If you support comments or quoted strings (e.g. (@)."@"@example) then you can get multiple at signs in an address, but in practice their only use these days is attacks where you confuse a badly-written server, which is part of why the web actively decided to disallow them.

> in practice their only use these days is attacks where you confuse a badly-written server

Or, you know, if you actually want to use an at sign in the manner allowed by the RFCs. That's the point of standards: if something is allowed, then … it is allowed.

Refusing to accept behaviour permitted by the standard is just broken.

Usage is what matters. Standards documents only have value as long as people follow them; once enough people deviate, they cease to have meaning. This is thoroughly the case with a lot of stuff pertaining to email: if you want to parse arbitrary emails, for example, the relevant MIME RFCs are altogether insufficient, because too many people have done it incorrectly de jure, often to the point where it’s no longer incorrect de facto because everyone supports the deviation. (This is a deviation from spec in the direction of permission.)

By hearsay alone, I say: comments and quoted strings in email addresses are, de facto, more or less dead. Comments especially, which were intended just as a compatibility measure for stuff from over forty years ago (when this was being developed organically, before there was any real spec). Most software whose direct business is email (that is, MUAs and MTAs) will more or less support them for historical reasons, but newer MUAs are likely not to support them in full (again especially comments), and I suspect the considerable majority of other software that deals in email addresses won’t support one or both of them. (This is a deviation from spec in the direction of restriction.)

One important aspect of this that I often see people forgetting is that this check is designed for working with ASCII, but the domain name at least can be non-ASCII. User interfaces should remember to support IDN in domain labels and convert it to punycode before validating, and if you store A-labels (which you probably do) then convert it back to IDN form when presenting it to users.

(Alas, <input type=email> still doesn’t support non-ASCII in the local part, which isn’t supported everywhere but is, I believe, fairly widely supported now. See https://github.com/whatwg/html/issues/4562 plus https://en.wikipedia.org/wiki/Email_address_internationaliza... for a little more background on what it is.)

I thought they've rolled back most of this because it made phishing domains indistinguishable from the real ones?

I used an-emoji.my.domain for a while until chrome changed it back to punycode

I erred slightly in what I wrote there; I said IDN form for user display, but should have said IDNA form, which is the set of restrictions to mitigate those hazards. That will allow things like उदाहरण.example, while leaving emoji as xn--n3h.example since they’re not valid in IDNA. See also https://en.wikipedia.org/wiki/Internationalized_domain_name and https://en.wikipedia.org/wiki/Emoji_domain.

Actually, I’ve got to add another note because my memory and experience was insufficient here: see also https://en.wikipedia.org/wiki/IDN_homograph_attack#Client-si... for descriptions of what further restrictions browsers do, with the most notable additional filter being disallowing mixing scripts. This is something that would be nice to formalise in some way, though I don’t know of any venue suitable for the task.

This is the sane approach. Nobody uses the bizarre address formats with quoted strings and embedded comments and whatnot. There are included in the RFC because of backwards compatibility with legacy email systems from before SMTP.

Quoted strings are actually used in the real world, though they're certainly not common.

Shitty front ends are definitely putting an end to that, though.

I'd say that's for the better.

What is the significance of [a-zA-Z0-9-]{0,61}? Some special limit on 61 characters?

Domain labels are limited to 63 bytes, with the [a-z..] before and after that makes 61 + 1 + 1 = 63.

Oh wow, that is awful.

I like your final sentence. Very to-the-point.

If you really want to take validation seriously, why not add a human element to it, and not design one line of code to try to fix everything?

One of the biggest problems with email input is typos -- and there are some very common typos that could easily be accounted for with code. For example foo@gmail.co, foo@gmial.com, foo@comcast, etc.

It should be common, when these types of typos occur, to prompt the user to fix them. Unfortunately, this is quite rare.

Everytime I input my .me domain in one of those well intended websites I get nagged about maybe being wrong and wantint to use .de or whatever.

No, I know my email, thx, it's bein prefilled from auto-complete. Don't fucking tell my Im typing it wrong when 1. Its my email, and I'm not even typing

I have the same issues with my .name domain. Most of the reason I have a gmail address as well is for the sites that don't think "first@last.name" is a real address.

Related: I used to use me@<myname>.com but gave up because Gmail uses "Me" as the name when displaying a user's own email address and it was confusing people enormously because a thread would look like:

>>>Me wrote: >>Me wrote: >Me wrote: Me wrote:

That would imply that programmers care about users. I've never met a single programmer who cared what the user experience was like. Today they even brag about this, saying "I only want to care about my code!" I see it here on HN all the time, and I hear it in companies.

The other thing is overengineering. Spending a week to implement email validation? Maybe, if it is REALLY important and results in lost revenue.

Otherwise I don't know what common typos are. I would have to do research on that. Then I have popular local email providers where I may come up with typos. but then again, it is only in my region - I would have to get some list of popular email providers in every country. Come up with a way to update that list after few years when new email providers come out. Or maybe develop a solution that matches "close enough" entries in my pre-defined list.

UX effort could be spent in an area which is used more often than only once. If there is a library for that, yeah, then slap it in, UX has been upgraded in no time. And again, only if revenue is on the other side of consideration, the development effort pays off.

Revenue focus is not customer focus. You don't get a second chance at a first impression. The difference between a great company and a mediocre one is how much they care. Growth-obsessed companies only care about revenue and the 80% case. Long lasting companies build relationships, listen closely, and feel deeply.

Perhaps you are only looking for that kind of message. I see concern for users expresses by many devs here and hear it from other devs that I talk to. I can tell you that we have extensive discussions about the user experience whenever planning a new feature or change to and existing one. If our customers are not happy, that affects their use of our product, so of course we make this a priority. Besides, we also use it and we want it to work for anyone using it.

OP has shared a solution for typo fix in the closing remarks of the blog: https://github.com/mailcheck/mailcheck

> If you really want to take validation seriously, why not add a human element to it, and not design one line of code to try to fix everything?

human: a customer telling a clerk, "yes, it's really gmial dot com," and the two sharing a laugh and trading stories about funny email addys

human element: the customer curses their phone because the backend folks never tested the edge case of your helpful button for confirming the suspicious case of "gmial.com" and just keep rejecting it. Then, after your helpful chat bot kept autocorrecting their chat input to "gmail.com" they threw their phone on the ground so hard it broke.

Don't be a human element.

I have a .con in the db

In fact, doesn't 'RFC' imply that it isn't a standard?

A pragmatic, domain-specific spec beats that.

edit: there's more nuance than that. Some RFCs are standards. 2822 is 'Category: Standards Track', so is a standard if referred to by its STD number? I can't find what its STD number is.