Real Email Validation | Hacker News Reader

Real Email Validation

djangotips.com

47 pointspythonist42 comments

The only E-Mail validation involves sending an actual email with a response link.

Because even if people happen to give you a functional email address, it isn't necessarily their email address.

And I say that as someone who has come to regret registering a first-initial-last-name gmail address. And it's not even a particularly common last name.

I think the purpose of this validation is to help people who mistype their emailadress, not to check if it is their emailaddress.

Right but on a large system it is possible to mistype it to a valid address that isn't yours.

>And I say that as someone who has come to regret registering a first-initial-last-name gmail address. And it's not even a particularly common last name.

It is amazing how common this problem is. I assumed it was incredibly rare, but I have 23 different people who have given my email address to someone thinking it was theirs somehow. Not like "I am just signing up for some forum" kind of stuff, logins to government websites, banks, car dealerships sending me stuff about someone else's financing, etc, etc. It is crazy how many places don't verify the owner of an email address before sending it sensitive info.

I used to get a lot of pictures of children related to a guy with my name. That was pretty creepy.

That's happened to me. I also occasionally get hotel, flight, and restaurant reservation emails. Apparently my alter ego likes to travel.

You wake up at Seatac, SFO, LAX. You wake up at O'Hare, Dallas-Fort Worth, BWI. Pacific, mountain, central. Lose an hour, gain an hour. This is your life, and it's ending one minute at a time. You wake up at Air Harbor International. If you wake up at a different time, in a different place, could you wake up as a different person?

Feel my pain. I have surname at gmail.

This is just awful. A quick scan of the code brings up the following problems:

* It fails to deal with the case where there is no MX record for the domain (fall back to A record)

* It fails to sort the MX records, potentially falling foul to tarpits

* It fails to connect to each A record lookup of the MX host on failures

* It fails to deal with transient failures (such as 4xx responses)

That was just from a quick scan.

Connecting to MX servers in a web environment (especially one using blocking I/O like Django) is generally a really bad idea. Many MX servers use delays and slow responses to combat spammers, and you're passing those slow responses on to your users.

Just check it looks vaguely like an email (the regexp fein posted is good enough most of the time) and send a confirmation email - it's the right thing to do.

Failing to deal with transient failures is especially bad when trying to deliver to a system that uses greylisting.

Additionally, doesn't it rely on the truthfulness of the SMTP server? That's not a good assumption - it's common to accept anything and null-routes bad addresses.

Indeed it does - the only way to truly validate is to get that confirmation email through.

On the flip side I do think there's some value in a service which provides a check on the domain - that way you can prevent someone typing in username@gmail.con by accident. But you'd have to actually implement it correctly.

Would people be interested in something like this as service?

And I'll still give you fakeaddress@mailinator.com, it will pass every check you can throw at it, including sending an email and getting me to click a link, and it still won't be a real email address.

Still your move, e-mail harvesters.

Checking that I haven't mistyped it or put the wrong thing in the wrong field is a basic sanity check. Beyond that, the only way to actually get a real email address that I read is to be a service I care about.

For me the trick isn't to get my real email address, I give that to anyone.

But kudos to you if you can make it into my "Important and unread" inbox and remain there. It's the only part of my email that I actually check.

Some services are so great I let their daily reminder emails go there and enjoy reading them. That's right, there are services out there (I only know of one) whose daily "You should use us" email is so awesome I enjoy reading it every day.

Out of curiosity, what service has such a great daily e-mail?

750words.com

Making your app connect to random SMTP servers every time it needs to validate an email address doesn't seem like a good idea.

Shared domains (gmail.com etc.) might even get you blacklisted if you flood the same SMTP servers over and over again.

Is there a work around ? How about using proxy but I guess that adds another layer of complexity

Use a queue processor - but that's probably going too far for simple email validation.

The simple work around - don't do it. This code is susceptible to Denial of Service problems similar to the URLField verify_exists option https://www.djangoproject.com/weblog/2011/sep/09/security-re... - a malicious SMTP server could tarpit all your SMTP connections from Django leaving your site with no workers to process other requests.

The email validation from an EmailField is designed to ensure that it could be a valid email address, not that it's a valid mailbox. Live with the limitation instead of trying to be too smart.

The best way to improve email delivery is to understand that email addresses represent humans. Address validation and long-term deliverability is primarily a problem of social engineering, not technical.

Ordinarily I'm in favour of things that can improve data quality with minimal user friction, but in this case while it looks like an attractive solution, it's both dangerous _and_ broken.

It's dangerous because if you repeatedly open empty SMTP sessions with major ISPs (and some neckbeard boxen) to validate addresses, you will rapidly fall onto blacklists. Furthermore existence of an address says nothing of the end user's ownership of that address.

It's broken because of the myriad crazy responses that mailservers return -: 5XX errors for soft-bounces, 4XX errors for permanent failures, deliberately dead primary MX server... The web's email infrastructure is so massively fragmented and quirkily non-RFC-compliant you just cannot rely on technical solutions to these problems except at scale of an ESP (disclaimer: I work at PostageApp.com, a transactional ESP, and we tackle this problem on a large scale)

Finally, it fails my 'Spammer Sniff Test': If you think of a clever trick to improve email delivery/opens/responses etc, it's been thought up 10 years ago by spammers and long since added to blocked behaviours in email protection infrastructure.

Check for '@', and craft your email verification process to incentivize following through. For long term delivery (to bypass the mailinator issue) provide value, pure and simple.

Greylisting is pretty common, and this would obviously fail:

http://en.wikipedia.org/wiki/Greylisting

As an aside, would there be some value in providing an email validator API?

Something exactly like this: http://mythic-beasts.com/~pdw/cgi-bin/emailvalidate

but which would respond in an easy-to-parse way (JSON|XML).

It could be enriched by detecting common spelling errors ('gmial' or 'g-a53'* instead of 'gmail' for example).

*: gmail when typed on a European laptop with numlock on.

Will also fail to allow addresses that purposely soft bounce (4xx) the first attempt (or attempts within a certain time limit) to deliver to them.

('SMPT' is used throughtout instead of 'SMTP'.)

What does django.core.validators.EmailValidator actually do?

Validating an email address with a regex is surprisingly hard: see http://www.ex-parrot.com/~pdw/Mail-RFC822-Address.html

I wonder if EmailValidator does this, or something simpler?

That validates RFC822 addresses, which is the full syntax of the From/To/CC headers. You don't want that for validating an email address on a web form.

Here's a secret:

regex: /^(.+)\@(.+)\.(.+)$/

maxlen: 254, minlen:5

Aside from sending your verification email, that's all you need.

http://www.ex-parrot.com/pdw/Mail-RFC822-Address.html

https://news.ycombinator.com/item?id=4794368

I believe that this part is using the Django's pattern matching:

super(EmailValidator, self).__call__(value)

Just tried it. It works!

Are single letter domain and tlds allowed?

yep, thats why we do a check for a min of 5 chars:

a@a.a

edit: the regex should pick this up by default anyway, given its looking for .+ and not .*, but my validation libs have the min and max length built in as well.

I meant to ask if they are valid in the IETF sense, not in your regex.

Turns out that single letter second level domains are allowed but they are reserved in some TLDs http://en.wikipedia.org/wiki/Single-letter_second-level_doma...

On the other hand, no single letter TLD exists http://data.iana.org/TLD/tlds-alpha-by-domain.txt

And I also just found out in that list that there are IDN TLDs (see punycode toward the end of the list), I didn’t know that. Full decoded list: http://mct.verisign-grs.com/convertServlet?input=XN--0ZWM56D...

I wonder how many validators fail on these.

That excludes TLD emails like postbox@com

I have never come across someone using one but it is valid. I would actually hate to see someone try to use one. I come across enough issues trying to use '+' in my gmail email address.

http://en.wikipedia.org/wiki/Email_address#Valid_email_addre...

It's not just TLDs. Machine aliases are also perfectly valid in e-mail addresses, e.g. "root@localhost", "fred@finance" etc.

This might not be practical in a majority of applications (you're hardly going to sign up to 3rd party services using an alias to a machine on your local network) but if you're building a generic e-mail address validation library, it's an edge case you cannot ignore.

n@ai (Ian Goldberg's real, valid email address) is rejected by this by your minlength and by the subdomain requirement. You're better off just checking for an @ and leaving the rest to your smtp library.

I just check for /.@./, which catches obvious errors like leaving out the email, or typing something in the wrong field. Beyond that, there's no point making assumptions (like "all domains have a dot").

Today all domains have at least one dot in them. If, at some point in the future, you're allowed to have "jimmy@nike", directly at a TLD, we'll have bigger fish to fry than that.

No they don't.

http://serverfault.com/questions/154991/why-do-some-tld-have...

Edge case city, there are perhaps a handful of people in the world without a dot in their domain, but yes, technically correct.

The thing is, for every legitimate bob@ai there are millions of erroneous bob@gmail mistakes that would slip through. At the very least a warning should be presented.

Could be easily modified to verify email lists too, very handy if you haven't sent for a while and want to avoid bounces.

This is pretty awesome! Wonder how much time would it take to validate. Last thing I would want is to make that signup process even slower. I guess you could still let the user pass and then run an async task to check "if the domain name exists, ask for MX server list from DNS, and verify that SMPT server will receive a message to that address" and then maybe set a flag somewhere.

Very helpful thanks !!

Reply on news.ycombinator.com