Ask HN: How does Experian determine whether a given Gmail address exists? | Hacker News Reader

Ask HN: How does Experian determine whether a given Gmail address exists?

15 pointsakdor115410 comments

Experian is selling an email validation service: https://www.experian.com.au/email-validation . This seems to, somehow, do live detection of whether a given gmail address exists.

To test this, I tested the result for a non-existing dummy address was 'invalid', then registered a gmail account with that dummy address, then re-checked the result to be 'valid'. As far as I could tell, there was nothing sent to the new account (so e.g. they aren't just checking for a bounce from a test email)

How can this work? This basically allows for user existence enumeration, which I would expect Google might frown upon. Reckon they are abusing some Google API in some way? (e.g. I'm having fuzzy thoughts about calendar invites, doc sharing, etc.) I am working under the assumption that they have special-cased detection for Google accounts; I haven't checked other email providers. Obviously their service could not possibly work in the general case.

I think you could validate an address by initiating an SMTP session to send an e-mail to that address, then aborting it. After issuing the RCPT (recipient) command, you'll get back a status code from the server telling you if the recipient is valid (250) or invalid (e.g., 550). At that point, you send QUIT without sending any DATA (so no message will be sent if the e-mail address is valid).

There are examples of successful and unsuccessful RCPT attempts here (scroll down to the second sample SMTP session):

https://en.wikipedia.org/wiki/List_of_SMTP_server_return_cod...

This is an entirely valid use of the SMTP protocol (getting back that status code is how you find out if a message to an address was undeliverable), but I suppose that if an e-mail provider wanted to prevent this kind of validation, they could stop accepting connections from subdomains of Experian that repeatedly aborted their SMTP sessions.

Just tried this with one of my servers and they definitely do this. They seem to be probing from 94.46.249.204, 94.46.121.19, 94.46.121.141 (and probably many more).

The setup behind it looks malicious. All of these reverse to subdomains of random, nonsensical two-word domains such as "motley-row.com" with no indication what-so-ever about the company behind it.

If they were operating a legitimate service I wonder why they go through all this trouble of obfuscation? Unless they're white-labelling some third-party service and that third-party might not only be used for legitimate purposes? I'd assume sysadmins might be more tolerating of these attempts if they were forthcoming about who they were and what their purpose was instead of looking like a black-hat scanner or spammer (in fact all these IPs are on spam blacklists already).

If the IP's do not match to these two-word domains in the forward and reverse DNS direction, my mail server will turf the connection.

Hypothesis: thus, my valid e-mail address will be declared invalid.

Experimental result: Experian fucking lies!

I tried two addresses, a valid one and invalid. Their user interface claims that it was not able to determine the status because "[t]he request timed out due to the host domain not responding in time".

This is complete horseshit.

Both requests resulted in a prompt access to my mail server and a swift and clear SMTP rejection. Nothing timed out.

This one was rejected because the IP address is listed by Spamhaus:

  2020-10-20 00:22:00 H=emv31.eistnesieu.com [64.17.42.224] F=<double-bounce@emv31.eistnesieu.com>

This one was rejected by a rule of my own, "too many digits in domain name":

  2020-10-20 00:23:16 H=smtp104.bordurecultures.com [192.121.0.117] F=<double-bounce@smtp104.BordureCultures.com>

Hiding the fact that their probe attempt was rejected due to looking like spam, by claiming that the request timed out, is egregiously dishonest. They are insinuating that they could validate an address in my domain, if only my domain would respond on time, covering up the fact that it did, and no they could not.

Given this is Experian, I wouldn't be surprised if the whole thing was outsourced to the lowest bidder in some third-world country and the code was something like try { check email } catch { return "timeout" } with no proper error handling, thus timeout is the generic error it will report for any failure.

Huh that's really interesting. I wonder if it would make sense for a mail server to always respond OK to RCPT, but respond with failure to DATA if any recipients are undeliverable? There would be a slight change in semantics here (there would be no way for the sending SMTP server to know which subset of the recipients are undeliverable) though, which might make it hard to notify users about specific bouncing addresses.

To add to this you can go into google documents and add random email addresses to the 'share with' option. This then validates google account email addresses + swaps the email address to their name.

I reported it as a bug once as its open to abuse but they came back and said its a feature.

In the old days you'd simply connect to the domains SMTP and do either a VRFY or start sending an email and then aborting. Google doesn't allow this. However when signing up to Gmail you are notified if the username is taken, so they might be using something like that.

Disclaimer: I don't work for Experian. I suspect only one of their devs could answer this affirmatively.

There are different approaches. As a sibling comment mentioned, you can use any login protocol that responds with a different message for users vs. non-users.

Off the top of my head, you have POP, IMAP, SMTP, OAuth, Gmail's undocumented HTTP API, Gmail's documented HTTP API, and web scraping.

You probably can't do any of those at scale.

To solve the scale issue, you could buy databases of verified email addresses, which are not expensive. For any address found in those databases, you know it's registered.

If it's not in your database, you can do a live check.

Alternatively, you could make a business arrangement with Google, which is also possible for a company like Experian.

I don't think you could use POP or IMAP, since these protocols are designed for a registered user to interact with a mail server - to connect, you'd first need to know the user's userID and password on that server (and the userID isn't necessarily related to the e-mail address, even though it may be on Gmail). And if you got back an error, you wouldn't know whether it was the userID or the password that was invalid. SMTP can be used, however, since that protocol is designed to be able to send messages to a user on any mail server. (See my earlier reply for an explanation of how this can be done with SMTP: https://news.ycombinator.com/item?id=24834067)

See the readme on https://www.npmjs.com/package/is-gmail-account-valid

Reply on news.ycombinator.com