Is sending email really that slow? In my experience, it just writes something to a local mail queue if your front ends can send email.
Now you could argue "that's true of anything!" but in general you are forced to depend on local IO/CPU/RAM otherwise your site is struggling in general, by adding DNS/SMTP/etc you're just adding more "stuff" that your site relies upon.
Plus SMTP servers are "allowed" to go offline for short periods. The client should try to resend the email 2-4 additional times before giving up. Clearly a PHP script cannot do that.
It isn't something you can control in the code.
php -r '$s = microtime(1);mail("jcampbell1@gmail.com","test","text");printf("%d\n ms",(microtime(1)-$s)*1000);'
> 20 ms
The reason I assume that it has the feeling of being slow, from the applications I've had to clean up, is because many times there's processing happening right before the mail() function. So while, yes, its relatively fast the the total time is what feels slow.
And for comparison I've ran your commandline test as well with two of my cheap-o VPS instances, one of which is a Linode instance:
Linode: 50ms, 13ms, 14ms
Crappier VPS: 677ms, 149ms, 188ms