This is the mail system | Hacker News Reader

This is the mail system

dewith.com

119 pointstbassetto74 comments

I completely disagree 100%. I think the mail system is exceptionally well-functioning. It's a massively-scalable, decentralized, asynchronous messaging system that supports large file attachments. At this, it excels. There's not one big evil company controlling all messages and unless you're sending massive amounts of emails every day, your messages almost always end up exactly where they're supposed to. I'd challenge anybody to do better.

The author's main gripe is Mailer Daemon messages. It's not email's fault that people write shitty clients that don't display a pretty failure message. If your car doesn't have anti-lock breaks, it's not the road's fault when you hit a patch of ice and slam into a tree.

Email is great not only because it works simply and beautifully, but because it's so extendable. People think it's broken. It's not. It's just simple...and it's this same simplicity that makes it easy to build on top of. Look at Gmail or Xobni. Both have incredible features that "email" doesn't have because they took the time to build extras on top that make it work nicely.

Email is NOT broken. There's nothing to fix. Whatever fixes you think it needs you can make yourself. The underlying platform works great. If you have a problem, write/patch a client.

Exactly. It would be no small matter to come up with an alternative system that has the simplicity, flexibility (including flexibility of archival), and most of all the freedom of email – and have it work anywhere near as well in practice as email does already.

(Google Wave showed promise... but it also introduced a ton of new complexity, and given the pattern of messages "living" on the sender's server, it wasn't clear how archival should actually work.)

It seems most people who advocate this notion that email is "broken" would rather us all use some centralized, proprietary system akin to Facebook messaging. This would be an unimaginable step backward for reasons that I hope do not require elaboration.

Federated XMPP seems rather nice alternative if it were extended to support asynchronous communications (eg servers would store messages when clients are offline). Yes, it is XML based and XML is not fashionable these days. But still I think it has potential if it were developed further.

btw Facebook messaging is partly based on XMPP so it's not completely proprietary.

Agreed.

There is one email "client" that does display a pretty failure message. It's Exchange. Email administrators hate it because when a user forwards the failure message so that a problem might be fixed, the message turns out to be useless.

Hi. This is the qmail-send program at yahoo.com. I'm afraid I wasn't able to deliver your message to the following addresses. This is a permanent error; I've given up. Sorry it didn't work out. : 72.1.169.10 does not like recipient. Remote host said: 550 : Recipient address rejected: undeliverable address: unknown user: "[list name]" Giving up on 72.1.169.10.

That's a pretty good error message. Most people glaze over as soon as they see it.

Most people ignore the content of error messages as soon as they realize what they're reading. An awfully large percentage of people ignore it entirely.

I don't know what to do about people.

For an overwhelming majority of nontechnical users, no, it is not a good error message. It is filled with confusing technical things that he/she shouldn't have to care about, like HTTP status codes and IP addresses. Even if these people chose to read the whole thing, they still would have no idea what is going on. So it's a terrible error message.

"I'm afraid I wasn't able to deliver your message to the following addresses: ..."

Seems reasonable to me.

It would be reasonable to someone reading HN, since we probably are not scared off by technical jargon. Even if the instructions were spelled out completely, the plain-text formatting and numbers (IP addresses) immediately tell the average user that this is over their head and they ignore it. It's not user-friendly for real-world users.

Yeah, but UIs aren't there for reading; they're for using. If I get an undeliverable because the recipient was bad, then the UI just needs to indicate, "Bad Recipient."

People are doing energy management all the time, and that's not a bad thing. They see something that looks technical at a glance (it's got numbers and slashes and run-on sentences) and that's it. They know they're not technical, and they don't aspire to be technical, so they don't waste energy trying to comprehend the message.

I know plenty of non technical people and they all understand the concept of bounced emails. Error 238B is terrible but "can't deliver email to ben.cornwalll@test.com" and 90% of people can understand they miss spelled the guys email address. It's only after that they their eyes gloss over and their mind shuts down.

It looks OK, but it's not for a single reason: who is "I" in this sentence? This is enough to confuse most of the users.

Those would be SMTP codes. I don't see why it's hard not to understand that "your message was not delivered to this user". The traceback should probably be sent to the postmaster rather than back to the user, but that's an anachronism of the mail servers, which were built back in a time when all email users were technical.

>Recipient address rejected: undeliverable address: unknown user //

Hmm, it's okay.

It's a person masquerading as a computer masquerading as a person. This doesn't really help IMO. Why can't it be

"Hi, this is the Yahoo! email support team. You sent a message to someone but when we tried to pass it on the email address didn't work so we can't send the email for you. Here's some things to try ... technical details and the text of the email you sent are below [or attached]."

The response could also include info like that the domain is live.

That sounds like someone at Yahoo is reading my email.

Well the chances are slim because they cant read that much at a time.

Probably a computer at the CIA is reading it though.

> Hi, this is the Yahoo! email support

Immediate deletion as a phishing scam before reading further.

> "Hi, this is the Yahoo! email support team...

At this point a large number of users will reply to the email expecting to reach the Yahoo! email support team, who of course can help with every individual bounced email. They will be disappointed when this doesn't work out.

Years ago one of the VPs at a client of mine demanded that I change the wording of Qmail bounce messages because he found them snarky and offensive.

(I didn't change it.)

Every MTA used to have their own way of reporting bounces. These days, the big ones (at least by volume) seem to all obey RFC 3464[1], which defines a MIME type and required and optional fields for delivery reports. Of course, the old MTAs are still around and still delivering their bounces in a myriad of different formats. The bounce processor I work on at work has about a dozen different variants on "The message could not be delivered. There is a copy of the original message/headers below this line". Those plus the RFC 3464-style messages cover roughly 95% of what we see day to day. The other five percent are so unrecognizable that we can't process them automatically.

[1]: http://tools.ietf.org/html/rfc3464

Please do no encourage admins to replace useful debugging output with useless HTML shiny.

But, by all means, improve the text itself.

What you call debugging output frequently isn't really debugging output - it's frequently a bunch of convoluted text that says 1 of 2 things, in 90% of cases: a) target email doesn't exist; b) your email was too large, dummy

The text can and should be improved and be void of HTML shiny - the mail client itself could parse the new, slimmer, standard error text and make it pleasing.

As somebody who administers an exchange server I see dozens of different types of failure messages and they are incredibly useful for troubleshooting. Lets please not dumb them down.

As somebody who spent 13 years doing tech support I can tell you that the vast majority of users don't read error messages no matter how shiny or well written they are.

The problem isn't the failure messages per se, it's the failure messages regarding user error.

If the user has made a mistake somewhere, the client should try to understand the error message and, if it's user error (email address doesn't exist, attachment too big), it should suggest solutions to the problem in a language the user can understand.

If the system itself has an issue, the client should provide enough information to tell the user either: a) How they can resolve it b) How they can contact whoever needs to resolve it

You can have both.

Easy-to-read message at the top, debug output underneath. Simple.

That's exactly what we have today. Bounce message on top, details of conversation underneath.

HTML definitely sucks to parse but it's definitely better then plain text. At least it's a standardized format, as well that it allows for a much better interpretation in terms of visuals without convoluting the DOM for a parser. Sort of the best of both worlds, again I know using any HTML lib sucks to a degree, even with xpath, but wouldn't it still be a step forward.

Let's standardize all email error messages to HTML. At least then a standard exists even though it will suck, most standards suck in many ways. But it's a total nightmare without any.

No, no, no, a thousand times, no. HTML is not easier to parse than plain text. Plain text is easily transferred, stored, parsed (with grep, or whatever you use for regular expressions), and, as a bonus, is easy for humans to look at. HTML helps with exactly none of those things. There's a reason that Erik S. Raymond emphasizes the power of plain text in The Art of Unix Programming. There's a reason that The Pragmatic Programmer has a chapter devoted to it. That's because plain text (more specifically plain ASCII text) is the closest thing we have to a universally understood format in computing. For something like e-mail, plain text is by far preferable to HTML, XML, JSON or some other structured format. You can't know that your target will have a parser for a structured format. The only thing you can assume is that your target is capable of reading and transferring plain text.

We could settle with markdown?

Does any mail client handle the mime-type text/markdown or text/x-markdown [0] or text/x-web-markdown [1] or something?

[0] http://www.mail-archive.com/markdown-discuss@six.pairlist.ne...

[1] http://codecraft.io/2011/07/18/ubuntu-markdown-files-mime-ty...

You can use multipart/alternative emails to send HTML and Text in the same email.

This means the OP's solution can leave useful debugging in place while providing the user with the HTML shiny they've come to expect.

You missed a crucial part of the post:

    Machines can read [these error messages] just fine, though.
    Here's an idea: machines shouldn’t slap us in the face. 
    They should help us along if they fail to do our bidding.

He's not encouraging admins to replace useful debugging output. In fact, it's the opposite: he's encouraging email client designers to actually use the debugging output to let people know what went wrong. His mockup is quite clearly of an email client showing users relevant recovery options based on an underlying error message the client has parsed.

In other words, he's proposing that email clients treat MTA error messages precisely as they already handle raw headers. Full headers contain useful debugging information, but you aren't normally debugging.

No, they can't. There's no standard format for these error messages that I know of.

Maybe there should be, but that's a problem that has to be solved before the shiny UI can be placed over it.

There's no standard format for email either, but that doesn't stop it from working. Email is worse than HTML in terms of what is expected to work. You're lucky if an email is all in one character set, and even luckier if that character set is the one it says it is.

If you've written many systems that deal with email you know that you could be processing almost anything.

Sure, there might not be a 'standard' format for email, but you can return a multipart message that perserves the ugly text messages and includes an HTML message for the rest of us.

Note, that by standard I mean most emails violate any particular email RFC in some egregious way, similar to the way most HTML pages are invalid, for example the page for this comment has no less than 20 errors.

There is no standard format for the plain text part of bounce that your MUA shows to you as it's just that: plaintext part to show when MUA knows nothing better to do. RFC 3464 specifies format of additional MIME parts of message that specify what happened in machine readable format. It's interesting to note that it's exactly same mechanism as is used for sender-requested notifications for delivered email in which case these messages are processed by MUAs just fine.

Yes, the email system could be better. But for something that was designed decades ago it works surprisingly well.

Should somebody try to do some smart parsing of failure notifications to make them more readable? Sure, I don't see why not. But don't pitch the baby out with the bathwater here. The underlying system is not that bad.

Actually, the underlying system is absolutely putrid, a sentiment that pg agrees with based on his latest talk at PyCon. Mail is, unequivocally, the worst part of my daily existence and you are kidding yourself if you're calling it "not that bad".

Using pg's words, mail is a to-do list that anybody can put shit on, and I don't have any control over who puts things on it. We live in an era of technology that can do e-mail so much better, and replacing it must happen in my lifetime. Everything, from the architecture to the messages themselves, need an overhaul. The extraneous shit we bolt on to e-mail (SPF, DKIM, PGP)...it's just making it a big, unwieldy mess.

> mail is a to-do list that anybody can put shit on, and I don't have any control over who puts things on it

That's a misuse of email on the receipient's side, not on the system's side. Email is about messages, not about tasks. Although there might be more comfortable (i.e., more automatic) ways, there's nothing wrong with maintaining a TODO list that is 100% separate from the INBOX.

For example, in my TODO list I describe shortly (and in my own words) what I consider to be important. Why should I waste my time scanning through the INBOX everytime I want to check what needs to be done?

In other words: It's me who decides what gets on my TODO list, nobody else! I'm in total control because I'm a free citizen living in a democracy, and as such it's my responsibility to organize my life. In particular, to decide what to do (to-do) in my life.

If you decide to give other people access to your TODO list (e.g. by defining your INBOX to be your TODO list), you're voluntarily giving up control of an important part of your life. Nobody forces you to do that. In fact, almost every organization guideline strongly advises against that kind of nonsense. (The most famous one being the "Inbox zero" series at http://www.43folders.com/izero)

I think that part of the idea is that reading a message, replying to a message & all other things that you do with mail can be considered as "tasks".

Email isn’t the problem, the problem is the client. Here’s just a few of the things we use email for today:

  Todo list
  Social networking notifications
  Text messaging
  Planning meetings
  Planning vacation
  Discussing issues with colleagues (mailing list)
  Bill notifications
  Newsletters / grey mail
  File sharing

We do all of those different things and shove them into a single client that was designed (with minor alterations) over 30 years ago. What were they using email for then? To send “holy shit, this is cool!” messages to other neckbeards.

We should be using a separate client for all of these uses. Email is nothing more than a messaging protocol. You could launch a group texting app and use email as the delivery protocol. Your user base would instantly be every person with an internet connection. It wouldn’t matter that person A has your app and person B doesn’t, you could make it transparent to the person that has the app and pitch to the person that doesn’t.

The biggest obstacle to this is that we all use regular old email clients today and increasing the already incredible noise would not be without backlash. I think you have to start out with apps that don’t require adding to your message count (a notification app that scrapes your inbox, for example).

Meanwhile startups are trying and failing to make every service its own silo that travels over port 80. This is what is most sickening about startup culture today, no one is creative about using the stuff that has been around forever and is used by everyone.

> mail is a to-do list that anybody can put shit on, and I don't have any control over who puts things on it.

If that's your complaint, then your beef isn't with email in particular, but with the mailbox paradigm in general. But I for one find that paradigm useful, and I respectfully disagree.

It is that bad. Email works and gets stuff done, yes, but at the expense of your time, server time, storage, network throughput. It is used for much more than it should, and is long past its expiration date. Sadly, we've all come to depend so much on email and how it works today that changing it would be like suddenly having to breathe underwater. It'll take a load of time to get used to whatever comes next.

Okay, email works at the expense of my time, server time, storage and network throughput. So what's your brilliant e-mail replacement? Heck, pg himself has talked about these very same issues and has noted that YCombinator would jump at funding any startup that came up with a credible replacement to e-mail.

E-mail is like democracy. It's the worst messaging system, except for all the others.

The underlying problem is that SMTP only has a few error codes, and so there is no way the MUA can understand (in a standard way) why the MTA rejected the message. HTTP ended up having richer errors, and so clients can provide nice messages if they feel like it. (IE does this for 404s and 500s, though it's not a nice message and it seems to upset most web developers.)

And an HTTP server can return a custom body for most errors, so the server can provide even more info if it wants.

SMTP MTAs can also "provide more info if it wants", but that's what the article considers a problem. What the author of the article wants is a way to dumb down error messages (or, less cynically, automatically correct errors), which is not easy with free-form responses. Imagine a web page that returns a page that says "404 Not found" with a 200 status code. Brokenness ensues.

What annoys me about SMTP is that it's poorly implemented. Have you tried sending a message to the postmaster at any large firm in recent years? Chances are nobody even reads that. I clicked on the link to submit a tip to my local news station last week; later I got a reply telling me that those mailboxes didn't exist. The largest broadcast media organization in the Bay Area can't even operate its own wesbite properly. So while SMTP is great on a technical level, it's only as good as the people who administrate it, and system administration is not as valued as it used to be because it's too labor-intensive.

Reminds me of the time I filled in a form on Microsoft.com and submitted it, I think it got sent to an internal nonexistant address, was autoforwarded, and then bounced to my email, containing the content of my original message and a reassurance to the internal person that their email address wouldn't be revealed to the outside world. It was.

I suggest returning the error in a .doc file so that people can read. At least some.

And yes I get you idea, this is actually a RFC http://tools.ietf.org/html/rfc6522

And the MUA (Mail Program) should just deal with it properly. But then you have all these Bloatus Notes, M-SExchange and other broken system in the wild...

.doc file? Why use a proprietary binary format when a suitably standardized plain text format would work just fine? I don't even open .doc files from unknown senders.

You obviously missed the sarcasm.

Sorry, sarcasm doesn't always come across clearly on the Internet.

Neither do email bounces!

I generally prefer email to any other form of electronic communication. It's sortable, archivable, easily formed into lists, decentralized, encryptable, supports attaching images, well-known, implemented in a number of protocols...

While you could implement a shiny-for-the-laypeople status code parser, email as a system works unbelievably well.

> archivable, ... decentralized, encryptable

This. I shudder every time someone mentions replacing e-mail attachments with Dropbox/iCloud/etc., which somebody already did in this thread. With e-mail, I am beholden to nobody; and if I pay some company to handle my e-mail, it's only because I voluntarily chose to do so. I could drop that company at any time, fire up my own mail server, and have a 5TB mailbox with 10, 20, 30 years of searchable history if I wanted to. I can also access it with a whole bunch of free and open-source software, from ESR's original fetchmail all the way to the latest alpha of Thunderbird.

Every other messaging system that I've heard about, on the other hand, is either held hostage to one company's proprietary platform or clearly not intended for people who need to look up conversations they had 10 years ago.

So please, folks, by all means fix what's currently wrong with e-mail. But please think twice before suggesting that everyone replace it wholesale with some proprietary cloud-based TODO app.

I was one who suggested Dropbox as a replacement for attachments (although it was really just an example, I don't even use Dropbox personally), and the point I was trying to make is that you can either have flexibility, or you can have standardization. The OP is suggesting we somehow standardize email error messages, but this doesn't seem possible while email remains a free-form, very simple protocol, which gets used for an infinite number of different purposes by different people.

"Fix what's currently wrong with email" is so overly vague as to be basically meaningless. Different people have completely different ideas about what is "wrong" with email, and it is such an entrenched technology that you will almost certainly never be able to make structural changes to it unless the problem set it addresses is radically diminished.

I don't see any reason why something like SMTP error messages can't be standardized if several major e-mail software vendors and related groups get together and agree to implement, over a period of 5-10 years, an extension to SMTP. Sure, it might not be as clean as rewriting the protocol from scratch, but it would work for all practical purposes. A lot of features of HTTP that we take for granted, like cookies and anti-clickjacking headers, were added through a similar process. The problem is political and economic, not technical.

I also don't see anything wrong with having a free-form, vendor-neutral, decentralized messaging protocol that can be used for a million different purposes. As long as the underlying protocol itself is reliable (e.g. it has proper error handling) and extensible (e.g. you can add custom headers and MIME types that other software won't mess with), innovators are free to implement whatever functionality they want on top of that protocol. Heck, Richard Stallman uses e-mail to browse the web! If you restrict the protocol to what you think should be its only purpose, you're unnecessarily limiting what other people can do with it. That would be like restricting HTTP to hypertext documents only, no AJAX, no API calls, no streaming.

IMO, e-mail doesn't need to become a TODO list or notification system. When I say "fix it", I mean we should clean up old cruft from the protocol (e.g. make it 8-bit clean) and make it extensible so that it is not too difficult for interested parties to implement a TODO list, notification system, etc. on top of it if they want to. People will obviously disagree about how exactly that should be done, but people disagree about everything anyway.

So please, folks, by all means fix what's currently wrong with e-mail.

I wonder if someone actually compiled a list of e-mail shortcomings? I tried to do a search but keyword space is filled with regular users searching for standard e-mail problems.

You want to strip out all of the useful information needed for debugging mail problems, and replace it with web 2.0 bullshit ajax and lickable icons that won't render in most mail clients?

Sorry you couldn't send your attachment but go back to your blog and leave us alone.

They are suggesting that the client generate those UIs based on the message in the email, not that the message have that UI in it.

All mail systems have different rules for what is "acceptable", what is perceived as "spam", etc. One mail server may allow attachments up to 10MB, another 20MB. Some mail server somewhere may allow attachments up to 50MB but will not allow attachments with a .exe extension, and due to the vast and wide uses of email it actually is important for this flexibility to be available.

Given the amount of variablility around mail handling, relays, spam filtering, etc, I'm not sure that it's entirely feasible to "standardise" error or bounceback messages. RFC 3464-style messages get us part way there but it just doesn't approach something as "nice" as HTTP error codes (which themselves still aren't handled in a standard way by web browsers, they are handled on a case by case basis, if at all, by web developers).

I think this problem is only going to approach being solved when some of the more edge-case uses of email (such as attaching/sending files) are deprecated by more targeted, more efficient services. For example, lets say everyone started using Dropbox or YouSendIt to send files instead of attaching them in emails. And alongside that perhaps a separate protocol such as iCal or similar takes over the "meeting arrangement" space. Eventually the usefulness of email may be narrowed down into a small enough band of "send and forget" messaging that someone can actually, finally, come up with a better, more efficient and usable alternative.

Until that happens, or something else equally radical, email is something we're just stuck with, worts and all.

>One mail server may allow attachments up to 10MB, another 20MB [...] not allow attachments with a .exe extension, [...] important for this flexibility to be available.

[...] I'm not sure that it's entirely feasible to "standardise" error or bounceback messages. //

Well couldn't we use something akin to SPF - ie a TXT field in the domain's DNS record with a shorthand for the acceptable receiving protocol in operation, kind of an RPF.

So you'd have something like:

alicious.com TXT v=rpf1 deny-ext:exe,reg,dll max-size:15MB req-field:sender ip4:xxx.xxx.xxx.xxx/16 req-format:txtonly req-other:spf ...

would tell the MTA (or the forwarding relay) that this address will likely reject emails containing exe, .reg, .dll files; emails over 15MB; emails without a sender field; emails apparently originating from servers outside the given address range; emails not in text format; emails from domains without a SPF; ...

Whoops got carried away there.

What would be the problems with this sort of scheme?

[FWIW there's already a milter called RPF but that's not what I'm proposing here I'm talking instead about a policy to give to a sender to tell the sender what you intend to accept so that they can choose to match your requirements and avoid having the mail rejected. Obviously an MTA/MUA can choose to receive mail that doesn't adhere to the policy if they wish.]

From a technical standpoint something like that could work to a limited extent (enough to be helpful in some situations, when both parties are using modern, up to date MTAs / mail clients). It still doesn't fix the error message from a bounce, it's more like a hint to the email author to say "this email might not be accepted for the following reasons: x, y and z".

Ultimately though, the inertia to implement such a thing on existing MTAs is somewhere close to the difficulty of implementing IPv6. Except the need for IPv6 is verging on dire, and not many people seem to be rushing to fix that one. How will we convince people to act with any expediency over something that is, at most, an inconvenience?

  > Except the need for IPv6 is verging on dire

But we just got all of Iran's blocks back because they are removing themselves from the Internet, right?

The MUA would do the work up front, query the TXT field for the rpf and tell yiu up front "I can't do that, $user wont accept attachments over 15MB". Then as the OP suggested the MUA can offer to zip, upload and link, or whatever.

Status codes:

http://www.iana.org/assignments/smtp-enhanced-status-codes/s...

http://tools.ietf.org/html/rfc3463

I'm guessing most e-mail client providers just haven't found a reason to prettify the responses. If anyone is up to the task, then can you explain PGP/GPG so the entire world can also understand and use that?

Client side parsing won't be as reliable as you expect.

SMTP bounce codes are notorious for changing on a per domain basis whenever someone feels like it.

I work with StrongMail supporting their Email Marketing Platform and their solution is to have a central database that is updated regularly for all major ISPs to basically be able to have a conversion process between the SMTP bounce codes and the 30 classifications for mail failures StrongMail groups failures by (such as User not existing, blocked due to spam complaint, service temporarily offline, etc). This conversion process has to be updated regularly to keep up with changes.

Server side generation of multi-part NDR reports including HTML email could be a solution, i am not sure if it violates the rfc to send NDRs in multi-part, it could perhaps need an amendment.

But if you want to show this then your in luck since all the major SMTP servers besides exchange are open source, go produce a patch set for postfix, sendmail and qmail.

My main problem is that spammers hacked the "sender unknown" error message long ago and use it to spam me (they send mails pretending to be me, so the error message gets delivered to me). So I had to filter them all away.

I wish Thunderbird's spam filter would check if I actually sent the message the error message refers to.

I really like this idea.

You can apply this to every transactional email, not just to bounce messages.

Off the top of my head:

- Vacation auto-replies

- Receipts

- Error reports (ok, that might be a bit coding specific)

Part of this would have to do with the fact that mail servers are never really updated with new "features."

- Sendmail was first released in 1983 initially. - Postfix was released in 1997 initially.

Though both are getting maintenance updates, and new processing features, none have improved the actual error path system. Maybe someone should write a better MTA?

It is a perfectly reasonable error message that mail clients should try to beautify. The message itself is absolutely necessary, but should perhaps be cleaned up by the client. All of the calls to write a new "protocol" by the posters on that blog make me wonder if they even understand what the word means.

This could be a good idea to improve bounce management from the client, if only for the reason that 80-90% of the world population simply don't understand english at all -- all bounces AFAIK are always in english, for a start.

Its old, it doesn't look nice. But it is surprisingly reliable.

Coming up with ideas to fix Email in 2012 is like coming up with ideas to fix IM - why bother?

Why not? If any of them turn out to be good, then we've improved something, haven't we?

Reply on news.ycombinator.com