Why it’s so hard to innovate in the e-mail space (2014)
medium.com
medium.com
I've considered merging mail and messaging, so that email protocols are used to do the things people do with instant messaging, Telegram and such. The first step is to get away from queuing mail.
A first step is a new kind of mail forwarder, one for machines that don't have local mailboxes (which is most non-mail servers). It acts as an SMTP server and client. When an email comes in, and the sender reaches the end of the DATA section, the forwarder checks its forwarding list to see where this mail is supposed to go. It opens a SMTP connection to the destination and sends the email. The incoming SMTP connection is held open while this is in progress. Any problems are reported back to the sender immediately with an SMTP status code. There are no bounce emails, no queuing, and no possibility of joe-jobs. All mail is handled immediately or rejected immediately. (This only applies to single-address messages, and only to "To" addresses. Everything else is bulk.)
The next step is an IMAP server which has the same immediate orientation. When it gets an incoming email, it sends a push notification to any connected clients, while holding the incoming connection open. If at least one client is responding, a SMTP success status is returned. Otherwise, you get some status such as 251 (User Not Local, Will Forward.)
Then, a threaded email client. It would help to have the convention that an email with subject but no text, text but no subject, or text the same as subject is displayed like an instant message.
All of these can be done independently, and are backwards-compatible. If you have all of them, email is equivalent to instant messaging - you can tell if it was delivered, it goes through immediately, and it works like a conversation.
Of course, there's no big-money startup in this.
Just to be sure: you are aware that that is a standard feature of email? There are headers that provide threading information, at least if you use non-crappy user agents.
> It would help to have the convention that an email with subject but no text, text but no subject, or text the same as subject is displayed like an instant message.
No, that would be a terrible idea, you never ever should redefine the semantics of existing messages. If you want to encode a new type of message, use metadata to mark it as such, that is what those extensibility mechanisms are for. Add a header to indicate that the sender is following some new convention, or add a multipart/alternative MIME part with a new MIME type that encodes such messages, with a fallback text/plain part for clients that don't understand the new convention. But whatever you do, never encode new semantics in a form that already has an established decoding/interpretation/meaning, and also always do it in a way that provides as much of the same functionality as possible with software that doesn't know about your new invention.
Plus the behavioral differences between email and instant message didn't blend well: was I going to wait in a mixed email/instant message conversation for a response, or was I going to move on to the other conversations in my inbox and send a reply or archive each so that I could move on to doing other things? And if I wanted to temporarily leave an instant-message-like conversation so that I could deal with some more email-like conversations, would the person I was chatting with know whether or not I would return?
Eventually I stepped back to look at what additional value a user gained by blurring instant message and email in this way. Fundamentally they're both asynchronous text-based communication, and the things you can actually do with them aren't strikingly different. The biggest difference is in the expectations that a user has for each.
Anyway I still wanted to "innovate in the email space", so I decided to go a more extreme route, and I made my own pull-based protocol (superficial similarities to IM2000 - http://cr.yp.to/im2000.html) that retains backwards compatibility with email while adding abilities like stateful plugins (JavaScript apps/widgets) that can share data through the new protocol, editing messages after they're sent (SMTP recipients receive the new version after the update), and inline replies to parts of messages to enable nested conversations. If you're interested, I posted it earlier this afternoon here (https://news.ycombinator.com/item?id=10693535).
Just to be sure: You know that that is really easy with email as it existed until about 20 years ago, including BBS networks, usenet, etc., and that it is still trivial with good email clients? That was a solved problem that made email really efficient, until some clueless developers and users came along and disinvented (is that a word?) the solution in order to make email easy to use (or so they claimed).
Traditionally, if you chose to reply to an email, the mail client would put the mail you were responding to into your editor, with > marks in front of every line. The only socially acceptable way[1] to write your answer was to insert your responses in between those quoted lines, without > marks, and to delete everything of the original mail that was irrelevant to establish the context for the reply.
That's how easy it is.
Usually, a good mail client would also color quoted lines so that it's easy to see and read only the response lines in a received email, so you'd only have to read the quoted text if you weren't sure about the context.
Clueless developers and users around the turn of the millenium then somehow didn't understand the purpose of providing you with a quoted version of the email you replied to, and started putting the reply above the quoted mail, thus ending up sending a copy of the mail they just received back to the person they received it from ... and nobody seemed to notice how idiotic an idea that actually is, and so it became sortof the new norm in large parts of the internet to attach large blobs of completely useless and often close to unreadable text to every mail, up to the point where some people even invented justifications for the behaviour that mostly tend to describe how this "feature" allows them to work around the lack of proper thread handling in their mail clients.
Obviously, easy quoting traditionally relies on plaintext only mails with fixed (short) line length, but format=flowed encoding has since been invented (the original RFC is from 1999) to accomodate screens and windows of variable width while maintaining backwards compatibility with old clients.
> Clueless developers and users around the turn of the millenium then somehow didn't understand the purpose of providing you with a quoted version of the email you replied to, and started putting the reply above the quoted mail, thus ending up sending a copy of the mail they just received back to the person they received it from ... and nobody seemed to notice how idiotic an idea that actually is, and so it became sortof the new norm in large parts of the internet to attach large blobs of completely useless and often close to unreadable text to every mail, up to the point where some people even invented justifications for the behaviour that mostly tend to describe how this "feature" allows them to work around the lack of proper thread handling in their mail clients.
Yes, I think the rationale must run something like "What if user A deletes the entire conversation they were having with user B, and user B replies to the deleted conversation, but user A doesn't know what the reply is about because they deleted the entire conversation, have a terrible memory, and are too embarrassed to ask user B to explain what's going on? Let's just send the entire conversation as quoted text every time."
Well, if it's supposed to be compatible with all the HTML crap out there, and even most MUAs' broken encoding of plain text email, it might be hard, but in principle, I'd think this should actually be rather simple:
Simply specify an algorithm that defines how to segment a text/plain body into paragraphs and how to thus generate MIME object-scope IDs per paragraph (by simply numbering them) and use that same structure to provide UI elements per paragraph. Then, if the user writes a per-paragraph reply, generate a multipart/alternative body, with one text/plain part in traditional quoting style, optionally with format=flowed, and one alternative application/fancy-paragraph-foo part which contains in some sort of serialization format references to the message+paragraph IDs, that paragraph's text, and the corresponding reply text. This way, a receiving MUA that doesn't support application/fancy-paragraph-foo will display an ordinary quoted plaintext body, and also, if a communication partner uses different MUAs, you can still reply to an email sent using an MUA without application/fancy-paragraph-foo support and still have the feature work if the reply is then read in an MUA that does understand it. Also, including the text of the original paragraph makes sure you can still display a message sensibly when the displaying MUA doesn't have the mail it's in reply to - you'd just have to be careful to not misrepresent the authorship information as reliable.
> What if user A deletes the entire conversation they were having with user B, and user B replies to the deleted conversation [...]
* g * (is there any way to escape asterisks to their literal meaning?!)
Well, the rationale I've heard is that it's easier to make sure new participants in a conversation can read up on what has happened so far ... which obviously would be solved far better by attaching a thread of message/rfc822 attachments when adding a new participant that the recipient could navigate using their MUA's usual UI instead of having to read a close to unreadable unstructured mess of emails.
1. Consistency. Okay, this is always going to be a problem, since rendering anything more than plain text in emails is a minefield that makes the bad old days of Internet Explorer vs Netscape Navigator look like a utopia. But people still expect their Amazon/eBay/Google/whatever account/confirmation emails to look decent, so you have to figure out a basic HTML parser as well. And then try and make it somewhat work with people who are designing their messages with stuff like Outlook's Microsoft Word engine in mind.
2. Spam. I'm surprised this wasn't really mentioned in the article, but it's arguably the biggest issue any email provider, client developer or email server software maker has to deal with. You have to make sure your software can figure out what messages are unwanted, then either send them to the spam folder or delete them. Most startups don't have to figure that out, since their systems are closed and spam prevention can be done on sign up. Not so much if you're building an email client where any Tom, Dick or Harry can send messages to anyone with no real way to verify if they're human or not.
So yeah, few other issues.
Some of the reasons why I think it's hard:
* I don't think security was a big deal to many of the mail applications when they were being built. I tried and gave up so many times trying to find a strong hashing algorithm that both dovecot and ruby supported. There was practically no documentation on this and it felt like I was venturing into territory never before seen.
* Parsing emails to be display correctly seems impossible. Sometimes emails have "<br>" in them and sometimes they have "\n". But what if the "\n" doesn't mean newline but it's just what someone wrote in the email? They come in a variety of mime types and formats. It sometimes seems impossible to do this.
* Not everyone uses the RFC standards! I thought the RFC said that a subject can only be 78 chars long. Yet I get emails all day long that go way beyond this which cause major problems in my code. AM I THE ONLY ONE AROUND HERE THAT CARES ABOUT RFCS?
Strict RFC implementation, email normalization. Bad emails won't kill your email client 30 years from now.
One of those should have an HTML MIME type, one should have a plaintext MIME type. The decoding is specified in detail in RFCs and W3C recommendations, please follow those rather than try to implement this using try and error.
> Not everyone uses the RFC standards!
That is unfortunately true, ...
> I thought the RFC said that a subject can only be 78 chars long.
... and one of the major reasons why is because people think but don't read. There exist all kinds of crazy myths about the content of RFCs, which is how all those broken implementations arise, this seems to be one of them--but feel free to point out which RFC says this where, in case I really have missed it.
We are a startup that just launched (https://tmail21.com) that aims to rethink the discussion itself. In our view, one of the major problems of e-mail (and chat for that matter) is that the discussions are not goal-oriented. So, discussions/threads can meander around with no outcome or accountability.
So, we enhance the notion of discussions to be goal-oriented.
Now, once one has the notion of goal-oriented discussions, a natural next step is to evolve a goal-oriented discussion into a Lean Process (which is basically a goal-oriented discussion with more structure) . Examples of a Lean Process might be a Blog Article process, a Product Deployment Process, a Feature Design Process, an Issue Escalation Process etc.
We've done all this in an email-LIKE interface. I guess we'll leave it to others to decide whether this constitutes innovation in the e-mail space :)
I actually propose instead of re-inventing email, create addons to extend what email message can be used/read/interacted with. For example, high school teaching today has been reliant on gmail for offline homework assignment and discuss between teacher and kids. There are some tools in that space aggressively exploited by schools to conduct their "innovative" teaching efforts.
Gmail, or microsoft outlook would be the best go-to destination for the majority of who can afford an in-house IT team, or shadow IT in large enterprise who's tired of the lack of evolution from SAP, IBM etc.
Still, Gmail and outlook are slow in adopting or providing email as an open extendable platform for BPM application purpose.
Then there are everyday use of email other than BPM, notification, online purchase receipt, offline discussion, even transmitting files, photo etc. via email are somewhat common with email client.
The longevity of email is exactly the lack of rigor of the intended use of it, BPM or not, email is flexible to conduct any discussion to an open goal, or without a goal.
To understand email better, one has to compare email to SMS, and social media along with how mass used these tools.
TMail21 does other email-like things like transmitting files, notifications, full search etc.
It also does interesting things like giving every thread a unique tracking number, which allows it to be tied into the broader enterprise ecosystem. The idea is that just like the URL transformed the Internet (i.e. the web), Tracking numbers can transform 'email'. Now, TMails can be referred to from Chat, Voice, IM, Apps, Spreadsheets etc.
Another thing we do that is very hard to do in email is things like Certified Mail, Certified Forwards, Transactional Guarantees, Certified Diffs, Non-repudiable audit trails etc.
All of these capabilities are however means to an end to make TMail21 the first true BPM tool for regular business users (rather than for coders and 'process analysts)
I guess it depends on what you mean by "primarily" but my inbox is overwhelmingly notifications and announcements, not discussions. There's remarkably little of my inbox dedicated to actually talking to people.
On the pure notification front, we have friendly tracking numbers (so that notifications can be referred to from other places like spreadsheets, chat, apps, voice etc.). These tracking numbers look something like 124-1234-1234. We also support Certified Mail, Certified Forwards, Certified Read Acks etc.
Having said this, we currently support outbound notifications (from TMail to TMail/EMail). On the inbound side we support TMail to TMail. We do not currently support EMail to TMail notifications although we may add this if we see sufficient demand.
* A 100x improvement over current email is needed for a switch.
* It has to be 100x cheaper than the current product.
While you can't multiply by 0 and I meant that facetiously, users won't pay for the software or only niche consumers. Which is why email providers that get paid focus on things like security, collaboration and and backup. These products are essentially 90% other things and 10% email. examples:
* dropbox * slack * telegram * silent circle * sms
obviously I am not going to name everything in the space. I can't think of a company that has email at it's core. maybe 'hushmail' or some small encrypted mail providers but what profitable business is at it's core an email provider
I think the focus on a particular technology (SMTP) rather than a service (messaging) is why email clients don't advance. Bizarrely, three other transports often are integrated into email clients: NNTP, voicemail, and fax. I can only guess that it's because of a completely arbitrary reason: They are older.
Probably chat clients should also be integrated. I should be able to send my contacts an instant or time-shifted message.
I look at iMessage and I think it works pretty perfectly, SMS when no internet or to a non apple user, iMessage whenever possible. What I want is for whatsapp (and I suppose facebook and plausibly email) to do the same all in my single messenger application.
But. Ignoring the need to get the tech owners to consent to this new "trillian", I'd also have to trust the app itself. Good luck making money on that.
I don't understand: Receiving messages on any of the transports is done every day and billions of platforms. After receiving, loading the messages into a common database, including mapping metadata such as sender, date, etc., seems relatively simple. Once in the database, sorting and filtering are simple.
What am I missing?
I agree with this completely--so much so that I decided to make my own communication protocol (and client-server implementation) that would do whatever I wanted and still be backwards compatible with email. You can see the results here: https://github.com/Zombition/nsmtp-monolith. So clearly it's not impossible.
> What am I missing?
The biggest difficulty isn't from pulling out metadata and mapping it to whatever you store locally. The specifications of different protocols determine the user behaviors that are possible with each protocol, and mapping behaviors between different protocols in a way that makes sense to end users is much more difficult if you want to retain quality communication with users who are not using your software to pull everything into one place. What I found from implementing a new protocol that aimed to be backwards compatible with SMTP was that maintaining backwards compatibility with SMTP guided 99% of my decisions for making the new protocol. If I wanted to add a feature, but it would be confusing for end users who might be trying to communicate with regular SMTP users (or the SMTP users that they were trying to communicate with), the only practical solution is to axe the feature or rehash it in a way that would be less confusing for communication between the two.
In practice this is quite difficult, because you have to balance awareness of both protocols with awareness of how data sent through one can be meaningfully translated to the other, and while you're considering those things, you also have to consider what the user experience is going to be like for people on both ends of the software. If you're trying to maintain compatibility with a widespread protocol like SMTP, there may be an enormous number of different client interfaces that can add a significant amount of variation to the user experience. It's a bit like unfolding a fractal.
Because email uniquely among communication protocols allows sending a message without any previous agreement from the recipient, will always be susceptible to spam (and getting rid of that feature would destroy emails benefits).
Plus people want to filter differently. People deal with their inboxes uniquely.
> Plus people want to filter differently. People deal with their inboxes uniquely.
That issue affects every messaging client's filtering. How does it affect a client any differently if it combines, for example, SMTP and SMS?
There's a huge long history of very smart people (not even including myself in that list - I've worked with some people MUCH smarter than myself on this) working on this problem, including the very creator of this site, pg himself, and even he doesn't consider this a solved problem. People are still paid well to combat spam every day on behalf of demanding customers (and I'm grateful I'm not one of them any more) and it will always require new techniques and new input. Hopefully you will contribute too.
Mailmate is my fav OS X client that focues more on power user features but it has a boring classic interface which doesn't bother me at all. Despite having integration support with SMIME/PGP, Markdown reply, powerful search features, and so on that makes me productive, the only thing I hear when I recommend this to other people is that its UI is ugly. :|
A better title would be: Why is it so hard for users to use a good email client?
Some of E-mail users are very conservative and specific about how they handle their E-mail. (Some of which are very risky behavior, which I actually tried to correct by explaining why that's bad idea, but ultmately given up...)
There are demography between novice and expert. Novices tend to not care about changes so much; some even won't notice changes, as long as features are somehow accessible. Experts may have some particular tastes, but they'd be quick to find solutions to change things around when they have to. But people in between tend to be very vocal about any changes and they can't (or not willing) to find and accept "solutions" -- they will be the first one to complain when UI elements like buttons shift around the screen. Unfortunately, as universal E-mail gets, a lot of E-mail users fall under category. (Though, in different ways, "experts" can be very particular about how they handle things can be very stubborn when there's no good reason presented to change -- but it's a bit different context.)
I can't even imagine how tough it will be to migrate some of people I know to anything other than what they are used to.
Unibox groups emails by person. It's different, but it really helps you staying organized.
Imagine being able to publish your email address online without worrying about spam; even in your profile you've been forced to obfuscate your own a little. Or to know that every message you send can only be read by the recipient and nobody in-between - without having to explain PGP to non-techs. Or being able to send a 50-100MB file without resorting to third party or public hosting - any of those features would be a bonus, in my view.
Not broken, but can be improved, don't you think?
I can't see that encryption will ever work end-to-end as we want. People are too stupid to learn most things. Technical solutions rely on big corporations that we trust to do the Right Thing(TM). The Right Thing(TM) is impossible from US companies and probably impossible from European companies.
Attachments. I don't remember the last time I sent a large one. As far as I know most of the size limitations are artificially implemented by companies that don't want to spend that much on storage. Other limitations such as Gmail's block on executables is because people are stupid and will run everything they get sent.
People will never not be stupid so we can never have nice things.
Google Wave was an attempt, but the chemistry wasn't right. Someone one will find the right mix...
Because email is dying and is already mostly optimized.
Email is dying. Its not dead, but it is no longer the go-forward internet communication platform.
What percentage of new internet users do not have any email accounts? I haven't been able to find any reliable data.
800 Million Messengers users as phone number only
20+ Apps with 100m + users as phone number only.
Together these PN based winners outstrip the combined usage of almost all web apps. I mean you are all downvoting me -- but its kind of funny how out of the loop the thinking is here.
In the author profile at the very bottom of the page, in small letters, there is a URL (but not a link) to
and the email client the author works on is apparently called Front. But why should her readers ever care about that?