Email sucks. This is how you fix it. | Hacker News Reader

Email sucks. This is how you fix it.

joncalhoun.posterous.com

54 pointsjoncalhoun65 comments

The problem is of course real, the root cause is incorrectly identified. The problem isn't that email isn't integrated with anything else, it's that it can't be integrated with anything else. And the reason for that is email is flat text (or gussied-up flat text in semi-HTML), and nothing else on your system can understand it.

If your bug request came wrapped in some metadata about how it's a bug request, then it would be easy to further integrate that into all sorts of things.

OK, that's great. So why doesn't that happen? Because defining such metadata formats is a non-trivial exercise, and even more difficult is getting everyone to use those formats. So it turns out this is really just another specific instance of the Great Semantic Divide, namely, how the heck can we actually share semantics in any reasonable manner on large scales? It's the exact same reason the Semantic Web hasn't happened.

Further, even if that hurdle were to be crossed, you'd have the problem that somebody, somewhere has to fill out the definition of a "bug report" before it can be wrapped in the relevant metadata. Right now this step occurs when you take the email and translate it into your todo system (or bug tracker or whatever). In this perfect world this supposedly is done by the sender... but that, too, is fraught with problems, because that takes a certain level of skill on the sender side and now you've lost the aspect of email where you just click "New Email", bash on the keyboard for a bit, and hit "Send", and it mostly works. There's also some important context issues, which is that even if the bug or task was perfectly filled out by the sender from their point of view you'll still need to tweak it because you have a different perspective on things. (Which is just the Great Semantic Divide popping up again.)

In summation, you're not just screwed, you're comprehensively screwed.

Metadata is a bit of a strawman.

There is sufficient information in the text to determine that the email is for a feature request - otherwise the author wouldn't know it was one.

What's needed is an intelligent engine for processing email. Improving email is an AI problem not one which requires an email to contain more information.

One difficulty I see is that developing a system for intelligently processing email is not necessarily the sort of thing a startup can tackle - the required data set for testing and validation is too large for a small company to obtain naturally and test under real world conditions. It's the sort of problem that lends itself to an "eat your own dogfood" approach - and thus justifiable at an existing large company.

My gut tells me that outside the corporate world, this wants to be a desktop application, not a web based one. The intelligent engine needs to know about the files and documents on my computer to make better sense of the contents of a message - I don't want to have to share all that with a web service and I don't want to enter a bunch of metadata to create rules.

The software should know it's a bug report because I write a lot of code for the product, have other bug reports on my computer, and posted ten bug fixes to the code repository. not because I spent four days writing a CLIPS program to handle bug reports.

It's a nasty problem if one tries to just grep it into submission. The answer to the email problem is a poem, not an equation.

"What's needed is an intelligent engine for processing email"

They already exist; the normal name for such a thing is "secretary" =D

I don't know about the inductive logic behind your "humans can solve this problem, therefore it is solvable by a computer". To create an algorithm that could classify email as well as a human, you'd need to create an algorithm as advanced as a human.

Furthermore it would not help much. You still would have to monitor the algorithm decisions. Otherwise you loose control over your work.

Never ever I would allow an algorithm to decide what is a bug and what is a feature request.

Your argument changes the premise. The email problem persists because it is hard (or impossible) to solve by humans.

Solving the email problem, in so far as it depends upon classifying email, must classify email significantly better than a human. One of the things computers do better than humans is working tirelessly, and at its heart the email problem is Sisyphean.

Google serves personalized advertising. Facebook customizes your feed. Your grocery store prints coupons at checkout.

All without homunculi.

So why not simply begin with a mail client plugin for that?

I imagine something like that:

When the user clicks "New Email" and types in the mail address the mail client would automatically connect to that domain and retrieve a list of the required metadata. Or you'd need to send an email that nobody except your server will ever see. But that may be too slow.

Then in the mail client you will get a notification (hopefully not an annoying popup) that the server supports "semantic mails" and that when using it you make the lives of people easier and probably get an answer faster.

What types of mail does it support? (Bugreport, Task, ...). That's the first thing you chose.

What fields are required? What fields are optional? What data type does each field take? You then can just draw appropriate widgets.

That would be a basic idea. Probably needs some tweaking, but doesn't seem that bad to me.

I agree some metadata would make life easier, but we are trying to simplify the process of extracting the data from an email and getting it where it needs to be quickly and easily.

We are experimenting with different ways to do make this easy for the user, which is why we need feedback from users who have a different background than our own.

Sadly you can't write email for programmers and expect everyone else to understand it :(

While I felt it defocused my post a bit to say this, it is true that you can make some things easier. In particular the Great Semantic Divide doesn't apply to yourself talking to yourself, so anything you do with custom scripts or code to make your particular life easier does indeed work. But I got to my point by reading about your specific complaints in your post. Making it easier to handle the flow mitigates the complaints in your post, but doesn't actually eliminate them. But mitigation is possible; life isn't hopeless.

I have seen a couple of cases of people (or machines) saying "Send an email to <address> with <some word> in the subject line." That seems easy for a human to carry out and for a machine to recognize. This seems to address your objection:

In this perfect world this supposedly is done by the sender... but that, too, is fraught with problems, because that takes a certain level of skill on the sender side and now you've lost the aspect of email where you just click "New Email", bash on the keyboard for a bit, and hit "Send"

It is indeed possible to define little machine interfaces like this and for users to learn and use them. A more complicated interface would of course be more difficult; maybe you would only try to ask technical users to do that, and have a machine send back "Did not parse correctly" if it did not parse correctly. If it becomes common and useful enough, perhaps you could make email applications that provided buttons or whatever to machine-generate machine-readable instructions, and big-business clients could instruct their employees that deal with your machine to use such an application. And maybe you could bug Gmail developers into implementing it as an optional feature, disabled by default--maybe in that "Gmail Labs" thing, although this has a different source. If that doesn't work, then maybe you could duplicate all of Gmail's nice functionality so that those employees feel fine using your email application. Eventually you might get something that could be called a standard out of this, but you would start with just a small use case.

It does seem likely that expecting an interface to work with no prior exchange of instructions ("put word in subject") between the receiver and the sender is, well, hubris. But I suspect you can get a fair amount done with tiny interfaces that are easy to learn. See also:

http://unqualified-reservations.blogspot.com/2009/07/wolfram...

You know, I hear that email sucks all of the time. It's all bunk.

Email has been around in it's present form for a good long time now. That staying power is a good example of why it is a "good enough" solution for just about anything.

It's like anything else... A general purpose tool will address 80% of your needs. The other 20% works best with something more specialized.

Like PHP.

Email is awesome. It's resilient, reliable, fast, flexible, open, and well documented. Stop fixing it.

I totally agree with you, except for the last sentence. Email is great, yes. However, please continue to innovate, because there could be something more awesome than email which would render email redundant.

continuing to innovate is good, yes, but all the 'innovations' so far seem to come at the expense of one of the good bits i mentioned. if your product makes email more awesome but breaks interoperability, it's not awesome.

Perhaps I should have said "email management sucks" but that isn't quite as succinct in the title of a blog post.

I am really focusing on ways to turn your email into something that is easy to act on.

Email is not designed to be a message queue. Applications (and business processes) should not be built on email. You have bugs in a program? Get a bug tracker. You need to handle your accounting? Get a web app for that. Email is not broken because it doesn't do enough. Email is broken because people use it for too much. Do you need to view every process that you interact with in your life in step-by-step flat text messages? No. While the usefulness of aggregating all of those messages in one place may be debatable, I don't think that it belongs in your email account. Email should be treated as an asynchronous communication medium for people only.

A logical extension of this idea would be the ability to send tasks. That is, say that I'm a teacher and I want to send out an assignment that's due by 6pm the next day to all my students. I should be able to send out a to-do task attached to my email telling my students when the assignment is due, and each student should be able to just click "add to my tasks" and have it integrate with their existing tasks without each student having to manually create a task.

This is actually something that Microsoft Outlook does reasonably well (and it has existed for quite a long time). You can mail around tasks, meeting requests, etc., and they can be automatically added to the recipients' task lists or calendars. Unfortunately, everyone in the organization has to be forced to use Outlook for this kind of stuff to work seamlessly.

This is something we are working on actually. I looked at it as an HR task that everyone in the company has to complete - you know its going to be a task so why not just send the task rather than the email?

With all due respect, email is not the problem. You are.

Your problems are not unique. Email is a great tool for delegation (to humans), and you should learn to do that.

I trust that my co-workers and employees will take care of the tasks I assign them. That lets me delegate and forget it.

Accounting? Pay this bill. Thanks.

Jr. Dev? File this as a bug. Thanks.

Co-founder? PG called, wants to invest. Meet him for lunch?

Technology is not always the answer. Many times behavior modification is the better solution.

True, but email really does suck too.

From pg, not long ago, the essay "Frighteningly Ambitious Startup Ideas":

http://paulgraham.com/ambitious.html

"2. Replace Email

"Email was not designed to be used the way we use it now. Email is not a messaging protocol. It's a todo list. Or rather, my inbox is a todo list, and email is the way things get onto it. But it is a disastrously bad todo list."

It would be nice to put the burden on the user to correctly apply tags/metadata to their email as they send it. The hope is the sender will think a little more deeper as to why and what their sending, and this just makes email organization easier all around, especially for the receiver. But I guess this opens up complexities on the universality of email...

Surprised nobody has mentioned Gmail filters yet. I use Gmail filters for my non-priority tasks. Things that I want to do in groups (like customer service emails) get labeled and archived immediately so I'm not distracted.

Gmail Filters are pretty damn powerful-- not sure if they support REGEXs on email subjects, but that'd be pretty amazing.

Omnifocus is developing a feature to allow "send to omnifocus" tasks where you can send your tasks to USERNAME@omnifocus.com and it will add them to your inbox. There's tons of potential with this, as it could be extended to assign tasks to specific projects, assign priorities, and set task due dates and gcal integration.

Do any email protocols accept XML or JSON? Not sure if that's even possible but I guess you could send a JSON object in the header and write a new client that handles it appropriately...

They're nothing that procmail can't do. And they're not a lot of things procmail can do, including revising content, and feeding mail directly into other processes (both for delivery and for action).

As with many GUI mail tools, one of the biggest limitations of GMail filters is that you can't reorder them. The priority in which you assign filters is really critical.

One other cool feature gmail has is the ability to let you put alternate email address names that you can then use to easily filter. Like if you go to the bug tool and enter your email address like this: bugs+justin.m.chase@gmail.com

Then gmail will know to send that to your address still but you can easily setup gmail rules to process it.

Heh, you could put something very capable and integrated really quickly using Emacs--all the building blocks are already there. I imagine combining Org-mode, an email client, your development environment and whatever else you happen to use in Emacs with a bit of elisp glue would do most of what the post proposes quickly and elegantly.

Coincidentally, can you use AucTeX's LaTeX previews in other buffers? Being able to see LaTeX equations in emails and chat buffers would be great.

That is great for me but not so great for those who aren't techies.

I've been working on a product also in this direction but it's more full featured and mature. It's called http://dynado.com and we're launching within a month (don't look at the design - the final main site isn't up yet).

It's basically a webmail where you can decide to either send somebody a task or an email. Everything goes through email though and you can send a task to anyone, even people from outside the system. They can then either collaborate via email or register to use the system themselves.

Of course it's integrated with an IM, file storage solution where you can attach those files, browse a stream of sent/received attachments. It also has a project tree and reports per-user or project, with really simple and integrated time tracking. AND it has a built in calendar with an innovative twist.

Basically it's an all-in-one business management tool tightly integrated with email in order to let you use the tools you know (email) but still capture all the knowledge and track all the tasks without switching to 10 different apps.

Yawn.

What you want is at task management system with a defined format which can be encapsulated in email messages.

Say, like, Debian's BTS: http://www.debian.org/Bugs/server-control

No, that's not something I'd expect the general public to use, but it's a good start and proof of concept.

Email itself provides you transport, storage, and overall format definitions.

MIME gives you attachment encapsulation for whatever format you care to provide.

MIME encoded signatures give you authentication and/or privacy (you care about that shit, right?)

Now "all you need to do" is define a task management standard and the interchange format that it uses. Drag'n'drop or procmail/maildrop filter attachments to your task management tool.

Wait for the world to beat a path to your door.

You've identified ONE use-case for mail. As with other use-cases, there are a few bits that could be tweaked to make it so (the standards adoption bit is the hard one). And while email / SMTP certainly has its warts, there's no need to scrap the entire system for your specific needs.

Interesting. I've thought about this very problem before. In fact, I even started building a solution. Most of my emails, like the OPs, ended up as todos, so it makes sense to skip the email part. The idea is that you have buckets of todos, and other people can assign todos directly to you, at which point they show up in your bucket, etc.

The project never grew past MVP, but lately I've been using Asana and it seems to be very close to what I was envisioning. I wonder if they're aiming to be an email replacement down the road. I could certainly see it happening, and they're already on the right path. There's no way they're planning on changing the world [1] with just a todo list app :)

[1] http://latimesblogs.latimes.com/technology/2011/11/facebook-...

> "We think it will be as impactful on the world as Facebook was." - Dustin Moskovitz (fb/asana)

Lots of programs already have this integration. Is it a bug report or feature request? Forward it to FogBugz. FogBugz figures out where it belongs (and if it gets it wrong, you tell it and it learns). Is it a task? Forward it to Flow and it gets entered as a task. Travel itinerary from your airline/car rental service/hotel? Forward it to TripIt. Works like magic.

This problem has, generally speaking, been addressed. You're just not using tools that address it. Which is not to say that there isn't room for improvement, or gaps in this toolchain (e.g., I can't think of a tool for bills/invoices off the top of my head, though I bet one exists). And addressing it by creating a solution for your specific combination of problems is certainly admirable.

But it's not email that's broken; if anything is broken, it's the endpoints (both the source and the destination). Saying email is broken is like saying the | operator in Unix is broken.

We kinda/sorta do this already with Asana. We have email addresses set for common projects and then just forward a lot of emails into the correct projects (assigning to other team members when appropriate). It's not exactly what the author of the article is looking for but pretty close.

I planned on just using it internally until all the bugs where worked out but what the heck it would be freaking awesome to get you using it now so you can tell me if you love it or it sucks.

For those of us considering signing up, do you personally find your product useful?

I'm working on this project as well so I can add a little insight.

I needed one place I could login to, add as many email accounts as I wanted and manage them from one location online. The email accounts run via the imap protocol.

I can't tell you how many times I've been working on a project and a client sends one team member a email and others don't get it, but we assume they did.

So I can create a list called "Roockit Bugs" share it with other users and drag and drop any email from any email account onto a task. A popup screen loads and auto-fills the task with the subject and inserts the email body into the description. I can now edit that task as needed and click save & archive. Now everyone can view it and can comment on it. The email is archived and out of my inbox so the right person can take care of it.

This process can work the same as above but I can drag a email to the messages section and it's converted to a message for all users to see. This makes sure everyone in a list stays on the same page without forwarding emails all over.

This is just scratching the surface but gives you the basic idea.

Seems like you need to go from communication into to-do into work in a way that is easier then we do now. Your solution seems interesting, but it has very limited scope. Maybe something like what pg talked about in #2 of http://www.paulgraham.com/ambitious.html would work. We need different protocols for different types of communication. A protocol for sending todo requests, a protocol for sending personal notes, etc. It would need work to figure out the details though (it's something I'm working on for my startup, but I haven't spent a lot of time on this particular problem yet). Thanks for putting your solution out there.

We just need to agree on universal MIME types for things like todos, events etc. Email already supports this, so the problem is really political and not technical.

The other big problem with email is it’s still never encrypted.

If anyone wants to experiment with such things, please contact me (alanhogan.com/contact), and we can trade public keys. Our experiences will inform a related blog post I am working on.

I forward any email that needs me to do something to Asana. Asana automatically creates a todo for me with the email content as an attached note.

I like this post, but to me, the gist of this is that we should have "droplets" that do different tasks for an email, so as to cut down on repetitive tasks.

I'm all in, but it seems to me that it would be best left to the end user -- if someone gets a bug report email, they might want to track the bug in Bugzilla, or Github Issues, etc. Not sure if signing into an external service is really necessary.

Have you tried ifttt?

e.g.

Create task in Asana when mail labeled Todo http://ifttt.com/recipes/18551

I want a solution, that, after I enter recipient's name, looks up that peson's defined tag cloud (or folder structure with keywords pulled automatically by analyzing content of such folders) and helps me (the sender) to tag my email, even pops up bug report form, if there is one, defined by recipient.

This reminds me a lot of Mitch Kapor's pet project, Chandler. It was really buggy but I guess after years of development, it finally stabilized. Any "note" can be morphed into a task, calendar event, or email. http://chandlerproject.org/

I am proposing that when you get an email from Company X about a new feature request, you should only need to drag the email into the "Company X Features" list. This should instantly create a todo task, share it with everyone with access to the "Company X Features" list, and everyone working on the task should be able to see relevant portions of the original email.

This is what mailing lists are for. Forward the email to the "Company X Features" list, and you've got it.

If you need something with more sophisticated group-wide activity tracking, emails and todolists are indeed poor choices- try filing a bug, and CC'ing the mailing list.

I know on HN it's not hip to not complain about email, and maybe these methods aren't the second coming of productivity, but they seem to work pretty darn well without having to reinvent the wheel.

The key issue with this is - I (representing majority) don't have these issues. I send some mails to RTM as tasks, label around, forward some, and, most importantly, resolve the ones I can resolve immediately. I receive ~50 new mails a day.

Email has some major problems but I'm not sure helping me file manually is a solution I'm looking for. But I'll be interested in trying Rockit out.

What I am really looking forward to is http://fluent.io

Power is out on my block, so sorry if I'm not answering emails or questions from everyone very quickly. I am currently running on laptop batter (1.5 hrs left) and my cell phone tethering.

sigh - Good ol' Silicon Valley Power.

I believe this title is how the Google Wave introduction began

Lets hope we don't end like Google Wave did. Shutting down a service is never fun.

There is a typo in the screenshot at http://www.roockit.com/: "Noifications"

Thanks :) That slipped through.

Does anyone know of a cache for this. It seems down, but it's Posterous? Other posterous blogs seem to be up.

Edit: Diregard -- working now.

Its possibly that Posterous had a hiccup. I have only written 2 posts there so they probably didn't expect 8000 views in a few hours.

I have the same problem...

Emacs org mode does this, I realize that its not a viable solution for most people but its pretty amasing.

See Chandler. Read 'Dreaming in Code.' Avoid that fate.

This is a solution but it isn't the solution. Email doesn't need to get better it needs replacing. I'll give everyone a hint: Basecamp almost has it right. If you focus on groups with the option to drill down to projects, individual schedules, and to-do's all in one then you're on the right path. On top of that the solution needs to be something everyone is using or can begin using quickly. Think Facebook. It's hard to explain but when you see it, you'll know it. I'm working on it myself.

Not that this isn't good. It really is but I can see it being unnatural for some. If only there was a way to do this without changing so much of your usual habits and kind of fitting yourself into a workflow someone else designed then it'd be massively popular. That said, this far from unimpressive. It's awesome and I'm sure a ton of people will find it super useful. But even the people who are getting Nigerian inheritances need some love. If someone built something that addressed the people that would use this but is also useful to the inheritance people I bet you anything those non-power users would actually find ways to use it even if they aren't busy a lot like how people who aren't social butterflies still log into Facebook and post status updates once a day.

Sorry, but no. I've been on dozens of Basecamp projects and the only way anyone reliably uses it is if they rely on the email notifications.

I probably wasn't as clear as I could've been. Obviously we already have Basecamp. That would only be one component and not every feature makes the cut.

I really truly believe that some form of project management can and will replace email and it can become integrated into our lives like Facebook my biggest qualm with project management is the project part. People aren't always necessarily working on projects but we sure do work with groups and keeping track of what's going on in all those groups is tough. I haven't seen anything that does it the way I feel it should yet. I'm not talking socially either. I'm talking about a tool that'll let you see what your family is up to or what your church group is doing etc. Yeah, I just described Facebook but the difference will be in how it lends itself to being used. Like if Facebook and Bascamp had a baby.

Well, the baby of Facebook and Basecamp would definitely be online, which makes it impossible for it to even want to compete with email.

If you cannot search, read and write messages OFFLINE it has failed in my eyes. Outside of silicon valley or whereever, there are places where you will not get a connection.

Furthermore, it'd have to be more of a protocol rather than a service itself. many companies will not rely on some third party for hosting their most valuable information and communication network. same goes for individuals, probably <1%, but that is the tech-savy crowd...

We are actually working on a way to allow companies to host all of this internally while still using our services. This would mean that any company with security concerns could host everything internally and we would still be able to provide support and features.

Along with that we also plan on providing offline support through standalone clients - though this isn't something we are focusing on at the moment. Right now we just want to get the user experience perfected.

The problem, in a nutshell, is that the OP has never managed an email server. The solution is no secret: postfix, exim, sendmail, or qmail (in that order) with spamassassin, amavis, spamhaus, dcc, razor, greylisting, clamav and occasional bayes db priming. Any ISP doing this will be able to tag and block and/or populate a spam folder with all but 5 to 10 false-negative spams per week.

Anyone getting more spam than should switch email ISPs. This is the real problem IMO, people who stay with incompetent email ISPs (including yahoo, aol, 1and1, att and several other large and small but technically challenged organizations).

I'm not trying to filter spam. Gmail does great at that. I'm trying to manage the non-spam emails I get. In fact Roockit's goal isn't to be an email provider - we simply want to help manager your current email.

People actually still use ISP provided email in 2012? Really?

Reply on news.ycombinator.com