Show HN: Switchboard – open-source email processing in Erlang
switchboard.spatch.co
switchboard.spatch.co
Our motivation in building and continuing to build Switchboard is to make it easier for developers to process emails. Looking forward to more ideas!
Since I sent my first email in the 80's, I've wanted to build business-logic systems with mail-based interfaces - things like, have an address such as "calendar@mydomain.com" which can be cc'ed on emails which might contain future scheduling events, and have the calendar@ client automatically work out the details of reminding everyone of the date-target that was set .. Seems to me that Switchboard would be the ideal platform upon which to develop such services .. among other things too. See any gotcha's?
Its always amazed me that automated email is not used more and more for basic business processes - sure, we have mailing lists and autoresponders and all sorts of things like that, but the fact that most language/tech stacks don't really treat email as the human-maintained message queue, with first class priorities, seems a little lacking .. I guess Switchboard is an attempt to make up for that. Anyway, thanks again - I'll certainly be devoting some research time to this tool, and I hope I can build something useful on top of it ..
One note: Switchboard is especially useful relative to a bare IMAP connection when you're monitoring many accounts. This is because it provides a lot of the boilerplate around restarting failed connections and issuing IMAP commands while an IMAP connection has an active IDLE continuation.
I agree that there's tons of room for more automation around emails. I think the lack of automation is due to not having high-level tools, which is where Switchboard comes in, and that emails are typically natural language. I'm personally excited to see Switchboard get used for email classification -- imagine if your business-logic could include the email "class", e.g. "sent by a human", or "marketing", ... etc. A bit like Gmail's categories, but open and accessible to developers.
I'll play with it .. hope you won't mind if I contact you in a week or so in case I have questions or maybe even to give you feedback/demo of what I've come up with in the meantime..
1. Building on top of IMAP is very difficult. It's not a technology that is really designed for the mobile world of today (e.g. streaming, sockets for chat)
2. Threading is a bitch. The old folder tree layout is formidable.
----------------
That said, it's so cool to see something like this in the wild. As an engineer, I would've used this as a boilerplate for my project years ago, and it would've saved me literally six months of efforts.
Kudos to this. It's gods work.
Free startup idea: Use this to help larger businesses collaborate more efficiently (much like if Yammer and GTasks had a baby).
2. Yes! We still have to work this into Switchboard, but conversations in the JMAP (http://jmap.io) spec give an idea of how it will be implemented.
However, it also serves as an example of some of the complexity that clients are expected to handle to provide a modern email experience. That algorithm isn't trivial, and reimplementing it with every client is expensive in development time, and error-prone.
As mentioned in a sibling comment, the Switchboard server's roadmap includes making it thread aware, leaving Switchboard clients lean.
On a side note - its funny how things flow on HN sometimes - yesterday I played with PlanLeaf, and had the thought 'nice, but it'd be even nicer to just have the code to process emails properly', and now .. along comes a stack that gives me a way to do just that ..
On another note, I think context.io provides a decent paid solution. How is this project different?
Switchboard is fairly different from context.io. Switchboard is open-source -- being able to keep in control of services accessing emails on your users' behalf is great. Also, Switchboard is intended for processing emails as they arrive using worker processes, as opposed to context.io which appears to be primarily a proxy.
Switchboard doesn't have any direct connection to SMTP. It monitors changes in your mail store using IMAP IDLE and issues events to workers which can then do basically whatever your heart desires.
(btw, we host our switchboard mailing lists on librelist.com which is a showcase for Lamson. cool project)
Additionally, there is Apache James (Java) which does something similar.
I'm not super familiar with Apache James, but it appears to be a mail server with pluggable components. Switchboard fits in as a layer between any mail server that supports IMAP, and its version of pluggable components [processes connection over the interface].
Thanks, this really helps us see where the documentation needs to be improved :)
We actually thought about using zeromq, but went with a simpler solution for now, i.e. using websockets.
If we were doing “server-side email processing” of emails as they were sent, SMTP would have fit in.
Switchboard uses a simplified interface based on JMAP[1] for client's and workers, which it does proxy into IMAP commands. One of the conveniences of Switchboard is it manages the IMAP connections: restarting them as they fail and allowing concurrent access.
[1] http://jmap.io/
If that's actually the case, how do you handle the case where someone read a new mail (and thus, mark it as read) before app processed it ?
Also, because it uses JSON over websockets, it's fairly easy to implement a new worker/client that follows the Switchboard interface -- here's the Python implementation in 132 LOC: https://github.com/jtmoulia/switchboard-python/blob/master/s...
Switchboard could be used to fetch the email, encrypt it, and then replace the unencrypted email with the encrypted email on the server. However, replacing emails is invasive if the user doesn’t expect it, and other than the email provider's word you have no guarantee that the unencrypted email was actually deleted.
If not, please let us know about any sticking points either through the mailing list, switchboarddev _at_ librelist.com, or the GitHub issues page.