IMAP API – self hosted daemon to access IMAP and SMTP accounts via HTTP API | Hacker News Reader

IMAP API – self hosted daemon to access IMAP and SMTP accounts via HTTP API

github.com

142 pointsandris948 comments

Wouldn't a JMAP bridge like [1] solve the same problems?

JMAP does solve the same problems, and then some -- looking at the specs, it's more of a 1:1 mapping between IMAP+SMTP. I'd say that if you're looking at implementing _all the functionality of a modern email client_ you might want to look at that. If your use-case is just wanting to fetch emails from a bunch of inbox folders in real-time, potentially with some querying/filtering, I'd say JMAP is a bit overwhelming. Sure, it has some clients/libraries available, but only a few.

imapapi, in my opinion, being a RESTful API, appears to have a much simpler surface, seems easier to play with using curl, and if you're developing a bunch of RESTful services that use Swagger, it fits better with your stuff. So, less dev-time to use :-)

JMAP is definitely more complicated, but that complexity enables some very useful features. You can search, you can fetch emails by thread. You can get text snippets of emails for preview. JMAP can also automatically convert any email into directly renderable HTML (or plain text).

I’m convinced if nothing else the jmap json format is what everyone should be using for passing around email objects since it’s so well designed and feature rich. Almost all hand rolled email json formats throw away data you really want when rendering or processing email messages.

IMAP api does look lovely though.

> If your use-case is just wanting to fetch emails from a bunch of inbox folders in real-time, potentially with some querying/filtering,

Nobody says you need to implement all of JMAP (from a client perspective). Just implement Email/query Email/get

I tried using the JMAP bridge to develop a modern client as you mention, but there were huge problems with it. It appears to be abandoned, and Fastmail’s demo install of it is entirely broken too.

I ended up using the underlying lib that this project uses (also by andris9) and have had basically no problems whatsoever, it was a night and day difference.

Also of note, FastMail uses JMAP. Their CEO was one of the drafters of the protocol.

In theory yes, but as I noted upthread, the reference JMAP bridge you linked is abandoned and Fastmail’s demo install is broken alas.

The existing JMAP bridge is broken. But it would cool if 'IMAP API' used JMAP instead of coming up with its own protocol. (Which covers a very similar feature set than JMAP)

This would allow you to use the api with any JMAP client like Ltt.rs for example.

This is great, but I really need a self hosted version of something like this for accessing calendars. Anybody know if something like this exists?

Far as I know, the major calendaring protocols all use HTTP as it is. Do you just want a mediator/self hosted backup? Or do you want to host your own calendars?

If it's the latter, Radicale is awesome.[1]

1: https://radicale.org/3.0.html

I think my biggest difficulty is that I'd like to be able to look at a duration of time and determine the events within that range.

Pulling raw iCal data over HTTP is possible but if you say, set up a recurring event several months back, you need to fetch that original event (even if it recurs every day) to include it in your calculations... I'd love a self-hosted version of this Nylas API [0] for example.

[0]: https://docs.nylas.com/reference#calendars-free-busy

We’ve working on this for one of our side projects. When you have hundreds of calendars and need to find availability for some of them on a timeslot, or fetching the slots for a period of time (I.e, a week) it becames hard really fast.

We are evaluating if it may be interest on it to release as a separate SaaS.

You can find my email on my profile if you want to discuss further your needs.

iCalendar files over HTTPS via OAuth?

https://tools.ietf.org/html/rfc5545#section-1

CalDAV is already built on HTTP, it's just implemented with a lot of gotchas and ancient history: https://tools.ietf.org/html/rfc4791#section-1

http://www1.udel.edu/CIS/software/dist/google/calendar/java.... for Google Calendar ics feed documentation (there might be better links, this just came up first)

Great to see this here. Andris9 is great! He also made wildduck which also allows IMAP api:

https://wildduck.email

I'd like to see an ms graph api version of this. A proper rest json api for mail, contacts and calendar is long overdue.

JMAP never took off. Maybe, if projects would copy the odata based graph api that ms documented we could collectively move there...

Have a look at https://kloudless.com/ - they have unified apis for mail /calendar / drive and a bunch more adapters.

If you just need email processing locally, I have found imapfilter to be quite nice. You can script it in Lua.

Personally I went a bit overboard with it and implemented a simple TCP server in node which queries GitLab APIs based on the email metadata and moves Mails into Folders based on assignees, Labels etc. So letting do imapfilter the heavy IMAP stuff, calling node via tcp and using the response to sort based on it.

[0]: https://github.com/lefcha/imapfilter

What if I have a SaaS platform and I want to configure the MX DNS and accept emails at `support@saas.com`? How would I set that up with say... docker-compose? Do I need both a STMP server and an IMAP server?

A with „locally“ meant like personal email accounts on Personal devices. If you have a Saas platform and want to send stuff, this OP sounds more reasonable.

I just wanted to give an alternative in case someone wants to programmatically interact with their email inbox without setting up and maintaining a service.

Nice job.

I just realized, would Elixir be the perfect platform for this since you could have a GenServer process that has a constantly alive IMAP connection for each server?

I'm not sure if this API is initiating an IMAP connection each time you request something, but keeping that connection alive could be really interesting.

Also can you use a regular connection pooling library to do this?

Or am I imagining a problem that doesn't exist? (overhead of reconnecting to IMAP server for each request)

Yes. And there was an Erlang library called switchboard I ported to Elixir that works as an IMAP client. It is not complete though.

A few issues are:

- it does not properly handle the case when you send something invalid in a continuation mode

- continuation commands from the command response queue (well it’s a gb tree actually) don’t actually get a response

- there is a weird bug where some pieces of mail don’t get processed - might be something wrong with me porting the code. It’s also not necessary since the mail parser from that Erlang smtp lib can be used

https://github.com/voughtdq/imap

IMAP API opens a single persistent connection for every registered IMAP account and then uses this connection for all operations. Unfortunately this also means that you can't do stuff in parallel, so if you pile up a bunch of requests against a single account then it might take some time until all these requests are processed as these need do be done one by one.

Awesome! I'm having to maintain some IMAP+SMTP handling code and having to deal with the intricacies / non-standards / quirks of different mail servers (Office365.......) has been such a time sink. This would be perfect in our infrastructure.

Any plans to provide it in a Docker container? I'd be willing to maintain one & push it to Docker Hub, for the AGPL version, if you don't have the time.

Someone kind of provided a Docker container already but I haven't really looked into it as I don't use Docker so much https://github.com/andris9/imapapi/issues/28 If there are more requests for it then I'll probably add it to IMAP API.

Docker support is now enabled https://github.com/andris9/imapapi#docker

nice!

Related: Here's a very simple one from a few years ago that does 1:1 sync via IMAP into CouchDB. CouchDB directly provides the local HTTP/REST UI & APIs: https://github.com/thomaswilley/yama

Given that CouchDB was inspired by Lotus Notes, using it to store emails is really going back to its roots.

I see only one reference to XOAUTH2 in the code - in an example script. Does this connect IMAP with XOAUTH2?

XOAUTH2 is supported but all the heavy lifting about refreshing access tokens etc must be done on your application side. Whenever IMAP API requires an OAuth2 token it makes a HTTP request to an URL you have defined and whatever is returned from that URL is the used to authenticate against the IMAP server. Here's an example of such an authentication server: https://github.com/andris9/imapapi/blob/master/examples/auth... Basically you would have to run your own and configure IMAP API to requests tokens from that server.

This is really cool. I’ve been a big fan of the Nodemailer project for how accessible it makes IMAP and SMTP. This is a significant step towards even more accessibility. I can imagine some pretty cool, lightweight email clients being built with this.

people interested in this should also take a look at rainloop and roundcube, two fully open source self-hostable webmail applications. Not exactly for the same purpose, but they both also implement access to IMAP-over-TLS stored mail.

Does Migadu run rainloop? My biggest annoyance with it is that I cannot bookmark a URL or add it to the TODO list.

Thanks for sharing. I have a reverse problem though. Would it be possible to access REST interface using IMAP a protocol? My use case is to access Office365 web mails using legacy apps such as mbsync and msmtp.

I think DavMail might meet your needs: http://davmail.sourceforge.net/

Thanks. I tried it once, and couldn't get it to work. I will take a look again.

Office365 already has an IMAP server. I forget the connection info, but I’ve used it successfully before

IIRC It has to be enabled by your Office365 admin. Office365's IMAP also doesn't support 2FA so if your org uses that, then IMAP is out.

Cool! I built something similar, albeit worse in every regard, also using nodemailer.

I might have to start using this instead. Mine basically just powers contact forms (send only).

Does the AGPL mean that you would also have to license any software that communicates with this daemon under the AGPL? Or just the daemon itself?

AGPL considers network use to be distribution, so if you have modified the AGPL-licensed software you have to make the source code available to users of that software. That does not mean you have to make the source of downstream systems available.

In fact I am not sure you have to make the source of your modified AGPL software available to end users if you are using it internally ("users of that server" from the license would be you), but IANAL.

This is correct.

It would only be polite to contribute back if you did find a bug fix anyway.

AGPL seems the most appropriate license for software like this.

For most organisations running AGPL licensed daemon application like IMAP API and interacting with it via HTTP does not mean that the license extends to their own code as well. For those that are not 100% sure or do not want to touch AGPL at all, there's an option to buy MIT licensed version from my paid module store: https://postalsys.com/packages

IANAL

It isn't clear exactly what is required here. I think the intention is that this is fine, but then what if you were just to modify it and stick a TCP proxy in front? Does that mean that you don't need to distribute your modifications?

The is ambiguous enough that Google with their army of lawyers forbids any use of AGPL software https://opensource.google/docs/using/agpl-policy/

Amazing, I will probably use it soon, thanks

Does this work with pop?

For now, no. POP3 is such a simple protocol you usually can just use it directly. If would be super difficult or in some parts impossible to emulate all the current features on top of a POP3 backend.

If you downloaded the messages locally you could.

Yes. Though IMAP API does not have a storage for emails. It only uses minimal storage for state caching, everything else is retrieved directly from the IMAP server. IMAP API also leverages IMAP server's capabilities for parsing, doing itself as less as possible.

Reply on news.ycombinator.com