Hacker News Reader

Comment by grinich | Hacker News Reader

Here's what you need to install to run the entire stack locally: https://github.com/nylas/sync-engine

Sorry-- launch day craziness :) More docs to follow soon!

Why do I need to run a whole stack for a mail client? It's a mail client! Why is everything JS web services :(

You don't have to do anything. Relax. Any open source project leverages other open source projects. How is this "stack" any different?

Because it's a mail client, and it needs a server that will talk to my SMTP server and the mail client. Which means that I either need to give all my most private and important data (my emails) to this service, or I need to set it up locally. For a mail client.

Not sure I see the point you are making. Why the concern it's running a server if it is running locally? You already give your email data to another server, and even if you run your own mail server, then what's the issue with running another server yourself?

I don't want to have to keep my computer on and connected to the internet 24/7 for something my mail client does anyway. I don't see why someone would architect a mail client like this.

I can't see why you would need to run your computer all the time. Why do you think this is necessary? You just turn on your computer and the sync service restarts where it left off. Just like Thunderbird, or any other MUA.

But I can see plenty of reasons the design was chosen. If you are bringing in lots of mail accounts, having it running as a daemon isn't necessarily a bad way to go. A sync program doesn't need to be associated with a tty, is better off run in the background in its own process, is arguably more extensible in design terms, doesn't necessarily take up any more memory than a regular executable, allows for multiple UIs (e.g. terminal program, Gtk+ GUI, WxWindows GUI), seperates processing from the UI, lots and lots of reasons.

Maybe the developers can step in here with a response?

You can make a UI to turn off the service or use the command prompt, if that's the concern.

If you can give me a valid technical reason running the sync part of the program as a daemon is bad, I'm definitely all ears though.

Are you suggesting this is a complete waste of time based on your own needs? Well, fine. But I'm sure there are plenty of valid use cases for it and at the very least the architecture and libraries involved are HN-worthy.

Seems like they've addressed at least one of your concerns, and it might actually turn out to be very simple to set up locally, so here's the privacy trade-off for not using it in your case: privacy > value || privacy > work_of_local_install

> Which means that I either need to give all my most private and important data (my emails) to this service

You are already doing this unless you run your own mailserver, or if any of your inbound/outbound emails aren't GPG encrypted (or encrypted during connections with other mail servers using TLS).

Something has to receive email for you when your computer is not running, unless you simply rely on a local mailserver, updating your MX and A records whenever your IP changes, and let other mail servers queue your email until your local desktop is back online to accept them.

I am not already giving my emails to this service, I'm giving them to my email provider. I don't see what value these guys provide that's worth giving them all my mail.

That's a fair argument. In that case, they should provide an MTA you can plop down on a virtual machine that'll store your emails for you and they interface with using JMAP [1].

[1] http://jmap.io/

"JMAP is a transport-agnostic, stateless JSON-based API for synchronising a mail client with a mail server. It is intended as a replacement for IMAP. The specification is based on the API currently used by FastMail’s web app. It aims to be compatible with the IMAP data model, so that it can be easily implemented on a server that currently supports IMAP, but also allows for reduced data usage and more efficient synchronisation, bundling of requests for latency mitigation and is generally much easier to work with than IMAP."

At least half of the blame should be not on them. IMAP doesn't know a thing about what happened to email in last 10 years (few minor exceptions apply).

They're merely implementing their own means to access email in convenient and modern way. Say, IMAP has no labels[1] or contacts[2] (and many other things) - there are proprietary extensions, MS Exchange being the most famous one, but they're... proprietary. And given that proprietary implementations differ, it makes sense to create an unifying wrapper.

And JS apps and web services are just an approach that's trendy today. Personally, I don't like it either, but can't blame for going with the flow. Especially since it's a business (and going against the flow has costs, both in development and adoption), not a private project where one can do things in a way that just feels right, ignoring anything else.

I believe it would absolutely make sense to bundle the "sync engine" within the mail client, though, to make it standalone. But I understand it's done this way just because there was only the engine (as a service, not a library) at the beginning.

____

[1] https://nylas.com/docs/#labels

[2] https://nylas.com/docs/#contacts

> but they're... proprietary

Everything you ever wanted to know about Exchange [1] (but were afraid to ask). I personally use SOGo [2] as a Exchange ActiveSync provider but I know OpenChange is a project to implement the entire Exchange stack. So I would not consider it proprietary.

[1] https://msdn.microsoft.com/en-us/library/cc307725%28v=exchg....

[2] http://www.sogo.nu/

>IMAP doesn't know a thing about what happened to email in last 10 years

Sounds like it's time for a new protocol, then. Is anything in the works?

JMAP?

http://jmap.io/

IMAP actually does have labels. They are called flags. Servers tend to not implement them well at all (sometimes allowing only a handful to exist, and almost never keeping an index for that metadata, meaning that searches by flag are either slow or often not even allowed :/), but that is not the fault of IMAP.

Apparently, most sane open source IMAP servers support PERMANENTFLAGS \ * which as I understand it means user-defined flags?:

http://serverfault.com/questions/115769/which-imap-flags-are...

I'm not entirely sure what's "happened with email in 10 years"? Search-defined virtual folders?

And, FWIW, search defined virtual folder is a client concept and would be trivially implemented over IMAP's incredibly powerful (and I believe I remember being even extensible) search commands, if only IMAP servers tended to provide the right kinds of index to make such queries fast. (I know this, as I was working on a highly scalable and very efficient IMAP server a few years ago that I had to force myself to stop working on as it was too much of a distraction from the things in life that are actually capable of putting food on the table.)

Mark Crispin was actually pretty-damned prescient when it came to IMAP: he believed that client operations should be offloaded to a server that would have much more disk space and CPU than the client, which is exactly the model everyone reaches for today, and thereby designed a thin client protocol that had asynchronous updates baked into its core. A few years ago, he passed away: here is what I wrote at the time about his mission for IMAP and history on the project.

https://news.ycombinator.com/item?id=4826113

Really, the only thing in IMAP to actively dislike is how the thin client view requires the server to have a "dense" (here I mean the mathematics term, not from the technical spec) numbering of messages the client is modeling in its view (this is the reason opening a large mailbox via Zimbra takes forever, and why many IMAP servers have dumb global locks), but I have come up with some techniques that should allow me to handle that with minimal overhead. Otherwise, with some of the modern extensions in place (QRESYNC, for example), IMAP is pretty damned awesome.

IIRC, IMAP flags/keywords were limited compared to folders (and a message can't belong to multiple folders). But maybe I got a wrong idea when I read the RFCs...

I would need a more concrete complaint to tell you why you are probably wrong here. The only thing I know of that tends to cause people to balk at flags is the frustrating idea that the name of the flag for some reason has to be the same as the name shown to the user, so the inability to rename flags or even use special characters causes people to give up on them... but you should treat them as opaque database tokens, not user-visible metadata, and map them to names. There was then an extension being worked on to store high-level names in the IMAP metadata store, but I don't know if it got completed (I havent paid much attention since Mark Crispin passed away: I felt like he was one of the few people keeping IMAP from losing its way, so I am highly concerned that I'm going to read through the mailing list and find people frantically pushing through all the stuff he had vetoed). But like, adding your own "documented but non-standard" blob to IMAP's metadata would still be worlds better than throwing away the entire protocol and starting over again from scratch :/. Honestly, I think the core issue with people working on IMAP is that people these days have very little data structure and algorithm training, particularly when it comes to implementations of parsers, and so when they stare into IMAP they miss the forest for the trees and barely scratch the surface of what it can accomplish before giving up and screaming for their JSON and HTTP. (That, and the dense numbering for message views, which is a truly-hard thing to get right, even with a lot of academic background.)

Bundling the sync engine with the client works if you're willing to download your entire mail archive onto every device that connects to your mail. An implementation of the sync engine that allowed only syncing partial mailboxes would be very difficult to implement on top of (all of) IMAP, Exchange, and Gmail. By abstracting that away into a new cloud service, clients end up being both simpler and more flexible.

Labels are over. And contacts could be done ousted of IMAP.

Nothing happened to email in the last 10 years, IMAP is fine.

I completely agree - things needs to be simpler not more complex. I believe a quote from Albert Einstein fits here:

> "Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius -- and a lot of courage -- to move in the opposite direction."

Though I think I get why this email client and Android's Outlook [1] will hoover your emails to some other server - the server may have more processing power than your device. And/or use an RDBMS to index the emails which is better suited for a server than a smartphone. And that's fine - but the email client should be able to do this on it's own without the need for a server (and explicitly show a warning/message allowing you to opt-in to using the third party service). It will be slower indexing emails on your phone but that is the price you will pay.

And to be fair - I've tried almost every email client I can get my hands on: Evolution, Foxmail, Pegasus, Thunderbird, Outlook, ElementaryOS. To date no one has gotten calendar, contacts, and email better integrated than Outlook. Trust me I REALLY REALLY REALLY don't want to use Outlook - but I've yet to use an open source alternative where in 5 clicks or less I am syncing all those items previously mentioned. Thunderbird comes in second place - but I have to install 2 different extensions for contact and calendar syncing. The contact syncing has been hit/miss and the calendar syncing was stable for the most part. I should note that I don't use Exchange but I use iredadmin with SOGo.

I think SyncML was supposed to solve the whole syncing problem - but never really took off.

[1] https://play.google.com/store/apps/details?id=com.microsoft....

Most mail clients have a "sync engine" of some kind, except traditionally they're not written in Python and don't communicate with the rest of the app using REST APIs.

But why is that a problem? It's open source, someone should just package everything up into one nice downloadable desktop application.

Working on it!

Front-end is gorgeous. Using it to try out some ideas re: email encryption.

Did you already start a plugin by yourself? If yes you should perhaps mention it on the roadmap (or vote for it otherwise): https://trello.com/b/hxsqB6vx/n1-open-source-roadmap / https://github.com/nylas/N1/issues/19

Why isn't this the default?

Reply on news.ycombinator.com