Show HN: I'm working on an open-source Gmail replacement | Hacker News Reader

Show HN: I'm working on an open-source Gmail replacement

khamidou.github.io

387 pointsvorador180 comments

I just put the finishing touches on my ansible playbook for my 'Goodbye Google' server (Mail via dovecot/postfix w/ dkim, dspam, greylist, sieve, Radicale for CardDAV, CalDAV, Prosody for xmpp). Works fine so far.

What I'm lacking right now is a decent webmail client. Roundcube isn't exactly my type of thing, mailpile might be interesting. This seems ambitious and interesting in general, but seems to come with too much strings attached (puppet? No, ansible. Comes with postfix? I already have that). So .. it is more than I'd need.

I do like the idea of ready-made, easy mail server setups though (obviously, given the first paragraph). Perhaps a project like this could integrate well into owncloud or arkos though?

Care to make that playbook publicly available? I would also be interested to try out hosting my own email, but am somehow afraid of setting it up wrong.

i found https://github.com/al3x/sovereign/blob/master/README.textile and http://yunohost.org/ from here actually...personally i also think https://library.linode.com/email/postfix is good.

Just pushed my current status to

  https://github.com/darklajid/ansible-communication

Careful though: This isn't deployed outside of test machines yet. I'm still working on the proper backup/restore scenarios, consider a couple improvements etc. - the readme outlines what I would like to do next.

Plus: While I self-host my mails for a while (without an automated setup of sorts, the one off snowflake sort of thing, so quite different from this here), I don't claim to be an expert here. Check what I did, take it as a template, don't use it without a basic understanding.

If there are some postfix/dovecot gurus out here and would like to point out obvious flaws in my setup - go ahead!

Here is the dovecot documentation for your "auto-create imap folders" issue: http://wiki2.dovecot.org/MailboxSettings

This guide works like a charm, takes about 2-3 hours to setup:

https://www.exratione.com/2012/05/a-mailserver-on-ubuntu-120...

I'm pretty sure I followed one of those previously.

I wouldn't do that again, though (hence my stumbling around with ansible). If the machine goes down/you want to recreate it or duplicate it for a friend's setup (without dd-ing everything and trying to figure out what to change later) you're back to square one. It's probably month or years after you read the guide that created your system, you don't have the link handy anymore, forgot all you learned about gotchas in the doc that were slightly different for your deployment.

These guides aren't bad per se. My current system is a heavily customized result of various "Here's how I do mails" guides as well and it works like a charm, for quite some time. But I could never reproduce it again, which caused my reluctance to jump from GMail as my primary account to the self-hosted one - even if that one works for me, for years.

IF all my critical data (password reset mails, banking information etc. etc.) end up on a host I'm responsible for, I want to be able to replace that thing ASAP if something goes wrong.

Currently (unfinished, see my link above), but:

  time ansible-playbook -K -i hosts site.yml
  real	6m29.708s
  user	0m9.623s
  sys	0m3.243s

(Most of the time is spent in downloading packages/installing them: Got a faster network? This will be faster as well)

That's from a bare bone installation (nothing on there but ssh, my unprivileged user with sudo privileges) to a running mail (smtp/imap), calender/contacts & xmpp server.

I don't claim that MY project is the way to go, quite frankly I'd be very uncomfortable to push others to use it. But I do believe that infrastructure that is this essential should be easy to recreate/reproduce, even for laymen/single persons.

Hello, I'm working on a similar thing based on mailboy[1], you might be interested in the admin interface.

Achiles heel now for DIY solutions is the web mail. Roundcube, the best candidate, is not really stellar (almost unusable on 1366x768 screens).

[1] - http://manurevah.com/blah/en/p/mailboy

mailbox looks suspiciously like postfixadmin, is it a fork?

I've similar setup on my server (except http://baikal-server.com/ for CardDAV/CalDAV and http://www.ejabberd.im/ for XMPP) and I was wondering if there is a simpler solution to email server for one person than postfix/dovecot/spam handling. I understand complexity of that subject but feels like there could be something that fits "personal server" scenario.

for a personal server, maybe try http://www.citadel.org/. One app that runs email, xmpp, calendar, contacts and notes, written in C++. I've installed just on one machine. The interface is awful, but the rest seems to work pretty well.

We have been working on such project in our own Opa technology for quite some time.

Do you think we should work on an open source solution for https://peps.mlstate.com ?

The project is a very clean webmail server, starting with our clean protocol implementations (SMTP, POP, IMAP) and does not require external projects.

Try with hn/hn...

Update: Please don't change the password for this hn account. As an admin, I can reset it, but have better things to do today ;)

Very nice interface. Kudos.

Thanks go to @tweetfr

This is pretty cool. Will you do any kind of personal variant for gmail refugees?

Thanks.

Yes, we are thinking about the following business model for PEPS, which was built for an important account but of which we own the IP.

1. Open source under the AGPL license, so that anyone can host its own instance easily and hopefully we can build a community around the product. 2. Sell support (together with a non-AGPL license for corporations).

1. Open source under the MIT license the current product. 2. Sell a complete packaged solution with add-ons for corporate users.

Any feedback between the two?

Voting for 2...AGPL is like biohazard level 4 viral.

How exactly would it be viral in this case? I ask because it's damned confusing. Since this would be an email server, wouldn't it be a drop-in component in most folks' infrastructure? If I understand it correctly, it would just require me to make the source code of the email server/client itself available to my users...not any of my other unrelated infrastructure apps, correct?

I was under the impression that AGPL requires the open-sourcing of everything that touches user data at any point in the pipeline.

The AGPL is exactly the same as the GPL, except that if there is a button to click to download the source code of the program, you are not allowed to remove that button.

From the AGPL FAQ:

If the Program as you received it is intended to interact with users through a computer network and if, in the version you received, any user interacting with the Program was given the opportunity to request transmission to that user of the Program's complete source code, you must not remove that facility from your modified version of the Program or work based on the Program, and must offer an equivalent opportunity for all users interacting with your Program through a computer network to request immediate transmission by HTTP of the complete source code of your modified version or other derivative work."

That last part--"derivative work"--is somewhat the rub.

If that part of the informal FAQ worries you that it might be interpreted in some strange way, just look at the actually license and read it.

Notwithstanding any other provision of this License, if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version

The definition of Corresponding Source code is listed earlier in this HN thread.

This is the problem with the GPL/AGPL- what exactly must be open sourced under what conditions has been debated many times, and different people/groups have come to very different conclusions.

The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work.

Please point out what exactly the different conclusions are to that text.

I am not an expert on the various points of the debate- I only know it exists. You can Duck Duck Go for "gpl derivative work definition" and find some references to this debate. Perhaps someone else has a specific answer for your question, sorry.

Correct, just the source code of the AGPL application, not of your server.

And WTFPL which you are using is the level 5 of foul language, and the death sentence to software names.

See, I can also use offensive language to describe a license. Doesn't that make for a nice conversation here on HN?

To be fair, if you use our code under the WTFPL (and don't, because it's like three versions and two test suites out of date, at this point!) we don't require that you open-source any other part of your stack.

AGPL would be perfect for such a product.

I'm getting an SSL error.

Yes, our certificate is self-signed for this demo. "Not to worry."

http://www.startssl.com/ No excuses its free :)

Mobile site needs some work.. But one suggestion is turn off autocomplete and autocapitalize for the username box... Makes logging in on iOS hard.

have you setup Exchange protocol support (for pushmail on iOS devices)? Gmail disabled Exchange protocol support for newer devices, that would be my main motivation to move away from Gmail.

Turns out we _do have_ an implementation of ActiveSync in Opa!

That said, as we need to make a living, and if we go the MIT way (see comments below), there are high chances that the Exchange support will be a paid extension.

> starting with our clean protocol implementations (SMTP, POP, IMAP) and does not require external projects.

I'm curious of the reasoning behind your decision to reimplement SMTP instead of just using Postfix, for example. I run mail servers for an ISP (and myself) and take advantage of a lot of the "advanced" functionality that Postfix has acquired in the last 15 years.

The initial goal of the project was to have a total control on the TCB (Trusted Computing Base) of a webmail for high security environments, in order to be able to pass certifications such as Common Criteria (EAL).

Our protocol implementations use a specific DSL, that generates Opa code which is strongly statically typed with high-level types such as variants/rows. Although we currently lack many features or optimizations, the approach is clean and much easier to certify.

This would simplify my life for sure.

Checkout https://github.com/al3x/sovereign , though xmpp hasn't been set up yet.

That looks awesome, thanks. You(?) went a lot further than me (and further than I want to go, admittedly).

Still, for people looking for a more complete package that looks like a much better alternative.

Not for me, because I'm trying to get away with a much smaller machine and less dependencies (i.e. I'm still really unsure if I want a webserver. I decided to go for clucene instead of solr because of the dependencies etc. etc.).

I will follow that repository though and think about recommending that to friends/coworkers though, depending on their needs/expectations.

I have only contributed a little on the project, most of the work has been done by its main author and others.

I'm very suspicious when MySQL finds its way into an open-source package, due to its ties to Oracle. Why not Postgresql?

Actually there's an issue open right now to switch to postgresql

Why not MariaDB and then you don't have to worry about fixing third party packages that depend on MySQL specific behavior?

Why would an email client need an RDBMS backend in the first place other than the fact that, like WordPress, the devs did not value KIS? Squirrelmail does fine without a db, as do all of the command line clients.

Have a look at Zarafa (http://www.zarafa.com/). It's free and open source, although Outlook integration is a paid option (via a plug-in). Comes with a modern webapp (http://www.zarafa.com/content/webapp).

> It's free and open source, although

There's also kolab.org which is completely free software (uses Roundcube for web interface since 3.1).

Zarafa is awesome! Is that a custom web-app? It resembles Outlook; just want to confirm that its actually not.

As an ex-zarafa employee I can confirm it is written in-house, from scratch. Uses Extjs.

Also see: https://community.zarafa.com/

It's a custom webapp that's modeled after Outlook, hence the resemblance. Try the demo, I'm sure you'll like it ;)

And also...Zimbra (http://www.zimbra.com). This is what I've used to replace gmail for now, and aside from some quirks, it's working well.

Zimbra is pretty damn nice. A bit resource hungry, but very functional.

Outlook integration is a paid option, but z-push (ActiveSync implementation) is free.

http://www.sogo.nu/downloads/zeg.html sounds interesting too.

Do you have a blog post of everything you've installed and getting them all to work well together? I've been interested in creating my own "Goodbye Google" server, but never get started:(

Not yet, but that _is_ planned.

That said: The point of these automated recipes is that you can run them _without_ reading a blog and copy/pasting things in a shell. I do plan to write about my experience. After I set up a blog. cough

Sort of related: how'd you get started with Ansible? Did you just dive in and hack around or follow some sort of guide or tutorial?

I've been looking to get into it but just haven't made a move on it yet...

I've started using it over the last month. The docs on the web site are good and this hands-on tutorial was also helpful:

https://github.com/leucos/ansible-tuto

I'm actually not sure it's better than the shell scripts it replaced so I may end up switching back.

The things I really like about ansible compared to shell scripts is:

- jinja2 templating for config files - roles & tags - many error-cases already thought of and coded around.

I've to disagree about the state of the ansible documentation, I guess. As far as I understand the website was restructured not too long ago and a lot of stuff is .. broken. I fell back to using the Google Site Search thingy to find anything, because ansible's documentation lacks any kind of index/reference (except for the modules page, which is nice). There's a good deal of magic involved, and passages in the documentation even state stuff like 'there are a couple of these magic variables, but not too much' (Note: No list is provided).

In general I hate the state of the documentation. It's barely usable.

I do like the simplicity of the format (.yml) and the architecture (no agent required, only dependency is python and I think even that can be bootstrapped via python).

My playbook (see above) certainly feels like a shell script packaged in .yml format, but .. I stumbled over and over again when I tried using ansible's modules (examples: No way to create postgresql roles without a password, no way to load data into postgresql => Two things I've to do via shell: or command: in my project).

Bottom line: I'm happy with the state I have, but I'm far from happy with ansible as a technology and probably wouldn't use it as it is for the next deployment project.

I welcome this as I welcome Mailpile and other efforts like it; I look for another mail solution every few years. Not with the idea of replacing gmail (might/might not do), but to see if anyone did anything useful yet. So far, no. All solutions, even considering the changes to gmail everyone seems to hate (I don't), are nothing compared.

Every time some new frontend/backend comes up, I say the same (and have done on HN before); i'm not a typical user; I had email since '95, I have been a heavy user since then and I took my mail with me since. In 2005 I was looking, like I had been then as well, for a better mail solution and stumbled upon gmail. I wrote an export script for the mail system I was using at the time and imported 10 years of mail into gmail.

I now have over a million mails in my mailbox (i'm not sure what is the total); I have over 50 mail addresses coming to my inbox, I receive 1000s (sometimes 10.000s) of spam messages per day which Google filters well. I guess this is due to the fact I have had/have businesses on those 50 mails since 1995. All clients I tried so far just simply hang when I try them, including outlook (exchange or imap), thunderbird and some free and commercial web versions (yahoo and outlook web/live simply don't work; I cannot even read my mail through the amount of spam and the clients are horrible for productivity imho. Slow as well).

I also need a solid spam solution; spamassassin simply doesn't cut it; not only does it run high processor on my server, it doesn't actually filter stuff like google does. Google almost never goes wrong for me; actually; I have had very few mistakes / missing mails. While with spamassassin, I'll be carefully inspecting 5k mails / day while still getting spam in my inbox.

I think my mailbox is a bit weird now, but it'll be quite normal as it'll be normal for people to have a mailbox since birth and taking it with them till they die. If you run a few businesses along the way, getting to 1 million messages is not hard; spam will find you as well. Media messages are getting common; I make a point of using dropbox/sftp for attachments, but not everyone does that, so I do get videos, huge blah megapixel cameras of birthdays of family etc. This is normal and will only grow; the current mail solutions don't handle it well. If you want to deliver a competitor to gmail, you need to make this work imho and it needs to be a test case.

Yeah, the gmail UI (which is forgettable) is about 5% of the product; the other 95% is:

* spam detection

* search

* mail header hacks

I don't think that can be approximated in "a few months".

> 50 mail addresses coming to my inbox

On the other hand, no product (open source or not) should optimize for the outliers (not to be confused with edge cases).

What do you mean with mail header hacks?

Mail headers and mail relays are notoriously inconsistent.

http://www.jwz.org/doc/threading.html

Thanks for posting the link!

Like I put in the tail of my post; I don't think i'm an outlier anymore soon. The 50 email addresses; sure, but the amount of mail, maybe for now, but not for long. And certainly not (already!) in size; there are plenty of people who get/send all their video/images/large docs via email.

Edit: also, the Google team was really good to me and my (free! well I pay more for storage obviously, but it's not a google apps one) mailbox; on several occasions they contacted me when my mailbox was not functioning and helped fix the issues with it. Compared to a few years ago, the gmail backend, for me has gotten a lot better; search used to just stall; now it's always fast and accurate. And when I deleted a significant amount of mails, my mailbox would be unreachable for at least 1 day (giving a Oops error code 14 or something like that); no it just works.

Edit2: I rather would like the spam AI from Google as a webservice for all clients to use, or an open source alternative. Spamassassin is a nice initiative, but it never worked for anyone I know. A lot of my colleagues just ended up routing their company mail through gmail to get a better spam filter.

You are an outlier and will always be an outlier. I've had email since I got my first modem in 1985. I've had hundreds of email addresses between BBSes, online services and the Internet. I've happily deleted all the mail and given up the addresses for all but a handful.

What might change at some point is A&E launching a Digital Hoarder reality show wherein people with such a storage proclivity can get the help they need.

That would be one massively boring show :)

Maybe you are right. I don't think so though; people are lazy and if the default status is that you can keep everything you have anyway (considering the allotted yearly storage growth and the plummeting prices of storage in general) you won't bother deleting anything.

You seem to come from the same time as I do ; I had my first modem around '85 and was very active on BBSes. I ran my BBSes, for most of the 80s, of 720 kb disks, so I do know what storage limitation was. It's just not there anymore. Gmail now easily handles all of this and so should some other solutions imho, especially the ones wanting to compete with gmail. Google said; 'never delete another mail' and i'm sure people got used to that by now for all cloud services.

The mail address and email hoarding came when I started making serious companies. And it's not like they bother me or whatever. Maybe I will throw everything away some time, but for now; I dodged an almost million euro gov fine a few years ago by having a complete mail conversations + attachments with my employees between 2002 and 2004. I would be in a whole load of crap if I wouldn't have those anymore.

You said the following two things which are mutually exclusive:

I wrote an export script for the mail system I was using at the time and imported 10 years of mail into gmail.

I don't think so though; people are lazy and if the default status is that you can keep everything you have anyway

The default status is that stuff goes away. I believe you're arguing that "this time it's different" but I see no evidence for that. In fact, there is a post almost every day on HN about Gmail users looking for an alternative. I myself can't wait to get off Gmail. When I do that I'll happily explicitly delete my Gmail account and all the email in it. Because I am lazy and it's easier to do that than it is to write a script to transfer it to what I use next.

You seem to come from the same time as I do ; I had my first modem around '85 and was very active on BBSes.

I got my first modem in 1985, got a phone line as my Christmas present from my parents in 1986 and started my own BBS. Started a consulting company in 1993. Started a small ISP in 1996, sold it in 1998 and started working for corporate America.

In the U.S., I find from a legal perspective that the less documentation you have the better. That's why every large U.S. company has an email retention policy that is very short. Anyone looking to sue you needs proof and often that proof comes from your own records as part of discovery since they don't have any proof. If as part of a standard process you delete everything you're not legally required to retain then you can't be forced into helping their case.

> I wrote an export script for the mail system I was using at the time and imported 10 years of mail into gmail. > I don't think so though; people are lazy and if the default status is that you can keep everything you have anyway

My, old person who grew up in the 80s, case versus people I know and talk to who are <= 20 y/o now.

> In the U.S., I find from a legal perspective that the less documentation you have the better. etc

This particular thing was a large project, paid for by a gov grant (not sure if that's the correct word, but I think it means the same) early and ran from 2001-2009; the grant was for millions. A few years ago, suddenly they came to check on the project; see if all paperwork is in order, hours written etc. Now during one of the office moves, someone misplaced the actual paperwork we prepared for this including CDs with source code, revision history etc (Now only if there was some kind of storage on internet to store this on...). We would have to pay that money back + fines if we cannot actually prove we did the work and who did it. In your case, we would've had to pay the money/fines, because outside my mail, there was no way to prove anything. And these gov checkups come years and years after the fact.

Granted; this was a stupid mistake of someone who thought the boxes were full of outdated crap, maybe even an external mover, however, I don't see the downside with keeping everything in this case. Especially if you have a company. In the US you have all kinds of legal issues; I understand that, we don't have that really, so worrying that someone will check your mail, even in courtcases, is not very likely.

I think you're absolutely right. We've actually been thought by Gmail not to remove messages but instead archive them and that's a good philosophy.

I've been on Gmail since 2005 and have about 100,000 messages at 20 GB. There's messages coming in from 25 different addresses at 300/day with 100 being spam.

I don't feel like I'm an email power user though, I imagine people in sales do about 100x more than this.

It's definitely a requirement to be able to do this and I feel even if we would be outliers, in a couple of years we won't be anymore.

And at least Google seems to think so; they worked with me to get my mail fast and working well. They asked me to turn stuff on and off, reroute addresses for a while and such until my mail (as it is now) was fast and solid. I think they do expect a lot of people to be like me in the future.

> On the other hand, no product (open source or not) should optimize for the outliers (not to be confused with edge cases).

I think (hope!) there's room for niche products. I would never try to make an email client "for everyone". All the ones I've tried are crap (because they're "for everyone").

If you'd said: define your market and optimize for that - then I'd agree. But to not aim for a niche if you're doing a small project sounds just crazy (and wrong).

> define your market and optimize for that

Right. But if you're writing a gmail clone, just don't get sidetracked by the token comment about email archives from 1985.

Add to that the speed. Roundcube looks pretty good, even Horde's could make a cut GUI-wise (judging by screenshots from the latest versions), but they both lag in the speed department, comparing to GMail or Fastmail.

My hunch is that Google's spam detection is not necessarily better than open-source solutions. It's just that they may have a larger corpus of data to train the system on.

If I host my own email server, my spam filter will only have my inputs to determine what is spam.

This leads me to an idea. There probably can be a service to build a dynamic corpus of spam samples. If I mark a message as spam, my email server can send a sample to the service, and periodically, my email server will download "top 100" samples to feed its filter.

This isn't necessarily a good idea. The type of SPAM you receive is different than the type of SPAM I receive. Moreover, it is very possible that I do (not) consider some messages as SPAM whereas you do (not). It is true that it takes some time to train the filter, but after that initial period you will get better results if you only train it with your own SPAM.

I've been hosting my own email on a small personal server for some ~7 years now. All my email addresses redirect here (I have 9 frequently used addresses, and a bunch of website-specific ones). I'm using spamassassin (plus amavisd-new) as a proxy content filter. These are my filter's numbers for the last three days:

      2 BAD-HEADER-2
      1 BAD-HEADER-8
     47 BANNED
    297 CLEAN
      6 INFECTED
    426 SPAM
     14 SPAMMY

Spammy mails are delivered to a special "Spam" folder (just like in GMail). All those were indeed SPAM messages. I've received no spam messages outside of those. The high number of CLEAN mails is because I'm subscribed to a few mailing lists and I receive notifications/tracking info/alerts from a number of services.

TL;DR: SpamAssassin is hard to setup and train correctly (just like the rest of a server's mail-stack). However, once you get over this part, the results can be as good as gmail's.

> I've received no spam messages outside of those.

How protective are you of your email address? My email is posted all over the internet, but I haven't seen spam in gmail for at least 6 months--probably longer.

I have a special "don't care" mail account whose mails are delivered to its own folder (included in the shown stats). I do get a spam message in that folder very occasionally (as in once per month), but it's really far off from becoming annoying.

I have a huge spam problem with gmail. Someone sold my email + first name. Googles spam filters are simply powerless if you include a first name in the email. I've reported these as spam (several a day) for months at this point.

Hell, it gets marked as important.

> Googles spam filters are simply powerless if you include a first name in the email

I think you're going to have to look elsewhere for a cause. Someone (probably several someones) sold out my email and first and full name. Gmail has no problem correctly categorizing all sorts of spam addressed to me by name.

Whatever the cause is, it's still Google's fault/problem.

I too have a big gmail spam problem (related to my email address being very short and guessable), and mark as spam on average about 10-20 emails a day.

> Whatever the cause is, it's still Google's fault/problem.

Why is it that no matter what the cause is, you feel that Google is to blame?

Err... because Google's spam filter is letting the message in? Seriously, it is a spam message, delivered to Google, and Google put it in the inbox instead of the spam filter. I don't know how much more clear cut it could be.

I may have an explanation for that. Gmail is very good at detecting waves of spam mails while the wave is being sent out. Basically, a (bunch of) hacked server(s) start sending SPAM mails. After some messages google realizes that these servers have been hijacked and starts rejecting mails from them.

If your address happens to be among the first of the spammer's list, you will be hit by these emails. Otherwise when they send to your address google has already stopped accepting mails from that (those) ip(s) and you don't receive anything.

It looks like enjo's address comes way before than yours...

Isn't that pretty much what Vipul's Razor does? http://en.wikipedia.org/wiki/Vipul%27s_Razor

Wasn't aware of that; thanks!

... and on top of all that is the simple fact that gmail is backed by a very robust infrastructure and supported by company that's very unlikely to go out of business anytime soon. [You can make jokes about Google reader etc, but gmail is far too important to Google's public image to make that a reasonable concern.]

Together with the very decent UI, great spam handling, and free space, this makes gmail a pretty good choice for a lot of people.

Even with a good free version of the software, duplicating the entire package is a huge task ...

I would happily pay someone non-google to get those 95% things right and then add in mobile sync, gmail-like interface, and you've got a customer.

What parts of gmail's webmail interface do you find indispensable?

For me, nothing; the huge thing about Gmail is it works flawlessly from my Android phone. No IMAP solution came close. These days I read most of my mail on the phone.

Once I was in a long check-in line at the airport. Silly me, I figured I wouldn't check in at home because I had to check bags. At the airport I got on my phone, pulled up the confirmation email, checked in, and went to the bag drop line. With the clumsiness of IMAP and dealing with it on phones I would never have managed to do this if I was hosting my own mail, and even if I had managed to do it, the cost would have been the hours and the hassle of maintaining mail.

It was at that moment that I realized I would never host my own mail again. Just not worth it.

Search is my biggest issue too. I switched out of gmail to my self-hosted server using roundcube and searching through IMAP is excruciatingly slow compared to gmail, even after I moved the mailboxes to an SSD.

I am not sure whether this is a limitation on the IMAP protocol, but I suspect a proper replacement to gmail might involve coding a new mailbox backend and protocol.

I may be wrong, but I believe IMAP, the protocol, just defines how IMAP clients/servers should issue and respond to search commands, but doesn't deal with mail indexing and things like that.

Cyrus IMAP, which I use, provides Squatter, which substantially speeds up search, but the IMAP client is still the interface. Performance I get is around a sec for a simple query, a few sec for a more complex one against a folder with 20K messages.

Last I checked, Roundcube had very limited search capabilities. Thunderbird is much better, but I'd prefer a terminal based search. Something like SQL. Does anyone know of a tool like that?

Just out of interest: do you know if mutt (plus maildir and, say, getmail) cope with the volume?

I used mutt on the server as email client before switching to gmail. Simply because no desktop client would cope. So at least till 2005 it was fine with mutt; I had to ssh (that was fine), spam issues (nothing to do with mutt, but still) and worse; I was running a business; attachments were a big issue. I had to save them and scp them over. Later I just created a mailbox binaries@... to which I forwarded the mails with attachments I wanted to see, which I then retrieved with a pop3 desktop mail client. It's just not very practical when running (a) business(es).

I would imagine there are better solutions for the attachments now; in 2005 you couldn't really find that much to solve it as I remember.

Thanks. I guess mutt plus bayesian spam filter might work reasonably well.

I wonder when filesystems became good enough for maildir to work with your volume of emails.

> I wonder when filesystems became good enough for maildir to work with your volume of emails.

'99-2001 I designed and deployed a webmail service where we handled ~2 million user accounts using Maildir's on a single SAN device formatted with ReiserFS (the fridge sized device in question was not much faster or higher capacity than my current laptop.... Actually the whole infrastructure was probably not much higher capacity than my laptop). Many of us working on it had individual folders with tens of thousands of messages.

"Pure" Maildir without the client doing additional indexing will quickly get very IO intensive, but storing hundreds of millions of small files in a single filesystem was a solved problem by then and with some simple indexing or caching its trivial to handle that volume.

We handled the indexing needed to reduce the IO load imposed by the clients by writing a custom POP server and delivery agent that maintained some more metadata - partially in the filename of the messages (read/unread status etc.) - saving us a stat() call per message on scanning the folders. Partially by maintaining cache files with even more details, saving us re-scanning the message headers (accessed by the web frontend via a couple of custom/non-standard POP commands - much easier than contemplating dealing with the horrors of IMAP; for actual non-web client access we put proxies in front of our "special" POP server).

Thanks for the report. Do you think maildir is the way to go for storing email (plus additional indices), or do you think a more custom format (eg in some relation database or something completely different) is better?

I prefer Maildir but mostly because it is trivial to manipulate because it is so simple. Our design back then was based on the assumption that we could blow away all the cache files at any time, and it could all be trivially rebuilt from just the raw messages.

I still have a soft spot for flat files when possible for that reason.

Also for our usage we saw a relational DB as a scalability hassle, as we needed to be prepared to scale this to many backends, and dealing with an RDBS when each set of objects was specific to a user was not very appealing. Today the challenge in that is much smaller due to faster hardware, and there are far more options in terms of database solutions.

But at the same time there's just not that much to gain.

Yes, I like maildir, too, because I can use all the well known unix tools (like grep) to manipulate and filter my emails.

I had to save them and scp them over.

I just mounted my ~/mail/attachments locally using SSHFS. Saving them with Mutt made them automatically "appear" on my machine. You could also install a few viewers on the server and then use X11 forwarding, which works fine if you have a decent connection.

I've been using mutt for 16 years and self hosting email. It's currently hosted on bigv.io. Spam is handled by spamd (ships with OpenBSD) and SMTP by OpenSMTPd. Local delivery is via procmail.

I have 40,000 messages archived with attachments etc.

Granted it's not pretty but it's fast, efficient, can read html mail fine and is pretty immune to spam.

The best thing is I haven't changed it for years.

I admire your purity. But what about viewing images in a mail?

Easy. Add to .mailcap:

   image/*; yourviewer %s

Then hit "v" on the image attachment and it pops up in a window. I use xv (still) to view it which runs under X. Hit "d" to download it.

I handle HTML mail, PDFs, word documents, everything with complete ease.

The insufficiency of the alternative spam solution is what I was afraid of. =/ It's easy to forget how awesome Gmail's spam filter is when you get spam in your inbox so rarely that you almost forget it exists at all.

I keep saying that Google could make a mint if they licensed an API for it. They probably just don't want to bother and would prefer that they keep its talents exclusive to Gmail. Also, I suppose that the people who'd want to utilize it would be leery of running their mail through Google's servers.

I have had very few mistakes / missing mails.

It's still pretty good, but I have been getting a lot more false positives in gmail lately than I used to. Even a security code from hotmail got redirected to spam, which I thought would be common enough (and important enough) that Google would have learned about it already.

Weird, I have exactly the opposite experience with spam -- my self hosted setup does better than gmail. Wrote this a while back, but still holds true: https://news.ycombinator.com/item?id=5441542

I don't have nearly the volume of email you do though. My inbox has 20K, probably another 70K in other folders. Thunderbird is fine for me, except I'd like a more powerful and faster search. Gmail search is not good enough, though much better than other webmail clients I've tried.

Collaboration or integration with Mailpile [1] might be a useful direction to consider. They are doing interesting work on the backend, but as far as I am aware, there isn't much frontend present yet.

[1]: https://github.com/pagekite/Mailpile

I really like it - good work, keep it up. It's a good thing that you guys (you, Mailpile, Roundcube) develop alternatives to gmail.

However, for me to switch (like most here, I am a heavy email user), I need a few things:

- PGP encryption

- Contacts

- Calendar

Have you thought about expanding the developer base? Maybe via Kickstarter/Indiegogo? The alpha already looks good, I'm sure a lot of us would like to contribute to the development.

It would be nice it supported PGP out of the box, I think it would be a strong selling point. Kudos for the initiative.

It's on my todo list.

See if you can do it in the browser.

http://openpgpjs.org/ could work? Never tried it.

How effective would it be to self host your mail regarding spam?

In gmail I guess that once a few peeps click "Report Spam" on a mail that passed filters, similar mails are also flagged as spam in other accounts. I've always found gmail extremely effective at this, and I practically never flag mail as spam myself.

Sure, there are automated spam filters to configure, but overall wouldn't going alone make things much less efficient on that topic?

There must be a spam filtering as a service? Send anything that wasn't signed and/or you're not 90%(?) sure isn't spam?

http://spamcheck.postmarkapp.com/

http://akismet.com/

There was Postini, which Google bought. You'd set your MX record to them, and they'd forward your non-spam email on to your SMTP server.

We use Postini at work and it is terrible. Normal Gmail filters are much better. I hear Google is replacing Postini with something they developed themselves.

Maybe having an online database of spam emails that every self hosted mail can access could be an answer.

https://wiki.apache.org/spamassassin/UsingRazor

It's less effective for sure, but depending on your situation it might be good enough. On email systems I maintain the only filtering I set up is AV with Clamav. One reason for this is my huge dislike for solutions that statistical filtering provides -- you end up having to check the spam folder or whatever anyway. I just don't see the point. The only solution I think is a real proper solution is for spam never to hit any user mailbox in the first place, but that will probably never happen. So just I limit spam filtering to checking strict SMTP compliance for the messages (message format) and Spam Haus filtering as the last measure. I can't imagine any legitimate email ever tripping format/protocol checks, and I was amazed how much spam just those checks eliminated. Spam Haus cuts on that a little more. Sure there is still spam hitting the inboxes, but I think it's manageable (haven't had complaints), say around 30 messages/day for a busy address (for example, the main address [relatively widely published and always unprotected] of an international organization). YMMV of course.

EDIT: Greylisting would probably cut down on this further drastically, but I'm just too lazy to implement because no one complained yet about too much spam. :)

I couldn't imagine not filtering my mail. I hate seeing spam too much. To a point where minimizing it is an obsession. I spent a considerable amount of time on this about 10 years ago, but very little since [1].

Anyway, just did a quick search through mail.log and in the past 24 hours, I got 4017 SPAMS that were rejected at SMTP time by Postfix [2], about 40 SPAMs got through those, but caught by SpamAssasin [3]. 0 spam delivered to inbox. 0 false positives. 126 legitimate mails.

I'd looked into greylisting some years ago, but never implemented because I didn't like the idea of delivery delays.

So I'm in complete agreement with you regarding how effective SMTP level checks + rbl are, but I disagree regarding statistical filtering -- Bayes was key for me to getting 100%. I had put off adding this layer for years, because I didn't fully understand or trust it, but now I'm a believer. Also, once trained, it just works. I rarely look at my spam folder. Retraining is easy, I just drag a rare false positive/negative to HamCaught/SpamMissed folders then nightly training script auto learns.

[1] I was an email admin at a local gov agency, then tried to get into email hosting biz, so it wasn't just personal obsession.

[2] Using things like standard helo/client/sender/recipient restrictions, with very few custom rules, and rbl check using spamhaus.

[3] SA + razor/pyzor/bayes, again with very few custom rules.

I do love the attention to detail, with the Nigerian's prince email.

Doing good software is one thing, doing it with a good touch of humor is what makes it stick. +1.

I'm looking forward to a day when a prominent model will be cloud-based applications storing data in private repositories.

Thanks for working on this.

I look forward to the day when it will be easy to set up and configure a mail server at home. By all accounts, it is currently a very painful process that is prone to error and interrupted service.

Doing it at home probably isn't a practical option unless you somehow get an IP that isn't blacklisted, but doing it on a VDS isn't that bad.

I host my email on a VDS. I would kind of like to be able to host other people's too, but I've always been freaked out by having the ability to read my friends' emails.

er. this is probably a dumb idea. works for 1 person only i guess. you can use google's smtp after signing up for google apps. http://emailrelay.sourceforge.net/ is 1 of the ways to relay(actually more forward than relay i guess) via smtp.gmail.com. now you're not blacklisted? even if you're running your server at home on dynamic ip? now u just need to be able to update enom or dyn whats your ip as it changes?

Happy to see more alternatives to Gmail. I am still using Gmail with the old HTML interface, as the new one is cluttered with all kinds of stuff I do not need. Stay with KISS, when it comes to functionality and interface, with optional integration of PGP. I wanted to quit using google services for long, but it is only now that I get active in that regard (e.g. duckduckgo as search engine). As such, I am searching an alternative for the

1. web based storage of emails 2. a good web interface.

As for 1., I stil have not found anything really interesting that comes for free.

As for 2., I was once using Mutt and found it quite efficient, and am now thinking to try out sup. However, there is still this feeling that there could be a better client when it comes to usability and ease of installation. Using Mutt on Windows can be really annoying.

Very cool. Looks like a really good web interface that would be a pleasure to use.

On a personal note, am I the only one that generally prefers a mail client? The ability to combine all my emails (work, gmail, @mydomain, etc) into one unified inbox is why I prefer it.

I do as well. That also opens up other possibilities like the ability to just drag and drop an email between IMAP accounts. The other day I was getting smtp errors from Google, so I sent a work email from a personal account, then dragged it to work sent items, because I won't remember to search for it in personal.

I think we'll see a lot more of this type of software in the near future.

It's easier and cheaper than ever to automate the setup of servers. Imagine being able to click a few buttons, in something like the webmin of yore, and suddenly having a private mail server/file sync node/document editor application, set up at the VPS provider of your choice.

This type of thing will encourage open standards, as the private servers will need to communicate with each other. It also ties in nicely with concerns about the implications of everything being hosted and controlled by major providers.

There are probably business opportunities at many points in this model.

The Distributed Everything blog post encouraged me to leave most of Google' services due to their closed ecosystem and the violation of privacy. I managed to leave everything (GTalk/Hangouts, Agenda, Contacts, Google+, etc.) except Gmail because of how great and useful filters and labels are. In my opinion, these two points are the reason why Gmail cannot be easily replaced. I thus would encourage working on these futures.

Also, I don't want to sound like a troll, but I wish it was build in a more popular language, such as Java (it's not my favorite) to encourage contributions. Good luck!

The main reason I'm using GMail is that it's available everywhere for free - both in money and my time.

However, I like your project and it would be nice to have an alternative if I decide one day that I value my privacy more.

How much of GMail does this do (or intend to do)? Off the top of my head, GMail does this for me (in this order):

- Email server that I can run for my domain.

- Search a large number of messages on the server

- Spam detection on the server

- Automatic labelling / categorisation on the server for incoming mail

- Address book

- Saving of drafts on the server so I can edit and send on several devices

- A webmail interface

Is this intended to be a GMail replacement or another webmail interface?

I want this to be a replacement. I haven't yet set up spam filtering and search but I'll get around it eventually.

This is just my MVP.

Cool. Well, good luck. (I don't think this is quite an MVP.)

Quite a minor point of feedback, and not altogether fair, but attempting to obscure your email (~) gives me a bit of a lack of confidence in the spam filters you're using. I have my gmail up in plaintext on my homepage, and don't experience too many problems.

If nothing else, putting your email in plaintext could help you debug your spam prevention?

I haven't setup spam detection yet, so that's why I obscured my email.

The expression "an ounce of prevention is worth a pound of cure" applies to spam too. The best spam filter is the one you don't need.

Kudos! I think the hardest part will come in trying to maintain and scale this from an interoperability perspective. I used to work at an ISP where 80%+ of the work on the mail product was getting our IPs un-blacklisted and other ISPs from not blocking our traffic anymore, as well as pruning out the bad/bot users.

Support keyboard shortcuts, at the earliest. It is one of the most heavily used features for heavy email users.

The biggest pain I've experienced with similar web interfaces was mobile. For the desktop there are a few that are decent, but on mobile they generally suck. So do the native email clients for Android and iOS.

Do you plan to stick with a desktop version? Will you always design 'destop first'?

Would be nice to see people get together and finalise a viable alternative. Keep up the good work! Meanwhile, here is another potential solution. https://assemblymade.com/amail

Afterlogic has a free lite version

I like it compared to other scripts like SquirrelMail or Horde.

webmail.afterlogic.com

This is pretty neat. Not sure if these are features available in the pro version but two things I think it's missing are (or that I didn't see in the demo): - multiple identities, so that when sending email you can pick the from email address + signature - setting filters for any header. This is useful for filtering mail based on the spam level header so that they go straight to the spam folder.

Really nice web interface though.

I think the demo is the pro version already. They have a decent forum: http://www.afterlogic.com/forum

I installed it on a hosting account and it works well. But I still fighting to get a tiny VPS running to host my email. I plan to co-run Afterlogic and alpine via SSH. Alpine allows for multiple identities - you can put whatever you want in the FROM field if you enable this in the configuration settings.

Great Job. Ignore all the other projects that are out there please (If they are not BSD, they are useless for me). This is what I was waiting for. I would recommend developing both mobile / desktop together.

check out https://github.com/roundcube/roundcubemail for a mature gmail alternative.

Tried out the demo; it's nice. Keyboard shortcuts and threaded display would be nice to have. Will check out the source for my learning.

Search is mentioned as a keep feature a lot in this thread. Would using elasticsearch be a feasible solution? It certainly scales easily.

XMPP would be cool too and suddenly you have a chatbox.

Would be cool to have in all the "home server" platforms (Freedom Box, ArkOS, ...) too!

Great, as far as I know Roundcube is the only decent open source webmail thing, and it could do with some competition.

No, there is Horde and Squirrel mail.

I myself prefer webmail.afterlogic.com It has a free light version.

I actually use squirrel mail but its interface is still stuck in 1999. Roundcube is pretty good but I didn't like it very much the last time I installed it.

Horde seems to be full of bugs regarding security. Several of my friends run it on their servers and had the server compromized. Looking at web server logs, the backdoor was always some part of Horde/Imp web interface.

They have all moved away from it since, so I can't tell if all of those have been fixed recently, but the whole architecture of it is not something I would trust.

have you seen/heard about mailpile?

Yes I did. I figured out that I should go ahead anyway. More choice is always good, right ?

Yes. Go ahead. Competition is the driver of innovation.

However I am afraid without the opportunity to put a man year work into it, it will be hard to tick all the boxes.

I really hope you succeed, though.

(Disclaimer I am a backer of Mailpile)

Not necessarily. Focusing talent on a collaborative effort can make one great product instead of many not so great ones.

Can you give me the 3 second overview of mailpile?

Would love a gmail alternative for hosting email on various domains. Look at fastmail - but it's not quite what I want yet.

I thought mailpile was a native application rather than a web thing?

It's webmail: http://www.mailpile.is/

It looks like it's a native app with a web UI.

It says 'web-mail' ; they wouldn't say that then would they? Maybe someone here knows more about that?

https://github.com/pagekite/Mailpile says:

> Mailpile has a built-in web server and will eventually include a proper web-based interface for searching, reading and composing e-mail.

I'll have to check it out and work out what it actually is/does sometime.

Is there a mailing list to track this project? I can't see any way of monitoring updates on the page.

Can anyone comment on Spamassassin's efficacy as compared to Google's spam filters?

I commented on it in another comment on OP.

It's fucking terrible by comparison.

I beg to differ.

This looks great. Just curious: Roundcube didn't cut it for you?

I appreciate the hard work. Nevertheless, IMHO it would be better if open source projects will bring original ideas to the market, instead of copying an existing product.

Yeah, RMS shouldn't have bothered making a free version of UNIX. Linus shouldn't have have made a free, sub-standard version of MINIX.

The Dovecot devs shouldn't have made Dovecot when we already had UW IMAP. Postfix is redundant, we've had Sendmail for decades.

They all should really have done something original instead.

I'd argue that post-PRISM, there's a big opportunity to re-create self-hosted options for services known to be open books to the government. It might not be creative, but it does provides users a familiar alternative to storing their data in the cloud.

> post-PRISM

Post? You optimist you. If anything we're smack in the middle of it.

Squirrelmail?

Nice client - but looks so 90s. No offense meant, but i have the mail client open all day. I need a fast, responsive client with a good search engine and spam detection. And a good look. Not sure if squirrelmail could do it.

What is the benefit of a web client if you keep it open the entire day? I find standalone clients, like Sylpheed, much easier to use and you're not constrained by what you can run in browser.

Agree, but all day means several different systems as well (Tabelt, Notebook etc pp). I prefer to have everything online

+1. Would use it.

Reply on news.ycombinator.com