Hacker News Reader

As far as I can tell from playing with many of them:

- Almost all (open source and proprietary) prefer mbox-style

- If not mbox, then it is monolithic file per folder, very few if any store email per file like maildir, despite it having many benefits [0] (if you are smarter than Bernstein on such things, please do write on these things; no one gives a shit to explain email out because it is not sexy)

- The index system always stores data separate of the email store; I have not seen a counter example, perhaps for Outlook, which I presume uses the Windows/OSX OS-level indexing

- To get back to my point re generic FTS, there it is, keep the OS ones so people can search in one place (I use recoll and notmuch for email specifically; it's annoying)

- What optimizations are there for indexing beyond header info? Again, good FTS systems can be extended (for what I define as good)[1]

I do not mean to criticize your points, but I do support work and all of these clients do good for non-technical users, but are ticking times bombs that shit themslves and require resyncing all data when they corrupt massive multi-gig files, let alone re-indexing it all.

Using them is one thing, having to fix them is what informs my opinions on this and why I sync to maildirs, use mutt-kz, and webmail depending on mood and time.

[0] https://cr.yp.to/proto/maildir.html

[1] https://bitbucket.org/medoc/recoll/wiki/IndexMailHeader apparently they have improved, perhaps I shall ditch notmuch out of notmuch discipline for email! Haha

Well you're right that we can have a generic FTS system with a general OS based search with specific handlers for mailbox types. I guess you're trying to say that the file formats have become too complex where the clients themselves can't implement them bug-free. About that, perhaps another way to tackle that is not to go back to a more "primitive" format but to improve the quality of email clients.

Personally, I never have IT problems, so I can't comment on the support burden. But I agree with you that if I had to re-prioritize, then reducing the support costs by trading performance and going back to a simpler format is probably worth it.

If a filesystem is a poor match for an email database, I'd appreciate the use of a well-established, open, and documented database format. That is, a format not only your unique email client can understand. Something with an independent manipulation tool. In the perfect case, something that can be interoperably used by several different email clients (not in parallel, of course) and / or third-party indexing / filtering / other processing tools.

SQLite is used in many cases to store smaller things. It's likely not a good fit for a few gigabytes of email with binary attachments, though. I wonder what would be? NoSQL document-storing databases seem to have been created for a purpose like this. Is any of these used by any remotely popular client?

Again, shoving everything into one file is mbox, which is a recognized file format (if not database), and it does not help things that much.

I have wondered the Sqlite thing myself beyond my one criticism. If anything, you would love K9 Mail on Android, which does this out of necessity.

And some people have made SQL and noSQL email stores.

http://nosql.mypopescu.com/post/3083070885/couchdb-as-email-...

https://www.npmjs.com/package/haraka-nosql

http://www.dbmail.org/

>I wonder what would be?

Well, maybe the people writing email clients have already thought quite a bit on this topic, likely created several prototypes, benchmarked them and chosen one that works the best given their constraints and features. Now all that's left is to open the format specification :)

There's nothing stopping a mail client from maintaining arbitrary indexes, caches, etc. in addition to the maildir.

Sure, every time you deviate from a targeted design you leave performance on the table. Whats acceptable to you, may not be to others.

Performance doing what? Email has several operations and none of these clients are performant at scale.

You know why mutt can be for me? I have a second program (isync/mbsync) sync the maildirs, a third to index it, and mutt is just for reading and sorting.

When you have one monolith doing all 3 at its will all the time, you get calls from users that Thunderbird and Mail.app are janky and the computer is hosed. It is interesting as FTS came to Thunderbird and they ran the indexer all the time, until they intelligently designed to have better controls around its execution as it destroyed general usability. Mail.app is better, but I see executive running it for days and it becomes a misbehaving pile of mess.

I am asking in a respectful tone and not to make light of your position, but what do you use? I want to get a better look into your view and I think this might help me understand you better, if you are willing to indulge me.

UPDATE: And when I say several things at once, ironically this is not by design! Email has changed a lot since the sendmail days (and that is cutting edge with respect to many veterans here) and thus there is well-known and accepted design I can think of beyond layers of addition on clients to meet definitions and changes in protocol, routinely made without any discussion of client operation beyond standards for file structures like mbox and Maildir, even they are formal standards, I would have to check,

>Performance doing what? Email has several operations and none of these clients are performant at scale.

Well OK, so the most important for me is search, and the second most important is being able to quickly apply labels based on filters on a bulk-scale. I have found mutt to not really be performant and the workflow that I achieved was rather brittle (complicated by other factors like having to deal with HTML email). I'm currently using Gmail, but I am experimenting with outlook and I quite like the performance. I don't have a ton of email, maybe around 3-4GB, 25K emails. Bulk operations like 'select all' - do X on several thousand emails takes seconds (which is still slow IMHO FWIW). I haven't seen such performance with other clients. Granted, I haven't broken down every single client and benchmarked different workloads, so I'm only speaking in generalities.

And don't worry about offending me. I was just trying to stir up some conversation. I don't take offense if someone disagrees with me on the internet.

No worries, then. You like me enjoy stirring up to learn.

As I suggested, I slip everything up into different applications. I use mutt-kz and not much with isync to sync maildir, view it mutt, and do label/filtering like Gmail locally with notmuch on a Xapian FTS subsystem backend. It's fun.

Then keep exactly your preferred high-performance database design, but derive its data from maildir, such that all mail is persisted in maildir first.

Yes, that would be a good way to go.

I am glad you made this point; I forgot to.