Comment by 616c | Hacker News Reader

Comment by 616c | Hacker News Reader

I will not beat a dead horse with my "mutt is superior because they know the best they cannot do is suck without confessing it" trope, but almost all email clients lack is Maildir.

Even if you are not a technical user, you should respect the fact that proprietary formats will bite you. Look how many this guy alone used. If you use opensource, even, the mbox and mh all email, one giant file paradigm is all nice until it is corrupted.

I wish more clients respected Maildir underneath, or tried to bend the truth on Windows FAT32 and NTFS ad the ; in path names limitation is a pain.

https://en.wikipedia.org/wiki/Maildir#Filesystem_Compatibili...

Sadly even Thunderbird has struggled to re-engineer a Maildir subsystem where it was never intended. Show me any email client devs who had the khutzpah to even dare that.

https://bugzilla.mozilla.org/show_bug.cgi?id=845952

But you have a lot of safety in spreading it across files, and you can throw dumber full text search at it, shell scripts, and a lot more flexibility over all.

Sadly, some Unixisms will never escape Unix culture. As someone who has watched people screw 10+ years of email in Thunderbird n>1 times, I wish for the love of God everyone would use Maildir in the client, regardless of interface, free or not, et cetera.

That's an implementation detail, best left to the developer of the app. What I care about is UI, speed, search, privacy, etc.

I don't agree with you. Like with anything else, less abstraction = more performance. Creating a specific/targeted way to index, store and manage email data has its benefits too.

I'd hazard a guess that most proprietary formats are similar to the maildir format. Its just that the individual files are opqaue at the filesystem level. If you create thousands of tiny files then you are hoping that the file system design - which like any other design can only be optimized for specific workloads - can handle it better than a database type format specifically designed for email.

Then again a generic full text search system (which probably won't even know that they're email files) over those thousands of files would have to be better than a targeted design for email data.

You would miss certain optimizations like.. (making one up now) extracting and storing just the email headers in a separate region to speed up indexing and having some indirection built in to lookup that specific email.

There's nothing stopping a mail client from maintaining arbitrary indexes, caches, etc. in addition to the maildir.

Sure, every time you deviate from a targeted design you leave performance on the table. Whats acceptable to you, may not be to others.

Then keep exactly your preferred high-performance database design, but derive its data from maildir, such that all mail is persisted in maildir first.

Yes, that would be a good way to go.

Performance doing what? Email has several operations and none of these clients are performant at scale.

You know why mutt can be for me? I have a second program (isync/mbsync) sync the maildirs, a third to index it, and mutt is just for reading and sorting.

When you have one monolith doing all 3 at its will all the time, you get calls from users that Thunderbird and Mail.app are janky and the computer is hosed. It is interesting as FTS came to Thunderbird and they ran the indexer all the time, until they intelligently designed to have better controls around its execution as it destroyed general usability. Mail.app is better, but I see executive running it for days and it becomes a misbehaving pile of mess.

I am asking in a respectful tone and not to make light of your position, but what do you use? I want to get a better look into your view and I think this might help me understand you better, if you are willing to indulge me.

UPDATE: And when I say several things at once, ironically this is not by design! Email has changed a lot since the sendmail days (and that is cutting edge with respect to many veterans here) and thus there is well-known and accepted design I can think of beyond layers of addition on clients to meet definitions and changes in protocol, routinely made without any discussion of client operation beyond standards for file structures like mbox and Maildir, even they are formal standards, I would have to check,

>Performance doing what? Email has several operations and none of these clients are performant at scale.

Well OK, so the most important for me is search, and the second most important is being able to quickly apply labels based on filters on a bulk-scale. I have found mutt to not really be performant and the workflow that I achieved was rather brittle (complicated by other factors like having to deal with HTML email). I'm currently using Gmail, but I am experimenting with outlook and I quite like the performance. I don't have a ton of email, maybe around 3-4GB, 25K emails. Bulk operations like 'select all' - do X on several thousand emails takes seconds (which is still slow IMHO FWIW). I haven't seen such performance with other clients. Granted, I haven't broken down every single client and benchmarked different workloads, so I'm only speaking in generalities.

And don't worry about offending me. I was just trying to stir up some conversation. I don't take offense if someone disagrees with me on the internet.

No worries, then. You like me enjoy stirring up to learn.

As I suggested, I slip everything up into different applications. I use mutt-kz and not much with isync to sync maildir, view it mutt, and do label/filtering like Gmail locally with notmuch on a Xapian FTS subsystem backend. It's fun.

I am glad you made this point; I forgot to.

If a filesystem is a poor match for an email database, I'd appreciate the use of a well-established, open, and documented database format. That is, a format not only your unique email client can understand. Something with an independent manipulation tool. In the perfect case, something that can be interoperably used by several different email clients (not in parallel, of course) and / or third-party indexing / filtering / other processing tools.

SQLite is used in many cases to store smaller things. It's likely not a good fit for a few gigabytes of email with binary attachments, though. I wonder what would be? NoSQL document-storing databases seem to have been created for a purpose like this. Is any of these used by any remotely popular client?

>I wonder what would be?

Well, maybe the people writing email clients have already thought quite a bit on this topic, likely created several prototypes, benchmarked them and chosen one that works the best given their constraints and features. Now all that's left is to open the format specification :)

Again, shoving everything into one file is mbox, which is a recognized file format (if not database), and it does not help things that much.

I have wondered the Sqlite thing myself beyond my one criticism. If anything, you would love K9 Mail on Android, which does this out of necessity.

And some people have made SQL and noSQL email stores.

http://nosql.mypopescu.com/post/3083070885/couchdb-as-email-...

https://www.npmjs.com/package/haraka-nosql

http://www.dbmail.org/

As far as I can tell from playing with many of them:

- Almost all (open source and proprietary) prefer mbox-style

- If not mbox, then it is monolithic file per folder, very few if any store email per file like maildir, despite it having many benefits [0] (if you are smarter than Bernstein on such things, please do write on these things; no one gives a shit to explain email out because it is not sexy)

- The index system always stores data separate of the email store; I have not seen a counter example, perhaps for Outlook, which I presume uses the Windows/OSX OS-level indexing

- To get back to my point re generic FTS, there it is, keep the OS ones so people can search in one place (I use recoll and notmuch for email specifically; it's annoying)

- What optimizations are there for indexing beyond header info? Again, good FTS systems can be extended (for what I define as good)[1]

I do not mean to criticize your points, but I do support work and all of these clients do good for non-technical users, but are ticking times bombs that shit themslves and require resyncing all data when they corrupt massive multi-gig files, let alone re-indexing it all.

Using them is one thing, having to fix them is what informs my opinions on this and why I sync to maildirs, use mutt-kz, and webmail depending on mood and time.

[0] https://cr.yp.to/proto/maildir.html

[1] https://bitbucket.org/medoc/recoll/wiki/IndexMailHeader apparently they have improved, perhaps I shall ditch notmuch out of notmuch discipline for email! Haha

Well you're right that we can have a generic FTS system with a general OS based search with specific handlers for mailbox types. I guess you're trying to say that the file formats have become too complex where the clients themselves can't implement them bug-free. About that, perhaps another way to tackle that is not to go back to a more "primitive" format but to improve the quality of email clients.

Personally, I never have IT problems, so I can't comment on the support burden. But I agree with you that if I had to re-prioritize, then reducing the support costs by trading performance and going back to a simpler format is probably worth it.

Reply on news.ycombinator.com