brongondwana

I'd be interested in knowing what kind of searches aren't working for you. Rebuilding our search infrastructure is on the roadmap (from the current Xapian to something which is easier to duplicate as tags into the client's local search database as well so online and offline searches don't have as many differences).

On behalf of all of us here, you're welcome!

Sorry to hear you had that experience.

There are a lot of delays and cross-checks built into our process for re-enabling access to your account, because it would be catastrophic to let the wrong person into an account because they claimed to be that person and to have "lost their password". So a "don't give access" failure is generally less costly than a "give access incorrectly" failure.

It sure sucks to be on the wrong side of one of those failures though. If you ever do give us another try, I hope it goes much more smoothly for you.

To respond to both of you above. Setting up two of these at once would have increased both the up-front cost (hardware for something like this is well north of a million USD) and the complexity and associated risks. Yes, it's partly a "people want it for their own reasons and we should offer it if it's viable" and partly a "it's good to have service in multiple jurisdictions and experience with them in advance of any further balkanization of the internet".

Re-launching might be costly, but dropping millions more on a second location and splitting our systems more would hav been much riskier (note: many of our customers with 'fastmail.com' addresses have chosen the EU region, we can't segment MX records at a tighter boundary than domain level).

So we do what we can - when you login with username and password, the password is only sent to your region's server (based on a lookup from the username which MUST be global, so it works regardless of which of our servers you hit). If you send a username which doesn't exist, we distribute you to a random region in the same percentage, so you can't use it as an existence oracle.

Tell the world you're REALLY fat without telling the world ...

Damn, $2000. I wish I'd kept my copy.

Did you use the 'litmus' test suite? I found it very useful when building Fastmail's (perl) WebDAV file server implementation.

There were also a bunch of fun things with quirks around unicode filename handling which made me sad (that was just a matter of testing against a ton of clients).

As for CalDAV and CardDAV - as others have said, JMAP Calendars/Contacts will make building clients a lot easier eventually... but yeah. My implementation of syncing as a client now is to look for sync-collection and fall back to collecting etags to know which URLs to fetch. Either way, sync-collection ALSO gives a set of URLs and then I multi-get those in batches; meaning both the primary and fallback codepath revert to the multi-get (or even individual GETs).

The good news is, you have less time to be annoyed by it

You're welcome! We're all very keen on keeping email open and something that everybody can build their own tools for.

We're working on it. We still use an unholy set of earlier versions of JMAP internally for our contacts and calendars; in particular the caldav_sync code is gnarly - I wrote it over 10 years ago when I knew less about calendars than I do now! It's still using an earlier branch of what became the perl Net::CalDAVTalk library interally, even though our frontend API is an almost-up-to-date version of what will become the JMAP Calendars spec eventually.

The downsides of developing and testing this stuff as we were writing it up!

We've finished rewriting the objectid generation to give smaller sized and more sortable IDs (they're an inverse of nanosecond internaldates now, plus some extra magic on the low bits for IMAP appends to avoid clashes)... which we wanted to speed up and reduce disk usage for the offline mode.

Next up is indeed updating to the latest spec on calendars and contacts. Files might take a bit longer, I really want to do some desktop clients for the files system, we have a really nice files backend at Fastmail which is only accessible via our interface or WebDAV right now.

That's not strictly true. IMAP has two things, UIDs which are mostly static (there's a UIDVALIDITY cache invalidation key to let a client know that UID information has been lost and recalculated); and message numbers.

UIDs don't change, but of course they can be deleted so it's a gappy list, meaning you can request even quite a large looking range of UIDs and get nothing back.

Message numbers change in every session, and also change every time you get an EXPUNGE. They're basically an ordered list without gaps, so you do a memmove at the offset of the EXPUNGE each time you get an expunge.

There are efforts like UIDONLY (RFC9586) to avoid having to keep that mapping at all, and there's OBJECTID (RFC8474) to let you cache a lot more even when UIDs are changed or when messages are moved between folders.

JMAP is only JSON over HTTP because that's what all the libraries support. Any data format which provides hashes and arrays would work fine, over any transport. So you could use CBOR or protocol buffers or whatever, over any channel you like.

Join the club, we have compulsive mouse habits.

(am a member of this select club)

This is why I trigger a segfault which dumps core at the spot where I had the printf when the conditions aren't what I want, so I can then open the debugger on the core (obviously: not if I have a copy of the input which can recreate it, if so then a debugger with a conditional breakpoint at the same spot is even better)

We tried very hard to convert FastMail to Fastmail and... it's been about 90% successful but there's definitely a bunch of things out there spelled the old way. We just joke about BIG M occasionally.

[TIL - it's not even as old as me!] https://australianfoodtimeline.com.au/1978-launch-of-big-m/

There are a few different threads on this and now that things are in a stable place I'm going to cross post this to all of them! The larger context is that we're making a major change to how we create IDs for email and mailboxes over the JMAP protocol. The old IDs are a UUID for mailboxId and the first 25 chars of the sha1 of the message for the emailId, prefixed by an 'M'. The new IDs are the createdmodseq for the mailbox prefixed by a 'P' (these are pretty short for most users) and a reverse counter of nanoseconds of the message internaldate (delivery time) for the emailId. This gives good storage density for offline and good data locality in databases for the email listings.

You can see all the code for that in a handful of merge requests in the public cyrus-imapd repository on github at https://github.com/cyrusimap/cyrus-imapd/

Over the past few weeks, I've been helping out with the last bits of code modification, largely the changes on https://github.com/cyrusimap/cyrus-imapd/pull/5539 if you're interested.

This morning we rolled out a build which we'd tested extensively on our staging and staff servers, but missed that for older v19 mailboxes which hadn't been upgraded to v20, the code to check if messages belonged in a thread incorrectly marked them all as missing.

This made MOST emails appear missing for most customers, clearly a very bad situation.

We immediately rolled back, but in the hurry missed that an unrelated change to correct subject matching for some languages (Japanese users had reported the issue, but possibly others as well) had changed the thread version, so new threads then had failed reads (making some, though many fewer, messages appear blank in the UI). There were about 50 million attempts to read those values over 15,000 users, because our UI was keeping on retrying thinking it was just a temporary synchronisation issue because the previous request told it there was a Thread to fetch data for. Ouch. https://github.com/cyrusimap/cyrus-imapd/pull/5527 contains those changes.

Anyway, since the only difference between the old and new records was normalisation of subjects, I wrote a tiny patch to let the old code read the newer records and just deployed that, which made all the emails re-appear for everyone again. This is the one bit of code from all this which isn't in a public repo, but it's two lines of: if (version == 2) version = 1;

Meanwhile, the real bug is fixed https://github.com/cyrusimap/cyrus-imapd/pull/5553 And a test has been written to prove it: https://github.com/cyrusimap/cyrus-imapd/pull/5554

But we'll wait until Monday to upgrade again, when we have fresh eyes available to watch that it's OK.

...

P.S. this is almost entirely unrelated to the UI changes. The underlying reason we're doing these changes IS related to UI changes, it's there to make offline mode use storage more efficiently on your device because the IDs are smaller and provide better data locality, but the timing is purely coincidental. The Cyrus changes have been done almost exclusively by the team in the USA and the UI changes by the team in Australia, and our deploy timelines were not synchronised.

There are a few different threads on this and now that things are in a stable place I'm going to cross post this to all of them!

The larger context is that we're making a major change to how we create IDs for email and mailboxes over the JMAP protocol. The old IDs are a UUID for mailboxId and the first 25 chars of the sha1 of the message for the emailId, prefixed by an 'M'. The new IDs are the createdmodseq for the mailbox prefixed by a 'P' (these are pretty short for most users) and a reverse counter of nanoseconds of the message internaldate (delivery time) for the emailId. This gives good storage density for offline and good data locality in databases for the email listings.

You can see all the code for that in a handful of merge requests in the public cyrus-imapd repository on github at https://github.com/cyrusimap/cyrus-imapd/

Over the past few weeks, I've been helping out with the last bits of code modification, largely the changes on https://github.com/cyrusimap/cyrus-imapd/pull/5539 if you're interested.

This morning we rolled out a build which we'd tested extensively on our staging and staff servers, but missed that for older v19 mailboxes which hadn't been upgraded to v20, the code to check if messages belonged in a thread incorrectly marked them all as missing.

This made MOST emails appear missing for most customers, clearly a very bad situation.

We immediately rolled back, but in the hurry missed that an unrelated change to correct subject matching for some languages (Japanese users had reported the issue, but possibly others as well) had changed the thread version, so new threads then had failed reads (making some, though many fewer, messages appear blank in the UI). There were about 50 million attempts to read those values over 15,000 users, because our UI was keeping on retrying thinking it was just a temporary synchronisation issue because the previous request told it there was a Thread to fetch data for. Ouch. https://github.com/cyrusimap/cyrus-imapd/pull/5527 contains those changes.

Anyway, since the only difference between the old and new records was normalisation of subjects, I wrote a tiny patch to let the old code read the newer records and just deployed that, which made all the emails re-appear for everyone again. This is the one bit of code from all this which isn't in a public repo, but it's two lines of: if (version == 2) version = 1;

Meanwhile, the real bug is fixed https://github.com/cyrusimap/cyrus-imapd/pull/5553 And a test has been written to prove it: https://github.com/cyrusimap/cyrus-imapd/pull/5554

But we'll wait until Monday to upgrade again, when we have fresh eyes available to watch that it's OK.

...

P.S. this is almost entirely unrelated to the UI changes. The underlying reason we're doing these changes IS related to UI changes, it's there to make offline mode use storage more efficiently on your device because the IDs are smaller and provide better data locality, but the timing is purely coincidental. The Cyrus changes have been done almost exclusively by the team in the USA and the UI changes by the team in Australia, and our deploy timelines were not synchronised.

Glad you asked! In a similar way to ARC (but better). The mailing list/group would add its own signature, and potentially a Delta-Body or Delta-Headers describing what changes it made (so that the verifier could undo the changes and verify the original signature, plus determine which changes were made by which hop).

So you'd have something like:

DKIM2: i=1; mf=sender@trusted.com; rt=accounts-payable@example.com; d=trusted.com

DKIM2: i=2; mf=bounce@example.com; rt=me@mydomain.com; d=example.com

So I could tell that the message came through example.com, and verify their signature on the message, as well as verify that trusted.com had intended the message to go to example.com in the previous hop.

Sadly, BCC is still a real thing, so the recipient system can't reliably tell that the recipient wasn't in the "to:" field and drop the message. That's one of the authentication holes that DKIM2 is planning to fix.

Bah, the datatracker doesn't list the candidate documents. Here's some direct links:

https://datatracker.ietf.org/doc/draft-chuang-dkim2-dns/

https://datatracker.ietf.org/doc/draft-gondwana-dkim2-header...

https://datatracker.ietf.org/doc/draft-gondwana-dkim2-modifi...

https://datatracker.ietf.org/doc/draft-robinson-dkim2-bounce...

https://datatracker.ietf.org/doc/draft-robinson-dkim2-messag...

I'm working on the solution to this (co-authors from Google and Yahoo, it's legit):

https://datatracker.ietf.org/doc/draft-gondwana-dkim2-motiva...

Note that it doesn't help avoid Google actually sending out a message with user-provided text in it, but it does stop it being replayed to you without Google intending it, because the SMTP FROM/TO are protected.

The motivation draft doesn't include technical detail, see early drafts of the technical detail in the various related docs at:

https://datatracker.ietf.org/wg/dkim/documents/

Thanks!

Also there's the problem that every human has to have perfect opsec or you get the problem we have now, where there are massive botnets out there of compromised home computers.

This kind of possibility is why Fastmail purchased fastmail.com and migrated away from our old 'fastmail.fm' domain. .fm was cool, but we ran into a couple of outages on the .fm servers meaning we went offline. No such issues since we've been on .com.

Just stayed at a couple of places in Portugal which were "adults only" and it was fine. Also stayed at places which were explicitly child friendly and designed with facilities for children. Seems perfectly reasonable so long as there are a mix of places in an area available for different needs.

(I guess a hotel which had children allowed and non-children allowed sections far enough apart that you could have a child-free area would be a possible workaround; in the same way that not EVERY room needs to be handicap-friendly even if you need a certain number which are)

Fantastic :) Great to hear. I really do hope to find some time to read through the code, I haven't written any Go, so it'll be a slog to understand everything, but reading code is good for you.

That definitely is an extract challenge with JMAP, keeping enough tombstone information to accurately calculate the `destroyed` ids.

Always interesting to see another implementation of IMAP4! Congrats.

I dunno if there's thousands. Maybe if you include ISPs! There's certainly quite a few though.

(and thanks for the Fastmail plug)

SPF has challenges with shared infrastructure - if you are sending from a large service and using SPF then anyone else on that service and spoof you unless the service has outbound controls to restrict which addresses you can send from.

Fastmail had to implement this a few years ago ourselves, after 20 years of allowing whatever, we had to start by auto-whitelisting all the addresses people were sending from for a while, then slowly start introducing a requirement to prove control of the sending address to add new sending addresses over time! Obviously hosting your domain with us gets you auto-approved for any address on that domain, but otherwise you either need to confirm that you can receive email at an address to send from it now.

But SPF by itself is pretty flawed. I'm keen to write more about DKIM2 when it gets chartered at IETF (hopefully) and we can post more public documents, but it should supersede SPF/DKIM for most uses.

Made it to NZ for the first time in my life just over a year ago. Love it, definitely gotta get back

(and visit Les Mills in Auckland again of course... always nice to get to the place where my OTHER job came from)