brongondwana

I dunno if there's thousands. Maybe if you include ISPs! There's certainly quite a few though.

(and thanks for the Fastmail plug)

SPF has challenges with shared infrastructure - if you are sending from a large service and using SPF then anyone else on that service and spoof you unless the service has outbound controls to restrict which addresses you can send from.

Fastmail had to implement this a few years ago ourselves, after 20 years of allowing whatever, we had to start by auto-whitelisting all the addresses people were sending from for a while, then slowly start introducing a requirement to prove control of the sending address to add new sending addresses over time! Obviously hosting your domain with us gets you auto-approved for any address on that domain, but otherwise you either need to confirm that you can receive email at an address to send from it now.

But SPF by itself is pretty flawed. I'm keen to write more about DKIM2 when it gets chartered at IETF (hopefully) and we can post more public documents, but it should supersede SPF/DKIM for most uses.

Made it to NZ for the first time in my life just over a year ago. Love it, definitely gotta get back

(and visit Les Mills in Auckland again of course... always nice to get to the place where my OTHER job came from)

Legitimately is, it's been a challenge at time but pretty happy with it. Pity LeBron came along and stole my googlability

Sadly, they weren't quite prescient enough to realise the importance of different initials, so we have to use TWO letters for card game scoresheets; but nah. Pity.

Me too!

My parents thought it was a cool word, so they named me after it. There's 5 of us, and we all have different surnames. Hippy parents, man.

I do believe we are talking past each other here. Of course ceph does data consistency, but it sure doesn't assert that a modseq is monotonically increasing or that an mailbox/uidvalidity/uid triple doesn't change digest, because it's not data-model aware.

sigh

IMAP is both excellent and, annoyingly inconsistent so it's much more of a pain to develop reliable parsers for. I'm pretty happy to use IMAP generally, but I'm also... MODSEQ is wrapped with a () in some places, not but STATUS HIGHESTMODSEQ:

  . SELECT INBOX.Archive
  * 1239 EXISTS
  * 0 RECENT
  * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $X-ME-Annot-2 $HasAttachment $IsNotification $IsMailingList $NotJunk $CanUnsubscribe)
  * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $X-ME-Annot-2 $HasAttachment $IsNotification $IsMailingList $NotJunk $CanUnsubscribe \*)] Ok
  * OK [UNSEEN 1221] Ok
  * OK [UIDVALIDITY 1108730350] Ok
  * OK [UIDNEXT 2231] Ok
  * OK [HIGHESTMODSEQ 40306873] Ok
  * OK [MAILBOXID (210306ee-5833-456b-bede-6d04757128b3)] Ok
  * OK [URLMECH INTERNAL] Ok
  * OK [ANNOTATIONS 65536] Ok
  . OK [READ-WRITE] Completed
  . FETCH 1 MODSEQ
  * OK [HIGHESTMODSEQ 40306873] CONDSTORE enabled by FETCH MODSEQ
  * 1 FETCH (MODSEQ (39570709))
  . OK Completed (0.000 sec)

And... the * operator ranges.

  . FETCH 2231:* (UID MODSEQ)
  * 1239 FETCH (UID 2230 MODSEQ (40306872))
  . OK Completed (0.001 sec)

Source: have also implemented both client and server side IMAP, and reviewed RFC9051 very closely.

The big benefit with JMAP is the state strings and /changes operation which allows even in the face of failures to recovery quite cleanly. It's really more a data synchronising protocol (RFC8620) with email support on top (RFC8621) and the new protocols coming in.

I mean, I'm happy to have this argument. CEPH is content agnostic and that's fantastic most of the time. Cyrus replication is data aware, so it's not just replicating the data, it's doing integrity checking and data model consistency handling.

Most of all, it's doing split brain recovery; which - if we wanted CP rather than AP then we wouldn't need, but that wasn't the original design.

If I was redoing this from scratch, I'd maybe do Ceph or similar and update Cyrus to work well with it, but that would be a big change from the current design.

Anyway, I'm happy to stipulate that Ceph is great tech, without going and telling other people that it's the only choice.

Yeah, if you don't have levels of redundancy, then you're pretty screwed. We could theoretically lose 2/3 of our systems and have sufficient capacity, because our metric is 2N primary plus N secondary, and we can run with half the racks switched off in the primary, or with the secondary entirely switched off, or (in theory, there's still some kinks with failover) with just secondary.

If I didn't see it, it didn't happen...

OK, I stand corrected. We lose one or two NVMe drives per year :)

There's various articles on our blog about our stack!

Ceph is most certainly not the only game in town. It's good and stuff, but it's just tech. We're using protocol level replication for each of our data stores.

We also did our own web framework :p

https://github.com/fastmail/overture

Yeah, we have good vendor relationships, good datacenter relationships, and we've made mis-steps along the way for sure. Own hardware isn't for everyone, but it's been great for us. YMMV

yeah, we use Cyrus replication still - it's protocol specific so it detects changes very efficiently as well, using the internal MODSEQ system also used for the JMAP /changes and IMAP CONDSTORE/QRESYNC.

Plus it has protocol consistency sanity checks built in.

Plus, I wrote it :p

1. hoping to have a JMAP archive format at some point which should cover that. I'd hope that normally you'd be fetching a delta update rather than the whole thing. We've got enough bandwidth for a few people do to it, but I wouldn't want every customer pulling their entire archive every week of 99% the same immutable data; that would be kinda sucky.

2. yeah, I'd love that too - we're keen to integrate with everything else that people are using. We have a basic in-house IdP thing for our own staff to authenticate against our hosted services, but haven't scaled it out. This will happen eventually, though I've been burned enough times I don't want to promise a timeframe.

PHL to STL these days, but same design:

https://www.fastmail.com/blog/moving-house-new-datacentre/

Yeah, we're only using ZFS replication for logs; we're using the Cyrus replication for emails because it has other sanity checks and data model consistency enforcement which is really valuable.

(And both are async. We'd need something like drbd for real synchronous replication)

Unlinking gets done asynchronously on the weekends from Cyrus, using the `cyr_expire` tool. Right now it only runs one unlinking process at a time on the whole machine due to historical ext4 issues ... but maybe we should revisit that now we're on ZFS and NVMe. Thanks for the reminder.

Emails are stored in cyrus-imapd.

For now, the "file storage" product is a Node tree in mysql, with content stored in a content-addressed blob store, which is some custom crap I wrote 15 years ago that is still going strong because it's so simple there's not much to go wrong.

We do plan to eventually move the blob storage into Cyrus as well though, because then we have a single replication and backup system rather than needing separate logic to maintain the blob store.

We're (very boring I know) just putting it all in a git repository with a Makefile which deploys it, plus some basic orchestration to run 'make diff' across the cluster and see what's out of sync, and 'make install' across hosts to deploy it into pl ace.

It's clunky, but simple, repeatable, and easily (vsfo) understood.

As for the bigger things, software etc - we have scripts that generate Debian packages which we store in our own private repo. You just install `fastmail-server` and the dependency management updates everything. There's a daily cronjob which checks if there are updated security packages or thing we failed to correctly deploy and emails us as well.

It's amazing what you can build on top of the OS provided tools with not too much complexity if you don't overthink it.

My beard isn't entirely grey yet!

The new NVMe drives we've only had for a few years, but so far there's only been a single failure across the whole fleet, and we keep spares in stock. It's been very reliable, not like the weeks back in (hmm, 2006? 2007?) the ancient past, when we were losing 15kRPM velociraptors every other day. They had a firmware fault and we eventually got an update which made them reliable, but it was a wild few months.

Yeah, we know about the ZFS encryption with send/receive bug, it's frustrating our attempts to get really nice HA support on our logging system... but so far it appears that just deleting the offsending snapshot and creating a new one works, and we're funding some research into the issue as well.

This is the current script - it runs every minute for each pool synced between the two log servers: https://gist.github.com/brong/6a23fee1480f2d62b8a18ade5aea66...

Jeremy is all the Fast things!

We adopted SSD for the current week's email and rust for the deeper storage many years ago. A few years ago we switched to everything on NVMe, so there's no longer two tiers of storage. That's when the pricing switched to make it worthwhile.

Sadly someone flagged the hilarious response "...spamming every online discussion about selfhosting email, [just go with Fastmail, it's greeaaaat!] or like that." before I got back to it.

True fact, many of us do love reading HN and chiming in. We don't mind self hosters at all. Eventually they get sick of it and move their email to Fastmail, or in extreme cases come work for us!

(it took me a good year of being a Fastmail staff member to finally decide to shut down my own self-hosting and just make it a Fastmail Family account... figured if I was going to get paged by my family for the email being broken I might as well be on the clock)

We have a much better account recovery process than, say, Protonmail where if you lose your encryption key then you really are screwed.

Obviously, the more information we have about you, the easier it is for you to prove your identity! Often the people who lose all their credentials and can't get the account back at all are the ones who used anonymous payment methods, clear all their cookies, use a fake name. Hard to identify those!

But generally, we help people regain access with a combination of payment method, backup communication methods they have registered with us like another email address or a phone number, and at the most extreme - Government ID. Provide enough of those and survive the lockout period in which we've informed the person with access to the account and given them a chance to object, we can get you back in.

Seriously the "24 hour lockout while we make sure nobody is accessing the account" is a pretty key part of slowing down attacks - generally if someone's identity is stolen then the attacker's main advantage is speed rushing through all their other accounts, so the slow-down is hugely advantageous.

Thanks! We're all pretty happy to keep doing what we're doing for a long time yet :) Have plenty of new stuff on the go (see the email work happening at IETF, a lot of it's being led or shepherded by someone from Fastmail)

AI is... yeah. I can see in theory being able to have an AI answer questions like "who sent me a question I haven't replied to yet?" or "who have I emailed something expecting a response to and not heard back from?" could be powerful - but the tech is really immature and it hallucinates the craziest things. I don't think it's something we need to do in-house - it's something that could be added on with an IMAP or JMAP connector for those who want it.