Migrate Wizard – IMAP Based Email Migration Tool | Hacker News Reader

Migrate Wizard – IMAP Based Email Migration Tool

migratewizard.com

41 pointstechstuff12327 comments

Open articleView on HN

I love imapsync:

https://imapsync.lamiral.info

This is the way projects used to be, and surprisingly excellent ones still are.

The amount of knowledge built into this is incredible:

https://imapsync.lamiral.info/S/news.shtml

// imapsync did 14M to 21M mailboxes transfers per month in 2024, or 0.22% of ALL email traffic

I haven't had excuse to use imapsync for awhile. I remember using it fondly in the past. I definitely hadn't looked at the site in years.

I see that there's a free (for up to 3GB, pay for more) migration service offered there now, too: https://imapsync.lamiral.info/X/

That's a pretty cool way to support the project.

imapsync is slower than mbsync[1] (both in startup and per message) and doesn't support bidirectional sync[2]. It however handles message identity better than mbsync and supports using different authorization and authentication. For administering our mailserver, I first used imapsync and then patched mbsync to support the Auth header. I still use imapsync to sync the flags afterwards. I think both have high-level race conditions, so maybe take the server offline during a sync.

    [1] https://isync.sourceforge.net/
    [2] https://imapsync.lamiral.info/FAQ.d/FAQ.Two_Ways_Sync.txt

Email migration is genuinely painful and I am sure there's a real market here, so I am not trying to discourage you. But why should I trust a third party with my IMAP credentials?

"Credentials encrypted in memory only and deleted immediately after migration".

I have no way to audit/verify this claim. You're essentially asking users to hand over the keys to their entire email history on faith.

Yep, and no ISO 27001 or SOC II, nor is the location of the country of operation disclosed, let alone any name associated with it.

I remember about 20 years ago writing a relatively simple tool in perl with IMAP::Client to migrate a Universities staff mail from Courier (I think) to Communigate Pro, and then another one to migrate from Communigate Pro to Microsoft Exchange a few years later.

I was at the beginning of my career. It was pretty easy. Went almost flawlessly, moving thousands of peoples email.

Where is the "painful" part? It's just moving blobs of text around.

It doesn’t explicitly state anything about the email contents in the privacy policy page. People generally trust their email providers to not snoop in their emails. I wonder why anyone should trust a cloud based service (such as this).

There's also no names listed of anyone associated with the project, nor they mention the country where they are located, so you don't even know which sets of laws applies.

I am in the process of migrating from GMail and Proton using imapsync, since Proton's built-in tool imported some 95% of emails only.

Turns out Proton is super picky about RFC compliance and will reject anything that doesn't met the criteria, which sucks because GMail does exactly the opposite and will take almost anything you throw at it.

So I have so far written about 7 different regexes to fix some specific mailer issues to make them RFC compliant, with plenty more to go. And even then it still somewhat sucks because I am, effectively, modifying the emails to a state they were not received/sent in.

I also had that issue when setting up a server. I decided to disable the enforcement for the initial sync and then enable it.

Email only has two really mandatory headers: Date and From. There were emails without a Date header, you can't make that shit up. Naturally MUAs also can't deal with this, e.g. Thunderbird shows the timestamp of the first sync.

I'm sure the MigrationWiz guys will appreciate your name.

I also can't imagine there is much demand for IMAP only email migration services these days.

Perhaps not much, but last year I was looking at using IMAP as an export/import path for a relative's old Eudora mails to some much-more-recent client. I insisted on it, after I found out it could no longer make a secure connection to their e-mail server.

That said, I was saved/pleased by Eudora2Unix [0], one of those projects that represents a very long slow burn of successive people with a similar niche struggle. You might think Thunderbird would have an import tool for that, but it's been so many years it didn't survive...

[0] https://github.com/jonabbey/eudora2unix

Out of curiosity, did you have much luck using Eudora's IMAP to do the export?

Yes, actually. I could (in Eudora) add an IMAP account and drag individual messages into it, and then later launch Thunderbird to pull from the same account. I am less-certain about issues around nested folders, which seem to be a magnet for oddities when it comes to e-mail clients and servers.

One of the reasons I kept looking for other approaches is due to how Eudora handles attachments. In this case, many of them had been "detached" from the message they came from. In other words, the message bodies no longer contained the data, but instead a placeholder pointing to a real file in a massive Attachments folder. (Not sure exactly how it handled name collisions.) Uploading such a message to IMAP would leave the attachment(s) behind.

The linked version of Eudora2Unix (unlike the Sourceforge version) tries to reverse this process and reconstruct the original messages, encoding and inserting attachments back into the body. It doesn't always work, but I suspect some of the failures I see are because the file content was deleted over the years by the user to free up disk space, without removing the e-mail trace.

Another plus is that I could just grab all the files for the user and then promise to come back with something better, as opposed to babysitting the upload/download process.

Thanks for sharing your experience and insights on this. Migrating out of Eudora is no walk in the park! I've often considered taking the IMAP route, but Eudora's support for IMAP feels a little patchy.

Specialised conversion/migration software is probably the better route to go, especially now that the Eudora source code was donated to the Computer History Museum. I'll give Eudora2Unix a shot though!

You mentioned attachment name clashes: Eudora simply appends an incrementing integer to the incoming attachment's filename, for example, it would retain file.jpg for the first email and use file1.jpg in the next email it encounters.

FWIW it wasn't took hard to make a little Dockerfile with python:2.7-alpine and have it bind-mount the a directory of data and run in there. (With a USER directive setting it to my UID.)

Note that its "pine" output format is... almost what Thunderbird wants, except for nested mailbox effects. That'll require renaming directories from foo to foo.sbd and sometimes adding a corresponding blank foo file.

I'm working on one this week, from a cPanel-hosted email to Google Workspace. There are probably still quite a number this category will still be happening.

Then there's all the people who want to exit Big Tech or US-based companies right now.

I was an early employee at MigrationWiz (BitTitan), always cool to see recognition of the co in the wild.

Geeman, the founding CEO would have definitely sent a C&D immediately, but BitTitan was acquired by a PE firm and I don't find it hard to imagine nobody cares anymore. [0][1][2]

[0] https://www.reddit.com/r/msp/comments/1mnelca/migrationwiz_w...

[1] https://www.reddit.com/r/msp/comments/1qnwkwl/do_not_use_bit...

[2] https://www.reddit.com/r/sysadmin/comments/1qswdub/bittitan_...

What is the advantage of this tool over, you know, just using Thunderbird or another MUA to copy your emails to the new mailbox?

You could use Thunderbird, but it’s slow for large mailboxes and multiple accounts. Being cloud-based, Migrate Wizard moves email faster, tracks progress, retries on errors, and supports incremental syncs - all without running a client locally.

does it matter how long it takes if you're going to do it once?

last month I moved 20 years of email

I started offlineimap before I went to bed and it finished before I woke up

> does it matter how long it takes if you're going to do it once?

I do it regularly for backups and updates. For an update I basically spin up a new server and then sync it. I do not want to wait that long, to see whether an update contained a bug or not.

I wonder if it includes workarounds for Google IMAP bugs etc. Some years ago I migrated over 10 years worth of emails from Google to Fastmail and I ran into a lot of issues with Google's buggy IMAP implementation. Things like a random 1 in every 1000 email failing to delete so I would have random emails still remaning. I found out it was a known issue which Google pretty much decided they WONTFIX. I later found Fastmail had its own migration tool that (I presume) has workarounds for various bugs in other providers IMAP implentation making it easier than using e.g. Thunderbird to do it. If I could go back in time and use a tool like this that has a way to work around Google's bugs that would have saved a lot of my time.

I moved from gmail to fastmail a few years ago

and being the cynical sort of person that I am, I didn't trust the fastmail importer

so I ran it, and also wrote my own implementation using the gmail api (NOT imap), and another using the fastmail jmap api, and reconciled them

100% match (bar the "Muted" folder, which fastmail ignored)

pretty much perfect

Thunderbird only supports moving from one mailbox to another, it doesn't support syncing whole namespaces. It also doesn't support syncing with namespace rewriting or syncing flags, timestamps or access rights. It does not allow using different authorization and authentication and is unsuitable for large syncs as you can't resume anything or have a progress bar. It is also pretty slow, because it does one mail at a time and is just not designed for this.

Note that I used the proper IMAP terms. A mailbox is what the layman probably calls a folder or directory, a namespace is what the laymen probably would call a mailbox or account.

Thunderbird stops the whole process on first failure. You won't even know from the log file which email is at fault.

Migrate Wizard is a fast, secure IMAP email migration service that helps developers and teams move email data between providers with zero downtime.

It supports large mailboxes, preserves full data integrity, requires no setup, and works with any IMAP-compatible email service.

Reply on news.ycombinator.com