By Switching a mail provider is often combined with one major thought holding one back: What will happen to all my e-mails? I don't want to lose my e-mails!
Sure, if you used POP3 to download the e-mails to your mail client, the e-mails are not stored on the server anymore. This worked fine in the 90s and 00s but since mobile Internet there are usually multiple devices connected to your mailbox: Notebook, Workstation, Smartphone or even a Tablet. In such a scenario, only IMAP (IMAPv4 to be precise) can do the job: All the e-mails remain on the server (in the mailbox) but are synced to the local mail client. Every change in the local mail client is synced back to the server. Hence all the devices see the same status.
I recently had to migrate more than 28'000 e-mails from a long-existing mailbox to another mailbox located on a different server (new mail provider). Perfect moment to play around with imapsync.
What is imapsync?
imapsync is a migration tool for mailboxes using the IMAP protocol. On the official website, imasync is described as:
Imapsync is an IMAP transfer tool. The purpose of imapsync is to migrate IMAP accounts or to backup IMAP accounts. IMAP, IMAP4 in fact (started December 1994), is one of the three current standard protocols used to access mailboxes, the two other being POP3 (started November 1988) and HTTP (started May 1996) with webmails, webmails are often tied to an IMAP server.
Gilles Lamiral
The best thing: imapsync is developed as Open Source Software. It is therefore free to use. However theauthor, Gilles Lamiral, offers additional support and quickly available imapsync for Windows (imapsync.exe), too. Windows users without knowledge how to build an executable exe file from a Perl source might want to consider this.
As this tutorial focuses on using imapsync on Linux, we won't cover the usage on Windows or on macOS.
Download and requirements
imapsync can be downloaded right from the public git repository on GitHub. It is recommended to use the latest release. At the time of this writing 1.977 was the newest available released version.
The newest tar.gz release is downloaded using wget into a directory called "build" and is then extracted using tar.
ck@linux ~ $ mkdir build
ck@linux ~ $ cd build/
ck@linux ~/build $ wget https://github.com/imapsync/imapsync/archive/1.977.tar.gz
ck@linux ~/build $ tar -xzf 1.977.tar.gz
ck@linux ~/build $ cd imapsync-1.977/Inside the extracted imapsync folder there are a couple of sub directories and files:
ck@linux ~/build/imapsync-1.977 $ ll
total 1912
drwxrwxr-x 10 ckadm ckadm 4096 Jun 17 2020 ./
drwxrwxr-x 7 ckadm ckadm 4096 Dec 29 13:53 ../
-rw-rw-r-- 1 ckadm ckadm 197011 Jun 17 2020 ChangeLog
-rw-rw-r-- 1 ckadm ckadm 29457 Jun 17 2020 CREDITS
lrwxrwxrwx 1 ckadm ckadm 5 Jun 17 2020 dist -> dist2
drwxrwxr-x 2 ckadm ckadm 4096 Jun 17 2020 doc/
drwxrwxr-x 2 ckadm ckadm 4096 Jun 17 2020 examples/
lrwxrwxrwx 1 ckadm ckadm 21 Jun 17 2020 FAQ -> FAQ.d/FAQ.General.txt
drwxrwxr-x 2 ckadm ckadm 4096 Jun 17 2020 FAQ.d/
drwxrwxr-x 2 ckadm ckadm 4096 Jun 17 2020 .github/
-rw-rw-r-- 1 ckadm ckadm 177 Jun 17 2020 .gitignore
-rwxrwxr-x 1 ckadm ckadm 152 Jun 17 2020 i3*
-rwxrwxr-x 1 ckadm ckadm 647598 Jun 17 2020 imapsync*
-rwxrwxr-x 1 ckadm ckadm 604052 Jun 17 2020 imapsync-1.945*
-rw-rw-r-- 1 ckadm ckadm 29958 Jun 17 2020 index.shtml
lrwxrwxrwx 1 ckadm ckadm 25 Jun 17 2020 INSTALL -> INSTALL.d/INSTALL.ANY.txt
drwxrwxr-x 2 ckadm ckadm 4096 Jun 17 2020 INSTALL.d/
-rw-rw-r-- 1 ckadm ckadm 320 Jun 17 2020 LICENSE
-rw-rw-r-- 1 ckadm ckadm 30815 Jun 17 2020 Makefile
-rw-rw-r-- 1 ckadm ckadm 320 Jun 17 2020 NOLIMIT
-rw-rw-r-- 1 ckadm ckadm 36936 Jun 17 2020 README
-rw-rw-r-- 1 ckadm ckadm 37780 Jun 17 2020 README.md
lrwxrwxrwx 1 ckadm ckadm 6 Jun 17 2020 README.txt -> README
-rw-rw-r-- 1 ckadm ckadm 5800 Jun 17 2020 README_Windows.txt
drwxrwxr-x 3 ckadm ckadm 4096 Jun 17 2020 S/
-rw-rw-r-- 1 ckadm ckadm 193577 Jun 17 2020 tests.sh
-rw-rw-r-- 1 ckadm ckadm 41612 Jun 17 2020 TODO
-rw-rw-r-- 1 ckadm ckadm 6 Jun 17 2020 VERSION
drwxrwxr-x 12 ckadm ckadm 4096 Jun 17 2020 W/
-rwxrwxr-x 1 ckadm ckadm 4725 Jun 17 2020 webserver*
drwxrwxr-x 2 ckadm ckadm 4096 Jun 17 2020 X/The most important file is already existing: imapsync, the program (a Perl script) itself. No need to compile anything. However there are a lot of Perl modules which must be installed as requirements of imapsync. Besides already having had a few Perl modules installed, the following modules needed to be installed additionally (package names for Debian and Ubuntu):
ck@linux ~ $ sudo apt-get install libio-tee-perl libmail-imapclient-perl libterm-readkey-perl libunicode-string-perl libreadonly-perl libsys-meminfo-perl libregexp-common-perl libfile-tail-perl libencode-imaputf7-perlIf there are still Perl modules missing, imapsync will exit with an error pointing to the missing module. Once all the required Perl modules are installed, imapsync can be launched and will by default show the help/usage information:
ck@linux ~/build/imapsync-1.977 $ ./imapsync
Name:
imapsync - Email IMAP tool for syncing, copying, migrating and archiving
email mailboxes between two imap servers, one way, and without duplicates.
Version:
This documentation refers to Imapsync $Revision: 1.977 $
Usage:
To synchronize the source imap account
"test1" on server "test1.lamiral.info" with password "secret1"
to the destination imap account
"test2" on server "test2.lamiral.info" with password "secret2"
do:
imapsync \
--host1 test1.lamiral.info --user1 test1 --password1 secret1 \
--host2 test2.lamiral.info --user2 test2 --password2 secret2
Options:
[...]How to run imapsync
If you know rsync, you quickly know how to use imapsync, too. Just as rsync, imapsync transfers (migrates) the mails from a source to a target. A source is defined by using the parameters –host1, –user1 and –password1. The target is defined by, you guessed it already, by the parameters –host2, –user2 and –password2. Additional options (such as verbose output) are available, too.
Simulating a mailbox migration
It is always wise to first start a mailbox migration with a simulation a.k.a. dry-run. The parameter –dry allows to simulate an IMAP migration. This also helps to verify the given credentials for the mailboxes. At the end of each imapsync run (with or without –dry), statistics of the migration will be shown.
ck@linux ~/build/imapsync-1.977 $ ./imapsync --dry --host1 source.example.com --user1 source@example.com --password1 12345678 --host2 target.example.com --user2 target@example.com --password2 87654321
[...]
++++ Statistics
Transfer started on : Wed Dec 30 10:07:32 2020
Transfer ended on : Wed Dec 30 10:08:13 2020
Transfer time : 40.9 sec
Folders synced : 262/262 synced
Messages transferred : 0 (could be 49 without dry mode)
Messages skipped : 0
Messages found duplicate on host1 : 0
Messages found duplicate on host2 : 0
Messages found crossduplicate on host2 : 0
Messages void (noheader) on host1 : 0
Messages void (noheader) on host2 : 0
Messages found in host1 not in host2 : 49 messages
Messages found in host2 not in host1 : 5 messages
Messages deleted on host1 : 0
Messages deleted on host2 : 0
Total bytes transferred : 0 (0.000 KiB)
Total bytes skipped : 0 (0.000 KiB)
Message rate : 0.0 messages/s
Average bandwidth rate : 0.0 KiB/s
Reconnections to host1 : 0
Reconnections to host2 : 0
Memory consumption at the end : 263.6 MiB (started with 221.6 MiB)
Load end is : 1.42 1.43 1.35 4/1434 on 4 cores
Biggest message : 0 bytes (0.000 KiB)
Memory/biggest message ratio : NA
Start difference host2 - host1 : -28558 messages, -8405244919 bytes (-7.828 GiB)
Final difference host2 - host1 : -28558 messages, -8405244919 bytes (-7.828 GiB)
The sync is not finished, there are 49 identified messages in host1 that are not on host2.
There is no unidentified message
The sync is not strict, there are 5 messages in host2 that are not on host1. Use --delete2 to delete them and have a strict sync. (5 identified messages in host2)
Detected 0 errorsThe statistics of the dry-run also show how many mailbox folders (262) and e-mails (28558) will be synced. The simulated run obviously runs pretty quickly – it took around 40s to complete.
Real mailbox migration
Time to do the real sync now. As, in this situation, more than 28k mails need to be transferred, this can take a while…
ck@linux ~/build/imapsync-1.977 $ ./imapsync --host1 source.example.com --user1 source@example.com --password1 12345678 --host2 target.example.com --user2 target@example.com --password2 87654321
Host1 Nb folders: 262 folders
Host2 Nb folders: 262 folders
Host1 Nb messages: 28568 messages
Host2 Nb messages: 28538 messages
Host1 Total size: 8405740214 bytes (7.828 GiB)
Host2 Total size: 8385683883 bytes (7.810 GiB)
Host1 Biggest message: 72919837 bytes (69.542 MiB)
Host2 Biggest message: 72919837 bytes (69.542 MiB)
Time spent on sizing: 21.5 seconds
++++ Statistics
Transfer started on : Wed Dec 30 10:09:36 2020
Transfer ended on : Wed Dec 30 11:51:20 2020
Transfer time : 6103.7 sec
Folders synced : 262/262 synced
Messages transferred : 28532
Messages skipped : 36
Messages found duplicate on host1 : 22
Messages found duplicate on host2 : 0
Messages found crossduplicate on host2 : 0
Messages void (noheader) on host1 : 14 in order to sync those 14 unidentified messages, add option --addheader
Messages void (noheader) on host2 : 0
Messages found in host1 not in host2 : 0 messages
Messages found in host2 not in host1 : 6 messages
Messages deleted on host1 : 0
Messages deleted on host2 : 0
Total bytes transferred : 8385026114 (7.809 GiB)
Total bytes skipped : 351682 (343.439 KiB)
Message rate : 4.7 messages/s
Average bandwidth rate : 1341.6 KiB/s
Reconnections to host1 : 0
Reconnections to host2 : 0
Memory consumption at the end : 411.2 MiB (started with 221.6 MiB)
Load end is : 1.41 1.60 1.85 1/1461 on 4 cores
Biggest message : 72919837 bytes (69.542 MiB)
Memory/biggest message ratio : 5.9
Start difference host2 - host1 : -28558 messages, -8405244919 bytes (-7.828 GiB)
Final difference host2 - host1 : -30 messages, -20056331 bytes (-19.127 MiB)
The sync looks good, all 28528 identified messages in host1 are on host2.
There are 14 unidentified messages (usually Sent or Draft messages). To sync them add option --addheader
The sync is not strict, there are 6 messages in host2 that are not on host1. Use --delete2 to delete them and have a strict sync. (28534 identified messages in host2)
Detected 0 errorsFinally after 6103 seconds (~1h 41min), the IMAP migration with more than 7.8 GB was completed and the target mailbox contained 28558 mails. Important to note: All e-mail headers remain unchanged. This also means that the dates of each e-mail (received, sent) remain the same. The same views and the same order of the e-mails are showing up in the new mailbox.
[…] to successfully share your photos over email or other services using the Internet, check your internet connection is successfully working. Make […]