This post explains how I used imapsync on Linux to successfully migrate all our user’s emails and includes a few tricks specific to Gmail and bulk transfers of multiple accounts.
Google Apps offers a great email service but if you want to move from an existing email provider then you may need some way of migrating existing emails into your new Gmail or Google Apps accounts. There are lots of tools out there to do this, and Google provide some excellent tools themselves for a wide variety of servers and clients including Microsoft Exchange and Outlook.
However, these tools aren’t always suitable. In our case we don’t have access to a Windows machine with sufficient bandwidth for running their IMAP tool, so we needed to find an alternative. I have previously used imapsync by Gilles Lamiral for moving between IMAP servers so naturally I turned to this option again.
Before we start
This method assumes you have a number of accounts with an existing email provider and that you can access these accounts using the IMAP protocol. It also assumes that you have already signed up to Google Apps and created your new Gmail users. Finally, it assumes you know your way around bash and a Linux terminal shell and that you have a Linux installation with a recent version of imapsync installed.
By the way, if you need any help with impasync or just find the tool useful then I highly recommend supporting the developer by buying the imapsync professional support package.
These instructions are not one-size-fits-all and so you should tweak and adjust where necessary. You are also strongly advised to do a few test migrations before the full live migration, and you should try to become familiar with the many options that imapsync provides as some may be useful for your specific environment. Also, I am not responsible if you lose all your emails or have any other problems. I am merely documenting this in the hope it will be useful to somebody else!
The method
The method uses a text file (users.txt) containing usernames and passwords of all your users containing their usernames and passwords for both the source IMAP server and the destination IMAP server (Google Apps). A simple bash script (migration.sh) then loops through the text file and uses impasync to connect and sync mail from the source IMAP server to the destination IMAP server.
The users text file
The users.txt file contains one user on each line with their source username, source password, destination username and destination password. All these fields are separated by semi-colons. Like this example:
user1@source.tld;user1sourcepassword;user1@destination.tld;user1destinationpassword
user2@source.tld;user2sourcepassword;user2@destination.tld;user2destinationpassword
user3@source.tld;user3sourcepassword;user3@destination.tld;user3destinationpassword
[...]
user99@source.tld;user99sourcepassword;user99@destination.tld;user99destinationpassword
The script
The migration.sh script is very simple but it is specific to my needs – yours may be different. At very least you will need to configure the SERVER1 (source) and SERVER2 (destination) server addresses. For testing you should pay attention to the JUSTCONNECT and DRY options that are commented out. For tweaking how it handles the mapping of your current folders to the new Gmail folders pay particular attention to the regextrans2 lines. Do some tests on a single or couple of dummy user accounts!
#!/bin/bash
#Configure servers
SERVER1=imap.source.com
SERVER2=imap.gmail.com
#Uncomment to hide folder sizes
#FAST="--nofoldersizes"
#Uncomment to do a dry run (no actual changes)
#DRY="--dry"
#Uncomment to just sync folders (no messages)
#JUSTFOLDERS="--justfolders"
#Uncomment to just connect (no syncs at all)
#JUSTCONNECT="--justconnect"
#Set the path to your imapsync binary
imapsync=imapsync
#Users file
if [ -z "$1" ]
then
echo "No users text file given."
exit
fi
if [ ! -f "$1" ]
then
echo "Given users text file \"$1\" does not exist"
exit
fi
{ while IFS=';' read u1 p1 u2 p2; do
$imapsync --usecache --tmpdir /var/tmp \
--host1 ${SERVER1} --user1 "$u1" \
--password1 "$p1" --ssl1 \
--host2 ${SERVER2} \
--port2 993 --user2 "$u2" \
--password2 "$p2" --ssl2 \
${FAST} ${DRY} ${JUSTFOLDERS} ${JUSTCONNECT} \
--regextrans2 's{Sent$}{[Gmail]/Sent Mail}' \
--regextrans2 's{Sent Items$}{[Gmail]/Sent Mail}' \
--regextrans2 's{Sent Messages$}{[Gmail]/Sent Mail}' \
--regextrans2 's{Drafts$}{[Gmail]/Drafts}' \
--exclude 'INBOX.Trash|INBOX.spam|INBOX.Apple Mail To Do'
done ; } < $1Running the script
In the simplest case, just run the script (called migration.sh) with the users filename as the first argument:
./migration.sh users.txtThe script can take a long time to run, so I suggest using nohup and redirecting the output to a log file:
nohup ./migration.sh users.txt > migrationlog.txt 2>&1 &Doing the migration
The script is designed to be run multiple times, and on subsequent runs it will ignore any messages that have already been transferred. This means that subsequent runs will be much faster than the initial run. In our migration of about 60 users I broke the users down into 6 batches of 10, then ran the script 6 times simultaneously on a Friday evening. By Sunday the scripts had finished running and I swapped over the MX records to point to our new Google Apps servers. I then ran the script again the following Wednesday to bring across any new messages that had been delivered to the old servers.
I'm not saying this is the correct way, but it worked for me. You should test, test, test before the migration and make sure you plan everything, including reducing the TTL on your DNS if necessary.
Examining the log files
The log files are a bit verbose so here is a nice awk for helping to examine them:
awk '($1 == "Host1:") || ($1 == "Host2:") || ($1 == "Transfer") || ($1 == "Messages") || ($1 == "Total") || ($1 == "Message") || ($1 == "Average") || ($1 == "Reconnections") || ($1 == "Memory") || ($1 == "Biggest") || ($1 == "Detected")' migrationlog.txt