====== Better Mobile Email ====== [[http://devconf.cz/filebrowser/download/440|Paul Frields' slides]] about his notebook's email setup show quite some similarities with my own one. The major differences are: * I moved from ''offlineimap'' to ''mbsync'', mostly for performance reasons. * Instead of having a full-fledged MTA running on my notebook, I prefer a small send-only MTA for the sake of small footprint and ease of configuration. Here's how the setup is done in particular: ===== The Big Picture ===== When it comes to using the same email account on multiple machines, IMAP is a great benefit. It allows for every box to see every incoming mail and housekeeping tasks such as deleting old mails or reorganizing them into subfolders will automatically be visible from the other machines, as well. By storing sent mail in an IMAP folder the same applies to outgoing mail, too. The only downside with typical IMAP usage is that the task of email synchronization and caching has to be done by the email client (MUA). This naturally involves delays during client usage, and often clients are not really usable without direct connectivity to the IMAP server. My personal MUA of choice is ''mutt'', which suffers these issues, too. Especially since handling mail in local maildirs is strikingly fast, the added delay of having to sync with IMAP feels much higher than with other email clients. In order to overcome this, an asynchronous approach serves well: A dedicated instance covers the task of bidirectional synchronization between remote IMAP and local maildirs, while ''mutt'' operates exclusively on the latter. Using ''mutt'''s SMTP support involves surprisingly similar issues: Without direct connectivity, no mail sending is possible at all (of course not, but this can be directly experienced by ''mutt'' first hanging for a while and then giving up with an error message). But even if connectivity is given, any delay and/or bandwidth limitation on the line to the MTA is directly appreciable. Here, an asynchronous approach is chosen as well: A dedicated instance takes care of queueing and ultimately delivering the email to be sent and ''mutt'' returns control to the user as soon as it has dispatched the job. ===== Bidirectional IMAP Synchronization ===== There are a number of tools to achieve this, but the only two which proved usable to me were ''offlineimap'' and ''mbsync''. Of those the latter is superior, not least due to the fact that it is written in C. ==== Mbsync Configuration File ==== ''mbsync'' is configured on a per user basis in //$HOME/.mbsyncrc//. In order to understand it's semantics, a number of concepts have to be known by the user: * A ''store'' is basically anything that is able to keep emails, like e.g. an mbox, a maildir or even an IMAP account. ''mbsync'' defines specific store types for these (except mbox, but who uses that anyway?) which hold type-dependent information. * A ''channel'' defines a path of synchronisation between two stores, optionally limited to a specific scope within both. With these two in mind, let's start by defining two stores. The first one defines the local maildir: MaildirStore local Path ~/ Inbox ~/Maildir/ SubFolders Maildir++ The way this is defined is a bit unintuitive: Instead of setting ''Path'' to the actual maildir path, it is set to it's parent and ''Inbox'' is set to point to the top-most maildir. I don't really remember why, but this was important to get my setup working. The second store defines the remote IMAP: IMAPStore remote Host mail.nwl.cc User myself SSLType IMAPS Obviously, this store defines the remote host running the IMAP service, the user name to use for authentication and that IMAPS should be used. If not using SASL authentication, one might want to additionally define the password using the ''Pass'' option. With both endpoints being defined, channels can be declared. The following defines two channels between those peers: One for the inbox only, the other for all subfolders. This way synchronisation of the INBOX can be triggered separately from the subfolders, which becomes useful later on: Channel inbox Master :remote:INBOX/ Slave :local:Maildir/ Channel folders Master :remote: Slave :local:Maildir/ Patterns * !INBOX !Chats !Contacts !Emailed\ Contacts !Junk A few things are noteworthy about the used semantics: * Each channel defines which of the peers is master and which is slave. This is important when defining how synchronisation should take place. Apart from that, these two keywords are interchangeable. * The IMAP server exposes the inbox as a subfolder on it's own, therefore the first channel specifies this explicitly. * The second channel makes use of the ''Patterns'' keyword for two purposes: - The asterisk (''*'') sign tells ''mbsync'' to recurse into subfolders. - The exclamation mark (''!'') prefixed folder names are excluded from synchronisation. In this case the remote is a Zimbra instance which seems to export (uninteresting) non-email data via IMAP, too. Finally, the above store and channel definitions are prepended by a few global statements: Expunge Both Create Both Remove Both SyncState * The ''Expunge'', ''Create'' and ''Remove'' statements control the synchronization process. Instead of ''Both'' one may also pass one of ''Master'', ''Slave'' or ''None''. Using the latter for ''Expunge'' is recommended measure against unwanted loss of data during testing. ''SyncState'' defines where the synchronization meta data should be kept. The asterisk (''*'') makes ''mbsync'' save it's meta data in the slave's store itself. ==== Calling Mbsync ==== To manually trigger synchronization of a defined channel, a direct call to ''mbsync'' suffices: mbsync ==== Looping Over Mbsync Calls ==== To integrate this nicely into the rest of my setup, I have written a shell script which provides a few more goodies: * Run indefinitely. * Before every round of synchronization, try to ping the defined IMAP host (see ''Host'' directive above) or go to sleep for 10sec then try again. * Synchronize the inbox every 10sec, the IMAP folders just once every 100sec. * After every folder sync run, fetch a list of existing maildir subfolders into //$HOME/.mutt/mbsyncloop.mailboxes// in a format suitable for including into //$HOME/.muttrc//. #!/bin/bash mailboxes="${HOME}/.mutt/mbsyncloop.mailboxes" do_mbsync() { # (channel) echo -e "synchronising $1 ..." mbsync $1 && echo -e "... done" || echo -e "... failed" } get_mailboxes() { # (maildir) echo -n "mailboxes \"+\"" find "$1" -type d -name cur | while read path; do mbox="$(basename $(dirname $path))" [[ $mbox == "."* ]] || continue mbox="+$mbox" echo -n " \"$mbox\"" done echo "" } update_mailboxes() { mboxes="$(get_mailboxes "$1")" [[ -f $mailboxes ]] || { echo "initially creating $mailboxes" echo -n $mboxes > "$mailboxes" return } old_mboxes="$(<$mailboxes)" [[ "$mboxes" == "$old_mboxes" ]] || { echo "updating $mailboxes" echo -n $mboxes > "$mailboxes" } } imaphost="$(awk '/^Host /{print $2}' ~/.mbsyncrc)" # sync INBOX every 10 seconds, # sync subfolders every minute cnt=0 while true; do ping -c 1 $imaphost >/dev/null 2>&1 || { echo "Host $imaphost not responding" sleep 10 continue } do_mbsync inbox [[ $((cnt % 10)) -eq 0 ]] && { cnt=0 do_mbsync folders update_mailboxes "${HOME}/.maildir" } ((cnt++)) sleep 10 done ===== Relay-Only MTA ===== My notebook does not receive email from external hosts. For local delivery to maildirs, ''procmail'' serves well as simple alternative. The only thing left for an MTA on my notebook to do is to get my user's email out to the right MX using appropriate credentials. And all that in background, without disturbing my workflow in case something (Wifi, uplink, VPN, MX, etc.) does not work as it should. Back then when I was searching for a relay-only MTA with queueing support, I didn't find any which felt comfortable. Therefore I sticked to ''esmtp'' I was using already and wrote a little shell-wrapper implementing the desired enhancement. This shell wrapper has been accepted into the official code repository back in 2007, but sadly the whole project was orphaned in 2011. Though, Gentoo luckily still ships it. Probably I should fork the project and revive it by implementing queueing support in C. Anyway, here's how to set things up using the latest release 1.2: ==== Esmtp Configuration File ==== The configuration file is quite straightforward. The only thing worth mentioning in beforehand is ''esmtp'''s support for identities: These allow to choose a different SMTP upstream based on sending address. Here's the config I use: mda = '/usr/bin/formail -a "Date: `date -R`" | /usr/bin/procmail -d %T' hostname = smtp.corp.example.com force reverse_path = myself@example.com identity = myself@nwl.cc preconnect = "ssh -f -L 25025:localhost:25 nwl.cc 'sleep 5'" hostname = localhost:25025 force reverse_path = myself@nwl.cc The first line (''mda'') defines the method for local delivery, i.e. for recipients not containing an @-sign. Piping through ''formail'' adds the useful ''Date'' header, which is otherwise taken care of by the external MTA. The default delivery method is via ''smtp.corp.example.com'' which requires no authentication (only reachable via VPN). Whatever ''From'' address I specify, ''esmtp'' sanitizes the reverse path for me. A second identity defines my private mail setup. It establishes an SSH tunnel prior to connecting, which is convenient for me as it relieves me from having to specify login credentials here and SSH Agent is running anyway. ==== Esmtp Queueing Wrapper ==== Now for queueing support: The whole magic is implemented by my shell wrapper, ''esmtp-wrapper''. I have it installed in //$HOME/bin// with appropriate symlinks (it changes it's operation mode depending the name by which it was called): % find bin -lname esmtp-wrapper bin/mailq bin/deliver bin/sendmail The commands to what their name suggests: ''mailq'' lists the local mail queue, ''deliver'' triggers mail delivery and ''sendmail'' just pretends to be ''sendmail'' itself. Queueing is implemented as follows: Upon invocation, ''sendmail'' writes the parameters it was called with along with the mail body passed via ''stdin'' into a temporary directory below //$HOME/.esmtp_queue// and then calls itself as ''deliver'' in background. Upon invocation, ''deliver'' first checks if it has been called by ''sendmail'' and in that case sleeps 5sec before doing anything (this is meant to catch mass mail delivery like e.g. git-send-email typically does). Then it continues to browse through any directories below //$HOME/.esmtp_queue// and calls ''esmtp'' exactly like ''sendmail'' has been called before. If that succeeds, the temporary directory is deleted, otherwise it's kept for a later try. ==== Periodic Mail Queue Flushing ==== If initial mail delivery fails, it has to be retried later. Since ''esmtp'' does not run as a daemon, there is no automatic way to achieve this. Instead, a simple cron job serves well: */10 * * * * /bin/ping -c1 smtp.corp.example.com >/dev/null 2>&1 && $HOME/bin/deliver This checks connectivity every 10min and calls ''deliver'' if given. ===== Wrapping Things Up ===== The last piece of the puzzle is ''mutt''. It's own setup is trivial, as it actually does not know about ''mbsync'' at all. And for ''esmtp'', all there is to specify is the path to ''sendmail'': set sendmail=~/bin/sendmail What is more interesting is how to conveniently run ''mutt'' and ''mbsyncloop'' together. For that purpose, I have a dedicated //.screenrc//: source .screenrc screen -t SYNC 0 mbsyncloop screen -t MUTT -h 0 1 mutt_then_killall_mbsyncloop bind c screen mutt bind ^c screen mutt When screen is invoked passing this config via it's ''-c'' parameter, it will automatically create two windows: * Window 0 runs ''mbsyncloop'', * Window 1 has no scrollback buffer (''-h 0'') and calls a wrapper to ''mutt'' with an admittedly quite verbose name. Scrollback is not needed since ''mutt'' is ncurses-based. Finally, it remaps the ''c'' command key which is normally used to create a new screen window to do just the same but automatically run another instance of ''mutt''. This makes it very convenient to quickly fire up a secondary ''mutt'' to search mails or whatever in case the first one should be occupied (e.g. with composing an email). Finally, the last puzzle piece's stud is that oddly named wrapper. Luckily, it does just what it's name (subtly) suggests: #!/bin/bash mutt; killall mbsyncloop The idea here is that once the initial instance of ''mutt'' ends, it also kills the running ''mbsyncloop'' instance and (in case no other windows (read: ''mutt'' instances) are running, the screen session ends. ===== Links ===== * [[http://esmtp.sourceforge.net/|esmtp project page at sourceforge]] * [[http://isync.sourceforge.net/|isync, the project containing mbsync]]