DokuWiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Better Mobile Email ======

[[http://devconf.cz/filebrowser/download/440|Paul Frields' slides]] about his
notebook's email setup show quite some similarities with my own one. The major
differences are:

  * I moved from ''offlineimap'' to  ''mbsync'', mostly for performance reasons.
  * Instead of having a full-fledged MTA running on my notebook, I prefer a small send-only MTA for the sake of small footprint and ease of configuration.

Here's how the setup is done in particular:

===== The Big Picture =====

When it comes to using the same email account on multiple machines, IMAP is a
great benefit. It allows for every box to see every incoming mail and
housekeeping tasks such as deleting old mails or reorganizing them into
subfolders will automatically be visible from the other machines, as well. By
storing sent mail in an IMAP folder the same applies to outgoing mail, too.

The only downside with typical IMAP usage is that the task of email
synchronization and caching has to be done by the email client (MUA). This
naturally involves delays during client usage, and often clients are not
really usable without direct connectivity to the IMAP server.

My personal MUA of choice is ''mutt'', which suffers these issues, too. Especially
since handling mail in local maildirs is strikingly fast, the added delay of
having to sync with IMAP feels much higher than with other email clients.

In order to overcome this, an asynchronous approach serves well: A dedicated
instance covers the task of bidirectional synchronization between remote IMAP
and local maildirs, while ''mutt'' operates exclusively on the latter.

Using ''mutt'''s SMTP support involves surprisingly similar issues: Without direct
connectivity, no mail sending is possible at all (of course not, but this can
be directly experienced by ''mutt'' first hanging for a while and then giving up
with an error message). But even if connectivity is given, any delay and/or
bandwidth limitation on the line to the MTA is directly appreciable.

Here, an asynchronous approach is chosen as well: A dedicated instance takes
care of queueing and ultimately delivering the email to be sent and ''mutt''
returns control to the user as soon as it has dispatched the job.

===== Bidirectional IMAP Synchronization =====

There are a number of tools to achieve this, but the only two which proved
usable to me were ''offlineimap'' and ''mbsync''. Of those the latter is
superior, not least due to the fact that it is written in C.

==== Mbsync Configuration File ====

''mbsync'' is configured on a per user basis in //$HOME/.mbsyncrc//. In order
to understand it's semantics, a number of concepts have to be known by the
user:

  * A ''store'' is basically anything that is able to keep emails, like e.g. an mbox, a maildir or even an IMAP account. ''mbsync'' defines specific store types for these (except mbox, but who uses that anyway?) which hold type-dependent information.

  * A ''channel'' defines a path of synchronisation between two stores, optionally limited to a specific scope within both.

With these two in mind, let's start by defining two stores. The first one
defines the local maildir:

<code>
MaildirStore local
Path ~/
Inbox ~/Maildir/
SubFolders Maildir++
</code>

The way this is defined is a bit unintuitive: Instead of setting ''Path'' to
the actual maildir path, it is set to it's parent and ''Inbox'' is set to
point to the top-most maildir. I don't really remember why, but this was
important to get my setup working. The second store defines the remote IMAP:

<code>
IMAPStore remote
Host mail.nwl.cc
User myself
SSLType IMAPS
</code>

Obviously, this store defines the remote host running the IMAP service, the
user name to use for authentication and that IMAPS should be used. If not
using SASL authentication, one might want to additionally define the password
using the ''Pass'' option.

With both endpoints being defined, channels can be declared. The following
defines two channels between those peers: One for the inbox only, the other
for all subfolders. This way synchronisation of the INBOX can be triggered
separately from the subfolders, which becomes useful later on:

<code>
Channel inbox
Master :remote:INBOX/
Slave :local:Maildir/

Channel folders
Master :remote:
Slave :local:Maildir/
Patterns * !INBOX !Chats !Contacts !Emailed\ Contacts !Junk
</code>

A few things are noteworthy about the used semantics:

  * Each channel defines which of the peers is master and which is slave. This is important when defining how synchronisation should take place. Apart from that, these two keywords are interchangeable.

  * The IMAP server exposes the inbox as a subfolder on it's own, therefore the first channel specifies this explicitly.

  * The second channel makes use of the ''Patterns'' keyword for two purposes:
    - The asterisk (''*'') sign tells ''mbsync'' to recurse into subfolders.
    - The exclamation mark (''!'') prefixed folder names are excluded from synchronisation. In this case the remote is a Zimbra instance which seems to export (uninteresting) non-email data via IMAP, too.

Finally, the above store and channel definitions are prepended by a few global
statements:

<code>
Expunge Both
Create Both
Remove Both

SyncState *
</code>

The ''Expunge'', ''Create'' and ''Remove'' statements control the
synchronization process. Instead of ''Both'' one may also pass one of
''Master'', ''Slave'' or ''None''. Using the latter for ''Expunge'' is
recommended measure against unwanted loss of data during testing.

''SyncState'' defines where the synchronization meta data should be kept. The
asterisk (''*'') makes ''mbsync'' save it's meta data in the slave's store
itself.

==== Calling Mbsync ====

To manually trigger synchronization of a defined channel, a direct call to
''mbsync'' suffices:

<code>
mbsync <channel_name>
</code>

==== Looping Over Mbsync Calls ====

To integrate this nicely into the rest of my setup, I have written a shell
script which provides a few more goodies:

  * Run indefinitely.
  * Before every round of synchronization, try to ping the defined IMAP host (see ''Host'' directive above) or go to sleep for 10sec then try again.
  * Synchronize the inbox every 10sec, the IMAP folders just once every 100sec.
  * After every folder sync run, fetch a list of existing maildir subfolders into //$HOME/.mutt/mbsyncloop.mailboxes// in a format suitable for including into //$HOME/.muttrc//.

<file bash mbsyncloop>
#!/bin/bash
mailboxes="${HOME}/.mutt/mbsyncloop.mailboxes"

do_mbsync() { # (channel)
	echo -e "synchronising $1 ..."
	mbsync $1 && echo -e "... done" || echo -e "... failed"
}

get_mailboxes() { # (maildir)
	echo -n "mailboxes \"+\""
	find "$1" -type d -name cur | while read path; do
		mbox="$(basename $(dirname $path))"
		[[ $mbox == "."* ]] || continue
		mbox="+$mbox"
		echo -n " \"$mbox\""
	done
	echo ""
}

update_mailboxes() {
	mboxes="$(get_mailboxes "$1")"
	[[ -f $mailboxes ]] || {
		echo "initially creating $mailboxes"
		echo -n $mboxes > "$mailboxes"
		return
	}
	old_mboxes="$(<$mailboxes)"
	[[ "$mboxes" == "$old_mboxes" ]] || {
		echo "updating $mailboxes"
		echo -n $mboxes > "$mailboxes"
	}
}

imaphost="$(awk '/^Host /{print $2}' ~/.mbsyncrc)"

# sync INBOX every 10 seconds,
# sync subfolders every minute
cnt=0
while true; do
	ping -c 1 $imaphost >/dev/null 2>&1 || {
		echo "Host $imaphost not responding"
		sleep 10
		continue
	}
	do_mbsync inbox
	[[ $((cnt % 10)) -eq 0 ]] && {
		cnt=0
		do_mbsync folders
		update_mailboxes "${HOME}/.maildir"
	}
	((cnt++))
	sleep 10
done
</file>

===== Relay-Only MTA =====

My notebook does not receive email from external hosts. For local delivery to
maildirs, ''procmail'' serves well as simple alternative. The only thing left
for an MTA on my notebook to do is to get my user's email out to the right MX
using appropriate credentials. And all that in background, without disturbing
my workflow in case something (Wifi, uplink, VPN, MX, etc.) does not work as
it should.

Back then when I was searching for a relay-only MTA with queueing support, I
didn't find any which felt comfortable. Therefore I sticked to ''esmtp'' I was
using already and wrote a little shell-wrapper implementing the desired
enhancement. This shell wrapper has been accepted into the official code
repository back in 2007, but sadly the whole project was orphaned in 2011.
Though, Gentoo luckily still ships it. Probably I should fork the project and
revive it by implementing queueing support in C. Anyway, here's how to set
things up using the latest release 1.2:

==== Esmtp Configuration File ====

The configuration file is quite straightforward. The only thing worth
mentioning in beforehand is ''esmtp'''s support for identities: These allow to
choose a different SMTP upstream based on sending address. Here's the config I
use:

<file esmtprc>
mda = '/usr/bin/formail -a "Date: `date -R`" | /usr/bin/procmail -d %T'

hostname = smtp.corp.example.com
force reverse_path = myself@example.com

identity = myself@nwl.cc
	preconnect = "ssh -f -L 25025:localhost:25 nwl.cc 'sleep 5'"
	hostname = localhost:25025
	force reverse_path = myself@nwl.cc
</file>

The first line (''mda'')  defines the method for local delivery, i.e. for
recipients not containing an @-sign. Piping through ''formail'' adds the
useful ''Date'' header, which is otherwise taken care of by the external MTA.

The default delivery method is via ''smtp.corp.example.com'' which requires no
authentication (only reachable via VPN). Whatever ''From'' address I specify,
''esmtp'' sanitizes the reverse path for me.

A second identity defines my private mail setup. It establishes an SSH tunnel
prior to connecting, which is convenient for me as it relieves me from having
to specify login credentials here and SSH Agent is running anyway.

==== Esmtp Queueing Wrapper ====

Now for queueing support: The whole magic is implemented by my shell wrapper,
''esmtp-wrapper''. I have it installed in //$HOME/bin// with appropriate
symlinks (it changes it's operation mode depending the name by which it was
called):

<code>
% find bin -lname esmtp-wrapper
bin/mailq
bin/deliver
bin/sendmail
</code>

The commands to what their name suggests: ''mailq'' lists the local mail
queue, ''deliver'' triggers mail delivery and ''sendmail'' just pretends to be
''sendmail'' itself. Queueing is implemented as follows:

Upon invocation, ''sendmail'' writes the parameters it was called with along
with the mail body passed via ''stdin'' into a temporary directory below
//$HOME/.esmtp_queue// and then calls itself as ''deliver'' in background.

Upon invocation, ''deliver'' first checks if it has been called by
''sendmail'' and in that case sleeps 5sec before doing anything (this is meant
to catch mass mail delivery like e.g. git-send-email typically does). Then it
continues to browse through any directories below //$HOME/.esmtp_queue// and
calls ''esmtp'' exactly like ''sendmail'' has been called before. If that
succeeds, the temporary directory is deleted, otherwise it's kept for a later
try.

==== Periodic Mail Queue Flushing ====

If initial mail delivery fails, it has to be retried later. Since ''esmtp''
does not run as a daemon, there is no automatic way to achieve this. Instead,
a simple cron job serves well:

<code>
*/10 * * * * /bin/ping -c1 smtp.corp.example.com >/dev/null 2>&1 && $HOME/bin/deliver
</code>

This checks connectivity every 10min and calls ''deliver'' if given.

===== Wrapping Things Up =====

The last piece of the puzzle is ''mutt''. It's own setup is trivial, as it
actually does not know about ''mbsync'' at all. And for ''esmtp'', all there
is to specify is the path to ''sendmail'':

<file>
set sendmail=~/bin/sendmail
</file>

What is more interesting is how to conveniently run ''mutt'' and
''mbsyncloop'' together. For that purpose, I have a dedicated //.screenrc//:

<file screenrc.mutt>
source .screenrc
screen -t SYNC	0 mbsyncloop
screen -t MUTT	-h 0 1 mutt_then_killall_mbsyncloop
bind c screen mutt
bind ^c screen mutt
</file>

When screen is invoked passing this config via it's ''-c'' parameter, it will
automatically create two windows:

  * Window 0 runs ''mbsyncloop'',
  * Window 1 has no scrollback buffer (''-h 0'') and calls a wrapper to ''mutt'' with an admittedly quite verbose name. Scrollback is not needed since ''mutt'' is ncurses-based.

Finally, it remaps the ''c'' command key which is normally used to create a
new screen window to do just the same but automatically run another instance
of ''mutt''. This makes it very convenient to quickly fire up a secondary
''mutt'' to search mails or whatever in case the first one should be occupied
(e.g. with composing an email).

Finally, the last puzzle piece's stud is that oddly named wrapper. Luckily, it
does just what it's name (subtly) suggests:

<file bash mutt_then_killall_mbsyncloop>
#!/bin/bash

mutt; killall mbsyncloop
</file>

The idea here is that once the initial instance of ''mutt'' ends, it also
kills the running ''mbsyncloop'' instance and (in case no other windows (read:
''mutt'' instances) are running, the screen session ends.

===== Links =====

  * [[http://esmtp.sourceforge.net/|esmtp project page at sourceforge]]
  * [[http://isync.sourceforge.net/|isync, the project containing mbsync]]