Differences

This shows you the differences between two versions of the page.

--- email:better_mobile_email [2016/02/01 21:13] – created phil
+++ email:better_mobile_email [Unknown date] (current) – removed - external edit (Unknown date) 127.0.0.1
@@ Line 1: / Line 1: @@
-====== Better Mobile Email ======
-[[http://devconf.cz/filebrowser/download/440|Paul Frields' slides]] about his
-notebook's email setup show quite some similarities with my own one. The major
-differences are:
-  * I moved from ''offlineimap'' to  ''mbsync'', mostly for performance reasons.
-  * Instead of having a full-fledged MTA running on my notebook, I prefer a small send-only MTA for the sake of small footprint and ease of configuration.
-Here's how the setup is done in particular:
-===== The Big Picture =====
-When it comes to using the same email account on multiple machines, IMAP is a
-great benefit. It allows for every box to see every incoming mail and
-housekeeping tasks such as deleting old mails or reorganizing them into
-subfolders will automatically be visible from the other machines, as well. By
-storing sent mail in an IMAP folder the same applies to outgoing mail, too.
-The only downside with typical IMAP usage is that the task of email
-synchronization and caching has to be done by the email client (MUA). This
-naturally involves delays during client usage, and often clients are not
-really usable without direct connectivity to the IMAP server.
-My personal MUA of choice is ''mutt'', which suffers these issues, too. Especially
-since handling mail in local maildirs is strikingly fast, the added delay of
-having to sync with IMAP feels much higher than with other email clients.
-In order to overcome this, an asynchronous approach serves well: A dedicated
-instance covers the task of bidirectional synchronization between remote IMAP
-and local maildirs, while ''mutt'' operates exclusively on the latter.
-Using ''mutt'''s SMTP support involves surprisingly similar issues: Without direct
-connectivity, no mail sending is possible at all (of course not, but this can
-be directly experienced by ''mutt'' first hanging for a while and then giving up
-with an error message). But even if connectivity is given, any delay and/or
-bandwidth limitation on the line to the MTA is directly appreciable.
-Here, an asynchronous approach is chosen as well: A dedicated instance takes
-care of queueing and ultimately delivering the email to be sent and ''mutt''
-returns control to the user as soon as it has dispatched the job.
-===== Bidirectional IMAP Synchronization =====
-There are a number of tools to achieve this, but the only two which proved
-usable to me were ''offlineimap'' and ''mbsync''. Of those the latter is
-superior, not least due to the fact that it is written in C.
-==== Mbsync Configuration File ====
-''mbsync'' is configured on a per user basis in //$HOME/.mbsyncrc//. In order
-to understand it's semantics, a number of concepts have to be known by the
-user:
-  * A ''store'' is basically anything that is able to keep emails, like e.g. an mbox, a maildir or even an IMAP account. ''mbsync'' defines specific store types for these (except mbox, but who uses that anyway?) which hold type-dependent information.
-  * A ''channel'' defines a path of synchronisation between two stores, optionally limited to a specific scope within both.
-With these two in mind, let's start by defining two stores. The first one
-defines the local maildir:
-<code>
-MaildirStore local
-Path ~/
-Inbox ~/Maildir/
-SubFolders Maildir++
-</code>
-The way this is defined is a bit unintuitive: Instead of setting ''Path'' to
-the actual maildir path, it is set to it's parent and ''Inbox'' is set to
-point to the top-most maildir. I don't really remember why, but this was
-important to get my setup working. The second store defines the remote IMAP:
-<code>
-IMAPStore remote
-Host mail.nwl.cc
-User myself
-SSLType IMAPS
-</code>
-Obviously, this store defines the remote host running the IMAP service, the
-user name to use for authentication and that IMAPS should be used. If not
-using SASL authentication, one might want to additionally define the password
-using the ''Pass'' option.
-With both endpoints being defined, channels can be declared. The following
-defines two channels between those peers: One for the inbox only, the other
-for all subfolders. This way synchronisation of the INBOX can be triggered
-separately from the subfolders, which becomes useful later on:
-<code>
-Channel inbox
-Master :remote:INBOX/
-Slave :local:Maildir/
-Channel folders
-Master :remote:
-Slave :local:Maildir/
-Patterns * !INBOX !Chats !Contacts !Emailed\ Contacts !Junk
-</code>
-A few things are noteworthy about the used semantics:
-  * Each channel defines which of the peers is master and which is slave. This is important when defining how synchronisation should take place. Apart from that, these two keywords are interchangeable.
-  * The IMAP server exposes the inbox as a subfolder on it's own, therefore the first channel specifies this explicitly.
-  * The second channel makes use of the ''Patterns'' keyword for two purposes:
-    - The asterisk (''*'') sign tells ''mbsync'' to recurse into subfolders.
-    - The exclamation mark (''!'') prefixed folder names are excluded from synchronisation. In this case the remote is a Zimbra instance which seems to export (uninteresting) non-email data via IMAP, too.
-Finally, the above store and channel definitions are prepended by a few global
-statements:
-<code>
-Expunge Both
-Create Both
-Remove Both
-SyncState *
-</code>
-The ''Expunge'', ''Create'' and ''Remove'' statements control the
-synchronization process. Instead of ''Both'' one may also pass one of
-''Master'', ''Slave'' or ''None''. Using the latter for ''Expunge'' is
-recommended measure against unwanted loss of data during testing.
-''SyncState'' defines where the synchronization meta data should be kept. The
-asterisk (''*'') makes ''mbsync'' save it's meta data in the slave's store
-itself.
-==== Calling Mbsync ====
-To manually trigger synchronization of a defined channel, a direct call to
-''mbsync'' suffices:
-<code>
-mbsync <channel_name>
-</code>
-==== Looping Over Mbsync Calls ====
-To integrate this nicely into the rest of my setup, I have written a shell
-script which provides a few more goodies:
-  * Run indefinitely.
-  * Before every round of synchronization, try to ping the defined IMAP host (see ''Host'' directive above) or go to sleep for 10sec then try again.
-  * Synchronize the inbox every 10sec, the IMAP folders just once every 100sec.
-  * After every folder sync run, fetch a list of existing maildir subfolders into //$HOME/.mutt/mbsyncloop.mailboxes// in a format suitable for including into //$HOME/.muttrc//.
-<file bash mbsyncloop>
-#!/bin/bash
-mailboxes="${HOME}/.mutt/mbsyncloop.mailboxes"
-do_mbsync() { # (channel)
-	echo -e "synchronising $1 ..."
-	mbsync $1 && echo -e "... done" || echo -e "... failed"
-}
-get_mailboxes() { # (maildir)
-	echo -n "mailboxes \"+\""
-	find "$1" -type d -name cur | while read path; do
-		mbox="$(basename $(dirname $path))"
-		[[ $mbox == "."* ]] || continue
-		mbox="+$mbox"
-		echo -n " \"$mbox\""
-	done
-	echo ""
-}
-update_mailboxes() {
-	mboxes="$(get_mailboxes "$1")"
-	[[ -f $mailboxes ]] || {
-		echo "initially creating $mailboxes"
-		echo -n $mboxes > "$mailboxes"
-		return
-	}
-	old_mboxes="$(<$mailboxes)"
-	[[ "$mboxes" == "$old_mboxes" ]] || {
-		echo "updating $mailboxes"
-		echo -n $mboxes > "$mailboxes"
-	}
-}
-imaphost="$(awk '/^Host /{print $2}' ~/.mbsyncrc)"
-# sync INBOX every 10 seconds,
-# sync subfolders every minute
-cnt=0
-while true; do
-	ping -c 1 $imaphost >/dev/null 2>&1 || {
-		echo "Host $imaphost not responding"
-		sleep 10
-		continue
-	}
-	do_mbsync inbox
-	[[ $((cnt % 10)) -eq 0 ]] && {
-		cnt=0
-		do_mbsync folders
-		update_mailboxes "${HOME}/.maildir"
-	}
-	((cnt++))
-	sleep 10
-done
-</file>
-===== Relay-Only MTA =====
-My notebook does not receive email from external hosts. For local delivery to
-maildirs, ''procmail'' serves well as simple alternative. The only thing left
-for an MTA on my notebook to do is to get my user's email out to the right MX
-using appropriate credentials. And all that in background, without disturbing
-my workflow in case something (Wifi, uplink, VPN, MX, etc.) does not work as
-it should.
-Back then when I was searching for a relay-only MTA with queueing support, I
-didn't find any which felt comfortable. Therefore I sticked to ''esmtp'' I was
-using already and wrote a little shell-wrapper implementing the desired
-enhancement. This shell wrapper has been accepted into the official code
-repository back in 2007, but sadly the whole project was orphaned in 2011.
-Though, Gentoo luckily still ships it. Probably I should fork the project and
-revive it by implementing queueing support in C. Anyway, here's how to set
-things up using the latest release 1.2:
-==== Esmtp Configuration File ====
-The configuration file is quite straightforward. The only thing worth
-mentioning in beforehand is ''esmtp'''s support for identities: These allow to
-choose a different SMTP upstream based on sending address. Here's the config I
-use:
-<file esmtprc>
-mda = '/usr/bin/formail -a "Date: `date -R`" | /usr/bin/procmail -d %T'
-hostname = smtp.corp.example.com
-force reverse_path = myself@example.com
-identity = myself@nwl.cc
-	preconnect = "ssh -f -L 25025:localhost:25 nwl.cc 'sleep 5'"
-	hostname = localhost:25025
-	force reverse_path = myself@nwl.cc
-</file>
-The first line (''mda'')  defines the method for local delivery, i.e. for
-recipients not containing an @-sign. Piping through ''formail'' adds the
-useful ''Date'' header, which is otherwise taken care of by the external MTA.
-The default delivery method is via ''smtp.corp.example.com'' which requires no
-authentication (only reachable via VPN). Whatever ''From'' address I specify,
-''esmtp'' sanitizes the reverse path for me.
-A second identity defines my private mail setup. It establishes an SSH tunnel
-prior to connecting, which is convenient for me as it relieves me from having
-to specify login credentials here and SSH Agent is running anyway.
-==== Esmtp Queueing Wrapper ====
-Now for queueing support: The whole magic is implemented by my shell wrapper,
-''esmtp-wrapper''. I have it installed in //$HOME/bin// with appropriate
-symlinks (it changes it's operation mode depending the name by which it was
-called):
-<code>
-% find bin -lname esmtp-wrapper
-bin/mailq
-bin/deliver
-bin/sendmail
-</code>
-The commands to what their name suggests: ''mailq'' lists the local mail
-queue, ''deliver'' triggers mail delivery and ''sendmail'' just pretends to be
-''sendmail'' itself. Queueing is implemented as follows:
-Upon invocation, ''sendmail'' writes the parameters it was called with along
-with the mail body passed via ''stdin'' into a temporary directory below
-//$HOME/.esmtp_queue// and then calls itself as ''deliver'' in background.
-Upon invocation, ''deliver'' first checks if it has been called by
-''sendmail'' and in that case sleeps 5sec before doing anything (this is meant
-to catch mass mail delivery like e.g. git-send-email typically does). Then it
-continues to browse through any directories below //$HOME/.esmtp_queue// and
-calls ''esmtp'' exactly like ''sendmail'' has been called before. If that
-succeeds, the temporary directory is deleted, otherwise it's kept for a later
-try.
-==== Periodic Mail Queue Flushing ====
-If initial mail delivery fails, it has to be retried later. Since ''esmtp''
-does not run as a daemon, there is no automatic way to achieve this. Instead,
-a simple cron job serves well:
-<code>
-*/10 * * * * /bin/ping -c1 smtp.corp.example.com >/dev/null 2>&1 && $HOME/bin/deliver
-</code>
-This checks connectivity every 10min and calls ''deliver'' if given.
-===== Wrapping Things Up =====
-The last piece of the puzzle is ''mutt''. It's own setup is trivial, as it
-actually does not know about ''mbsync'' at all. And for ''esmtp'', all there
-is to specify is the path to ''sendmail'':
-<file>
-set sendmail=~/bin/sendmail
-</file>
-What is more interesting is how to conveniently run ''mutt'' and
-''mbsyncloop'' together. For that purpose, I have a dedicated //.screenrc//:
-<file screenrc.mutt>
-source .screenrc
-screen -t SYNC	0 mbsyncloop
-screen -t MUTT	-h 0 1 mutt_then_killall_mbsyncloop
-bind c screen mutt
-bind ^c screen mutt
-</file>
-When screen is invoked passing this config via it's ''-c'' parameter, it will
-automatically create two windows:
-  * Window 0 runs ''mbsyncloop'',
-  * Window 1 has no scrollback buffer (''-h 0'') and calls a wrapper to ''mutt'' with an admittedly quite verbose name. Scrollback is not needed since ''mutt'' is ncurses-based.
-Finally, it remaps the ''c'' command key which is normally used to create a
-new screen window to do just the same but automatically run another instance
-of ''mutt''. This makes it very convenient to quickly fire up a secondary
-''mutt'' to search mails or whatever in case the first one should be occupied
-(e.g. with composing an email).
-Finally, the last puzzle piece's stud is that oddly named wrapper. Luckily, it
-does just what it's name (subtly) suggests:
-<file bash mutt_then_killall_mbsyncloop>
-#!/bin/bash
-mutt; killall mbsyncloop
-</file>
-The idea here is that once the initial instance of ''mutt'' ends, it also
-kills the running ''mbsyncloop'' instance and (in case no other windows (read:
-''mutt'' instances) are running, the screen session ends.
-===== Links =====
-  * [[http://esmtp.sourceforge.net/|esmtp project page at sourceforge]]
-  * [[http://isync.sourceforge.net/|isync, the project containing mbsync]]