[doc] update documentation

I'm updating (1) some very outdated doc from when the program was not yet
written, and (2) some small stuff inside classes docstrings.

6 files changed, 120 insertions, 202 deletions

diff --git a/README.md b/README.md
index 03b0ade..ca58bca 100644
--- a/README.md
+++ b/README.md
@@ -1,82 +1,43 @@
-leap_mx
+Leap MX
 =======
-**Note:** Currently in development. Feel free to test, and please [report
- bugs on our tracker](https://we.riseup.net/leap/mx) or [by email](mailto:isis@leap.se).
+
+**Note:** Currently in development. Feel free to test, and please [report bugs
+on our tracker](https://we.riseup.net/leap/mx) or [by
+email](mailto:discuss@leap.se).
 
 An asynchronous, transparently-encrypting remailer for the LEAP platform,
 using BigCouch/CouchDB and PGP/GnuPG, written in Twisted Python.
 
-## [install](#install) ##
-
-### [virtualenv](#virtualenv) ###
-=================================
-Impatient? Don't like virtualenvs? [tl;dr](#tl;dr)
-
-Virtualenv is somewhat equivalent to fakeroot for python packages, and -- due
-to being packaged with copies of pip and python -- can be used to bootstrap
-its own install process, allowing pip and python to be used with sudo.
-
-#### installing without sudo ####
+## Installing
 
-To install without using sudo, a bootstrap script to handle the setup process
-is provided. It does the following:
-
- 1. Download, over SSL, the latest tarballs for virtualenv and
- virtualenvwrapper from pypi.
- 2. Unpack the tarballs, use the system python interpreter to call the
- virtualenv.py script to setup a bootstrap virtual environment.
- 3. Use the pip installed in the bootstrap virtualenv to install
- virtualenvwrapper in the bootstrap virtualenv.
- 4. Obtain a copy of leap_mx with git clone.
- 5. Use ```mkvirtualenv``` included in the virtualenvwrapper inside the
- bootstrap virtualenv to install a project virtualenv for leap_mx.
-
-To use the bootstrap script, do:
-~~~
-$ wget -O bootstrap https://raw.github.com/isislovecruft/leap_mx/fix/no-suid-for-virtualenv/bootstrap
-$ ./bootstrap
-$ workon leap_mx
-~~~
+  * Leap MX is available as a debian package in [Leap
+    repository](http://deb.leap.se/repository/).
+  * A python package is available in
+    [pypi](https://pypi.python.org/pypi/leap.mx). Use ./pkg/requirements.pip
+    to install requirements.
+  * Source code is available in [github](https://github.com/leapcode/leap_mx).
 
-#### installing in a regular virtualenv ###
-To install python, virtualenv, and get started, do:
+## Configuring
 
-~~~
-$ sudo apt-get install python2.7 python-dev python-virtualenv virtualenvwrapper
-$ git clone https://github.com/leapcode/leap_mx.git leap_mx
-$ export WORKON_LEAPMX=${PWD}/leap_mx
-$ source /usr/local/bin/virtualenvwrapper.sh
-$ mkvirtualenv -a $WORKON_LEAPMX -r ${WORKON_LEAPMX}/pkg/requirements.pip \
-      --no-site-packages --setuptools --unzip-setuptools leap_mx
-~~~
-
-### [tl;dr](#tl;dr) ###
-To get started quickly, without virtualenv, do:
-~~~
-$ sudo apt-get install python git
-$ git clone https://github.com/leapcode/leap_mx.git
-# pip install -r ./leap_mx/pkg/requirements.pip
-~~~
-Although, **it is advised** to install inside a python virtualenv.
-
-## [configuration](#configuration) ##
 A sample config file can be found in pkg/mx.conf.sample
 
-## [running](#running) ##
-=========================
+## Running
 
-To get running, clone this repo, and (assuming you've already set up your
-virtualenv and obtained all the requirements) do:
+The debian package contains an initscript for the service. If you want to run
+from source or from the python package, maybe setup a virtualenv and do:
 
 ~~~
-$ twistd -ny mx.tac
+# git clone or unzip the python package, change into the dir, and do:
+$ python setup.py install
+# copy ./pkg/mx.conf.sample to /etc/leap/mx.conf and edit that file, then run:
+$ twistd -ny pkg/mx.tac
 ~~~
 
-## [hacking](#hacking) ##
-=========================
-Please see the HACKING and DESIGN docs.
+## Hacking
+
+Please see the doc/DESIGN docs.
 
-Our bugtracker is [here](https://leap.se/code/projects/eip/issue/new).
+Our bugtracker is [here](https://leap.se/code/projects/mx).
 
 Please use that for bug reports and feature requests instead of github's
 tracker. We're using github for code commenting and review between
diff --git a/doc/DESIGN.md b/doc/DESIGN.md
index 2d9fe82..e98976d 100644
--- a/doc/DESIGN.md
+++ b/doc/DESIGN.md
@@ -1,7 +1,7 @@
-# design #
+# design
+
+## overview
 
-## overview #
-----------------------
 This page pertains to the incoming mail exchange servers of the provider.
 
 General overview of how incoming email will work:
@@ -27,39 +27,36 @@ General overview of how incoming email will work:
  9. Soledad, in the background, will then re-encrypt this email (now a soledad
  document), and sync to the cloud.
 
-## postfix pipeline ##
----------------------------
+## postfix pipeline
+
 incoming mx servers will run postfix, configured in a particular way:
 
  1. postscreen: before accepting an incoming message, checks RBLs, checks RFC
  validity, checks for spam pipelining.
-    (pass) proceed to next step.
-    (fail) return SMTP error, which bounces email.
+   * (pass) proceed to next step.
+   * (fail) return SMTP error, which bounces email.
  2. more SMTP checks: valid hostnames, etc.
-    (pass) accepted, proceed to next step.
-    (fail) return SMTP error, which bounces email.
+   * (pass) accepted, proceed to next step.
+   * (fail) return SMTP error, which bounces email.
  3. check_recipient_access -- look up each recipient and ensure they are
  allowed to receive messages.
-    (pass) empty result, proceed to next step.
-    (fail) return SMTP error code and error comment, bounce message.
+   * (pass) empty result, proceed to next step.
+   * (fail) return SMTP error code and error comment, bounce message.
  4. milter processessing (spamassassin & clamav)
-    (pass) continue
-    (fail) bounce message, flag as spam, or silently kill.
+   * (pass) continue
+   * (fail) bounce message, flag as spam, or silently kill.
  5. virtual_alias_maps -- map user defined aliases and forwards
-    (local address) continue if new address is for this mx
-    (remote address) continue. normally, postfix would relay to the remote domain, but we don't want that.
+   * (local address) continue if new address is for this mx
+   * (remote address) continue. normally, postfix would relay to the remote domain, but we don't want that.
  6. deliver message to spool
-    (write) save the message to disk on the mx.
+   * (write) save the message to disk on the mx.
  7. postfix's job is done, mail_receiver picks up email from spool directory
 
 Questions:
 
- * what is the best way to have postfix write a message to a spool directory?
-   There is a built-in facility for saving to a maildir, so we could just
-   specify a common maildir for everyone. alternately, we could pipe to a
-   simple command that was responsible for safely saving the file to disk. a
-   third possibility would be to have a local long running daemon that spoke
-   lmtp that postfix forward the message on to for delivery.
+ * postfix uses a built-in facility and saves all messages to a common
+   maildir. as an alternative, we could have a local long running daemon that
+   spoke lmtp that postfix forward the message on to for delivery.
  * if virtual_alias_maps comes after check_recipient_access, then a user with
    aliases set but who is over quota will not be able to forward email. i think
    this is fine.
@@ -80,41 +77,42 @@ Considerations:
  somewhere, and that copy should not be deleted until there is confirmation
  that the next stage has succeeded.
  
-## alias_resolver ##
-------------------------------
-The alias_resolver will be a daemon running on MX servers that handles lookups
-in the user database of email aliases, forwards, quota, and account status.
+## TCP Maps
+
+MX runs TCP maps that handle lookups in the user database of email aliases,
+forwards, quota, and account status.
 
 Communication with:
 
- 1. postfix:: alias_resolver will be bound to localhost and speak postfix's
- very simple [tcp map protocol -> http://www.postfix.org/tcp_table.5.html].
+ 1. postfix: bind to localhost and speak postfix's very simple [tcp map
+    protocol -> http://www.postfix.org/tcp_table.5.html].
 
- 2. couchdb:: alias_resolver will make couchdb queries to a local http load
- balancer that connects to a couchdb/bigcouch
- cluster. [directly accessing the couch->https://we.riseup.net/leap+platform/querying-the-couchdb]
- might help getting started.
+ 2. couchdb: make couchdb queries to a local http load  balancer that connects
+    to a couchdb/bigcouch cluster.
 
-### Discussion: ###
+### Discussion
 
  1. we want the lookups to be fast. using views in couchdb, these should be
-   very fast. when using bigcouch, we can make it faster by specifying a read
-   quorum of 1 (instead of the default 2). this will make it so that only a
-   single couchdb needs to be queried to find the result. i don't know if this
-   would cause problems, but aliases don't change very often.
+    very fast. when using bigcouch, we can make it faster by specifying a read
+    quorum of 1 (instead of the default 2). this will make it so that only a
+    single couchdb needs to be queried to find the result. i don't know if
+    this would cause problems, but aliases don't change very often.
+
+TCP map is responsible for two map lookups in postfix: ```check_recipient```
+and ```virtual_alias_map```.
 
-alias_resolver will be responsible for two map lookups in postfix:
+#### check_recipient
 
-#### check_recipient ####
--------------------------
 postfix config:
 
-@check_recipient_access tcp:localhost:1000@
+```
+check_recipient_access tcp:localhost:2244
+```
 
-postfix will send "get username@domain.org" and alias_resolver should return an
-empty result ("200 \n", i think) if postfix should deliver email to the
-user. otherwise, it should return an error. here is example response, verbatim,
-that can be used to bounce over quota users:
+postfix sends "get username@domain.org" and alias_resolver returns an empty
+result ("200 \n", i think) if postfix should deliver email to the user.
+otherwise, it returns an error. here is example response, verbatim, that
+can be used to bounce over quota users:
 
 ```
 200 DEFER_IF_PERMIT Sorry, your message cannot be delivered because the
@@ -126,34 +124,33 @@ to tell them of this problem.
 that they should try again soon. Typically, an MX will try repeatedly, at
 longer and longer intervals, for four days before giving up.
 
-#### virtual alias map ####
----------------------------
+#### virtual_alias_map
+
 postfix config:
 
-@virtual_alias_map tcp:localhost:1001@
+```
+virtual_alias_map tcp:localhost:4242
+```
 
-postfix will send "get alias-address@domain.org" and alias_resolver should
-return "200 id_123456\n", where 123456 is the unique id of the user that has
+postfix sends "get alias-address@domain.org" and alias_resolver returns "200
+123456\n", where 123456 is the unique id of the user that has
 alias-address@domain.org.
 
-couchdb should have a view that will let us query on an (alias) address and
-return the user id.
+couchdb has a view that lets us query on an (alias) address and return the
+user id.
 
-note: if the result of the alias map (e.g. id_123456) does not have a domain
-suffix, i think postfix will use the 'default transport'. if we want it to use
-the virtual transport instead, we should append the domain (eg
-id_123456@example.org). see
+note: if the result of the alias map (e.g. 123456) does not have a domain
+suffix, postfix will use the 'default transport'. if we want it to use the
+virtual transport instead, we should append the domain (eg
+123456@example.org). see
 http://www.postfix.org/ADDRESS_REWRITING_README.html#resolve
 
 
-### Current status: ### 
+### Current status
+
 The current implementation of alias_resolver is in
 leap-mx/src/leap/mx/alias_resolver.py.
 
-The class ```alias_resolver.StatusCodes``` deals with creating SMTP-like
-response messages for Postfix, speaking Postfix's TCP Map protocol (from item
-#1).
-
 As for Discussion item #1: 
 
 It might be possible to use
@@ -165,37 +162,8 @@ handling Memcached servers, this is in ```twisted.protocols.memcache```. This
 should be prioritised for later, if it is decided that querying the CouchDB is
 too expensive or time-consuming.
 
-Thus far, to speed up alias lookup, an in-memory mapping of alias<->resolution
-pairs is created by ```alias_resolver.AliasResolverFactory()```, which can be
-optionally seeded with a dictionary of ```{ 'alias': 'resolution' }``` pairs
-by doing:
-~~~~~~
->>> from leap.mx import alias_resolver
->>> aliasResolverFactory = alias_resolver.AliasResolverFactory(
-...      addr='1.2.3.4', port=4242, data={'isis': 'isis@leap.se',
-...                                       'drebs': 'drebs@leap.se'})
->>> aliasResolver = aliasResolverFactory.buildProtocol()
->>> aliasResolver.check_recipient_access('isis')
-200 OK Others might say 'HELLA AWESOME'...but we're not convinced.
-~~~~~~
-
-TODO:
- 1. The AliasResolverFactory needs to be connected to the CouchDB. The
- classmethod in which this should occur is ```AliasResolverFactory.get()```.
-
- 2. I am not sure where to get the user's UUID from (Soledad?). Wherever we get
- it from, it will need to be returned in
- ```AliasResolver.virtual_alias_map()```, and if we want Postfix to hear about
- it, then that response will need to be fed into ```AliasResolver.sendCode```.
- 
- 3. Other than those two things, I think everything is done. The only potential
- other thing I can think of is that the codes in
- ```alias_resolver.StatusCodes``` might need to be urlencoded for Postfix to
- accept them, but this is like two lines of code from urllib.
-
-
 
-## mail_receiver ##
+## mail_receiver
 
 the mail_receiver is a daemon that runs on incoming MX servers and is
 responsible for encrypting incoming email to the user's public key and saving
@@ -203,36 +171,36 @@ the email to an incoming queue database for that user.
 
 communicates with:
 
- * message spool directory:: mail_reciever sits and waits for new email to be
+ * message spool directory: mail_reciever sits and waits for new email to be
    written to the spool directory (maybe using this
    https://github.com/seb-m/pyinotify, i think it is better than FAM). when a
    new file is dumped into the spool, mail_receiver reads the file, encrypts
    the entire thing using the public key of the recipient, and saves to
    couchdb.
- * couchdb get:: mail_receiver does a query on user id to get back user's
+ * couchdb get: mail_receiver does a query on user id to get back user's
    public openpgp key. read quorum of 1 is probably ok.
- * couchdb put:: mail_receiver communicates with couchdb for storing encrypted
+ * couchdb put: mail_receiver communicates with couchdb for storing encrypted
    email for each user (eventually, mail_receiver will communicate with a local
    http proxy, that communicates with a bigcouch cluster, but the api is
    identical)
 
-discussion:
- * i am not sure if postfix adds a header to indicate to whom a message was
-   actually delivered. if not, this is a problem, because then how do we know
-   what db to put it in or what public key to use? this is perhaps a good
-   reason to not let postfix handle writing the message to disk, but instead
-   pipe it to another command (because postfix sets env variables for stuff
-   like recipient).
+### Current Status
 
- * should the incoming message queue be a separate database or should it be
-   just documents in the user's main database with special flags?
+ * Postfix adds a "Delivered-To" header to indicate to whom a message was
+   actually delivered. There may be more than one header of this kind, and we
+   should use the topmost one. That value of that header is "<uuid>@<domain>",
+   and we can extract the user's uid from there to fetch the pgp public key to
+   which encrypt the message. 
+
+ * currently, the incoming message is put into the user's database with some
+   special flags. should we have a different way to deal with that?
 
  * whenever possible, we should refer to the user by a fixed id, not their
    username, because we want to support the ability to change usernames. so,
-   for example, database names should not be based on usernames.
-
-### Current Status: ###
-None of this is done, although having it be a separate daemon sound weird.
+   for example, database names should not be based on usernames. The alias
+   resolver then resolves to the user's uuid and from that point on we can
+   deal only with the user's uid.
 
-You would probably want to use ```twisted.mail.mail.FileMonitoringService``` to
-watch the mailbox (is the mailbox virtual or a maildir or mbox or?)
+ * We currently use ```twisted.internet.inotify.INotiry()``` to watch the
+   maildir for creation of new files. Alternatelly, we could possibly use
+   ```twisted.mail.mail.FileMonitoringService```.
diff --git a/doc/NOTES.md b/doc/NOTES.md
index a53f49d..337aa96 100644
--- a/doc/NOTES.md
+++ b/doc/NOTES.md
@@ -28,32 +28,3 @@ a viable protocol for this, and how would it interact with CouchDB?
 
 4. What lib should we use for Python + Twisted + GPG/PGP ?
    4.a. It looks like most people are using python-gnupg...
-
-
-## Tickets ##
--------------
-
-'''To be created:'''
-
-ticket for feature-alias_resolver_couchdb_support:
-
-    o The alias resolver needs to speak to a couchdb/bigcouch
-    instance(s). Currently, it merely creates an in-memory dictionary
-    mapping. It seems like paisley is the best library for this.
-
-ticket for feature-check_recipient: 
-
-    o Need various errors for anything that could go wrong, e.g. the recipient
-      address is malformed, sender doesn't have permissions to send to such
-      address, etc.
-    o These errcodes need to follow the SMTP server transport code spec.
-
-ticket for feature-virtual_alias_map: 
-
-    o Get the recipient's userid from couchdb.
-
-ticket for feature-evaluate_python_gnupg:
-
-    o Briefly audit library in order to assess if it has the necessary
-    features, as well as its general code quality.
-
diff --git a/src/leap/mx/alias_resolver.py b/src/leap/mx/alias_resolver.py
index 752eac4..c6f2acc 100644
--- a/src/leap/mx/alias_resolver.py
+++ b/src/leap/mx/alias_resolver.py
@@ -19,6 +19,12 @@
 """
 Classes for resolving postfix aliases.
 
+The resolver is queried by the mail server before delivery to the mail spool
+directory, and should return the user uuid. This way, we get rid from the user
+address early and the mail server will delivery the message to
+"<uuid>@<domain>". Later, the mail receiver part of MX will parse the
+"Delivered-To" header to extract the uuid and fetch the user's pgp public key.
+
 Test this with postmap -v -q "foo" tcp:localhost:4242
 
 TODO:
diff --git a/src/leap/mx/check_recipient_access.py b/src/leap/mx/check_recipient_access.py
index 9f79dfe..55460a6 100644
--- a/src/leap/mx/check_recipient_access.py
+++ b/src/leap/mx/check_recipient_access.py
@@ -17,7 +17,12 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 """
-Classes for resolving postfix recipient access
+Classes for resolving postfix recipient access.
+
+The resolver is queried by the mail server before delivery to the mail spool
+directory, and should check if the address is able to receive messages.
+Examples of reasons for denying delivery would be that the user is out of
+quota, is user, or have no pgp public key in the server.
 
 Test this with postmap -v -q "foo" tcp:localhost:2244
 """
@@ -44,6 +49,9 @@ class LEAPPostFixTCPMapAccessServer(postfix.PostfixTCPMapServer):
         Return a code and message depending on the result of the factory's
         get().
 
+        If there's no pgp public key for the user, we currently return a
+        temporary failure saying that the user account is disabled.
+
         For more info, see: http://www.postfix.org/access.5.html
 
         :param value: The uuid and public key.
diff --git a/src/leap/mx/tcp_map.py b/src/leap/mx/tcp_map.py
index 108c2aa..597c830 100644
--- a/src/leap/mx/tcp_map.py
+++ b/src/leap/mx/tcp_map.py
@@ -30,6 +30,10 @@ TCP_MAP_CODE_TEMPORARY_FAILURE = 400
 TCP_MAP_CODE_PERMANENT_FAILURE = 500
 
 
+# we have to also extend from object here to make the class a new-style class.
+# If we don't, we get a TypeError because "new-style classes can't have only
+# classic bases". This has to do with the way abc.ABCMeta works and the old
+# and new style of python classes.
 class LEAPPostfixTCPMapServerFactory(ServerFactory, object):
     """
     A factory for postfix tcp map servers.