summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
-rw-r--r--README.md87
-rw-r--r--doc/DESIGN.md186
-rw-r--r--doc/NOTES.md29
-rw-r--r--src/leap/mx/alias_resolver.py6
-rw-r--r--src/leap/mx/check_recipient_access.py10
-rw-r--r--src/leap/mx/tcp_map.py4
6 files changed, 120 insertions, 202 deletions
diff --git a/README.md b/README.md
index 03b0ade..ca58bca 100644
--- a/README.md
+++ b/README.md
@@ -1,82 +1,43 @@
-leap_mx
+Leap MX
=======
-**Note:** Currently in development. Feel free to test, and please [report
- bugs on our tracker](https://we.riseup.net/leap/mx) or [by email](mailto:isis@leap.se).
+
+**Note:** Currently in development. Feel free to test, and please [report bugs
+on our tracker](https://we.riseup.net/leap/mx) or [by
+email](mailto:discuss@leap.se).
An asynchronous, transparently-encrypting remailer for the LEAP platform,
using BigCouch/CouchDB and PGP/GnuPG, written in Twisted Python.
-## [install](#install) ##
-
-### [virtualenv](#virtualenv) ###
-=================================
-Impatient? Don't like virtualenvs? [tl;dr](#tl;dr)
-
-Virtualenv is somewhat equivalent to fakeroot for python packages, and -- due
-to being packaged with copies of pip and python -- can be used to bootstrap
-its own install process, allowing pip and python to be used with sudo.
-
-#### installing without sudo ####
+## Installing
-To install without using sudo, a bootstrap script to handle the setup process
-is provided. It does the following:
-
- 1. Download, over SSL, the latest tarballs for virtualenv and
- virtualenvwrapper from pypi.
- 2. Unpack the tarballs, use the system python interpreter to call the
- virtualenv.py script to setup a bootstrap virtual environment.
- 3. Use the pip installed in the bootstrap virtualenv to install
- virtualenvwrapper in the bootstrap virtualenv.
- 4. Obtain a copy of leap_mx with git clone.
- 5. Use ```mkvirtualenv``` included in the virtualenvwrapper inside the
- bootstrap virtualenv to install a project virtualenv for leap_mx.
-
-To use the bootstrap script, do:
-~~~
-$ wget -O bootstrap https://raw.github.com/isislovecruft/leap_mx/fix/no-suid-for-virtualenv/bootstrap
-$ ./bootstrap
-$ workon leap_mx
-~~~
+ * Leap MX is available as a debian package in [Leap
+ repository](http://deb.leap.se/repository/).
+ * A python package is available in
+ [pypi](https://pypi.python.org/pypi/leap.mx). Use ./pkg/requirements.pip
+ to install requirements.
+ * Source code is available in [github](https://github.com/leapcode/leap_mx).
-#### installing in a regular virtualenv ###
-To install python, virtualenv, and get started, do:
+## Configuring
-~~~
-$ sudo apt-get install python2.7 python-dev python-virtualenv virtualenvwrapper
-$ git clone https://github.com/leapcode/leap_mx.git leap_mx
-$ export WORKON_LEAPMX=${PWD}/leap_mx
-$ source /usr/local/bin/virtualenvwrapper.sh
-$ mkvirtualenv -a $WORKON_LEAPMX -r ${WORKON_LEAPMX}/pkg/requirements.pip \
- --no-site-packages --setuptools --unzip-setuptools leap_mx
-~~~
-
-### [tl;dr](#tl;dr) ###
-To get started quickly, without virtualenv, do:
-~~~
-$ sudo apt-get install python git
-$ git clone https://github.com/leapcode/leap_mx.git
-# pip install -r ./leap_mx/pkg/requirements.pip
-~~~
-Although, **it is advised** to install inside a python virtualenv.
-
-## [configuration](#configuration) ##
A sample config file can be found in pkg/mx.conf.sample
-## [running](#running) ##
-=========================
+## Running
-To get running, clone this repo, and (assuming you've already set up your
-virtualenv and obtained all the requirements) do:
+The debian package contains an initscript for the service. If you want to run
+from source or from the python package, maybe setup a virtualenv and do:
~~~
-$ twistd -ny mx.tac
+# git clone or unzip the python package, change into the dir, and do:
+$ python setup.py install
+# copy ./pkg/mx.conf.sample to /etc/leap/mx.conf and edit that file, then run:
+$ twistd -ny pkg/mx.tac
~~~
-## [hacking](#hacking) ##
-=========================
-Please see the HACKING and DESIGN docs.
+## Hacking
+
+Please see the doc/DESIGN docs.
-Our bugtracker is [here](https://leap.se/code/projects/eip/issue/new).
+Our bugtracker is [here](https://leap.se/code/projects/mx).
Please use that for bug reports and feature requests instead of github's
tracker. We're using github for code commenting and review between
diff --git a/doc/DESIGN.md b/doc/DESIGN.md
index 2d9fe82..e98976d 100644
--- a/doc/DESIGN.md
+++ b/doc/DESIGN.md
@@ -1,7 +1,7 @@
-# design #
+# design
+
+## overview
-## overview #
-----------------------
This page pertains to the incoming mail exchange servers of the provider.
General overview of how incoming email will work:
@@ -27,39 +27,36 @@ General overview of how incoming email will work:
9. Soledad, in the background, will then re-encrypt this email (now a soledad
document), and sync to the cloud.
-## postfix pipeline ##
----------------------------
+## postfix pipeline
+
incoming mx servers will run postfix, configured in a particular way:
1. postscreen: before accepting an incoming message, checks RBLs, checks RFC
validity, checks for spam pipelining.
- (pass) proceed to next step.
- (fail) return SMTP error, which bounces email.
+ * (pass) proceed to next step.
+ * (fail) return SMTP error, which bounces email.
2. more SMTP checks: valid hostnames, etc.
- (pass) accepted, proceed to next step.
- (fail) return SMTP error, which bounces email.
+ * (pass) accepted, proceed to next step.
+ * (fail) return SMTP error, which bounces email.
3. check_recipient_access -- look up each recipient and ensure they are
allowed to receive messages.
- (pass) empty result, proceed to next step.
- (fail) return SMTP error code and error comment, bounce message.
+ * (pass) empty result, proceed to next step.
+ * (fail) return SMTP error code and error comment, bounce message.
4. milter processessing (spamassassin & clamav)
- (pass) continue
- (fail) bounce message, flag as spam, or silently kill.
+ * (pass) continue
+ * (fail) bounce message, flag as spam, or silently kill.
5. virtual_alias_maps -- map user defined aliases and forwards
- (local address) continue if new address is for this mx
- (remote address) continue. normally, postfix would relay to the remote domain, but we don't want that.
+ * (local address) continue if new address is for this mx
+ * (remote address) continue. normally, postfix would relay to the remote domain, but we don't want that.
6. deliver message to spool
- (write) save the message to disk on the mx.
+ * (write) save the message to disk on the mx.
7. postfix's job is done, mail_receiver picks up email from spool directory
Questions:
- * what is the best way to have postfix write a message to a spool directory?
- There is a built-in facility for saving to a maildir, so we could just
- specify a common maildir for everyone. alternately, we could pipe to a
- simple command that was responsible for safely saving the file to disk. a
- third possibility would be to have a local long running daemon that spoke
- lmtp that postfix forward the message on to for delivery.
+ * postfix uses a built-in facility and saves all messages to a common
+ maildir. as an alternative, we could have a local long running daemon that
+ spoke lmtp that postfix forward the message on to for delivery.
* if virtual_alias_maps comes after check_recipient_access, then a user with
aliases set but who is over quota will not be able to forward email. i think
this is fine.
@@ -80,41 +77,42 @@ Considerations:
somewhere, and that copy should not be deleted until there is confirmation
that the next stage has succeeded.
-## alias_resolver ##
-------------------------------
-The alias_resolver will be a daemon running on MX servers that handles lookups
-in the user database of email aliases, forwards, quota, and account status.
+## TCP Maps
+
+MX runs TCP maps that handle lookups in the user database of email aliases,
+forwards, quota, and account status.
Communication with:
- 1. postfix:: alias_resolver will be bound to localhost and speak postfix's
- very simple [tcp map protocol -> http://www.postfix.org/tcp_table.5.html].
+ 1. postfix: bind to localhost and speak postfix's very simple [tcp map
+ protocol -> http://www.postfix.org/tcp_table.5.html].
- 2. couchdb:: alias_resolver will make couchdb queries to a local http load
- balancer that connects to a couchdb/bigcouch
- cluster. [directly accessing the couch->https://we.riseup.net/leap+platform/querying-the-couchdb]
- might help getting started.
+ 2. couchdb: make couchdb queries to a local http load balancer that connects
+ to a couchdb/bigcouch cluster.
-### Discussion: ###
+### Discussion
1. we want the lookups to be fast. using views in couchdb, these should be
- very fast. when using bigcouch, we can make it faster by specifying a read
- quorum of 1 (instead of the default 2). this will make it so that only a
- single couchdb needs to be queried to find the result. i don't know if this
- would cause problems, but aliases don't change very often.
+ very fast. when using bigcouch, we can make it faster by specifying a read
+ quorum of 1 (instead of the default 2). this will make it so that only a
+ single couchdb needs to be queried to find the result. i don't know if
+ this would cause problems, but aliases don't change very often.
+
+TCP map is responsible for two map lookups in postfix: ```check_recipient```
+and ```virtual_alias_map```.
-alias_resolver will be responsible for two map lookups in postfix:
+#### check_recipient
-#### check_recipient ####
--------------------------
postfix config:
-@check_recipient_access tcp:localhost:1000@
+```
+check_recipient_access tcp:localhost:2244
+```
-postfix will send "get username@domain.org" and alias_resolver should return an
-empty result ("200 \n", i think) if postfix should deliver email to the
-user. otherwise, it should return an error. here is example response, verbatim,
-that can be used to bounce over quota users:
+postfix sends "get username@domain.org" and alias_resolver returns an empty
+result ("200 \n", i think) if postfix should deliver email to the user.
+otherwise, it returns an error. here is example response, verbatim, that
+can be used to bounce over quota users:
```
200 DEFER_IF_PERMIT Sorry, your message cannot be delivered because the
@@ -126,34 +124,33 @@ to tell them of this problem.
that they should try again soon. Typically, an MX will try repeatedly, at
longer and longer intervals, for four days before giving up.
-#### virtual alias map ####
----------------------------
+#### virtual_alias_map
+
postfix config:
-@virtual_alias_map tcp:localhost:1001@
+```
+virtual_alias_map tcp:localhost:4242
+```
-postfix will send "get alias-address@domain.org" and alias_resolver should
-return "200 id_123456\n", where 123456 is the unique id of the user that has
+postfix sends "get alias-address@domain.org" and alias_resolver returns "200
+123456\n", where 123456 is the unique id of the user that has
alias-address@domain.org.
-couchdb should have a view that will let us query on an (alias) address and
-return the user id.
+couchdb has a view that lets us query on an (alias) address and return the
+user id.
-note: if the result of the alias map (e.g. id_123456) does not have a domain
-suffix, i think postfix will use the 'default transport'. if we want it to use
-the virtual transport instead, we should append the domain (eg
-id_123456@example.org). see
+note: if the result of the alias map (e.g. 123456) does not have a domain
+suffix, postfix will use the 'default transport'. if we want it to use the
+virtual transport instead, we should append the domain (eg
+123456@example.org). see
http://www.postfix.org/ADDRESS_REWRITING_README.html#resolve
-### Current status: ###
+### Current status
+
The current implementation of alias_resolver is in
leap-mx/src/leap/mx/alias_resolver.py.
-The class ```alias_resolver.StatusCodes``` deals with creating SMTP-like
-response messages for Postfix, speaking Postfix's TCP Map protocol (from item
-#1).
-
As for Discussion item #1:
It might be possible to use
@@ -165,37 +162,8 @@ handling Memcached servers, this is in ```twisted.protocols.memcache```. This
should be prioritised for later, if it is decided that querying the CouchDB is
too expensive or time-consuming.
-Thus far, to speed up alias lookup, an in-memory mapping of alias<->resolution
-pairs is created by ```alias_resolver.AliasResolverFactory()```, which can be
-optionally seeded with a dictionary of ```{ 'alias': 'resolution' }``` pairs
-by doing:
-~~~~~~
->>> from leap.mx import alias_resolver
->>> aliasResolverFactory = alias_resolver.AliasResolverFactory(
-... addr='1.2.3.4', port=4242, data={'isis': 'isis@leap.se',
-... 'drebs': 'drebs@leap.se'})
->>> aliasResolver = aliasResolverFactory.buildProtocol()
->>> aliasResolver.check_recipient_access('isis')
-200 OK Others might say 'HELLA AWESOME'...but we're not convinced.
-~~~~~~
-
-TODO:
- 1. The AliasResolverFactory needs to be connected to the CouchDB. The
- classmethod in which this should occur is ```AliasResolverFactory.get()```.
-
- 2. I am not sure where to get the user's UUID from (Soledad?). Wherever we get
- it from, it will need to be returned in
- ```AliasResolver.virtual_alias_map()```, and if we want Postfix to hear about
- it, then that response will need to be fed into ```AliasResolver.sendCode```.
-
- 3. Other than those two things, I think everything is done. The only potential
- other thing I can think of is that the codes in
- ```alias_resolver.StatusCodes``` might need to be urlencoded for Postfix to
- accept them, but this is like two lines of code from urllib.
-
-
-## mail_receiver ##
+## mail_receiver
the mail_receiver is a daemon that runs on incoming MX servers and is
responsible for encrypting incoming email to the user's public key and saving
@@ -203,36 +171,36 @@ the email to an incoming queue database for that user.
communicates with:
- * message spool directory:: mail_reciever sits and waits for new email to be
+ * message spool directory: mail_reciever sits and waits for new email to be
written to the spool directory (maybe using this
https://github.com/seb-m/pyinotify, i think it is better than FAM). when a
new file is dumped into the spool, mail_receiver reads the file, encrypts
the entire thing using the public key of the recipient, and saves to
couchdb.
- * couchdb get:: mail_receiver does a query on user id to get back user's
+ * couchdb get: mail_receiver does a query on user id to get back user's
public openpgp key. read quorum of 1 is probably ok.
- * couchdb put:: mail_receiver communicates with couchdb for storing encrypted
+ * couchdb put: mail_receiver communicates with couchdb for storing encrypted
email for each user (eventually, mail_receiver will communicate with a local
http proxy, that communicates with a bigcouch cluster, but the api is
identical)
-discussion:
- * i am not sure if postfix adds a header to indicate to whom a message was
- actually delivered. if not, this is a problem, because then how do we know
- what db to put it in or what public key to use? this is perhaps a good
- reason to not let postfix handle writing the message to disk, but instead
- pipe it to another command (because postfix sets env variables for stuff
- like recipient).
+### Current Status
- * should the incoming message queue be a separate database or should it be
- just documents in the user's main database with special flags?
+ * Postfix adds a "Delivered-To" header to indicate to whom a message was
+ actually delivered. There may be more than one header of this kind, and we
+ should use the topmost one. That value of that header is "<uuid>@<domain>",
+ and we can extract the user's uid from there to fetch the pgp public key to
+ which encrypt the message.
+
+ * currently, the incoming message is put into the user's database with some
+ special flags. should we have a different way to deal with that?
* whenever possible, we should refer to the user by a fixed id, not their
username, because we want to support the ability to change usernames. so,
- for example, database names should not be based on usernames.
-
-### Current Status: ###
-None of this is done, although having it be a separate daemon sound weird.
+ for example, database names should not be based on usernames. The alias
+ resolver then resolves to the user's uuid and from that point on we can
+ deal only with the user's uid.
-You would probably want to use ```twisted.mail.mail.FileMonitoringService``` to
-watch the mailbox (is the mailbox virtual or a maildir or mbox or?)
+ * We currently use ```twisted.internet.inotify.INotiry()``` to watch the
+ maildir for creation of new files. Alternatelly, we could possibly use
+ ```twisted.mail.mail.FileMonitoringService```.
diff --git a/doc/NOTES.md b/doc/NOTES.md
index a53f49d..337aa96 100644
--- a/doc/NOTES.md
+++ b/doc/NOTES.md
@@ -28,32 +28,3 @@ a viable protocol for this, and how would it interact with CouchDB?
4. What lib should we use for Python + Twisted + GPG/PGP ?
4.a. It looks like most people are using python-gnupg...
-
-
-## Tickets ##
--------------
-
-'''To be created:'''
-
-ticket for feature-alias_resolver_couchdb_support:
-
- o The alias resolver needs to speak to a couchdb/bigcouch
- instance(s). Currently, it merely creates an in-memory dictionary
- mapping. It seems like paisley is the best library for this.
-
-ticket for feature-check_recipient:
-
- o Need various errors for anything that could go wrong, e.g. the recipient
- address is malformed, sender doesn't have permissions to send to such
- address, etc.
- o These errcodes need to follow the SMTP server transport code spec.
-
-ticket for feature-virtual_alias_map:
-
- o Get the recipient's userid from couchdb.
-
-ticket for feature-evaluate_python_gnupg:
-
- o Briefly audit library in order to assess if it has the necessary
- features, as well as its general code quality.
-
diff --git a/src/leap/mx/alias_resolver.py b/src/leap/mx/alias_resolver.py
index 752eac4..c6f2acc 100644
--- a/src/leap/mx/alias_resolver.py
+++ b/src/leap/mx/alias_resolver.py
@@ -19,6 +19,12 @@
"""
Classes for resolving postfix aliases.
+The resolver is queried by the mail server before delivery to the mail spool
+directory, and should return the user uuid. This way, we get rid from the user
+address early and the mail server will delivery the message to
+"<uuid>@<domain>". Later, the mail receiver part of MX will parse the
+"Delivered-To" header to extract the uuid and fetch the user's pgp public key.
+
Test this with postmap -v -q "foo" tcp:localhost:4242
TODO:
diff --git a/src/leap/mx/check_recipient_access.py b/src/leap/mx/check_recipient_access.py
index 9f79dfe..55460a6 100644
--- a/src/leap/mx/check_recipient_access.py
+++ b/src/leap/mx/check_recipient_access.py
@@ -17,7 +17,12 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
-Classes for resolving postfix recipient access
+Classes for resolving postfix recipient access.
+
+The resolver is queried by the mail server before delivery to the mail spool
+directory, and should check if the address is able to receive messages.
+Examples of reasons for denying delivery would be that the user is out of
+quota, is user, or have no pgp public key in the server.
Test this with postmap -v -q "foo" tcp:localhost:2244
"""
@@ -44,6 +49,9 @@ class LEAPPostFixTCPMapAccessServer(postfix.PostfixTCPMapServer):
Return a code and message depending on the result of the factory's
get().
+ If there's no pgp public key for the user, we currently return a
+ temporary failure saying that the user account is disabled.
+
For more info, see: http://www.postfix.org/access.5.html
:param value: The uuid and public key.
diff --git a/src/leap/mx/tcp_map.py b/src/leap/mx/tcp_map.py
index 108c2aa..597c830 100644
--- a/src/leap/mx/tcp_map.py
+++ b/src/leap/mx/tcp_map.py
@@ -30,6 +30,10 @@ TCP_MAP_CODE_TEMPORARY_FAILURE = 400
TCP_MAP_CODE_PERMANENT_FAILURE = 500
+# we have to also extend from object here to make the class a new-style class.
+# If we don't, we get a TypeError because "new-style classes can't have only
+# classic bases". This has to do with the way abc.ABCMeta works and the old
+# and new style of python classes.
class LEAPPostfixTCPMapServerFactory(ServerFactory, object):
"""
A factory for postfix tcp map servers.