diff options
| -rw-r--r-- | README.md | 87 | ||||
| -rw-r--r-- | doc/DESIGN.md | 186 | ||||
| -rw-r--r-- | doc/NOTES.md | 29 | ||||
| -rw-r--r-- | src/leap/mx/alias_resolver.py | 6 | ||||
| -rw-r--r-- | src/leap/mx/check_recipient_access.py | 10 | ||||
| -rw-r--r-- | src/leap/mx/tcp_map.py | 4 |
6 files changed, 120 insertions, 202 deletions
@@ -1,82 +1,43 @@ -leap_mx +Leap MX ======= -**Note:** Currently in development. Feel free to test, and please [report - bugs on our tracker](https://we.riseup.net/leap/mx) or [by email](mailto:isis@leap.se). + +**Note:** Currently in development. Feel free to test, and please [report bugs +on our tracker](https://we.riseup.net/leap/mx) or [by +email](mailto:discuss@leap.se). An asynchronous, transparently-encrypting remailer for the LEAP platform, using BigCouch/CouchDB and PGP/GnuPG, written in Twisted Python. -## [install](#install) ## - -### [virtualenv](#virtualenv) ### -================================= -Impatient? Don't like virtualenvs? [tl;dr](#tl;dr) - -Virtualenv is somewhat equivalent to fakeroot for python packages, and -- due -to being packaged with copies of pip and python -- can be used to bootstrap -its own install process, allowing pip and python to be used with sudo. - -#### installing without sudo #### +## Installing -To install without using sudo, a bootstrap script to handle the setup process -is provided. It does the following: - - 1. Download, over SSL, the latest tarballs for virtualenv and - virtualenvwrapper from pypi. - 2. Unpack the tarballs, use the system python interpreter to call the - virtualenv.py script to setup a bootstrap virtual environment. - 3. Use the pip installed in the bootstrap virtualenv to install - virtualenvwrapper in the bootstrap virtualenv. - 4. Obtain a copy of leap_mx with git clone. - 5. Use ```mkvirtualenv``` included in the virtualenvwrapper inside the - bootstrap virtualenv to install a project virtualenv for leap_mx. - -To use the bootstrap script, do: -~~~ -$ wget -O bootstrap https://raw.github.com/isislovecruft/leap_mx/fix/no-suid-for-virtualenv/bootstrap -$ ./bootstrap -$ workon leap_mx -~~~ + * Leap MX is available as a debian package in [Leap + repository](http://deb.leap.se/repository/). + * A python package is available in + [pypi](https://pypi.python.org/pypi/leap.mx). Use ./pkg/requirements.pip + to install requirements. + * Source code is available in [github](https://github.com/leapcode/leap_mx). -#### installing in a regular virtualenv ### -To install python, virtualenv, and get started, do: +## Configuring -~~~ -$ sudo apt-get install python2.7 python-dev python-virtualenv virtualenvwrapper -$ git clone https://github.com/leapcode/leap_mx.git leap_mx -$ export WORKON_LEAPMX=${PWD}/leap_mx -$ source /usr/local/bin/virtualenvwrapper.sh -$ mkvirtualenv -a $WORKON_LEAPMX -r ${WORKON_LEAPMX}/pkg/requirements.pip \ - --no-site-packages --setuptools --unzip-setuptools leap_mx -~~~ - -### [tl;dr](#tl;dr) ### -To get started quickly, without virtualenv, do: -~~~ -$ sudo apt-get install python git -$ git clone https://github.com/leapcode/leap_mx.git -# pip install -r ./leap_mx/pkg/requirements.pip -~~~ -Although, **it is advised** to install inside a python virtualenv. - -## [configuration](#configuration) ## A sample config file can be found in pkg/mx.conf.sample -## [running](#running) ## -========================= +## Running -To get running, clone this repo, and (assuming you've already set up your -virtualenv and obtained all the requirements) do: +The debian package contains an initscript for the service. If you want to run +from source or from the python package, maybe setup a virtualenv and do: ~~~ -$ twistd -ny mx.tac +# git clone or unzip the python package, change into the dir, and do: +$ python setup.py install +# copy ./pkg/mx.conf.sample to /etc/leap/mx.conf and edit that file, then run: +$ twistd -ny pkg/mx.tac ~~~ -## [hacking](#hacking) ## -========================= -Please see the HACKING and DESIGN docs. +## Hacking + +Please see the doc/DESIGN docs. -Our bugtracker is [here](https://leap.se/code/projects/eip/issue/new). +Our bugtracker is [here](https://leap.se/code/projects/mx). Please use that for bug reports and feature requests instead of github's tracker. We're using github for code commenting and review between diff --git a/doc/DESIGN.md b/doc/DESIGN.md index 2d9fe82..e98976d 100644 --- a/doc/DESIGN.md +++ b/doc/DESIGN.md @@ -1,7 +1,7 @@ -# design # +# design + +## overview -## overview # ----------------------- This page pertains to the incoming mail exchange servers of the provider. General overview of how incoming email will work: @@ -27,39 +27,36 @@ General overview of how incoming email will work: 9. Soledad, in the background, will then re-encrypt this email (now a soledad document), and sync to the cloud. -## postfix pipeline ## ---------------------------- +## postfix pipeline + incoming mx servers will run postfix, configured in a particular way: 1. postscreen: before accepting an incoming message, checks RBLs, checks RFC validity, checks for spam pipelining. - (pass) proceed to next step. - (fail) return SMTP error, which bounces email. + * (pass) proceed to next step. + * (fail) return SMTP error, which bounces email. 2. more SMTP checks: valid hostnames, etc. - (pass) accepted, proceed to next step. - (fail) return SMTP error, which bounces email. + * (pass) accepted, proceed to next step. + * (fail) return SMTP error, which bounces email. 3. check_recipient_access -- look up each recipient and ensure they are allowed to receive messages. - (pass) empty result, proceed to next step. - (fail) return SMTP error code and error comment, bounce message. + * (pass) empty result, proceed to next step. + * (fail) return SMTP error code and error comment, bounce message. 4. milter processessing (spamassassin & clamav) - (pass) continue - (fail) bounce message, flag as spam, or silently kill. + * (pass) continue + * (fail) bounce message, flag as spam, or silently kill. 5. virtual_alias_maps -- map user defined aliases and forwards - (local address) continue if new address is for this mx - (remote address) continue. normally, postfix would relay to the remote domain, but we don't want that. + * (local address) continue if new address is for this mx + * (remote address) continue. normally, postfix would relay to the remote domain, but we don't want that. 6. deliver message to spool - (write) save the message to disk on the mx. + * (write) save the message to disk on the mx. 7. postfix's job is done, mail_receiver picks up email from spool directory Questions: - * what is the best way to have postfix write a message to a spool directory? - There is a built-in facility for saving to a maildir, so we could just - specify a common maildir for everyone. alternately, we could pipe to a - simple command that was responsible for safely saving the file to disk. a - third possibility would be to have a local long running daemon that spoke - lmtp that postfix forward the message on to for delivery. + * postfix uses a built-in facility and saves all messages to a common + maildir. as an alternative, we could have a local long running daemon that + spoke lmtp that postfix forward the message on to for delivery. * if virtual_alias_maps comes after check_recipient_access, then a user with aliases set but who is over quota will not be able to forward email. i think this is fine. @@ -80,41 +77,42 @@ Considerations: somewhere, and that copy should not be deleted until there is confirmation that the next stage has succeeded. -## alias_resolver ## ------------------------------- -The alias_resolver will be a daemon running on MX servers that handles lookups -in the user database of email aliases, forwards, quota, and account status. +## TCP Maps + +MX runs TCP maps that handle lookups in the user database of email aliases, +forwards, quota, and account status. Communication with: - 1. postfix:: alias_resolver will be bound to localhost and speak postfix's - very simple [tcp map protocol -> http://www.postfix.org/tcp_table.5.html]. + 1. postfix: bind to localhost and speak postfix's very simple [tcp map + protocol -> http://www.postfix.org/tcp_table.5.html]. - 2. couchdb:: alias_resolver will make couchdb queries to a local http load - balancer that connects to a couchdb/bigcouch - cluster. [directly accessing the couch->https://we.riseup.net/leap+platform/querying-the-couchdb] - might help getting started. + 2. couchdb: make couchdb queries to a local http load balancer that connects + to a couchdb/bigcouch cluster. -### Discussion: ### +### Discussion 1. we want the lookups to be fast. using views in couchdb, these should be - very fast. when using bigcouch, we can make it faster by specifying a read - quorum of 1 (instead of the default 2). this will make it so that only a - single couchdb needs to be queried to find the result. i don't know if this - would cause problems, but aliases don't change very often. + very fast. when using bigcouch, we can make it faster by specifying a read + quorum of 1 (instead of the default 2). this will make it so that only a + single couchdb needs to be queried to find the result. i don't know if + this would cause problems, but aliases don't change very often. + +TCP map is responsible for two map lookups in postfix: ```check_recipient``` +and ```virtual_alias_map```. -alias_resolver will be responsible for two map lookups in postfix: +#### check_recipient -#### check_recipient #### -------------------------- postfix config: -@check_recipient_access tcp:localhost:1000@ +``` +check_recipient_access tcp:localhost:2244 +``` -postfix will send "get username@domain.org" and alias_resolver should return an -empty result ("200 \n", i think) if postfix should deliver email to the -user. otherwise, it should return an error. here is example response, verbatim, -that can be used to bounce over quota users: +postfix sends "get username@domain.org" and alias_resolver returns an empty +result ("200 \n", i think) if postfix should deliver email to the user. +otherwise, it returns an error. here is example response, verbatim, that +can be used to bounce over quota users: ``` 200 DEFER_IF_PERMIT Sorry, your message cannot be delivered because the @@ -126,34 +124,33 @@ to tell them of this problem. that they should try again soon. Typically, an MX will try repeatedly, at longer and longer intervals, for four days before giving up. -#### virtual alias map #### ---------------------------- +#### virtual_alias_map + postfix config: -@virtual_alias_map tcp:localhost:1001@ +``` +virtual_alias_map tcp:localhost:4242 +``` -postfix will send "get alias-address@domain.org" and alias_resolver should -return "200 id_123456\n", where 123456 is the unique id of the user that has +postfix sends "get alias-address@domain.org" and alias_resolver returns "200 +123456\n", where 123456 is the unique id of the user that has alias-address@domain.org. -couchdb should have a view that will let us query on an (alias) address and -return the user id. +couchdb has a view that lets us query on an (alias) address and return the +user id. -note: if the result of the alias map (e.g. id_123456) does not have a domain -suffix, i think postfix will use the 'default transport'. if we want it to use -the virtual transport instead, we should append the domain (eg -id_123456@example.org). see +note: if the result of the alias map (e.g. 123456) does not have a domain +suffix, postfix will use the 'default transport'. if we want it to use the +virtual transport instead, we should append the domain (eg +123456@example.org). see http://www.postfix.org/ADDRESS_REWRITING_README.html#resolve -### Current status: ### +### Current status + The current implementation of alias_resolver is in leap-mx/src/leap/mx/alias_resolver.py. -The class ```alias_resolver.StatusCodes``` deals with creating SMTP-like -response messages for Postfix, speaking Postfix's TCP Map protocol (from item -#1). - As for Discussion item #1: It might be possible to use @@ -165,37 +162,8 @@ handling Memcached servers, this is in ```twisted.protocols.memcache```. This should be prioritised for later, if it is decided that querying the CouchDB is too expensive or time-consuming. -Thus far, to speed up alias lookup, an in-memory mapping of alias<->resolution -pairs is created by ```alias_resolver.AliasResolverFactory()```, which can be -optionally seeded with a dictionary of ```{ 'alias': 'resolution' }``` pairs -by doing: -~~~~~~ ->>> from leap.mx import alias_resolver ->>> aliasResolverFactory = alias_resolver.AliasResolverFactory( -... addr='1.2.3.4', port=4242, data={'isis': 'isis@leap.se', -... 'drebs': 'drebs@leap.se'}) ->>> aliasResolver = aliasResolverFactory.buildProtocol() ->>> aliasResolver.check_recipient_access('isis') -200 OK Others might say 'HELLA AWESOME'...but we're not convinced. -~~~~~~ - -TODO: - 1. The AliasResolverFactory needs to be connected to the CouchDB. The - classmethod in which this should occur is ```AliasResolverFactory.get()```. - - 2. I am not sure where to get the user's UUID from (Soledad?). Wherever we get - it from, it will need to be returned in - ```AliasResolver.virtual_alias_map()```, and if we want Postfix to hear about - it, then that response will need to be fed into ```AliasResolver.sendCode```. - - 3. Other than those two things, I think everything is done. The only potential - other thing I can think of is that the codes in - ```alias_resolver.StatusCodes``` might need to be urlencoded for Postfix to - accept them, but this is like two lines of code from urllib. - - -## mail_receiver ## +## mail_receiver the mail_receiver is a daemon that runs on incoming MX servers and is responsible for encrypting incoming email to the user's public key and saving @@ -203,36 +171,36 @@ the email to an incoming queue database for that user. communicates with: - * message spool directory:: mail_reciever sits and waits for new email to be + * message spool directory: mail_reciever sits and waits for new email to be written to the spool directory (maybe using this https://github.com/seb-m/pyinotify, i think it is better than FAM). when a new file is dumped into the spool, mail_receiver reads the file, encrypts the entire thing using the public key of the recipient, and saves to couchdb. - * couchdb get:: mail_receiver does a query on user id to get back user's + * couchdb get: mail_receiver does a query on user id to get back user's public openpgp key. read quorum of 1 is probably ok. - * couchdb put:: mail_receiver communicates with couchdb for storing encrypted + * couchdb put: mail_receiver communicates with couchdb for storing encrypted email for each user (eventually, mail_receiver will communicate with a local http proxy, that communicates with a bigcouch cluster, but the api is identical) -discussion: - * i am not sure if postfix adds a header to indicate to whom a message was - actually delivered. if not, this is a problem, because then how do we know - what db to put it in or what public key to use? this is perhaps a good - reason to not let postfix handle writing the message to disk, but instead - pipe it to another command (because postfix sets env variables for stuff - like recipient). +### Current Status - * should the incoming message queue be a separate database or should it be - just documents in the user's main database with special flags? + * Postfix adds a "Delivered-To" header to indicate to whom a message was + actually delivered. There may be more than one header of this kind, and we + should use the topmost one. That value of that header is "<uuid>@<domain>", + and we can extract the user's uid from there to fetch the pgp public key to + which encrypt the message. + + * currently, the incoming message is put into the user's database with some + special flags. should we have a different way to deal with that? * whenever possible, we should refer to the user by a fixed id, not their username, because we want to support the ability to change usernames. so, - for example, database names should not be based on usernames. - -### Current Status: ### -None of this is done, although having it be a separate daemon sound weird. + for example, database names should not be based on usernames. The alias + resolver then resolves to the user's uuid and from that point on we can + deal only with the user's uid. -You would probably want to use ```twisted.mail.mail.FileMonitoringService``` to -watch the mailbox (is the mailbox virtual or a maildir or mbox or?) + * We currently use ```twisted.internet.inotify.INotiry()``` to watch the + maildir for creation of new files. Alternatelly, we could possibly use + ```twisted.mail.mail.FileMonitoringService```. diff --git a/doc/NOTES.md b/doc/NOTES.md index a53f49d..337aa96 100644 --- a/doc/NOTES.md +++ b/doc/NOTES.md @@ -28,32 +28,3 @@ a viable protocol for this, and how would it interact with CouchDB? 4. What lib should we use for Python + Twisted + GPG/PGP ? 4.a. It looks like most people are using python-gnupg... - - -## Tickets ## -------------- - -'''To be created:''' - -ticket for feature-alias_resolver_couchdb_support: - - o The alias resolver needs to speak to a couchdb/bigcouch - instance(s). Currently, it merely creates an in-memory dictionary - mapping. It seems like paisley is the best library for this. - -ticket for feature-check_recipient: - - o Need various errors for anything that could go wrong, e.g. the recipient - address is malformed, sender doesn't have permissions to send to such - address, etc. - o These errcodes need to follow the SMTP server transport code spec. - -ticket for feature-virtual_alias_map: - - o Get the recipient's userid from couchdb. - -ticket for feature-evaluate_python_gnupg: - - o Briefly audit library in order to assess if it has the necessary - features, as well as its general code quality. - diff --git a/src/leap/mx/alias_resolver.py b/src/leap/mx/alias_resolver.py index 752eac4..c6f2acc 100644 --- a/src/leap/mx/alias_resolver.py +++ b/src/leap/mx/alias_resolver.py @@ -19,6 +19,12 @@ """ Classes for resolving postfix aliases. +The resolver is queried by the mail server before delivery to the mail spool +directory, and should return the user uuid. This way, we get rid from the user +address early and the mail server will delivery the message to +"<uuid>@<domain>". Later, the mail receiver part of MX will parse the +"Delivered-To" header to extract the uuid and fetch the user's pgp public key. + Test this with postmap -v -q "foo" tcp:localhost:4242 TODO: diff --git a/src/leap/mx/check_recipient_access.py b/src/leap/mx/check_recipient_access.py index 9f79dfe..55460a6 100644 --- a/src/leap/mx/check_recipient_access.py +++ b/src/leap/mx/check_recipient_access.py @@ -17,7 +17,12 @@ # along with this program. If not, see <http://www.gnu.org/licenses/>. """ -Classes for resolving postfix recipient access +Classes for resolving postfix recipient access. + +The resolver is queried by the mail server before delivery to the mail spool +directory, and should check if the address is able to receive messages. +Examples of reasons for denying delivery would be that the user is out of +quota, is user, or have no pgp public key in the server. Test this with postmap -v -q "foo" tcp:localhost:2244 """ @@ -44,6 +49,9 @@ class LEAPPostFixTCPMapAccessServer(postfix.PostfixTCPMapServer): Return a code and message depending on the result of the factory's get(). + If there's no pgp public key for the user, we currently return a + temporary failure saying that the user account is disabled. + For more info, see: http://www.postfix.org/access.5.html :param value: The uuid and public key. diff --git a/src/leap/mx/tcp_map.py b/src/leap/mx/tcp_map.py index 108c2aa..597c830 100644 --- a/src/leap/mx/tcp_map.py +++ b/src/leap/mx/tcp_map.py @@ -30,6 +30,10 @@ TCP_MAP_CODE_TEMPORARY_FAILURE = 400 TCP_MAP_CODE_PERMANENT_FAILURE = 500 +# we have to also extend from object here to make the class a new-style class. +# If we don't, we get a TypeError because "new-style classes can't have only +# classic bases". This has to do with the way abc.ABCMeta works and the old +# and new style of python classes. class LEAPPostfixTCPMapServerFactory(ServerFactory, object): """ A factory for postfix tcp map servers. |