summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authordrebs <drebs@leap.se>2015-04-21 17:02:47 -0300
committerdrebs <drebs@leap.se>2015-04-21 17:02:47 -0300
commitad8c61d00fb7d2474d129a5bb553dd0f5206f279 (patch)
treef6b6ee915ebe28b61655bbbd2c8ed57e1bc72020
parent5d0fffda31e0f07f7f8396fda961d3d0fac33733 (diff)
parent799703cf884191d097eb5d5316fa964e421683fd (diff)
Merge tag '0.6.2' into debian/release-0.6.2
Tag leap.mx version 0.6.2 Conflicts: CHANGELOG src/leap/mx/mail_receiver.py
-rw-r--r--CHANGELOG7
-rw-r--r--README.md93
-rwxr-xr-xdeb_release.sh2
-rw-r--r--debian/changelog6
-rw-r--r--debian/copyright12
-rw-r--r--doc/DESIGN.md186
-rw-r--r--doc/NOTES.md29
-rw-r--r--doc/leap-commit-template7
-rw-r--r--doc/leap-commit-template.README47
-rw-r--r--pkg/leap_mx.tac4
-rw-r--r--src/leap/mx/alias_resolver.py91
-rw-r--r--src/leap/mx/bounce.py526
-rw-r--r--src/leap/mx/check_recipient_access.py68
-rw-r--r--src/leap/mx/couchdbhelper.py148
-rw-r--r--src/leap/mx/mail_receiver.py151
-rw-r--r--src/leap/mx/tcp_map.py72
16 files changed, 949 insertions, 500 deletions
diff --git a/CHANGELOG b/CHANGELOG
index 6f58b5e..c2f3662 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,3 +1,10 @@
+0.6.2 Apr 21, 2015:
+ o Add PGP key lookup on access check server and reject mail if no PGP key
+ was found for the user. Closes #6795.
+ o Fix bounce message recipient. Closes #6854.
+ o Implement bouncing as per RFCs 6522, 3834 and 3464. Closes #6858.
+ o Correctly return async bouncer deferred.
+
0.6.1 Feb 11, 2015:
o Process unprocessed mail when MX starts (closes #2591).
o Log to syslog. Closes: #6307
diff --git a/README.md b/README.md
index 03b0ade..d0b5018 100644
--- a/README.md
+++ b/README.md
@@ -1,83 +1,56 @@
-leap_mx
+Leap MX
=======
-**Note:** Currently in development. Feel free to test, and please [report
- bugs on our tracker](https://we.riseup.net/leap/mx) or [by email](mailto:isis@leap.se).
+
+**Note:** Currently in development. Feel free to test, and please [report bugs
+on our tracker](https://we.riseup.net/leap/mx) or [by
+email](mailto:discuss@leap.se).
An asynchronous, transparently-encrypting remailer for the LEAP platform,
using BigCouch/CouchDB and PGP/GnuPG, written in Twisted Python.
-## [install](#install) ##
+## Installing
-### [virtualenv](#virtualenv) ###
-=================================
-Impatient? Don't like virtualenvs? [tl;dr](#tl;dr)
+ * Leap MX is available as a debian package in [Leap
+ repository](http://deb.leap.se/repository/).
+ * A python package is available in
+ [pypi](https://pypi.python.org/pypi/leap.mx). Use ./pkg/requirements.pip
+ to install requirements.
+ * Source code is available in [github](https://github.com/leapcode/leap_mx).
-Virtualenv is somewhat equivalent to fakeroot for python packages, and -- due
-to being packaged with copies of pip and python -- can be used to bootstrap
-its own install process, allowing pip and python to be used with sudo.
+## Configuring
-#### installing without sudo ####
+A sample config file can be found in pkg/mx.conf.sample
-To install without using sudo, a bootstrap script to handle the setup process
-is provided. It does the following:
+## Running
- 1. Download, over SSL, the latest tarballs for virtualenv and
- virtualenvwrapper from pypi.
- 2. Unpack the tarballs, use the system python interpreter to call the
- virtualenv.py script to setup a bootstrap virtual environment.
- 3. Use the pip installed in the bootstrap virtualenv to install
- virtualenvwrapper in the bootstrap virtualenv.
- 4. Obtain a copy of leap_mx with git clone.
- 5. Use ```mkvirtualenv``` included in the virtualenvwrapper inside the
- bootstrap virtualenv to install a project virtualenv for leap_mx.
+The debian package contains an initscript for the service. If you want to run
+from source or from the python package, maybe setup a virtualenv and do:
-To use the bootstrap script, do:
~~~
-$ wget -O bootstrap https://raw.github.com/isislovecruft/leap_mx/fix/no-suid-for-virtualenv/bootstrap
-$ ./bootstrap
-$ workon leap_mx
+# git clone or unzip the python package, change into the dir, and do:
+$ python setup.py install
+# copy ./pkg/mx.conf.sample to /etc/leap/mx.conf and edit that file, then run:
+$ twistd -ny pkg/mx.tac
~~~
-#### installing in a regular virtualenv ###
-To install python, virtualenv, and get started, do:
+## Hacking
-~~~
-$ sudo apt-get install python2.7 python-dev python-virtualenv virtualenvwrapper
-$ git clone https://github.com/leapcode/leap_mx.git leap_mx
-$ export WORKON_LEAPMX=${PWD}/leap_mx
-$ source /usr/local/bin/virtualenvwrapper.sh
-$ mkvirtualenv -a $WORKON_LEAPMX -r ${WORKON_LEAPMX}/pkg/requirements.pip \
- --no-site-packages --setuptools --unzip-setuptools leap_mx
-~~~
+Please see the doc/DESIGN docs.
-### [tl;dr](#tl;dr) ###
-To get started quickly, without virtualenv, do:
-~~~
-$ sudo apt-get install python git
-$ git clone https://github.com/leapcode/leap_mx.git
-# pip install -r ./leap_mx/pkg/requirements.pip
-~~~
-Although, **it is advised** to install inside a python virtualenv.
+Our bugtracker is [here](https://leap.se/code/projects/mx).
-## [configuration](#configuration) ##
-A sample config file can be found in pkg/mx.conf.sample
+Please use that for bug reports and feature requests instead of github's
+tracker. We're using github for code commenting and review between
+collaborators.
-## [running](#running) ##
-=========================
+## Issues
-To get running, clone this repo, and (assuming you've already set up your
-virtualenv and obtained all the requirements) do:
+* see the [Changelog](./CHANGELOG) for details of all major changes in the different versions
-~~~
-$ twistd -ny mx.tac
-~~~
+### 0.6.1
-## [hacking](#hacking) ##
-=========================
-Please see the HACKING and DESIGN docs.
+* Bouncing messages can get into a bouncing loop (#6858)
-Our bugtracker is [here](https://leap.se/code/projects/eip/issue/new).
+### 0.6.0
-Please use that for bug reports and feature requests instead of github's
-tracker. We're using github for code commenting and review between
-collaborators.
+* leap-mx needs to get restarted after the first incoming mail is delivered (#6687)
diff --git a/deb_release.sh b/deb_release.sh
index 8c116ab..a38a405 100755
--- a/deb_release.sh
+++ b/deb_release.sh
@@ -5,4 +5,4 @@ rm ${VERSION_FILE}
python setup.py freeze_debianver
sed -i 's/-dirty//g' ${VERSION_FILE}
git add ${VERSION_FILE}
-git ci -m "freeze debian version"
+git ci -m "freeze debian version"
diff --git a/debian/changelog b/debian/changelog
index a2782a3..c473f3c 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -26,7 +26,7 @@ leap-mx (0.5.0) unstable; urgency=medium
leap-mx (0.3.5) unstable; urgency=low
- * Update to 0.3.5
+ * Update to 0.3.5
-- Ben Carrillo <ben@futeisha.org> Tue, 10 Dec 2013 16:02:10 -0400
@@ -39,7 +39,7 @@ leap-mx (0.3.4) unstable; urgency=low
leap-mx (0.3.3.2) unstable; urgency=low
* Cherry-pick e108a2ffec444c09b3661379a1051fda1f9952cf
-
+
-- Micah Anderson <micah@debian.org> Sat, 09 Nov 2013 14:32:08 -0500
leap-mx (0.3.3.1) unstable; urgency=low
@@ -64,7 +64,7 @@ leap-mx (0.3.2) unstable; urgency=low
leap-mx (0.3.1.5) unstable; urgency=low
- * Cherry pick fix from
+ * Cherry pick fix from
https://github.com/chiiph/leap_mx/commit/78f6ca775dc42eba69f2dc1e134ca360c0813aff
to keep file watcher in memory to prevent losing file events
diff --git a/debian/copyright b/debian/copyright
index 520cf16..0c2eab5 100644
--- a/debian/copyright
+++ b/debian/copyright
@@ -10,7 +10,7 @@ License: AGPL
Files: debian/*
Copyright: Copyright 2013 Micah Anderson <micah@leap.se>
License: GPL-3+
-
+
License: GPL-3+
On Debian systems, the complete text of the GNU General
Public License can be found in `/usr/share/common-licenses/GPL'.
@@ -368,7 +368,7 @@ License: AGPL
Additional terms, permissive or non-permissive, may be stated in the form of a
separately written license, or stated as exceptions; the above requirements
apply either way.
- .
+ .
8. Termination.
.
You may not propagate or modify a covered work except as expressly provided
@@ -492,7 +492,7 @@ License: AGPL
Nothing in this License shall be construed as excluding or limiting any implied
license or other defenses to infringement that may otherwise be available to
you under applicable patent law.
- .
+ .
12. No Surrender of Others' Freedom.
.
If conditions are imposed on you (whether by court order, agreement or
@@ -538,7 +538,7 @@ License: AGPL
published by the Free Software Foundation. If the Program does not specify a
version number of the GNU Affero General Public License, you may choose any
version ever published by the Free Software Foundation.
- .
+ .
If the Program specifies that a proxy can decide which future versions of the
GNU Affero General Public License can be used, that proxy's public statement of
acceptance of a version permanently authorizes you to choose that version for
@@ -549,7 +549,7 @@ License: AGPL
copyright holder as a result of your choosing to follow a later version.
.
15. Disclaimer of Warranty.
- .
+ .
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE
LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER
PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER
@@ -576,4 +576,4 @@ License: AGPL
be given local legal effect according to their terms, reviewing courts shall
apply local law that most closely approximates an absolute waiver of all civil
liability in connection with the Program, unless a warranty or assumption of
- liability accompanies a copy of the Program in return for a fee. \ No newline at end of file
+ liability accompanies a copy of the Program in return for a fee.
diff --git a/doc/DESIGN.md b/doc/DESIGN.md
index 2d9fe82..e98976d 100644
--- a/doc/DESIGN.md
+++ b/doc/DESIGN.md
@@ -1,7 +1,7 @@
-# design #
+# design
+
+## overview
-## overview #
-----------------------
This page pertains to the incoming mail exchange servers of the provider.
General overview of how incoming email will work:
@@ -27,39 +27,36 @@ General overview of how incoming email will work:
9. Soledad, in the background, will then re-encrypt this email (now a soledad
document), and sync to the cloud.
-## postfix pipeline ##
----------------------------
+## postfix pipeline
+
incoming mx servers will run postfix, configured in a particular way:
1. postscreen: before accepting an incoming message, checks RBLs, checks RFC
validity, checks for spam pipelining.
- (pass) proceed to next step.
- (fail) return SMTP error, which bounces email.
+ * (pass) proceed to next step.
+ * (fail) return SMTP error, which bounces email.
2. more SMTP checks: valid hostnames, etc.
- (pass) accepted, proceed to next step.
- (fail) return SMTP error, which bounces email.
+ * (pass) accepted, proceed to next step.
+ * (fail) return SMTP error, which bounces email.
3. check_recipient_access -- look up each recipient and ensure they are
allowed to receive messages.
- (pass) empty result, proceed to next step.
- (fail) return SMTP error code and error comment, bounce message.
+ * (pass) empty result, proceed to next step.
+ * (fail) return SMTP error code and error comment, bounce message.
4. milter processessing (spamassassin & clamav)
- (pass) continue
- (fail) bounce message, flag as spam, or silently kill.
+ * (pass) continue
+ * (fail) bounce message, flag as spam, or silently kill.
5. virtual_alias_maps -- map user defined aliases and forwards
- (local address) continue if new address is for this mx
- (remote address) continue. normally, postfix would relay to the remote domain, but we don't want that.
+ * (local address) continue if new address is for this mx
+ * (remote address) continue. normally, postfix would relay to the remote domain, but we don't want that.
6. deliver message to spool
- (write) save the message to disk on the mx.
+ * (write) save the message to disk on the mx.
7. postfix's job is done, mail_receiver picks up email from spool directory
Questions:
- * what is the best way to have postfix write a message to a spool directory?
- There is a built-in facility for saving to a maildir, so we could just
- specify a common maildir for everyone. alternately, we could pipe to a
- simple command that was responsible for safely saving the file to disk. a
- third possibility would be to have a local long running daemon that spoke
- lmtp that postfix forward the message on to for delivery.
+ * postfix uses a built-in facility and saves all messages to a common
+ maildir. as an alternative, we could have a local long running daemon that
+ spoke lmtp that postfix forward the message on to for delivery.
* if virtual_alias_maps comes after check_recipient_access, then a user with
aliases set but who is over quota will not be able to forward email. i think
this is fine.
@@ -80,41 +77,42 @@ Considerations:
somewhere, and that copy should not be deleted until there is confirmation
that the next stage has succeeded.
-## alias_resolver ##
-------------------------------
-The alias_resolver will be a daemon running on MX servers that handles lookups
-in the user database of email aliases, forwards, quota, and account status.
+## TCP Maps
+
+MX runs TCP maps that handle lookups in the user database of email aliases,
+forwards, quota, and account status.
Communication with:
- 1. postfix:: alias_resolver will be bound to localhost and speak postfix's
- very simple [tcp map protocol -> http://www.postfix.org/tcp_table.5.html].
+ 1. postfix: bind to localhost and speak postfix's very simple [tcp map
+ protocol -> http://www.postfix.org/tcp_table.5.html].
- 2. couchdb:: alias_resolver will make couchdb queries to a local http load
- balancer that connects to a couchdb/bigcouch
- cluster. [directly accessing the couch->https://we.riseup.net/leap+platform/querying-the-couchdb]
- might help getting started.
+ 2. couchdb: make couchdb queries to a local http load balancer that connects
+ to a couchdb/bigcouch cluster.
-### Discussion: ###
+### Discussion
1. we want the lookups to be fast. using views in couchdb, these should be
- very fast. when using bigcouch, we can make it faster by specifying a read
- quorum of 1 (instead of the default 2). this will make it so that only a
- single couchdb needs to be queried to find the result. i don't know if this
- would cause problems, but aliases don't change very often.
+ very fast. when using bigcouch, we can make it faster by specifying a read
+ quorum of 1 (instead of the default 2). this will make it so that only a
+ single couchdb needs to be queried to find the result. i don't know if
+ this would cause problems, but aliases don't change very often.
+
+TCP map is responsible for two map lookups in postfix: ```check_recipient```
+and ```virtual_alias_map```.
-alias_resolver will be responsible for two map lookups in postfix:
+#### check_recipient
-#### check_recipient ####
--------------------------
postfix config:
-@check_recipient_access tcp:localhost:1000@
+```
+check_recipient_access tcp:localhost:2244
+```
-postfix will send "get username@domain.org" and alias_resolver should return an
-empty result ("200 \n", i think) if postfix should deliver email to the
-user. otherwise, it should return an error. here is example response, verbatim,
-that can be used to bounce over quota users:
+postfix sends "get username@domain.org" and alias_resolver returns an empty
+result ("200 \n", i think) if postfix should deliver email to the user.
+otherwise, it returns an error. here is example response, verbatim, that
+can be used to bounce over quota users:
```
200 DEFER_IF_PERMIT Sorry, your message cannot be delivered because the
@@ -126,34 +124,33 @@ to tell them of this problem.
that they should try again soon. Typically, an MX will try repeatedly, at
longer and longer intervals, for four days before giving up.
-#### virtual alias map ####
----------------------------
+#### virtual_alias_map
+
postfix config:
-@virtual_alias_map tcp:localhost:1001@
+```
+virtual_alias_map tcp:localhost:4242
+```
-postfix will send "get alias-address@domain.org" and alias_resolver should
-return "200 id_123456\n", where 123456 is the unique id of the user that has
+postfix sends "get alias-address@domain.org" and alias_resolver returns "200
+123456\n", where 123456 is the unique id of the user that has
alias-address@domain.org.
-couchdb should have a view that will let us query on an (alias) address and
-return the user id.
+couchdb has a view that lets us query on an (alias) address and return the
+user id.
-note: if the result of the alias map (e.g. id_123456) does not have a domain
-suffix, i think postfix will use the 'default transport'. if we want it to use
-the virtual transport instead, we should append the domain (eg
-id_123456@example.org). see
+note: if the result of the alias map (e.g. 123456) does not have a domain
+suffix, postfix will use the 'default transport'. if we want it to use the
+virtual transport instead, we should append the domain (eg
+123456@example.org). see
http://www.postfix.org/ADDRESS_REWRITING_README.html#resolve
-### Current status: ###
+### Current status
+
The current implementation of alias_resolver is in
leap-mx/src/leap/mx/alias_resolver.py.
-The class ```alias_resolver.StatusCodes``` deals with creating SMTP-like
-response messages for Postfix, speaking Postfix's TCP Map protocol (from item
-#1).
-
As for Discussion item #1:
It might be possible to use
@@ -165,37 +162,8 @@ handling Memcached servers, this is in ```twisted.protocols.memcache```. This
should be prioritised for later, if it is decided that querying the CouchDB is
too expensive or time-consuming.
-Thus far, to speed up alias lookup, an in-memory mapping of alias<->resolution
-pairs is created by ```alias_resolver.AliasResolverFactory()```, which can be
-optionally seeded with a dictionary of ```{ 'alias': 'resolution' }``` pairs
-by doing:
-~~~~~~
->>> from leap.mx import alias_resolver
->>> aliasResolverFactory = alias_resolver.AliasResolverFactory(
-... addr='1.2.3.4', port=4242, data={'isis': 'isis@leap.se',
-... 'drebs': 'drebs@leap.se'})
->>> aliasResolver = aliasResolverFactory.buildProtocol()
->>> aliasResolver.check_recipient_access('isis')
-200 OK Others might say 'HELLA AWESOME'...but we're not convinced.
-~~~~~~
-
-TODO:
- 1. The AliasResolverFactory needs to be connected to the CouchDB. The
- classmethod in which this should occur is ```AliasResolverFactory.get()```.
-
- 2. I am not sure where to get the user's UUID from (Soledad?). Wherever we get
- it from, it will need to be returned in
- ```AliasResolver.virtual_alias_map()```, and if we want Postfix to hear about
- it, then that response will need to be fed into ```AliasResolver.sendCode```.
-
- 3. Other than those two things, I think everything is done. The only potential
- other thing I can think of is that the codes in
- ```alias_resolver.StatusCodes``` might need to be urlencoded for Postfix to
- accept them, but this is like two lines of code from urllib.
-
-
-## mail_receiver ##
+## mail_receiver
the mail_receiver is a daemon that runs on incoming MX servers and is
responsible for encrypting incoming email to the user's public key and saving
@@ -203,36 +171,36 @@ the email to an incoming queue database for that user.
communicates with:
- * message spool directory:: mail_reciever sits and waits for new email to be
+ * message spool directory: mail_reciever sits and waits for new email to be
written to the spool directory (maybe using this
https://github.com/seb-m/pyinotify, i think it is better than FAM). when a
new file is dumped into the spool, mail_receiver reads the file, encrypts
the entire thing using the public key of the recipient, and saves to
couchdb.
- * couchdb get:: mail_receiver does a query on user id to get back user's
+ * couchdb get: mail_receiver does a query on user id to get back user's
public openpgp key. read quorum of 1 is probably ok.
- * couchdb put:: mail_receiver communicates with couchdb for storing encrypted
+ * couchdb put: mail_receiver communicates with couchdb for storing encrypted
email for each user (eventually, mail_receiver will communicate with a local
http proxy, that communicates with a bigcouch cluster, but the api is
identical)
-discussion:
- * i am not sure if postfix adds a header to indicate to whom a message was
- actually delivered. if not, this is a problem, because then how do we know
- what db to put it in or what public key to use? this is perhaps a good
- reason to not let postfix handle writing the message to disk, but instead
- pipe it to another command (because postfix sets env variables for stuff
- like recipient).
+### Current Status
- * should the incoming message queue be a separate database or should it be
- just documents in the user's main database with special flags?
+ * Postfix adds a "Delivered-To" header to indicate to whom a message was
+ actually delivered. There may be more than one header of this kind, and we
+ should use the topmost one. That value of that header is "<uuid>@<domain>",
+ and we can extract the user's uid from there to fetch the pgp public key to
+ which encrypt the message.
+
+ * currently, the incoming message is put into the user's database with some
+ special flags. should we have a different way to deal with that?
* whenever possible, we should refer to the user by a fixed id, not their
username, because we want to support the ability to change usernames. so,
- for example, database names should not be based on usernames.
-
-### Current Status: ###
-None of this is done, although having it be a separate daemon sound weird.
+ for example, database names should not be based on usernames. The alias
+ resolver then resolves to the user's uuid and from that point on we can
+ deal only with the user's uid.
-You would probably want to use ```twisted.mail.mail.FileMonitoringService``` to
-watch the mailbox (is the mailbox virtual or a maildir or mbox or?)
+ * We currently use ```twisted.internet.inotify.INotiry()``` to watch the
+ maildir for creation of new files. Alternatelly, we could possibly use
+ ```twisted.mail.mail.FileMonitoringService```.
diff --git a/doc/NOTES.md b/doc/NOTES.md
index a53f49d..337aa96 100644
--- a/doc/NOTES.md
+++ b/doc/NOTES.md
@@ -28,32 +28,3 @@ a viable protocol for this, and how would it interact with CouchDB?
4. What lib should we use for Python + Twisted + GPG/PGP ?
4.a. It looks like most people are using python-gnupg...
-
-
-## Tickets ##
--------------
-
-'''To be created:'''
-
-ticket for feature-alias_resolver_couchdb_support:
-
- o The alias resolver needs to speak to a couchdb/bigcouch
- instance(s). Currently, it merely creates an in-memory dictionary
- mapping. It seems like paisley is the best library for this.
-
-ticket for feature-check_recipient:
-
- o Need various errors for anything that could go wrong, e.g. the recipient
- address is malformed, sender doesn't have permissions to send to such
- address, etc.
- o These errcodes need to follow the SMTP server transport code spec.
-
-ticket for feature-virtual_alias_map:
-
- o Get the recipient's userid from couchdb.
-
-ticket for feature-evaluate_python_gnupg:
-
- o Briefly audit library in order to assess if it has the necessary
- features, as well as its general code quality.
-
diff --git a/doc/leap-commit-template b/doc/leap-commit-template
new file mode 100644
index 0000000..8a5c7cd
--- /dev/null
+++ b/doc/leap-commit-template
@@ -0,0 +1,7 @@
+[bug|feat|docs|style|refactor|test|pkg|i18n] ...
+...
+
+- Resolves: #XYZ
+- Related: #XYZ
+- Documentation: #XYZ
+- Releases: XYZ
diff --git a/doc/leap-commit-template.README b/doc/leap-commit-template.README
new file mode 100644
index 0000000..ce8809e
--- /dev/null
+++ b/doc/leap-commit-template.README
@@ -0,0 +1,47 @@
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+HOW TO USE THIS TEMPLATE:
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Run `git config commit.template docs/leap-commit-template` or
+edit the .git/config for this project and add
+`template = docs/leap-commit-template`
+under the [commit] block
+
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+COMMIT TEMPLATE FORMAT EXPLAINED
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+[type] <subject>
+
+<body>
+<footer>
+
+Type should be one of the following:
+- bug (bug fix)
+- feat (new feature)
+- docs (changes to documentation)
+- style (formatting, pep8 violations, etc; no code change)
+- refactor (refactoring production code)
+- test (adding missing tests, refactoring tests; no production code change)
+- pkg (packaging related changes; no production code change)
+- i18n translation related changes
+
+Subject should use imperative tone and say what you did.
+For example, use 'change', NOT 'changed' or 'changes'.
+
+The body should go into detail about changes made.
+
+The footer should contain any issue references or actions.
+You can use one or several of the following:
+
+- Resolves: #XYZ
+- Related: #XYZ
+- Documentation: #XYZ
+- Releases: XYZ
+
+The Documentation field should be included in every new feature commit, and it
+should link to an issue in the bug tracker where the new feature is analyzed
+and documented.
+
+For a full example of how to write a good commit message, check out
+https://github.com/sparkbox/how_to/tree/master/style/git
diff --git a/pkg/leap_mx.tac b/pkg/leap_mx.tac
index 75d2405..7da59cf 100644
--- a/pkg/leap_mx.tac
+++ b/pkg/leap_mx.tac
@@ -46,8 +46,8 @@ password = config.get("couchdb", "password")
server = config.get("couchdb", "server")
port = config.get("couchdb", "port")
-bounce_from = "bounce"
-bounce_subject = "Delivery failure"
+bounce_from = "Mail Delivery Subsystem <MAILER-DAEMON>"
+bounce_subject = "Undelivered Mail Returned to Sender"
try:
bounce_from = config.get("bounce", "from")
diff --git a/src/leap/mx/alias_resolver.py b/src/leap/mx/alias_resolver.py
index 45a3ed2..c6f2acc 100644
--- a/src/leap/mx/alias_resolver.py
+++ b/src/leap/mx/alias_resolver.py
@@ -19,73 +19,62 @@
"""
Classes for resolving postfix aliases.
+The resolver is queried by the mail server before delivery to the mail spool
+directory, and should return the user uuid. This way, we get rid from the user
+address early and the mail server will delivery the message to
+"<uuid>@<domain>". Later, the mail receiver part of MX will parse the
+"Delivered-To" header to extract the uuid and fetch the user's pgp public key.
+
Test this with postmap -v -q "foo" tcp:localhost:4242
TODO:
o Look into using twisted.protocols.postfix.policies classes for
controlling concurrent connections and throttling resource consumption.
+ o We should probably use twisted.mail.alias somehow.
"""
-try:
- # TODO: we should probably use the system alias somehow
- # from twisted.mail import alias
- from twisted.protocols import postfix
- from twisted.python import log
- from twisted.internet import defer
-except ImportError:
- print "This software requires Twisted. Please see the README file"
- print "for instructions on getting required dependencies."
-
-class LEAPPostFixTCPMapserver(postfix.PostfixTCPMapServer):
- def _cbGot(self, value):
- if value is None:
- self.sendCode(500, postfix.quote("NOT FOUND SRY"))
- else:
- self.sendCode(200, postfix.quote(value))
+from twisted.protocols import postfix
+from leap.mx.tcp_map import LEAPPostfixTCPMapServerFactory
+from leap.mx.tcp_map import TCP_MAP_CODE_SUCCESS
+from leap.mx.tcp_map import TCP_MAP_CODE_PERMANENT_FAILURE
-class AliasResolverFactory(postfix.PostfixTCPMapDeferringDictServerFactory):
- protocol = LEAPPostFixTCPMapserver
+class LEAPPostfixTCPMapAliasServer(postfix.PostfixTCPMapServer):
+ """
+ A postfix tcp map alias resolver server.
+ """
- def __init__(self, couchdb, *args, **kwargs):
- postfix.PostfixTCPMapDeferringDictServerFactory.__init__(
- self, *args, **kwargs)
- self._cdb = couchdb
-
- def _to_str(self, result):
- """
- Properly encodes the result string if any.
+ def _cbGot(self, user_data):
"""
- if isinstance(result, unicode):
- result = result.encode("utf8")
- if result is None:
- log.msg("Result not found")
- return result
+ Return a code and message depending on the result of the factory's
+ get().
- def spit_result(self, result):
+ :param user_data: The user's uuid and pgp public key.
+ :type user_data: list
"""
- Formats the return codes in a postfix friendly format.
- """
- if result is None:
- return None
+ uuid, _ = user_data
+ if uuid is None:
+ self.sendCode(
+ TCP_MAP_CODE_PERMANENT_FAILURE,
+ postfix.quote("NOT FOUND SRY"))
else:
- return defer.succeed(result)
+ # properly encode uuid, otherwise twisted complains when replying
+ if isinstance(uuid, unicode):
+ uuid = uuid.encode("utf8")
+ self.sendCode(
+ TCP_MAP_CODE_SUCCESS,
+ postfix.quote(uuid))
- def get(self, key):
- """
- Looks up the passed key, but only up to the username id of the key.
- At some point we will have to consider the domain part too.
- """
- try:
- log.msg("Query key: %s" % (key,))
- d = self._cdb.queryByAddress(key)
+class AliasResolverFactory(LEAPPostfixTCPMapServerFactory):
+ """
+ A factory for postfix tcp map alias resolver servers.
+ """
+
+ protocol = LEAPPostfixTCPMapAliasServer
- d.addCallback(self._to_str)
- d.addCallback(self.spit_result)
- d.addErrback(log.err)
- return d
- except Exception as e:
- log.err('exception in get: %r' % e)
+ @property
+ def _query_message(self):
+ return "Resolving alias for"
diff --git a/src/leap/mx/bounce.py b/src/leap/mx/bounce.py
new file mode 100644
index 0000000..2ece6df
--- /dev/null
+++ b/src/leap/mx/bounce.py
@@ -0,0 +1,526 @@
+#!/usr/bin/env python
+# -*- encoding: utf-8 -*-
+# bounce.py
+# Copyright (C) 2015 LEAP
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+
+"""
+Everything you need to correctly bounce a message!
+
+This is built from the following RFCs:
+
+ * The Multipart/Report Media Type for the Reporting of Mail System
+ Administrative Messages
+ https://tools.ietf.org/html/rfc6522
+
+ * Recommendations for Automatic Responses to Electronic Mail
+ https://tools.ietf.org/html/rfc3834
+
+ * An Extensible Message Format for Delivery Status Notifications
+ https://tools.ietf.org/html/rfc3464
+"""
+
+
+import re
+import socket
+
+from StringIO import StringIO
+from textwrap import wrap
+
+from email.errors import MessageError
+from email.message import Message
+from email.utils import formatdate
+from email.utils import parseaddr
+from email.mime.base import MIMEBase
+from email.mime.multipart import MIMEMultipart
+from email.mime.text import MIMEText
+from email.generator import Generator
+from email.generator import NL
+
+from twisted.internet import defer
+from twisted.internet import protocol
+from twisted.internet import reactor
+from twisted.internet.error import ProcessDone
+from twisted.python import log
+
+
+EMAIL_ADDRESS_REGEXP = re.compile("[^@]+@[^@]+\.[^@]+")
+HOSTNAME = socket.gethostbyaddr(socket.gethostname())[0]
+
+
+def _valid_address(address):
+ """
+ Return whether address is a valid email address.
+
+ :param address: An email address candidate.
+ :type address: str
+
+ :return: Whether address is valid.
+ :rtype: bool
+ """
+ return bool(EMAIL_ADDRESS_REGEXP.match(address))
+
+
+def bounce_message(bounce_from, bounce_subject, orig_msg, reason):
+ """
+ Bounce a message.
+
+ :param bounce_from: The sender of the bounce message.
+ :type bounce_from: str
+ :param bounce_subject: The subject of the bounce message.
+ :type bounce_subject: str
+ :param orig_msg: The original message that will be bounced.
+ :type orig_msg: email.message.Message
+ :param reason: The reason for bouncing the message.
+ :type reason: str
+
+ :return: A deferred that will fire with the output of the sendmail process
+ if it was successful or with a failure containing the reason for
+ the end of the process if it failed.
+ :rtype: Deferred
+ """
+ orig_rpath = orig_msg.get("Return-Path")
+
+ # do not bounce if sender address is invalid
+ _, addr = parseaddr(orig_rpath)
+ if not _valid_address(addr):
+ log.msg(
+ "Will not send a bounce message to an invalid address: %s"
+ % orig_rpath)
+ return
+
+ msg = _build_bounce_message(
+ bounce_from, bounce_subject, orig_msg, reason)
+ return _async_check_output(["/usr/sbin/sendmail", "-t"], msg.as_string())
+
+
+def _check_valid_return_path(return_path):
+ """
+ Check if a certain return path is valid.
+
+ From RFC 3834:
+
+ Responders MUST NOT generate any response for which the
+ destination of that response would be a null address (e.g., an
+ address for which SMTP MAIL FROM or Return-Path is <>), since the
+ response would not be delivered to a useful destination.
+ Responders MAY refuse to generate responses for addresses commonly
+ used as return addresses by responders - e.g., those with local-
+ parts matching "owner-*", "*-request", "MAILER-DAEMON", etc.
+ Responders are encouraged to check the destination address for
+ validity before generating the response, to avoid generating
+ responses that cannot be delivered or are unlikely to be useful.
+
+ :return: Whether the return_path is valid.
+ :rtype: bool
+ """
+ _, addr = parseaddr(return_path)
+
+ # check null address
+ if not addr:
+ return False
+
+ # check addresses commonly used as return addresses by responders
+ local, _ = addr.split("@", 1)
+ if local.startswith("owner-") \
+ or local.endswith("-request") \
+ or local.startswith("MAILER-DAEMON"):
+ return False
+
+ return True
+
+
+class DeliveryStatusNotificationMessage(MIMEBase):
+ """
+ A delivery status message, as per RFC 3464.
+ """
+
+ def __init__(self, orig_msg):
+ """
+ Initialize the DSN.
+ """
+ MIMEBase.__init__(self, "message", "delivery-status")
+ self.__delitem__("MIME-Version")
+ self._build_dsn(orig_msg)
+
+ def _build_dsn(self, orig_msg):
+ """
+ Build an RFC 3464 compliant delivery status message.
+
+ :param orig_msg: The original bouncing message.
+ :type orig_msg: email.message.Message
+ """
+ content = []
+
+ # Per-Message DSN fields
+ # ======================
+
+ # Original-Envelope-Id (optional)
+ envelope_id = orig_msg.get("Envelope-Id")
+ if envelope_id:
+ content.append("Original-Envelope-Id: %s" % envelope_id)
+
+ # Reporting-MTA (required)
+ content.append("Reporting-MTA: dns; %s" % HOSTNAME)
+
+ # XXX add Arrival-Date DSN field? (optional).
+
+ content.append("")
+
+ # Per-Recipient DSN fields
+ # ========================
+
+ # Original-Recipient (optional)
+ orig_to = orig_msg.get("X-Original-To") # added by postfix
+ _, orig_addr = parseaddr(orig_to)
+ if orig_addr:
+ content.append("Original-Recipient: rfc822; %s" % orig_addr)
+
+ # Final-Recipient (required)
+ delivered_to = orig_msg.get("Delivered-To")
+ content.append("Final-Recipient: rfc822; %s" % delivered_to)
+
+ # Action (required)
+ content.append("Action: failed")
+
+ # Status (required)
+ content.append("Status: 5.0.0") # permanent failure
+
+ # XXX add other optional fields? (Remote-MTA, Diagnostic-Code,
+ # Last-Attempt-Date, Final-Log-ID, Will-Retry-Until)
+
+ # return a "message/delivery-status" message
+ msg = Message()
+ msg.set_payload("\n".join(content))
+ self.attach(msg)
+
+
+class RFC822Headers(MIMEText):
+ """
+ A text/rfc822-headers mime message as defined in RFC 6522.
+ """
+
+ def __init__(self, _text, **kwargs):
+ """
+ Initialize the message.
+
+ :param _text: The contents of the message.
+ :type _text: str
+ """
+ MIMEText.__init__(
+ self, _text,
+ # set "text/rfc822-headers" mime type
+ _subtype='rfc822-headers',
+ **kwargs)
+
+
+BOUNCE_TEMPLATE = """
+This is the mail system at {0}.
+
+I'm sorry to have to inform you that your message could not
+be delivered to one or more recipients. It's attached below.
+
+For further assistance, please send mail to postmaster.
+
+If you do so, please include this problem report. You can
+delete your own text from the attached returned message.
+
+ The mail system
+
+{1}
+""".strip()
+
+
+class InvalidReturnPathError(MessageError):
+ """
+ Exception raised when the return path is invalid.
+ """
+
+
+def _build_bounce_message(bounce_from, bounce_subject, orig_msg, reason):
+ """
+ Build a bounce message.
+
+ :param bounce_from: The sender address of the bounce message.
+ :type bounce_from: str
+ :param bounce_subject: The subject of the bounce message.
+ :type bounce_subject: str
+ :param orig_msg: The original bouncing message.
+ :type orig_msg: email.message.Message
+ :param reason: The reason for the bounce.
+ :type reason: str
+
+ :return: The bounce message.
+ :rtype: MIMEMultipartReport
+
+ :raise InvalidReturnPathError: Raised when the "Return-Path" header of the
+ original message is invalid for creating a
+ bounce message.
+ """
+ # abort creation if "Return-Path" header is invalid
+ orig_rpath = orig_msg.get("Return-Path")
+ if not _check_valid_return_path(orig_rpath):
+ raise InvalidReturnPathError
+
+ msg = MIMEMultipartReport()
+ msg['From'] = bounce_from
+ msg['To'] = orig_rpath
+ msg['Date'] = formatdate(localtime=True)
+ msg['Subject'] = bounce_subject
+ msg['Return-Path'] = "<>" # prevent bounce message loop, see RFC 3834
+
+ # create and attach first required part
+ orig_to = orig_msg.get("X-Original-To") # added by postfix
+ wrapped_reason = wrap(("<%s>: " % orig_to) + reason, 74)
+ for i in xrange(1, len(wrapped_reason)):
+ wrapped_reason[i] = " " + wrapped_reason[i]
+ wrapped_reason = "\n".join(wrapped_reason)
+ text = BOUNCE_TEMPLATE.format(HOSTNAME, wrapped_reason)
+ msg.attach(MIMEText(text))
+
+ # create and attach second required part
+ msg.attach(DeliveryStatusNotificationMessage(orig_msg))
+
+ # attach third (optional) part.
+ #
+ # XXX From RFC 6522:
+ #
+ # When 8-bit or binary data not encoded in a 7-bit form is to be
+ # returned, and the return path is not guaranteed to be 8-bit or
+ # binary capable, two options are available. The original message
+ # MAY be re-encoded into a legal 7-bit MIME message or the
+ # text/rfc822-headers media type MAY be used to return only the
+ # original message headers.
+ #
+ # This is not implemented yet, we should detect if content is 7bit and
+ # use the class RFC822Headers if it is not.
+# try:
+# payload = orig_msg.get_payload()
+# payload.encode("ascii")
+# except UnicodeError:
+# headers = []
+# for k in orig_msg.keys():
+# headers.append("%s: %s" % (k, orig_msg[k]))
+# orig_msg = RFC822Headers("\n".join(headers))
+ msg.attach(orig_msg)
+
+ return msg
+
+
+class BouncerSubprocessProtocol(protocol.ProcessProtocol):
+ """
+ Bouncer subprocess protocol that will feed the msg contents to be
+ bounced through stdin
+ """
+
+ def __init__(self, msg):
+ """
+ Constructor for the BouncerSubprocessProtocol
+
+ :param msg: Message to send to stdin when the process has
+ launched
+ :type msg: str
+ """
+ self._msg = msg
+ self._outBuffer = ""
+ self._errBuffer = ""
+ self._d = defer.Deferred()
+
+ @property
+ def deferred(self):
+ return self._d
+
+ def connectionMade(self):
+ self.transport.write(self._msg)
+ self.transport.closeStdin()
+
+ def outReceived(self, data):
+ self._outBuffer += data
+
+ def errReceived(self, data):
+ self._errBuffer += data
+
+ def processEnded(self, reason):
+ if reason.check(ProcessDone):
+ self._d.callback(self._outBuffer)
+ else:
+ self._d.errback(reason)
+
+
+def _async_check_output(args, msg):
+ """
+ Async spawn a process and return a defer to be able to check the
+ output with a callback/errback
+
+ :param args: the command to execute along with the params for it
+ :type args: list of str
+ :param msg: string that will be send to stdin of the process once
+ it's spawned
+ :type msg: str
+
+ :rtype: defer.Deferred
+ """
+ pprotocol = BouncerSubprocessProtocol(msg)
+ reactor.spawnProcess(pprotocol, args[0], args)
+ return pprotocol.deferred
+
+
+class DSNGenerator(Generator):
+ """
+ A slightly modified generator to correctly parse delivery status
+ notifications.
+ """
+
+ def _handle_message_delivery_status(self, msg):
+ """
+ Handle a message of type "message/delivery-status".
+
+ This is modified from upstream version in that it also removes empty
+ lines in the beginning of each part.
+
+ :param msg: The message to be handled.
+ :type msg: Message
+ """
+ # We can't just write the headers directly to self's file object
+ # because this will leave an extra newline between the last header
+ # block and the boundary. Sigh.
+ blocks = []
+ for part in msg.get_payload():
+ s = StringIO()
+ g = self.clone(s)
+ g.flatten(part, unixfrom=False)
+ text = s.getvalue()
+ lines = text.split('\n')
+ # Strip off the unnecessary trailing empty line
+ if lines:
+ if lines[0] == '':
+ lines.pop(0)
+ if lines[-1] == '':
+ lines.pop()
+ blocks.append(NL.join(lines))
+ else:
+ blocks.append(text)
+ # Now join all the blocks with an empty line. This has the lovely
+ # effect of separating each block with an empty line, but not adding
+ # an extra one after the last one.
+ self._fp.write(NL.join(blocks))
+
+
+class MIMEMultipartReport(MIMEMultipart):
+ """
+ Implement multipart/report MIME type as defined in RFC 6522.
+
+ The syntax of multipart/report is identical to the multipart/mixed
+ content type defined in https://tools.ietf.org/html/rfc2045.
+
+ The multipart/report media type contains either two or three sub-
+ parts, in the following order:
+
+ 1. (REQUIRED) A human-readable message.
+ 2. (REQUIRED) A machine-parsable body part containing an account of
+ the reported message handling event.
+ 3. (OPTIONAL) A body part containing the returned message or a
+ portion thereof.
+ """
+
+ def __init__(
+ self, report_type="message/delivery-status", boundary=None,
+ _subparts=None):
+ """
+ Initialize the message.
+
+ As per RFC 6522, boundary and report_type are required parameters.
+
+ :param report_type: The type of report. This is set as a
+ "Content-Type" parameter, and should match the
+ MIME subtype of the second body part.
+ :type report_type: str
+
+ """
+ MIMEMultipart.__init__(
+ self,
+ # set mime type to "multipart/report"
+ _subtype="report",
+ boundary=boundary,
+ _subparts=_subparts,
+ # add "report-type" as a "Content-Type" parameter
+ report_type=report_type)
+ self._report_type = report_type
+
+ def attach(self, payload):
+ """
+ Add the given payload to the current payload, but first verify if it's
+ valid according to RFC6522.
+
+ :param payload: The payload to be attached.
+ :type payload: Message
+
+ :raise MessageError: Raised if the payload is invalid.
+ """
+ idx = len(self.get_payload()) + 1
+ self._check_valid_payload(idx, payload)
+ MIMEMultipart.attach(self, payload)
+
+ def _check_valid_payload(self, idx, payload):
+ """
+ Check that an attachment is valid according to RFC6522.
+
+ :param payload: The payload to be attached.
+ :type payload: Message
+
+ :raise MessageError: Raised if the payload is invalid.
+ """
+ if idx == 1:
+ # The text in the first section can use any IANA-registered MIME
+ # media type, charset, or language.
+ cond = lambda payload: isinstance(payload, MIMEBase)
+ error_msg = "The first attachment must be a MIME message."
+ elif idx == 2:
+ # RFC 6522 requires that the report-type parameter is equal to the
+ # MIME subtype of the second body type of the multipart/report.
+ cond = lambda payload: \
+ payload.get_content_type() == self._report_type
+ error_msg = "The second attachment's subtype must be %s." \
+ % self._report_type
+ elif idx == 3:
+ # A body part containing the returned message or a portion thereof.
+ cond = lambda payload: isinstance(payload, Message)
+ error_msg = "The third attachment must be a message."
+ else:
+ # The multipart/report media type contains either two or three sub-
+ # parts.
+ cond = lambda _: False
+ error_msg = "The multipart/report media type contains either " \
+ "two or three sub-parts."
+ if not cond(payload):
+ raise MessageError("Invalid attachment: %s" % error_msg)
+
+ def as_string(self, unixfrom=False):
+ """
+ Return the entire formatted message as string.
+
+ This is modified from upstream to use our own generator.
+
+ :param as_string: Whether to include the Unix From envelope heder.
+ :type as_string: bool
+
+ :return: The entire formatted message.
+ :rtype: str
+ """
+ fp = StringIO()
+ g = DSNGenerator(fp)
+ g.flatten(self, unixfrom=unixfrom)
+ return fp.getvalue()
diff --git a/src/leap/mx/check_recipient_access.py b/src/leap/mx/check_recipient_access.py
index b80ccfd..55460a6 100644
--- a/src/leap/mx/check_recipient_access.py
+++ b/src/leap/mx/check_recipient_access.py
@@ -17,26 +17,72 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""
-Classes for resolving postfix recipient access
+Classes for resolving postfix recipient access.
+
+The resolver is queried by the mail server before delivery to the mail spool
+directory, and should check if the address is able to receive messages.
+Examples of reasons for denying delivery would be that the user is out of
+quota, is user, or have no pgp public key in the server.
Test this with postmap -v -q "foo" tcp:localhost:2244
"""
from twisted.protocols import postfix
-from leap.mx.alias_resolver import AliasResolverFactory
+from leap.mx.tcp_map import LEAPPostfixTCPMapServerFactory
+from leap.mx.tcp_map import TCP_MAP_CODE_SUCCESS
+from leap.mx.tcp_map import TCP_MAP_CODE_TEMPORARY_FAILURE
+from leap.mx.tcp_map import TCP_MAP_CODE_PERMANENT_FAILURE
+
+
+class LEAPPostFixTCPMapAccessServer(postfix.PostfixTCPMapServer):
+ """
+ A postfix tcp map recipient access checker server.
+ The server potentially receives the uuid and a PGP key for the user, which
+ are looked up by the factory, and will return a permanent or a temporary
+ failure in case either the user or the key don't exist, respectivelly.
+ """
-class LEAPPostFixTCPMapserverAccess(postfix.PostfixTCPMapServer):
def _cbGot(self, value):
- # For more info, see:
- # http://www.postfix.org/tcp_table.5.html
- # http://www.postfix.org/access.5.html
- if value is None:
- self.sendCode(500, postfix.quote("REJECT"))
+ """
+ Return a code and message depending on the result of the factory's
+ get().
+
+ If there's no pgp public key for the user, we currently return a
+ temporary failure saying that the user account is disabled.
+
+ For more info, see: http://www.postfix.org/access.5.html
+
+ :param value: The uuid and public key.
+ :type value: list
+ """
+ uuid, pubkey = value
+ if uuid is None:
+ self.sendCode(
+ TCP_MAP_CODE_PERMANENT_FAILURE,
+ postfix.quote("REJECT"))
+ elif pubkey is None:
+ self.sendCode(
+ TCP_MAP_CODE_TEMPORARY_FAILURE,
+ postfix.quote("4.7.13 USER ACCOUNT DISABLED"))
else:
- self.sendCode(200, postfix.quote("OK"))
+ self.sendCode(
+ TCP_MAP_CODE_SUCCESS,
+ postfix.quote("OK"))
+
+
+class CheckRecipientAccessFactory(LEAPPostfixTCPMapServerFactory):
+ """
+ A factory for the recipient access checker.
+
+ When queried, the factory looks up the user's uuid and a PGP key for that
+ user and returns the result to the server's _cbGot() method.
+ """
+
+ protocol = LEAPPostFixTCPMapAccessServer
+ @property
+ def _query_message(self):
+ return "Checking recipient access for"
-class CheckRecipientAccessFactory(AliasResolverFactory):
- protocol = LEAPPostFixTCPMapserverAccess
diff --git a/src/leap/mx/couchdbhelper.py b/src/leap/mx/couchdbhelper.py
index f20f1dd..1752b4e 100644
--- a/src/leap/mx/couchdbhelper.py
+++ b/src/leap/mx/couchdbhelper.py
@@ -15,24 +15,15 @@
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
"""
Classes for working with CouchDB or BigCouch instances which store email alias
maps, user UUIDs, and GPG keyIDs.
"""
-from functools import partial
-
-try:
- from paisley import client
-except ImportError:
- print "This software requires paisley. Please see the README file"
- print "for instructions on getting required dependencies."
-try:
- from twisted.python import log
-except ImportError:
- print "This software requires Twisted. Please see the README file"
- print "for instructions on getting required dependencies."
+from paisley import client
+from twisted.python import log
class ConnectedCouchDB(client.CouchDB):
@@ -66,24 +57,8 @@ class ConnectedCouchDB(client.CouchDB):
username=username,
password=password,
*args, **kwargs)
-
self._cache = {}
- if dbName is None:
- databases = self.listDB()
- databases.addCallback(self._print_databases)
-
- def _print_databases(self, data):
- """
- Callback for listDB that prints the available databases
-
- :param data: response from the listDB command
- :type data: array
- """
- log.msg("Available databases:")
- for database in data:
- log.msg(" * %s" % (database,))
-
def createDB(self, dbName):
"""
Overrides ``paisley.client.CouchDB.createDB``.
@@ -96,110 +71,63 @@ class ConnectedCouchDB(client.CouchDB):
"""
pass
- def queryByAddress(self, address):
+ def getUuidAndPubkey(self, address):
"""
- Check to see if a particular email or alias exists.
+ Query couch and return a deferred that will fire with the uuid and pgp
+ public key for address.
- :param alias: A string representing the email or alias to check.
- :type alias: str
- :return: a deferred for this query
+ :param address: A string representing the email or alias to check.
+ :type address: str
+ :return: A deferred that will fire with the user's uuid and pgp public
+ key.
:rtype twisted.defer.Deferred
"""
- assert isinstance(address, (str, unicode)), "Email or alias queries must be string"
-
# TODO: Cache results
-
d = self.openView(docId="Identity",
viewId="by_address/",
key=address,
reduce=False,
include_docs=True)
- d.addCallbacks(partial(self._get_uuid, address), log.err)
-
+ def _get_uuid_and_pubkey_cbk(result):
+ uuid = None
+ pubkey = None
+ if result["rows"]:
+ doc = result["rows"][0]["doc"]
+ uuid = doc["user_id"]
+ if "keys" in doc:
+ pubkey = doc["keys"]["pgp"]
+ return uuid, pubkey
+
+ d.addCallback(_get_uuid_and_pubkey_cbk)
return d
- def _get_uuid(self, address, result):
- """
- Parses the result of the by_address query and gets the uuid
-
- :param address: alias looked up
- :type address: string
- :param result: result dictionary
- :type result: dict
- :return: The uuid for alias if available
- :rtype: str
- """
- for row in result["rows"]:
- if row["key"] == address:
- uuid = row["doc"].get("user_id", None)
- if uuid is None:
- log.msg("ERROR: Found doc for %s but there's not user_id!"
- % (address,))
- return uuid
- return None
-
- def getPubKey(self, uuid):
+ def getPubkey(self, uuid):
"""
- Returns a deferred that will return the pubkey for the uuid provided
+ Query couch and return a deferred that will fire with the pgp public
+ key for user with given uuid.
- :param uuid: uuid for the user to query
+ :param uuid: The uuid of a user
:type uuid: str
+ :return: A deferred that will fire with the pgp public key for
+ the user.
:rtype: Deferred
"""
d = self.openView(docId="Identity",
- viewId="pgp_key_by_email/",
- user_id=uuid,
+ viewId="by_user_id/",
+ key=uuid,
reduce=False,
include_docs=True)
- d.addCallbacks(partial(self._get_pgp_key, uuid), log.err)
+ def _get_pubkey_cbk(result):
+ pubkey = None
+ try:
+ doc = result["rows"][0]["doc"]
+ pubkey = doc["keys"]["pgp"]
+ except (KeyError, IndexError):
+ pass
+ return pubkey
+ d.addCallbacks(_get_pubkey_cbk, log.err)
return d
-
- def _get_pgp_key(self, uuid, result):
- """
- Callback used to filter the correct pubkey from the result of
- the query to the couchdb
-
- :param uuid: uuid for the user that was queried
- :type uuid: str
- :param result: result dictionary for the db query
- :type result: dict
-
- :rtype: str or None
- """
- for row in result["rows"]:
- user_id = row["doc"].get("user_id")
- if not user_id:
- print("User %s is in an inconsistent state")
- continue
- if user_id == uuid:
- return row["value"]
- return None
-
-if __name__ == "__main__":
- from twisted.internet import reactor
- cdb = ConnectedCouchDB("localhost",
- port=6666,
- dbName="users",
- username="",
- password="")
-
- d = cdb.queryByLoginOrAlias("test1")
-
- @d.addCallback
- def right(result):
- print "Should be an actual uuid:", result
- print "Public Key:"
- print cdb.getPubKey(result)
-
- d2 = cdb.queryByLoginOrAlias("asdjaoisdjoiqwjeoi")
-
- @d2.addCallback
- def wrong(result):
- print "Should be None:", result
-
- reactor.callLater(5, reactor.stop)
- reactor.run()
diff --git a/src/leap/mx/mail_receiver.py b/src/leap/mx/mail_receiver.py
index 630c982..446fd38 100644
--- a/src/leap/mx/mail_receiver.py
+++ b/src/leap/mx/mail_receiver.py
@@ -39,13 +39,8 @@ import signal
import json
import email.utils
-import socket
from email import message_from_string
-from email.MIMEMultipart import MIMEMultipart
-from email.MIMEText import MIMEText
-from email.Utils import formatdate
-from email.header import decode_header
from twisted.application.service import Service, IService
from twisted.internet import inotify, defer, task, reactor
@@ -53,84 +48,15 @@ from twisted.python import filepath, log
from zope.interface import implements
-from leap.soledad.common.crypto import (
- EncryptionSchemes,
- ENC_JSON_KEY,
- ENC_SCHEME_KEY,
-)
+from leap.soledad.common.crypto import EncryptionSchemes
+from leap.soledad.common.crypto import ENC_JSON_KEY
+from leap.soledad.common.crypto import ENC_SCHEME_KEY
from leap.soledad.common.couch import CouchDatabase, CouchDocument
-from leap.keymanager import openpgp
-
-BOUNCE_TEMPLATE = """
-Delivery to the following recipient failed:
- {0}
-
-Reasons:
- {1}
-
-Original message:
- {2}
-""".strip()
-
-
-from twisted.internet import protocol
-from twisted.internet.error import ProcessDone
-
-
-class BouncerSubprocessProtocol(protocol.ProcessProtocol):
- """
- Bouncer subprocess protocol that will feed the msg contents to be
- bounced through stdin
- """
-
- def __init__(self, msg):
- """
- Constructor for the BouncerSubprocessProtocol
-
- :param msg: Message to send to stdin when the process has
- launched
- :type msg: str
- """
- self._msg = msg
- self._outBuffer = ""
- self._errBuffer = ""
- self._d = None
-
- def connectionMade(self):
- self._d = defer.Deferred()
-
- self.transport.write(self._msg)
- self.transport.closeStdin()
-
- def outReceived(self, data):
- self._outBuffer += data
- def errReceived(self, data):
- self._errBuffer += data
-
- def processEnded(self, reason):
- if reason.check(ProcessDone):
- self._d.callback(self._outBuffer)
- else:
- self._d.errback(reason)
-
-
-def async_check_output(args, msg):
- """
- Async spawn a process and return a defer to be able to check the
- output with a callback/errback
-
- :param args: the command to execute along with the params for it
- :type args: list of str
- :param msg: string that will be send to stdin of the process once
- it's spawned
- :type msg: str
+from leap.keymanager import openpgp
- :rtype: defer.Deferred
- """
- pprotocol = BouncerSubprocessProtocol(msg)
- reactor.spawnProcess(pprotocol, args[0], args)
- return pprotocol.d
+from leap.mx.bounce import bounce_message
+from leap.mx.bounce import InvalidReturnPathError
class MailReceiver(Service):
@@ -177,8 +103,6 @@ class MailReceiver(Service):
self._directories = directories
self._bounce_from = bounce_from
self._bounce_subject = bounce_subject
-
- self._domain = socket.gethostbyaddr(socket.gethostname())[0]
self._processing_skipped = False
def startService(self):
@@ -353,7 +277,7 @@ class MailReceiver(Service):
def _get_owner(self, mail):
"""
- Given an email, returns the uuid of the owner.
+ Given an email, return the uuid of the owner.
:param mail: mail to analyze
:type mail: email.message.Message
@@ -361,25 +285,24 @@ class MailReceiver(Service):
:returns: uuid
:rtype: str or None
"""
- uuid = None
-
+ # we expect the topmost "Delivered-To" header to indicate the correct
+ # final delivery address. It should consist of <uuid>@<domain>, as the
+ # earlier alias resolver query should have translated the username to
+ # the user id. See https://leap.se/code/issues/6858 for more info.
delivereds = mail.get_all("Delivered-To")
if delivereds is None:
+ # XXX this should not happen! see the comment above
return None
- for to in delivereds:
- name, addr = email.utils.parseaddr(to)
- parts = addr.split("@")
- if len(parts) > 1 and parts[1] == self._domain:
- uuid = parts[0]
- break
-
+ final_address = delivereds.pop(0)
+ _, addr = email.utils.parseaddr(final_address)
+ uuid, _ = addr.split("@")
return uuid
@defer.inlineCallbacks
- def _bounce_mail(self, orig_msg, filepath, reason):
+ def _bounce_message(self, orig_msg, filepath, reason):
"""
- Bounces the email contained in orig_msg to it's sender and
- removes it from the queue.
+ Bounce the message contained in orig_msg to it's sender and
+ remove it from the queue.
:param orig_msg: Message that is going to be bounced
:type orig_msg: email.message.Message
@@ -388,22 +311,12 @@ class MailReceiver(Service):
:param reason: Brief explanation about why it's being bounced
:type reason: str
"""
- to = orig_msg.get("From")
-
- msg = MIMEMultipart()
- msg['From'] = self._bounce_from
- msg['To'] = to
- msg['Date'] = formatdate(localtime=True)
- msg['Subject'] = self._bounce_subject
-
- decoded_to = " ".join([x[0] for x in decode_header(to)])
- text = BOUNCE_TEMPLATE.format(decoded_to,
- reason,
- orig_msg.as_string())
-
- msg.attach(MIMEText(text))
-
- yield async_check_output(["/usr/sbin/sendmail", "-t"], msg.as_string())
+ try:
+ yield bounce_message(
+ self._bounce_from, self._bounce_subject, orig_msg, reason)
+ except InvalidReturnPathError:
+ # give up bouncing this message!
+ log.msg("Will not bounce message because of invalid return path.")
yield self._conditional_remove(True, filepath)
def sleep(self, secs):
@@ -478,7 +391,7 @@ class MailReceiver(Service):
(filepath.path,))
bounce_reason = "Missing UUID: There was a problem " \
"locating the user in our database."
- yield self._bounce_mail(msg, filepath, bounce_reason)
+ yield self._bounce_message(msg, filepath, bounce_reason)
defer.returnValue(None)
log.msg("Mail owner: %s" % (uuid,))
@@ -486,13 +399,15 @@ class MailReceiver(Service):
log.msg("BUG: There was no uuid!")
defer.returnValue(None)
- pubkey = yield self._users_cdb.getPubKey(uuid)
+ pubkey = yield self._users_cdb.getPubkey(uuid)
if pubkey is None or len(pubkey) == 0:
- log.msg("No public key, stopping the processing chain")
- bounce_reason = "Missing PubKey: There was a problem " \
- "locating the user's public key in our " \
- "database."
- yield self._bounce_mail(msg, filepath, bounce_reason)
+ log.msg(
+ "No public key for %s, stopping the processing chain."
+ % uuid)
+ bounce_reason = "Missing PGP public key: There was a " \
+ "problem locating the user's public key in " \
+ "our database."
+ yield self._bounce_message(msg, filepath, bounce_reason)
defer.returnValue(None)
log.msg("Encrypting message to %s's pubkey" % (uuid,))
diff --git a/src/leap/mx/tcp_map.py b/src/leap/mx/tcp_map.py
new file mode 100644
index 0000000..597c830
--- /dev/null
+++ b/src/leap/mx/tcp_map.py
@@ -0,0 +1,72 @@
+#!/usr/bin/env python
+# -*- encoding: utf-8 -*-
+# tcpmap.py
+# Copyright (C) 2015 LEAP
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+
+from abc import ABCMeta
+from abc import abstractproperty
+
+from twisted.internet.protocol import ServerFactory
+from twisted.python import log
+
+
+# For info on codes, see: http://www.postfix.org/tcp_table.5.html
+TCP_MAP_CODE_SUCCESS = 200
+TCP_MAP_CODE_TEMPORARY_FAILURE = 400
+TCP_MAP_CODE_PERMANENT_FAILURE = 500
+
+
+# we have to also extend from object here to make the class a new-style class.
+# If we don't, we get a TypeError because "new-style classes can't have only
+# classic bases". This has to do with the way abc.ABCMeta works and the old
+# and new style of python classes.
+class LEAPPostfixTCPMapServerFactory(ServerFactory, object):
+ """
+ A factory for postfix tcp map servers.
+ """
+
+ __metaclass__ = ABCMeta
+
+
+ def __init__(self, couchdb):
+ """
+ Initialize the factory.
+
+ :param couchdb: A CouchDB client.
+ :type couchdb: leap.mx.couchdbhelper.ConnectedCouchDB
+ """
+ self._cdb = couchdb
+
+ @abstractproperty
+ def _query_message(self):
+ pass
+
+ def get(self, lookup_key):
+ """
+ Look up user based on lookup_key.
+
+ :param lookup_key: The lookup key.
+ :type lookup_key: str
+
+ :return: A deferred that will be fired with the user's address, uuid
+ and pgp key.
+ :rtype: Deferred
+ """
+ log.msg("%s %s" % (self._query_message, lookup_key,))
+ d = self._cdb.getUuidAndPubkey(lookup_key)
+ d.addErrback(log.err)
+ return d