diff options
178 files changed, 4715 insertions, 2690 deletions
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..3e5d9cbe --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,78 @@ +# How to contribute + +Here's the brief: + +* We welcome contributions of all kinds, including but not limited to features, bug fixes, quality assurance, documentation, security review or asking questions +* Pull requests are based off, integrated into & rebased against master +* Write automated tests, ideally using TDD. CI needs to be green in order to merge. +* Tablero board for ready to play work +* Contact us for questions & suggestions: + * IRC: #pixelated @ chat.freenode.org ([join via webchat](https://webchat.freenode.net/)) + * Email: [team@pixelated-project.org](mailto:team@pixelated-project.org) + * Twitter: [@pixelatedteam](https://twitter.com/pixelatedteam) + + + This document outlines our way of working, gives hints and outlines the steps to make your contribution to Pixelated as smooth as possible. You're not required to read this before getting started. We're explaining the way we work to make sure you're having a good experience and can make best use of the time you're contributing to our project. + +## Contributions steps + +This is the lifecycle of a contribution. See our [README](README.md) for details on how to set up your development environment. + +We follow a simplified fork + pull request workflow: + +* To start, fork this repository and create a branch that's based off the latest commit in the `master` branch +* Implement the change +* Send a pull request against the master branch. Please make sure the automated tests are passing, as indicated by GitHub on the pull requests. +* Please keep your feature branch updated. Rebase your branch against upstream changes on the master branch, resolve any conflicts and make sure the tests are staying green. +* Your pull request will reviewed and merged + +### What to work on + +Our [Kanban board "Tablero"](https://pixboard.herokuapp.com) provides an overview of issues that are ready to play or awaiting QA. If you're just getting familiar with Pixelated, see the [issues labeled 'Beginner'](labels/Beginners). + +Generally, all issues that have no user assigned are awaiting work and free to play. If you want to make sure, or you think it will take more than a couple of days to complete your work, please reach out to us using the contact info above. + +### Guidelines + +When implementing your change, please follow this advice: + +* Your change should be described in an issue, or latest in the pull request description. +* For bugs, please describe, in an issue or pull request: + 1. Steps to reproduce the behavior + 2. Expected behavior + 3. Actual behavior. Please also include as much meta-information as reasonable, e.g. time & date, software version etc. +* Pull requests need not to be finished work only; you can also submit changes in consecutive Pull Requests as long as CI stays green. Also, please send a PR with the intention of discussion & feedback. Please mark those Pull Requests appropriately. +* We review your pull request. This review is prioritised and done as part of our priotisation. During this time, we ask you to keep it up to date by frequently rebasing against master + +### Review Criteria + +When reviewing your contribution, we apply the following criteria: + +* Test must be green. This usually includes an automatic check of the style guide using e.g. pep8 or jshint. All tests should be executed locally before you push, as well as on CI. If you struggle to reproduce a failure on CI locally, please notify us on IRC so we can resolve the issue. +* Your change should be in-line with Pixelated's direction. Chances that it is are good if, in descending priority: + * It is described by an issue that is labelled as 'ready' + * It is described by an issue that is labelled as 'backlog' + * You've fixed a bug for which no issue existed yet, but described it in the pull request as explained in the section *Steps* + * You've implemented a feature for which no issue existed yet. While we don't require up-front consensus, we strongly encourage you to describe feature suggestions in issues to get feedback before you spent significant time on implementation. +* We won't tolerate abusive, exploitative or harassing behavior in every context of our project and refuse collaboration with any individual who exposes such behavior. + +## Types of contributions + +Pixelated evolves upon the source code it's made of. This evolution is fueled by a variety of tasks; some of which require developing source code, some of which do not. + +Contributions we appreciate: + +* Features: New functionality is described in issues in the form of a user story to capture the end-user benefit. +* Bug fixes: Things go wrong from time to time. +* Quality Assurance: While every software change should be covered by automated tests, there are certain types of errors that are best spotted by a human. We apply QA to dev-complete changes. +* Documentation: Feedback & improvements of our guides & tutorial copywritings. +* Security review +* Asking questions on IRC + +## ThoughtWork's role + +ThoughtWorks seeds the community that builds pixelated. We seed the development, investing our own resources: We provide a team of 10+ software delivery experts to lay the foundation for the project. We use our network and contacts to approach customers and users. + +ThoughtWorks started building Pixelated because it is right. In combining our passion for defending a free internet and our capability to deliver software, we build software to counter widespread mass surveillance of email communication. + +It is not ThoughWork's goal to make money from Pixelated. The reasons are multiple, but at the end of the day, we believe that our goals of mass adoption and decentralization can best be achieved if Pixelated puts end users and providers before a revenue stream. @@ -1,7 +1,7 @@ Pixelated User Agent ==================== -[](https://snap-ci.com/pixelated-project/pixelated-user-agent/branch/master) [ +[](https://snap-ci.com/pixelated/pixelated-user-agent/branch/master) [ ](https://coveralls.io/r/pixelated-project/pixelated-user-agent?branch=master) The Pixelated User Agent is the mail client of the Pixelated ecosystem. It is composed of two parts, a web interface written in JavaScript ([FlightJS](https://flightjs.github.io/)) and a Python API that interacts with a LEAP Provider, the e-mail platform that Pixelated is built on. @@ -76,8 +76,6 @@ For people that just want to try the user agent, we have debian packages availab ```shell echo "deb http://packages.pixelated-project.org/debian wheezy-snapshots main" > /etc/apt/sources.list.d/pixelated.list -echo "deb http://packages.pixelated-project.org/debian wheezy-backports main" >> /etc/apt/sources.list.d/pixelated.list -echo "deb http://packages.pixelated-project.org/debian wheezy main" >> /etc/apt/sources.list.d/pixelated.list apt-key adv --keyserver pool.sks-keyservers.net --recv-key 287A1542472DC0E3 diff --git a/debian/control b/debian/control index 35d4af94..b05e7d67 100644 --- a/debian/control +++ b/debian/control @@ -2,7 +2,7 @@ Source: pixelated-user-agent Maintainer: Pixelated Team <pixelated-team@thoughtworks.com> Section: python Priority: optional -Build-Depends: python-setuptools (>= 0.6b3), python-all (>= 2.6.6-3), debhelper (>= 7.4.3), ruby-compass, nodejs +Build-Depends: python-setuptools (>= 0.6b3), python-all (>= 2.6.6-3), debhelper (>= 7.4.3), ruby-compass, nodejs, nodejs-legacy, build-essential, libffi-dev, python-dev, pixpybuild Standards-Version: 3.9.1 Homepage: http://pixelated-project.org Vcs-Git: https://github.com/pixelated-project/pixelated-user-agent.git @@ -11,7 +11,7 @@ X-Python-Version: >= 2.7 Package: pixelated-user-agent Architecture: all -Depends: python (>= 2.7), python (<< 2.8), leap-keymanager, soledad-common, soledad-client, leap-mail, leap-auth, python-dirspec, python-u1db, python-whoosh (>=2.5.7), python-sqlcipher, python-twisted-web (>= 12.3.0), python-pyasn1, python-gnupg, python-requests (>= 2.0.0), python-twisted (>= 12.3.0) +Depends: python (>= 2.7), python (<< 2.8), libffi5, python-gnupg Description: API to serve the pixelated front-end requests Pixelated User Agent Service ============================ diff --git a/debian/pixelated-user-agent.links b/debian/pixelated-user-agent.links new file mode 100644 index 00000000..d5bfa8ed --- /dev/null +++ b/debian/pixelated-user-agent.links @@ -0,0 +1,3 @@ +/usr/share/python/pixelated-user-agent/bin/pixelated-user-agent /usr/bin/pixelated-user-agent +/usr/share/python/pixelated-user-agent/bin/pixelated-maintenance /usr/bin/pixelated-maintenance + diff --git a/debian/rules b/debian/rules index 742f40f1..195871cf 100755 --- a/debian/rules +++ b/debian/rules @@ -7,6 +7,9 @@ DPKG_EXPORT_BUILDFLAGS = 1 include /usr/share/dpkg/default.mk +ifdef GO_BUILDDIR + export DH_BUILDOPTIONS=--builddirectory=$(GO_BUILDDIR) +endif # see FEATURE AREAS in dpkg-buildflags(1) #export DEB_BUILD_MAINT_OPTIONS = hardening=+all @@ -19,7 +22,7 @@ include /usr/share/dpkg/default.mk # main packaging script based on dh7 syntax %: - dh $@ --sourcedirectory=service + dh $@ --buildsystem pixpybuild --sourcedirectory=service $(DH_BUILDOPTIONS) # debmake generated override targets # This is example for Cmake (See http://bugs.debian.org/641051 ) diff --git a/doc/maintenance.md b/doc/maintenance.md index 38c18bf8..f2b482b7 100644 --- a/doc/maintenance.md +++ b/doc/maintenance.md @@ -46,9 +46,9 @@ optional arguments: The commands you can run are: * reset - Use this to remove all mails from your account. Existing encryption keys like your GnuPG key is not affected -* sync - Sync your soledad database * load-mails - Loads existing mails into your account -* dump-soledad- Get a soledad database dump. Mostly for debugging use cases +* dump-soledad - Get a soledad database dump. Mostly for debugging use cases +* sync - Sync your soledad database Like with other such tools, to get detailed help for a single command, call it with the --help option. diff --git a/provisioning/Dockerfile b/provisioning/Dockerfile index 930ed875..4955dd49 100644 --- a/provisioning/Dockerfile +++ b/provisioning/Dockerfile @@ -16,7 +16,7 @@ # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. -FROM debian:jessie +FROM debian:wheezy MAINTAINER fbernitt@thoughtworks.com @@ -25,16 +25,7 @@ RUN echo "APT::Get::Assume-Yes true;" >>/etc/apt/apt.conf # Install Pixelated User Packages RUN echo "deb http://packages.pixelated-project.org/debian wheezy-snapshots main" >> /etc/apt/sources.list -RUN echo "deb http://packages.pixelated-project.org/debian wheezy-backports main" >> /etc/apt/sources.list -RUN echo "deb http://packages.pixelated-project.org/debian wheezy main" >> /etc/apt/sources.list -RUN echo "deb http://deb.bitmask.net/debian/ wheezy main" >> /etc/apt/sources.list -RUN echo "deb http://deb.leap.se/experimental wheezy main" >> /etc/apt/sources.list -RUN apt-key adv --keyserver pool.sks-keyservers.net --recv-key 1E34A1828E207901 && \ - apt-key adv --keyserver pool.sks-keyservers.net --recv-key 287A1542472DC0E3 - -RUN echo "Package: python-sqlcipher" > /etc/apt/preferences.d/python-sqlcipher -RUN echo "Pin: version 2.6.3.3+0~20141111222659.14+wheezy~1.gbp2d164a+pix1" >> /etc/apt/preferences.d/python-sqlcipher -RUN echo "Pin-Priority: 1000" >> /etc/apt/preferences.d/python-sqlcipher +RUN apt-key adv --keyserver pool.sks-keyservers.net --recv-key 287A1542472DC0E3 # Update packages lists RUN apt-get update -y --force-yes @@ -53,12 +44,9 @@ ENV LC_ALL C.UTF-8 ENV LANG en_US.UTF-8 ENV LANGUAGE en_US.UTF-8 -# Install pip for taskthread dependency (no backport yet) -RUN apt-get install python-pip python-all-dev libssl-dev -RUN pip install taskthread - -RUN apt-get install -y --force-yes --allow-unauthenticated soledad-client soledad-common -# Install Pixelated User Agent +RUN apt-get install -y --force-yes python-gnupg RUN apt-get install -y --force-yes pixelated-user-agent +RUN ln -s /mnt/user/ /.config + EXPOSE 4567 diff --git a/provisioning/modules/pixelated/files/activate_custom_node_modules.sh b/provisioning/modules/pixelated/files/activate_custom_node_modules.sh index f46b67e1..5ad419e3 100755 --- a/provisioning/modules/pixelated/files/activate_custom_node_modules.sh +++ b/provisioning/modules/pixelated/files/activate_custom_node_modules.sh @@ -27,6 +27,9 @@ fi source /home/vagrant/user-agent-venv/bin/activate +pip install --upgrade pip +pip install --upgrade setuptools + cd /vagrant/service echo "running python setup" ./go setuppy > /dev/null diff --git a/service/.coveragerc b/service/.coveragerc index 6f970475..e411cd6e 100644 --- a/service/.coveragerc +++ b/service/.coveragerc @@ -2,6 +2,3 @@ branch = True source = pixelated omit = test/* - -[report] -ingore-errors = True diff --git a/service/README.md b/service/README.md index 9da458f2..da1b334f 100644 --- a/service/README.md +++ b/service/README.md @@ -60,3 +60,13 @@ pixelated-user-agent ``` pixelated-user-agent --config=<config_file_path> ``` + +Run a single unit or integration test +``` +trial +``` +Run a single functional test + +``` +behave test/functional/features/name_of_feature.feature +``` diff --git a/service/develop_requirements.txt b/service/develop_requirements.txt new file mode 100644 index 00000000..e4c2dce3 --- /dev/null +++ b/service/develop_requirements.txt @@ -0,0 +1,10 @@ +--allow-external u1db --allow-unverified u1db +--allow-external dirspec --allow-unverified dirspec +-e 'git+https://github.com/pixelated-project/leap_pycommon.git@develop#egg=leap.common' +-e 'git+https://github.com/pixelated-project/leap_auth.git#egg=leap.auth' +-e 'git+https://github.com/pixelated-project/soledad.git@develop#egg=leap.soledad.common&subdirectory=common/' +-e 'git+https://github.com/pixelated-project/soledad.git@develop#egg=leap.soledad.client&subdirectory=client/' +-e 'git+https://github.com/pixelated-project/soledad.git@develop#egg=leap.soledad.server&subdirectory=server/' +-e 'git+https://github.com/pixelated-project/keymanager.git@develop#egg=leap.keymanager' +-e 'git+https://github.com/pixelated-project/leap_mail.git@develop#egg=leap.mail' + @@ -17,10 +17,11 @@ function resolveNumOfCores { function setuppy { echo "Installing Pixelated User Agent." + pip install -r develop_requirements.txt pip install -r test_requirements.txt python setup.py develop $* pip uninstall -y scrypt; pip install scrypt - pip uninstall -y gnupg; pip install gnupg==1.2.5 + pip uninstall -y gnupg; pip install gnupg==2.0.2 echo "Done." } @@ -43,6 +44,10 @@ function runIntegrationTests { function runUnitTests { echo "Executing Unit Tests." + if [ -f ~/.config/leap/events/zmq_certificates/public_keys/server.key ] ; then + echo "Removing zmq server.key before running tests." + rm ~/.config/leap/events/zmq_certificates/public_keys/server.key + fi trial --reporter=text $* test.unit echo "Done." } @@ -77,7 +82,7 @@ function runCoverageIntegration { getTrialAbsolutePath coverage run -p --source=pixelated $TRIAL_PATH --reporter=text $* test.integration coverage combine - coverage html + coverage html --ignore-errors echo "Done." } diff --git a/service/pixelated/adapter/errors/__init__.py b/service/pixelated/adapter/errors/__init__.py new file mode 100644 index 00000000..31ad4947 --- /dev/null +++ b/service/pixelated/adapter/errors/__init__.py @@ -0,0 +1,3 @@ +class DuplicatedDraftException(Exception): + def __init__(self, message): + super(Exception, self).__init__(message) diff --git a/service/pixelated/adapter/listeners/mailbox_indexer_listener.py b/service/pixelated/adapter/listeners/mailbox_indexer_listener.py index d8e0f81e..8896d742 100644 --- a/service/pixelated/adapter/listeners/mailbox_indexer_listener.py +++ b/service/pixelated/adapter/listeners/mailbox_indexer_listener.py @@ -13,6 +13,11 @@ # # You should have received a copy of the GNU Affero General Public License # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +import logging +from twisted.internet import defer + + +logger = logging.getLogger(__name__) class MailboxIndexerListener(object): @@ -21,22 +26,31 @@ class MailboxIndexerListener(object): SEARCH_ENGINE = None @classmethod - def listen(cls, account, mailbox_name, soledad_querier): - listener = MailboxIndexerListener(mailbox_name, soledad_querier) - if listener not in account.getMailbox(mailbox_name).listeners: - account.getMailbox(mailbox_name).addListener(listener) + @defer.inlineCallbacks + def listen(cls, account, mailbox_name, mail_store): + listener = MailboxIndexerListener(mailbox_name, mail_store) + if listener not in (yield account.getMailbox(mailbox_name)).listeners: + mbx = yield account.getMailbox(mailbox_name) + mbx.addListener(listener) + + defer.returnValue(listener) - def __init__(self, mailbox_name, soledad_querier): + def __init__(self, mailbox_name, mail_store): self.mailbox_name = mailbox_name - self.querier = soledad_querier + self.mail_store = mail_store + @defer.inlineCallbacks def newMessages(self, exists, recent): - indexed_idents = set(self.SEARCH_ENGINE.search('tag:' + self.mailbox_name.lower(), all_mails=True)) - soledad_idents = self.querier.idents_by_mailbox(self.mailbox_name) + try: + indexed_idents = set(self.SEARCH_ENGINE.search('tag:' + self.mailbox_name.lower(), all_mails=True)) + soledad_idents = yield self.mail_store.get_mailbox_mail_ids(self.mailbox_name) + soledad_idents = set(soledad_idents) - missing_idents = soledad_idents.difference(indexed_idents) + missing_idents = soledad_idents.difference(indexed_idents) - self.SEARCH_ENGINE.index_mails(self.querier.mails(missing_idents)) + self.SEARCH_ENGINE.index_mails((yield self.mail_store.get_mails(missing_idents))) + except Exception, e: # this is a event handler, don't let exceptions escape + logger.error(e) def __eq__(self, other): return other and other.mailbox_name == self.mailbox_name @@ -46,3 +60,11 @@ class MailboxIndexerListener(object): def __repr__(self): return 'MailboxListener: ' + self.mailbox_name + + +@defer.inlineCallbacks +def listen_all_mailboxes(account, search_engine, mail_store): + MailboxIndexerListener.SEARCH_ENGINE = search_engine + mailboxes = yield account.account.list_all_mailbox_names() + for mailbox_name in mailboxes: + yield MailboxIndexerListener.listen(account, mailbox_name, mail_store) diff --git a/service/pixelated/adapter/mailstore/__init__.py b/service/pixelated/adapter/mailstore/__init__.py new file mode 100644 index 00000000..978df45d --- /dev/null +++ b/service/pixelated/adapter/mailstore/__init__.py @@ -0,0 +1,20 @@ +# +# Copyright (c) 2015 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. + +from pixelated.adapter.mailstore.mailstore import MailStore, underscore_uuid +from pixelated.adapter.mailstore.leap_mailstore import LeapMailStore + +__all__ = ['MailStore', 'LeapMailStore', 'underscore_uuid'] diff --git a/service/pixelated/adapter/mailstore/body_parser.py b/service/pixelated/adapter/mailstore/body_parser.py new file mode 100644 index 00000000..a6017833 --- /dev/null +++ b/service/pixelated/adapter/mailstore/body_parser.py @@ -0,0 +1,68 @@ +# +# Copyright (c) 2015 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. + +from email.parser import Parser +import re +import logging + +logger = logging.getLogger(__name__) + + +def _parse_charset_header(content_type_and_charset_header, default_charset='us-ascii'): + try: + return re.compile('.*charset="?([a-zA-Z0-9-]+)"?', re.MULTILINE | re.DOTALL).match(content_type_and_charset_header).group(1) + except: + return default_charset + + +class BodyParser(object): + + def __init__(self, content, content_type='text/plain; charset="us-ascii"', content_transfer_encoding=None): + self._content = content + self._content_type = content_type + self._content_transfer_encoding = content_transfer_encoding + + def parsed_content(self): + charset = _parse_charset_header(self._content_type) + text = self._serialize_for_parser(charset) + + decoded_body = self._parse_and_decode(text) + + return unicode(decoded_body, encoding=charset) + + def _parse_and_decode(self, text): + parsed_body = Parser().parsestr(text) + decoded_body = self._unwrap_content_transfer_encoding(parsed_body) + return decoded_body + + def _unwrap_content_transfer_encoding(self, parsed_body): + return parsed_body.get_payload(decode=True) + + def _serialize_for_parser(self, charset): + text = u'Content-Type: %s\n' % self._content_type + if self._content_transfer_encoding is not None: + text += u'Content-Transfer-Encoding: %s\n' % self._content_transfer_encoding + + text += u'\n' + encoded_text = text.encode(charset) + if isinstance(self._content, unicode): + try: + return encoded_text + self._content.encode(charset) + except UnicodeError, e: + logger.warn('Failed to encode content for charset %s. Ignoring invalid chars: %s' % (charset, e)) + return encoded_text + self._content.encode(charset, 'ignore') + else: + return encoded_text + self._content diff --git a/service/pixelated/adapter/mailstore/leap_mailstore.py b/service/pixelated/adapter/mailstore/leap_mailstore.py new file mode 100644 index 00000000..2754c624 --- /dev/null +++ b/service/pixelated/adapter/mailstore/leap_mailstore.py @@ -0,0 +1,413 @@ +# +# Copyright (c) 2015 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +import base64 +from email.header import decode_header +from email.utils import parseaddr +import quopri +from uuid import uuid4 + +import re +from leap.mail.adaptors.soledad import SoledadMailAdaptor, ContentDocWrapper +from twisted.internet import defer +from pixelated.adapter.mailstore.body_parser import BodyParser +from pixelated.adapter.mailstore.mailstore import MailStore, underscore_uuid +from leap.mail.mail import Message +from pixelated.adapter.model.mail import Mail, InputMail + + +class AttachmentInfo(object): + def __init__(self, ident, name, encoding): + self.ident = ident + self.name = name + self.encoding = encoding + + +class LeapMail(Mail): + + def __init__(self, mail_id, mailbox_name, headers=None, tags=set(), flags=set(), body=None, attachments=[]): + self._mail_id = mail_id + self._mailbox_name = mailbox_name + self._headers = headers if headers is not None else {} + self._body = body + self.tags = set(tags) # TODO test that asserts copy + self._flags = set(flags) # TODO test that asserts copy + self._attachments = attachments + + @property + def headers(self): + cpy = dict(self._headers) + + for name in set(self._headers.keys()).intersection(['To', 'Cc', 'Bcc']): + cpy[name] = self._headers[name].split(',') if self._headers[name] else [] + + return cpy + + @property + def ident(self): + return self._mail_id + + @property + def mail_id(self): + return self._mail_id + + @property + def body(self): + return self._body + + @property + def flags(self): + return self._flags + + @property + def mailbox_name(self): + return self._mailbox_name + + @property + def security_casing(self): + casing = dict(imprints=self._signature_information(), locks=[]) + if self._encrypted() == "decrypted": + casing["locks"] = [{"state": "valid"}] + return casing + + def _encrypted(self): + return self.headers.get("X-Leap-Encryption", "false") + + def _signature_information(self): + signature = self.headers.get("X-Leap-Signature", None) + if signature is None or signature.startswith("could not verify"): + return [{"state": "no_signature_information"}] + else: + if signature.startswith("valid"): + return [{"state": "valid", "seal": {"validity": "valid"}}] + else: + return [] + + @property + def raw(self): + result = u'' + for k, v in self._headers.items(): + content, encoding = decode_header(v)[0] + if encoding: + result += '%s: %s\n' % (k, unicode(content, encoding=encoding)) + else: + result += '%s: %s\n' % (k, v) + result += '\n' + + if self._body: + result = result + self._body + + return result + + def _decoded_header_utf_8(self, header_value): + if isinstance(header_value, list): + return self.remove_duplicates([self._decoded_header_utf_8(v) for v in header_value]) + elif header_value is not None: + def encode_chunk(content, encoding): + return unicode(content, encoding=encoding or 'ascii', errors='ignore') + + try: + encoded_chunks = [encode_chunk(content, encoding) for content, encoding in decode_header(header_value)] + return ' '.join(encoded_chunks) # decode_header strips whitespaces on all chunks, joining over ' ' is only a workaround, not a proper fix + except UnicodeEncodeError: + return unicode(header_value.encode('ascii', errors='ignore')) + + def as_dict(self): + return { + 'header': {k.lower(): self._decoded_header_utf_8(v) for k, v in self.headers.items()}, + 'ident': self._mail_id, + 'tags': self.tags, + 'status': list(self.status), + 'body': self._body, + 'security_casing': self.security_casing, + 'textPlainBody': self._body, + 'replying': self._replying_dict(), + 'mailbox': self._mailbox_name.lower(), + 'attachments': [{'ident': attachment.ident, 'name': attachment.name, 'encoding': attachment.encoding} for attachment in self._attachments] + } + + def _replying_dict(self): + result = {'single': None, 'all': {'to-field': [], 'cc-field': []}} + + sender_mail = self._decoded_header_utf_8(self.headers.get('Reply-To', self.headers.get('From'))) + # Issue #215: Fix for existing mails without any from address. + if sender_mail is None: + sender_mail = InputMail.FROM_EMAIL_ADDRESS + + recipients = self._decoded_header_utf_8(self._reply_recipient('To')) + if not self._parsed_mail_matches(sender_mail, InputMail.FROM_EMAIL_ADDRESS) or len(recipients) == 0: + recipients.append(sender_mail) + recipients = self.remove_duplicates(recipients) + ccs = self._decoded_header_utf_8(self._reply_recipient('Cc')) + + result['single'] = self._single_reply_recipient(recipients, sender_mail) + result['all']['to-field'] = recipients + result['all']['cc-field'] = ccs + return result + + def _single_reply_recipient(self, recipients, sender_mail): + """ + Currently the domain model expects only one single recipient for reply action. But it should support an array, + or even better: there should not be any conceptual difference between reply and reply all for this logic + """ + if self._parsed_mail_matches(sender_mail, InputMail.FROM_EMAIL_ADDRESS): + return recipients[0] + else: + return sender_mail + + def remove_duplicates(self, recipients): + return list(set(recipients)) + + def _reply_recipient(self, kind): + recipients = self.headers.get(kind, []) + if not recipients: + recipients = [] + + return [recipient for recipient in recipients if not self._parsed_mail_matches(recipient, InputMail.FROM_EMAIL_ADDRESS)] + + def _parsed_mail_matches(self, to_parse, expected): + if InputMail.FROM_EMAIL_ADDRESS is None: + return False + return parseaddr(self._decoded_header_utf_8(to_parse))[1] == expected + + @staticmethod + def from_dict(mail_dict): + # TODO: implement this method and also write tests for it + headers = {key.capitalize(): value for key, value in mail_dict.get('header', {}).items()} + headers['Date'] = date.mail_date_now() + body = mail_dict.get('body', '') + tags = set(mail_dict.get('tags', [])) + status = set(mail_dict.get('status', [])) + attachments = [] + + # mail_id, mailbox_name, headers=None, tags=set(), flags=set(), body=None, attachments=[] + return LeapMail(None, None, headers, tags, set(), body, attachments) + + +def _extract_filename(content_disposition): + match = re.compile('.*name=\"(.*)\".*').search(content_disposition) + filename = '' + if match: + filename = match.group(1) + return filename + + +class LeapMailStore(MailStore): + __slots__ = ('soledad') + + def __init__(self, soledad): + self.soledad = soledad + + @defer.inlineCallbacks + def get_mail(self, mail_id, include_body=False): + message = yield self._fetch_msg_from_soledad(mail_id) + if not _is_empty_message(message): + leap_mail = yield self._leap_message_to_leap_mail(mail_id, message, include_body) + else: + leap_mail = None + + defer.returnValue(leap_mail) + + def get_mails(self, mail_ids): + deferreds = [] + for mail_id in mail_ids: + deferreds.append(self.get_mail(mail_id)) + + return defer.gatherResults(deferreds, consumeErrors=True) + + @defer.inlineCallbacks + def get_mail_attachment(self, attachment_id): + results = yield self.soledad.get_from_index('by-type-and-payloadhash', 'cnt', attachment_id) if attachment_id else [] + if len(results): + content = ContentDocWrapper(**results[0].content) + defer.returnValue({'content-type': content.content_type, 'content': self._try_decode( + content.raw, content.content_transfer_encoding)}) + else: + raise ValueError('No attachment with id %s found!' % attachment_id) + + def _try_decode(self, raw, encoding): + encoding = encoding.lower() + if encoding == 'base64': + data = base64.decodestring(raw) + elif encoding == 'quoted-printable': + data = quopri.decodestring(raw) + else: + data = str(raw) + + return bytearray(data) + + @defer.inlineCallbacks + def update_mail(self, mail): + message = yield self._fetch_msg_from_soledad(mail.mail_id) + message.get_wrapper().set_tags(tuple(mail.tags)) + message.get_wrapper().set_flags(tuple(mail.flags)) + yield self._update_mail(message) # TODO assert this is yielded (otherwise asynchronous) + + @defer.inlineCallbacks + def all_mails(self): + mdocs = yield self.soledad.get_from_index('by-type', 'meta') + + mail_ids = map(lambda doc: doc.doc_id, mdocs) + + mails = yield self.get_mails(mail_ids) + defer.returnValue(mails) + + @defer.inlineCallbacks + def add_mailbox(self, mailbox_name): + mailbox = yield self._get_or_create_mailbox(mailbox_name) + defer.returnValue(mailbox) + + @defer.inlineCallbacks + def get_mailbox_names(self): + mbox_map = set((yield self._mailbox_uuid_to_name_map()).values()) + + defer.returnValue(mbox_map.union({'INBOX'})) + + @defer.inlineCallbacks + def _mailbox_uuid_to_name_map(self): + map = {} + mbox_docs = yield self.soledad.get_from_index('by-type', 'mbox') + for doc in mbox_docs: + map[underscore_uuid(doc.content.get('uuid'))] = doc.content.get('mbox') + + defer.returnValue(map) + + @defer.inlineCallbacks + def add_mail(self, mailbox_name, raw_msg): + mailbox = yield self._get_or_create_mailbox(mailbox_name) + message = SoledadMailAdaptor().get_msg_from_string(Message, raw_msg) + message.get_wrapper().set_mbox_uuid(mailbox.uuid) + + yield SoledadMailAdaptor().create_msg(self.soledad, message) + + # add behavious from insert_mdoc_id from mail.py + mail = yield self._leap_message_to_leap_mail(message.get_wrapper().mdoc.doc_id, message, include_body=True) # TODO test that asserts include_body + defer.returnValue(mail) + + @defer.inlineCallbacks + def delete_mail(self, mail_id): + message = yield self._fetch_msg_from_soledad(mail_id) + yield message.get_wrapper().delete(self.soledad) + + @defer.inlineCallbacks + def get_mailbox_mail_ids(self, mailbox_name): + mailbox = yield self._get_or_create_mailbox(mailbox_name) + fdocs = yield self.soledad.get_from_index('by-type-and-mbox-uuid', 'flags', underscore_uuid(mailbox.uuid)) + + mail_ids = map(lambda doc: _fdoc_id_to_mdoc_id(doc.doc_id), fdocs) + + defer.returnValue(mail_ids) + + @defer.inlineCallbacks + def delete_mailbox(self, mailbox_name): + mbx_wrapper = yield self._get_or_create_mailbox(mailbox_name) + yield SoledadMailAdaptor().delete_mbox(self.soledad, mbx_wrapper) + + @defer.inlineCallbacks + def copy_mail_to_mailbox(self, mail_id, mailbox_name): + message = yield self._fetch_msg_from_soledad(mail_id, load_body=True) + mailbox = yield self._get_or_create_mailbox(mailbox_name) + copy_wrapper = yield message.get_wrapper().copy(self.soledad, mailbox.uuid) + + leap_message = Message(copy_wrapper) + + mail = yield self._leap_message_to_leap_mail(copy_wrapper.mdoc.doc_id, leap_message, include_body=False) + + defer.returnValue(mail) + + @defer.inlineCallbacks + def move_mail_to_mailbox(self, mail_id, mailbox_name): + mail_copy = yield self.copy_mail_to_mailbox(mail_id, mailbox_name) + yield self.delete_mail(mail_id) + defer.returnValue(mail_copy) + + def _update_mail(self, message): + return message.get_wrapper().update(self.soledad) + + @defer.inlineCallbacks + def _leap_message_to_leap_mail(self, mail_id, message, include_body): + if include_body: + # TODO use body from message if available + body = yield self._raw_message_body(message) + else: + body = None + + # fetch mailbox name by mbox_uuid + mbox_uuid = message.get_wrapper().fdoc.mbox_uuid + mbox_name = yield self._mailbox_name_from_uuid(mbox_uuid) + + mail = LeapMail(mail_id, mbox_name, message.get_wrapper().hdoc.headers, set(message.get_tags()), set(message.get_flags()), body=body, attachments=self._extract_attachment_info_from(message)) # TODO assert flags are passed on + + defer.returnValue(mail) + + @defer.inlineCallbacks + def _raw_message_body(self, message): + content_doc = (yield message.get_wrapper().get_body(self.soledad)) + parser = BodyParser(content_doc.raw, content_type=content_doc.content_type, content_transfer_encoding=content_doc.content_transfer_encoding) + defer.returnValue(parser.parsed_content()) + + @defer.inlineCallbacks + def _mailbox_name_from_uuid(self, uuid): + map = (yield self._mailbox_uuid_to_name_map()) + defer.returnValue(map.get(uuid, '')) + + @defer.inlineCallbacks + def _get_or_create_mailbox(self, mailbox_name): + mailbox_name_upper = mailbox_name.upper() + mbx = yield SoledadMailAdaptor().get_or_create_mbox(self.soledad, mailbox_name_upper) + if mbx.uuid is None: + mbx.uuid = str(uuid4()) + yield mbx.update(self.soledad) + defer.returnValue(mbx) + + def _fetch_msg_from_soledad(self, mail_id, load_body=False): + return SoledadMailAdaptor().get_msg_from_mdoc_id(Message, self.soledad, mail_id, get_cdocs=load_body) + + @defer.inlineCallbacks + def _dump_soledad(self): + gen, docs = yield self.soledad.get_all_docs() + for doc in docs: + print '\n%s\n' % doc + + def _extract_attachment_info_from(self, message): + wrapper = message.get_wrapper() + part_maps = wrapper.hdoc.part_map + return self._extract_part_map(part_maps) + + def _extract_part_map(self, part_maps): + result = [] + + for nr, part_map in part_maps.items(): + if 'headers' in part_map and 'phash' in part_map: + headers = {header[0]: header[1] for header in part_map['headers']} + phash = part_map['phash'] + if 'Content-Disposition' in headers: + disposition = headers['Content-Disposition'] + if 'attachment' in disposition: + filename = _extract_filename(disposition) + encoding = headers.get('Content-Transfer-Encoding', None) + result.append(AttachmentInfo(phash, filename, encoding)) + if 'part_map' in part_map: + result += self._extract_part_map(part_map['part_map']) + + return result + + +def _is_empty_message(message): + return (message is None) or (message.get_wrapper().mdoc.doc_id is None) + + +def _fdoc_id_to_mdoc_id(fdoc_id): + return 'M' + fdoc_id[1:] diff --git a/service/pixelated/adapter/mailstore/mailstore.py b/service/pixelated/adapter/mailstore/mailstore.py new file mode 100644 index 00000000..60716dfe --- /dev/null +++ b/service/pixelated/adapter/mailstore/mailstore.py @@ -0,0 +1,60 @@ +# +# Copyright (c) 2015 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. + + +class MailStore(object): + def get_mail(self, mail_id): + pass + + def get_mail_attachment(self, attachment_id): + pass + + def get_mails(self, mail_ids): + pass + + def all_mails(self): + pass + + def delete_mail(self, mail_id): + pass + + def update_mail(self, mail): + pass + + def add_mail(self, mailbox_name, mail): + pass + + def get_mailbox_names(self): + pass + + def add_mailbox(self, mailbox_name): + pass + + def delete_mailbox(self, mailbox_name): + pass + + def get_mailbox_mail_ids(self, mailbox_name): + pass + + def copy_mail_to_mailbox(self, mail_id, mailbox_name): + pass + + def move_mail_to_mailbox(self, mail_id, mailbox_name): + pass + + +def underscore_uuid(uuid): + return uuid.replace('-', '_') diff --git a/service/pixelated/adapter/mailstore/maintenance/__init__.py b/service/pixelated/adapter/mailstore/maintenance/__init__.py new file mode 100644 index 00000000..edc442c2 --- /dev/null +++ b/service/pixelated/adapter/mailstore/maintenance/__init__.py @@ -0,0 +1,99 @@ +# +# Copyright (c) 2015 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +from leap.keymanager.keys import KEY_TYPE_KEY, KEY_PRIVATE_KEY, KEY_ID_KEY, KEY_ADDRESS_KEY +from leap.keymanager.openpgp import OpenPGPKey + +from twisted.internet import defer +import logging + + +TYPE_OPENPGP_KEY = 'OpenPGPKey' +TYPE_OPENPGP_ACTIVE = 'OpenPGPKey-active' + +KEY_DOC_TYPES = {TYPE_OPENPGP_ACTIVE, TYPE_OPENPGP_KEY} + +logger = logging.getLogger(__name__) + + +def _is_key_doc(doc): + return doc.content.get(KEY_TYPE_KEY, None) in KEY_DOC_TYPES + + +def _is_private_key_doc(doc): + return _is_key_doc(doc) and doc.content.get(KEY_PRIVATE_KEY, False) + + +def _is_active_key_doc(doc): + return _is_key_doc(doc) and doc.content.get(KEY_TYPE_KEY, None) == TYPE_OPENPGP_ACTIVE + + +def _is_public_key(doc): + return _is_key_doc(doc) and not doc.content.get(KEY_PRIVATE_KEY, False) + + +def _key_id(doc): + return doc.content.get(KEY_ID_KEY, None) + + +def _address(doc): + return doc.content.get(KEY_ADDRESS_KEY, None) + + +class SoledadMaintenance(object): + def __init__(self, soledad): + self._soledad = soledad + + @defer.inlineCallbacks + def repair(self): + _, docs = yield self._soledad.get_all_docs() + + private_key_ids = self._key_ids_with_private_key(docs) + + for doc in docs: + if _is_key_doc(doc) and _key_id(doc) not in private_key_ids: + logger.warn('Deleting doc %s for key %s of <%s>' % (doc.doc_id, _key_id(doc), _address(doc))) + yield self._soledad.delete_doc(doc) + + yield self._repair_missing_active_docs(docs, private_key_ids) + + @defer.inlineCallbacks + def _repair_missing_active_docs(self, docs, private_key_ids): + missing = self._missing_active_docs(docs, private_key_ids) + for key_id in missing: + emails = self._emails_for_key_id(docs, key_id) + for email in emails: + logger.warn('Re-creating active doc for key %s, email %s' % (key_id, email)) + yield self._soledad.create_doc_from_json(OpenPGPKey(email, key_id=key_id, private=False).get_active_json(email)) + + def _key_ids_with_private_key(self, docs): + return [doc.content[KEY_ID_KEY] for doc in docs if _is_private_key_doc(doc)] + + def _missing_active_docs(self, docs, private_key_ids): + active_doc_ids = self._active_docs_for_key_id(docs) + + return set([private_key_id for private_key_id in private_key_ids if private_key_id not in active_doc_ids]) + + def _emails_for_key_id(self, docs, key_id): + for doc in docs: + if _is_private_key_doc(doc) and _key_id(doc) == key_id: + email = _address(doc) + if isinstance(email, list): + return email + else: + return [email] + + def _active_docs_for_key_id(self, docs): + return [doc.content[KEY_ID_KEY] for doc in docs if _is_active_key_doc(doc) and _is_public_key(doc)] diff --git a/service/pixelated/adapter/mailstore/searchable_mailstore.py b/service/pixelated/adapter/mailstore/searchable_mailstore.py new file mode 100644 index 00000000..0c5310eb --- /dev/null +++ b/service/pixelated/adapter/mailstore/searchable_mailstore.py @@ -0,0 +1,80 @@ +# +# Copyright (c) 2015 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +from twisted.internet import defer +from types import FunctionType +from pixelated.adapter.mailstore import MailStore + + +class SearchableMailStore(object): # implementes MailStore + + def __init__(self, delegate, search_engine): + self._delegate = delegate + self._search_engine = search_engine + + @classmethod + def _create_delegator(cls, method_name): + def delegator(self, *args, **kw): + return getattr(self._delegate, method_name)(*args, **kw) + + setattr(cls, method_name, delegator) + + @defer.inlineCallbacks + def add_mail(self, mailbox_name, mail): + stored_mail = yield self._delegate.add_mail(mailbox_name, mail) + self._search_engine.index_mail(stored_mail) + defer.returnValue(stored_mail) + + @defer.inlineCallbacks + def delete_mail(self, mail_id): + yield self._delegate.delete_mail(mail_id) + self._search_engine.remove_from_index(mail_id) + + @defer.inlineCallbacks + def update_mail(self, mail): + yield self._delegate.update_mail(mail) + self._search_engine.index_mail(mail) + + @defer.inlineCallbacks + def move_mail_to_mailbox(self, mail_id, mailbox_name): + moved_mail = yield self._delegate.move_mail_to_mailbox(mail_id, mailbox_name) + self._search_engine.remove_from_index(mail_id) + self._search_engine.index_mail(moved_mail) + defer.returnValue(moved_mail) + + @defer.inlineCallbacks + def copy_mail_to_mailbox(self, mail_id, mailbox_name): + copied_mail = yield self._delegate.copy_mail_to_mailbox(mail_id, mailbox_name) + self._search_engine.index_mail(copied_mail) + defer.returnValue(copied_mail) + + def delete_mailbox(self, mailbox_name): + raise NotImplementedError() + + def __getattr__(self, name): + """ + Acts like method missing. If a method of MailStore is not implemented in this class, + a delegate method is created. + + :param name: attribute name + :return: method or attribute + """ + methods = ([key for key, value in MailStore.__dict__.items() if type(value) == FunctionType]) + + if name in methods: + SearchableMailStore._create_delegator(name) + return super(SearchableMailStore, self).__getattribute__(name) + else: + raise NotImplementedError('No attribute %s' % name) diff --git a/service/pixelated/adapter/model/mail.py b/service/pixelated/adapter/model/mail.py index 464e0343..b89e511a 100644 --- a/service/pixelated/adapter/model/mail.py +++ b/service/pixelated/adapter/model/mail.py @@ -21,21 +21,38 @@ import dateutil.parser as dateparser from uuid import uuid4 from email import message_from_file from email.mime.text import MIMEText -from email.header import decode_header +from email.header import decode_header, Header from email.MIMEMultipart import MIMEMultipart from pycryptopp.hash import sha256 -from leap.mail.imap.fields import fields +from leap.mail.adaptors import soledad_indexes as fields import leap.mail.walk as walk from pixelated.adapter.model.status import Status from pixelated.support import date from pixelated.support.functional import compact +from twisted.internet import defer + logger = logging.getLogger(__name__) +TYPE_KEY = 'type' +CONTENT_HASH_KEY = 'chash' +HEADERS_KEY = 'headers' +DATE_KEY = 'date' +SUBJECT_KEY = 'subject' +PARTS_MAP_KEY = 'part_map' +BODY_KEY = 'body' +MSGID_KEY = 'msgid' +MULTIPART_KEY = 'multi' +SIZE_KEY = 'size' + class Mail(object): @property + def from_sender(self): + return self.headers['From'] + + @property def to(self): return self.headers['To'] @@ -48,6 +65,10 @@ class Mail(object): return self.headers['Bcc'] @property + def subject(self): + return self.headers['Subject'] + + @property def date(self): return self.headers['Date'] @@ -61,7 +82,17 @@ class Mail(object): @property def mailbox_name(self): - return self.fdoc.content.get('mbox') + # FIXME mbox is no longer available, instead we now have mbox_uuid + return self.fdoc.content.get('mbox', 'INBOX') + + def _encode_header_value_list(self, header_value_list): + return [self._encode_header_value(v) for v in header_value_list] + + def _encode_header_value(self, header_value): + if isinstance(header_value, unicode): + return str(Header(header_value, 'utf-8')) + else: + return str(header_value) @property def _mime_multipart(self): @@ -69,7 +100,10 @@ class Mail(object): return self._mime mime = MIMEMultipart() for key, value in self.headers.items(): - mime[str(key)] = str(value) + if isinstance(value, list): + mime[str(key)] = ', '.join(self._encode_header_value_list(value)) + else: + mime[str(key)] = self._encode_header_value(value) try: body_to_use = self.body @@ -128,19 +162,19 @@ class InputMail(Mail): return self._fd fd = {} - fd[fields.MBOX_KEY] = mailbox - fd[fields.UID_KEY] = next_uid - fd[fields.CONTENT_HASH_KEY] = self._get_chash() - fd[fields.SIZE_KEY] = len(self.raw) - fd[fields.MULTIPART_KEY] = True - fd[fields.RECENT_KEY] = True - fd[fields.TYPE_KEY] = fields.TYPE_FLAGS_VAL - fd[fields.FLAGS_KEY] = Status.to_flags(self._status) + fd[fields.MBOX] = mailbox + fd[fields.MBOX_UUID] = next_uid + fd[fields.CONTENT_HASH] = self._get_chash() + fd[SIZE_KEY] = len(self.raw) + fd[MULTIPART_KEY] = True + fd[fields.RECENT] = True + fd[fields.TYPE] = fields.FLAGS + fd[fields.FLAGS] = Status.to_flags(self._status) self._fd = fd return fd def _get_body_phash(self): - return walk.get_body_phash_multi(walk.get_payloads(self._mime_multipart)) + return walk.get_body_phash(self._mime_multipart) def _hdoc(self): if self._hd: @@ -151,15 +185,15 @@ class InputMail(Mail): headers['From'] = InputMail.FROM_EMAIL_ADDRESS hd = {} - hd[fields.HEADERS_KEY] = headers - hd[fields.DATE_KEY] = headers['Date'] - hd[fields.CONTENT_HASH_KEY] = self._get_chash() - hd[fields.MSGID_KEY] = '' - hd[fields.MULTIPART_KEY] = True - hd[fields.SUBJECT_KEY] = headers.get('Subject') - hd[fields.TYPE_KEY] = fields.TYPE_HEADERS_VAL - hd[fields.BODY_KEY] = self._get_body_phash() - hd[fields.PARTS_MAP_KEY] = \ + hd[HEADERS_KEY] = headers + hd[DATE_KEY] = headers['Date'] + hd[CONTENT_HASH_KEY] = self._get_chash() + hd[MSGID_KEY] = '' + hd[MULTIPART_KEY] = True + hd[SUBJECT_KEY] = headers.get('Subject') + hd[TYPE_KEY] = fields.HEADERS + hd[BODY_KEY] = self._get_body_phash() + hd[PARTS_MAP_KEY] = \ walk.walk_msg_tree(walk.get_parts(self._mime_multipart), body_phash=self._get_body_phash())['part_map'] self._hd = hd @@ -172,12 +206,15 @@ class InputMail(Mail): mime_multipart = MIMEMultipart() for header in ['To', 'Cc', 'Bcc']: - if self.headers[header]: + if self.headers.get(header): mime_multipart[header] = ", ".join(self.headers[header]) - if self.headers['Subject']: + if self.headers.get('Subject'): mime_multipart['Subject'] = self.headers['Subject'] + if self.headers.get('From'): + mime_multipart['From'] = self.headers['From'] + mime_multipart['Date'] = self.headers['Date'] if type(self.body) is list: for part in self.body: @@ -207,13 +244,10 @@ class InputMail(Mail): input_mail = InputMail() input_mail.headers = {key.capitalize(): value for key, value in mail_dict.get('header', {}).items()} - # XXX this is overriding the property in PixelatedMail - input_mail.headers['Date'] = date.iso_now() + input_mail.headers['Date'] = date.mail_date_now() - # XXX this is overriding the property in PixelatedMail input_mail.body = mail_dict.get('body', '') - # XXX this is overriding the property in the PixelatedMail input_mail.tags = set(mail_dict.get('tags', [])) input_mail._status = set(mail_dict.get('status', [])) @@ -222,292 +256,18 @@ class InputMail(Mail): @staticmethod def from_python_mail(mail): input_mail = InputMail() - input_mail.headers = {key.capitalize(): value for key, value in mail.items()} - input_mail.headers['Date'] = date.iso_now() - input_mail.headers['Subject'] = mail['Subject'] - input_mail.headers['To'] = InputMail.FROM_EMAIL_ADDRESS - input_mail._mime = MIMEMultipart() + input_mail.headers = {unicode(key.capitalize()): unicode(value) for key, value in mail.items()} + input_mail.headers[u'Date'] = unicode(date.mail_date_now()) + input_mail.headers[u'To'] = [u''] + for payload in mail.get_payload(): - input_mail._mime.attach(payload) + input_mail._mime_multipart.attach(payload) if payload.get_content_type() == 'text/plain': - input_mail.body = payload.as_string() + input_mail.body = unicode(payload.as_string()) + input_mail._mime = input_mail.to_mime_multipart() return input_mail -class PixelatedMail(Mail): - - @staticmethod - def from_soledad(fdoc, hdoc, bdoc, parts=None, soledad_querier=None): - mail = PixelatedMail() - mail.parts = parts - mail.boundary = str(uuid4()).replace('-', '') - mail.bdoc = bdoc - mail.fdoc = fdoc - mail.hdoc = hdoc - mail.querier = soledad_querier - mail._mime = None - return mail - - def _decode_part(self, part): - encoding = part['headers'].get('Content-Transfer-Encoding', '') - content_type = self._parse_charset_header(part['headers'].get('Content-Type')) - - try: - decoding_func = self._decoding_function_for_encoding(encoding) - return self._decode_content_with_fallback(part['content'], decoding_func, content_type) - except Exception: - logger.error('Failed to decode mail part with:') - logger.error('Content-Transfer-Encoding: %s' % encoding) - logger.error('Content-Type: %s' % part['headers'].get('Content-Type')) - raise - - def _decoding_function_for_encoding(self, encoding): - decoding_map = { - 'quoted-printable': lambda content, content_type: content.decode('quopri').decode(content_type), - 'base64': lambda content, content_type: content.decode('base64').decode('utf-8'), - '7bit': lambda content, content_type: content.encode(content_type), - '8bit': lambda content, content_type: content.encode(content_type) - } - if encoding in decoding_map: - return decoding_map[encoding] - else: - return decoding_map['8bit'] - - def _decode_content_with_fallback(self, content, decode_func, content_type): - try: - return decode_func(content, content_type) - # return content.encode(content_type) - except ValueError: - return content.encode('ascii', 'ignore') - - @property - def alternatives(self): - return self.parts.get('alternatives') - - @property - def text_plain_body(self): - if self.parts and len(self.alternatives) >= 1: - return self._decode_part(self.alternatives[0]) - else: - return self.bdoc.content['raw'] # plain - - @property - def html_body(self): - if self.parts and len(self.alternatives) > 1: - html_parts = [e for e in self.alternatives if re.match('text/html', e['headers'].get('Content-Type', ''))] - if len(html_parts): - return self._decode_part(html_parts[0]) - - @property - def headers(self): - _headers = { - 'To': [], - 'Cc': [], - 'Bcc': [] - } - hdoc_headers = self.hdoc.content['headers'] - - for header in ['To', 'Cc', 'Bcc']: - header_value = self._decode_header(hdoc_headers.get(header)) - if not header_value: - continue - _headers[header] = header_value if type(header_value) is list else header_value.split(',') - _headers[header] = [head.strip() for head in compact(_headers[header])] - - for header in ['From', 'Subject']: - _headers[header] = self._decode_header(hdoc_headers.get(header)) - - try: - _headers['Date'] = self._get_date() - except Exception: - _headers['Date'] = date.iso_now() - - if self.parts and len(self.parts['alternatives']) > 1: - _headers['content_type'] = 'multipart/alternative; boundary="%s"' % self.boundary - elif self.hdoc.content['headers'].get('Content-Type'): - _headers['content_type'] = hdoc_headers.get('Content-Type') - - if hdoc_headers.get('Reply-To'): - _headers['Reply-To'] = hdoc_headers.get('Reply-To') - - return _headers - - def _decode_header_with_fallback(self, entry): - try: - return decode_header(entry)[0][0] - except Exception: - return entry.encode('ascii', 'ignore') - - def _decode_header(self, header): - if not header: - return None - if isinstance(header, list): - return [self._decode_header_with_fallback(entry) for entry in header] - else: - return self._decode_header_with_fallback(header) - - def _get_date(self): - date = self.hdoc.content.get('date', None) - try: - if not date: - received = self.hdoc.content.get('received', None) - if received: - date = received.split(";")[-1].strip() - else: - # we can't get a date for this mail, so lets just use now - logger.warning('Encountered a mail with missing date and received header fields. ID %s' % self.fdoc.content.get('uid', None)) - date = date.iso_now() - return dateparser.parse(date).isoformat() - except (ValueError, TypeError): - date = date.iso_now() - return dateparser.parse(date).isoformat() - - @property - def security_casing(self): - casing = {"imprints": [], "locks": []} - casing["imprints"] = self.signature_information - if self.encrypted == "true": - casing["locks"] = [{"state": "valid"}] - elif self.encrypted == "fail": - casing["locks"] = [{"state": "failure"}] - return casing - - @property - def tags(self): - _tags = self.fdoc.content.get('tags', '[]') - return set(_tags) if type(_tags) is list or type(_tags) is set else set(json.loads(_tags)) - - @property - def ident(self): - return self.fdoc.content.get('chash') - - @property - def mailbox_name(self): - return self.fdoc.content.get('mbox') - - @property - def is_recent(self): - return Status('recent') in self.status - - @property - def uid(self): - return self.fdoc.content['uid'] - - @property - def flags(self): - return self.fdoc.content['flags'] - - def save(self): - return self.querier.save_mail(self) - - def set_mailbox(self, mailbox_name): - self.fdoc.content['mbox'] = mailbox_name - - def remove_all_tags(self): - self.update_tags(set([])) - - def update_tags(self, tags): - self._persist_mail_tags(tags) - return self.tags - - def mark_as_read(self): - if Status.SEEN in self.flags: - return self - self.flags.append(Status.SEEN) - self.save() - return self - - def mark_as_unread(self): - if Status.SEEN in self.flags: - self.flags.remove(Status.SEEN) - self.save() - return self - - def mark_as_not_recent(self): - if Status.RECENT in self.flags: - self.flags.remove(Status.RECENT) - self.save() - return self - - def _persist_mail_tags(self, current_tags): - self.fdoc.content['tags'] = json.dumps(list(current_tags)) - self.save() - - def has_tag(self, tag): - return tag in self.tags - - @property - def signature_information(self): - signature = self.hdoc.content["headers"].get("X-Leap-Signature", None) - if signature is None or signature.startswith("could not verify"): - return [{"state": "no_signature_information"}] - else: - if signature.startswith("valid"): - return [{"state": "valid", "seal": {"validity": "valid"}}] - else: - return [] - - @property - def encrypted(self): - return self.hdoc.content["headers"].get("X-Pixelated-encryption-status", "false") - - @property - def bounced(self): - content_type = self.hdoc.content["headers"].get("Content-Type", '') - if re.compile('delivery-status').search(content_type): - bounce_recipient = self._extract_bounced_address(self.hdoc.content) - bounce_daemon = self.headers["From"] - return [bounce_recipient, bounce_daemon] if bounce_recipient else False - - return False - - def _extract_bounced_address(self, part): - part_header = dict(part.get('headers', {})) - if 'Final-Recipient' in part_header: - if self._bounce_permanent(part_header): - return part_header['Final-Recipient'].split(';')[1].strip() - else: - return False - elif 'part_map' in part: - for subpart in part['part_map'].values(): - result = self._extract_bounced_address(subpart) - if result: - return result - else: - continue - return False - - def _bounce_permanent(self, part_headers): - status = part_headers.get('Status', '') - return status.startswith('5') - - def as_dict(self): - dict_mail = {'header': {k.lower(): v for k, v in self.headers.items()}, - 'ident': self.ident, - 'tags': list(self.tags), - 'status': list(self.status), - 'security_casing': self.security_casing, - 'textPlainBody': self.text_plain_body, - 'htmlBody': self.html_body, - 'mailbox': self.mailbox_name.lower(), - 'attachments': self.parts['attachments'] if self.parts else []} - dict_mail['replying'] = {'single': None, 'all': {'to-field': [], 'cc-field': []}} - - sender_mail = self.headers.get('Reply-To', self.headers.get('From')) - # Issue #215: Fix for existing mails without any from address. - if sender_mail is None: - sender_mail = InputMail.FROM_EMAIL_ADDRESS - - recipients = [recipient for recipient in self.headers['To'] if recipient != InputMail.FROM_EMAIL_ADDRESS] - recipients.append(sender_mail) - ccs = [cc for cc in self.headers['Cc'] if cc != InputMail.FROM_EMAIL_ADDRESS] - - dict_mail['replying']['single'] = sender_mail - dict_mail['replying']['all']['to-field'] = recipients - dict_mail['replying']['all']['cc-field'] = ccs - return dict_mail - - def welcome_mail(): current_path = os.path.dirname(os.path.abspath(__file__)) with open(os.path.join(current_path, '..', '..', 'assets', 'welcome.mail')) as mail_template_file: diff --git a/service/pixelated/adapter/search/__init__.py b/service/pixelated/adapter/search/__init__.py index b8d3e7ca..56ab2255 100644 --- a/service/pixelated/adapter/search/__init__.py +++ b/service/pixelated/adapter/search/__init__.py @@ -18,6 +18,8 @@ from pixelated.support.encrypted_file_storage import EncryptedFileStorage import os import re +import dateutil.parser +import time from pixelated.adapter.model.status import Status from pixelated.adapter.search.contacts import contacts_suggestions from whoosh.index import FileIndex @@ -27,7 +29,6 @@ from whoosh.qparser import MultifieldParser from whoosh.writing import AsyncWriter from whoosh import sorting from pixelated.support.functional import unique -from pixelated.support.date import milliseconds import traceback @@ -102,7 +103,6 @@ class SearchEngine(object): to=KEYWORD(stored=False, commas=True), cc=KEYWORD(stored=False, commas=True), bcc=KEYWORD(stored=False, commas=True), - bounced=KEYWORD(stored=False, commas=True), subject=TEXT(stored=False), date=NUMERIC(stored=False, sortable=True, bits=64, signed=False), body=TEXT(stored=False), @@ -121,32 +121,38 @@ class SearchEngine(object): def _index_mail(self, writer, mail): mdict = mail.as_dict() header = mdict['header'] - tags = mdict.get('tags', []) - tags.append(mail.mailbox_name.lower()) - bounced = mail.bounced if mail.bounced else [''] + tags = set(mdict.get('tags', {})) + tags.add(mail.mailbox_name.lower()) index_data = { - 'sender': self._unicode_header_field(header.get('from', '')), - 'subject': self._unicode_header_field(header.get('subject', '')), - 'date': milliseconds(header.get('date', '')), - 'to': u','.join([h.decode('utf-8') for h in header.get('to', [''])]), - 'cc': u','.join([h.decode('utf-8') for h in header.get('cc', [''])]), - 'bcc': u','.join([h.decode('utf-8') for h in header.get('bcc', [''])]), + 'sender': self._empty_string_to_none(header.get('from', '')), + 'subject': self._empty_string_to_none(header.get('subject', '')), + 'date': self._format_utc_integer(header.get('date', '')), + 'to': self._format_recipient(header, 'to'), + 'cc': self._format_recipient(header, 'cc'), + 'bcc': self._format_recipient(header, 'bcc'), 'tag': u','.join(unique(tags)), - 'bounced': u','.join(bounced), - 'body': unicode(mdict['textPlainBody']), + 'body': unicode(mdict['textPlainBody'] if 'textPlainBody' in mdict else mdict['body']), 'ident': unicode(mdict['ident']), 'flags': unicode(','.join(unique(mail.flags))), - 'raw': unicode(mail.raw.decode('utf-8')) + 'raw': unicode(mail.raw) } writer.update_document(**index_data) - def _unicode_header_field(self, field_value): + def _format_utc_integer(self, date): + timetuple = dateutil.parser.parse(date).utctimetuple() + return time.strftime('%s', timetuple) + + def _format_recipient(self, headers, name): + list = headers.get(name, ['']) + return u','.join(list) if list else u'' + + def _empty_string_to_none(self, field_value): if not field_value: return None - - return unicode(field_value.decode('utf-8')) + else: + return field_value def index_mails(self, mails, callback=None): try: diff --git a/service/pixelated/adapter/search/contacts.py b/service/pixelated/adapter/search/contacts.py index 0dfeb15b..0729e146 100644 --- a/service/pixelated/adapter/search/contacts.py +++ b/service/pixelated/adapter/search/contacts.py @@ -31,21 +31,12 @@ def address_duplication_filter(contacts): return contacts_by_mail.values() -def bounced_addresses_filter(searcher, contacts): - query = QueryParser('bounced', searcher.schema).parse('*') - bounced_addresses = searcher.search(query, - limit=None, - groupedby=sorting.FieldFacet('bounced', - allow_overlap=True)).groups() - return set(contacts) - set(flatten([bounced_addresses])) - - def extract_mail_address(text): return parseaddr(text)[1] def contacts_suggestions(query, searcher): - return address_duplication_filter(bounced_addresses_filter(searcher, search_addresses(searcher, query))) if query else [] + return address_duplication_filter(search_addresses(searcher, query)) if query else [] def search_addresses(searcher, query): diff --git a/service/pixelated/adapter/search/index_storage_key.py b/service/pixelated/adapter/search/index_storage_key.py new file mode 100644 index 00000000..b2761849 --- /dev/null +++ b/service/pixelated/adapter/search/index_storage_key.py @@ -0,0 +1,42 @@ +# +# Copyright (c) 2015 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +import base64 +from twisted.internet import defer +import os + + +class SearchIndexStorageKey(object): + __slots__ = '_soledad' + + def __init__(self, soledad): + self._soledad = soledad + + @defer.inlineCallbacks + def get_or_create_key(self): + docs = yield self._soledad.get_from_index('by-type', 'index_key') + + if len(docs): + key = docs[0].content['value'] + else: + key = self._new_index_key() + yield self._store_key_in_soledad(key) + defer.returnValue(key) + + def _new_index_key(self): + return os.urandom(64) # 32 for encryption, 32 for hmac + + def _store_key_in_soledad(self, index_key): + return self._soledad.create_doc(dict(type='index_key', value=base64.encodestring(index_key))) diff --git a/service/pixelated/adapter/services/draft_service.py b/service/pixelated/adapter/services/draft_service.py index c8df0a05..65794f04 100644 --- a/service/pixelated/adapter/services/draft_service.py +++ b/service/pixelated/adapter/services/draft_service.py @@ -13,19 +13,36 @@ # # You should have received a copy of the GNU Affero General Public License # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +from twisted.internet import defer +from pixelated.adapter.errors import DuplicatedDraftException class DraftService(object): - __slots__ = '_mailboxes' + __slots__ = '_mail_store' - def __init__(self, mailboxes): - self._mailboxes = mailboxes + def __init__(self, mail_store): + self._mail_store = mail_store + @defer.inlineCallbacks def create_draft(self, input_mail): - pixelated_mail = self._mailboxes.drafts.add(input_mail) - return pixelated_mail + mail = yield self._mail_store.add_mail('DRAFTS', input_mail.raw) + defer.returnValue(mail) + # pixelated_mail = yield (yield self._mailboxes.drafts).add(input_mail) + # defer.returnValue(pixelated_mail) + @defer.inlineCallbacks def update_draft(self, ident, input_mail): - pixelated_mail = self.create_draft(input_mail) - self._mailboxes.drafts.remove(ident) - return pixelated_mail + new_draft = yield self.create_draft(input_mail) + try: + yield self._mail_store.delete_mail(ident) + defer.returnValue(new_draft) + except Exception as error: + errorMessage = error.args[0].getErrorMessage() + + if errorMessage == 'Need to create doc before deleting': + yield self._mail_store.delete_mail(new_draft.ident) + raise DuplicatedDraftException(errorMessage) + + # pixelated_mail = yield self.create_draft(input_mail) + # yield (yield self._mailboxes.drafts).remove(ident) + # defer.returnValue(pixelated_mail) diff --git a/service/pixelated/adapter/services/feedback_service.py b/service/pixelated/adapter/services/feedback_service.py new file mode 100644 index 00000000..5200a9ff --- /dev/null +++ b/service/pixelated/adapter/services/feedback_service.py @@ -0,0 +1,20 @@ +import os +import requests + + +class FeedbackService(object): + FEEDBACK_URL = os.environ.get('FEEDBACK_URL') + + def __init__(self, leap_session): + self.leap_session = leap_session + + def open_ticket(self, feedback): + account_mail = self.leap_session.account_email() + data = { + "ticket[comments_attributes][0][body]": feedback, + "ticket[subject]": "Feedback user-agent from {0}".format(account_mail), + "ticket[email]": account_mail, + "ticket[regarding_user]": account_mail + } + + return requests.post(self.FEEDBACK_URL, data=data, verify=False) diff --git a/service/pixelated/adapter/services/mail_service.py b/service/pixelated/adapter/services/mail_service.py index 233d4d4a..44c4c145 100644 --- a/service/pixelated/adapter/services/mail_service.py +++ b/service/pixelated/adapter/services/mail_service.py @@ -13,37 +13,50 @@ # # You should have received a copy of the GNU Affero General Public License # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +from twisted.internet import defer from pixelated.adapter.model.mail import InputMail +from pixelated.adapter.model.status import Status from pixelated.adapter.services.tag_service import extract_reserved_tags +from email import message_from_file +import os class MailService(object): - def __init__(self, mailboxes, mail_sender, soledad_querier, search_engine): - self.mailboxes = mailboxes - self.querier = soledad_querier + def __init__(self, mail_sender, mail_store, search_engine): + self.mail_store = mail_store self.search_engine = search_engine self.mail_sender = mail_sender + @defer.inlineCallbacks def all_mails(self): - return self.querier.all_mails() + mails = yield self.mail_store.all_mails() + defer.returnValue(mails) + @defer.inlineCallbacks def mails(self, query, window_size, page): mail_ids, total = self.search_engine.search(query, window_size, page) - return self.querier.mails(mail_ids), total + try: + mails = yield self.mail_store.get_mails(mail_ids) + defer.returnValue((mails, total)) + except Exception, e: + import traceback + traceback.print_exc() + raise + @defer.inlineCallbacks def update_tags(self, mail_id, new_tags): new_tags = self._filter_white_space_tags(new_tags) reserved_words = extract_reserved_tags(new_tags) if len(reserved_words): raise ValueError('None of the following words can be used as tags: ' + ' '.join(reserved_words)) new_tags = self._favor_existing_tags_casing(new_tags) - mail = self.mail(mail_id) - mail.update_tags(set(new_tags)) - self.search_engine.index_mail(mail) + mail = yield self.mail(mail_id) + mail.tags = set(new_tags) + yield self.mail_store.update_mail(mail) - return mail + defer.returnValue(mail) def _filter_white_space_tags(self, tags): return [tag.strip() for tag in tags if not tag.isspace()] @@ -58,53 +71,65 @@ class MailService(object): return [_use_current_casing(new_tag.lower()) if new_tag.lower() in current_tags_lower else new_tag for new_tag in new_tags] def mail(self, mail_id): - return self.querier.mail(mail_id) + return self.mail_store.get_mail(mail_id, include_body=True) - def attachment(self, attachment_id, encoding): - return self.querier.attachment(attachment_id, encoding) + def attachment(self, attachment_id): + return self.mail_store.get_mail_attachment(attachment_id) + @defer.inlineCallbacks def mail_exists(self, mail_id): - return not(not(self.querier.get_header_by_chash(mail_id))) + try: + mail = yield self.mail_store.get_mail(mail_id, include_body=False) + defer.returnValue(mail is not None) + except Exception, e: + defer.returnValue(False) + @defer.inlineCallbacks def send_mail(self, content_dict): mail = InputMail.from_dict(content_dict) draft_id = content_dict.get('ident') - def move_to_sent(_): - return self.move_to_sent(draft_id, mail) - - deferred = self.mail_sender.sendmail(mail) - deferred.addCallback(move_to_sent) - return deferred + yield self.mail_sender.sendmail(mail) + sent_mail = yield self.move_to_sent(draft_id, mail) + defer.returnValue(sent_mail) + @defer.inlineCallbacks def move_to_sent(self, last_draft_ident, mail): if last_draft_ident: - self.mailboxes.drafts.remove(last_draft_ident) - return self.mailboxes.sent.add(mail) + try: + yield self.mail_store.delete_mail(last_draft_ident) + except Exception as error: + pass + + sent_mail = yield self.mail_store.add_mail('SENT', mail.raw) + sent_mail.flags.add(Status.SEEN) + yield self.mail_store.update_mail(sent_mail) + defer.returnValue(sent_mail) + @defer.inlineCallbacks def mark_as_read(self, mail_id): - mail = self.mail(mail_id) - mail.mark_as_read() - self.search_engine.index_mail(mail) + mail = yield self.mail(mail_id) + mail.flags.add(Status.SEEN) + yield self.mail_store.update_mail(mail) + @defer.inlineCallbacks def mark_as_unread(self, mail_id): - mail = self.mail(mail_id) - mail.mark_as_unread() - self.search_engine.index_mail(mail) + mail = yield self.mail(mail_id) + mail.flags.remove(Status.SEEN) + yield self.mail_store.update_mail(mail) + @defer.inlineCallbacks def delete_mail(self, mail_id): - mail = self.mail(mail_id) - if mail.mailbox_name == 'TRASH': - self.delete_permanent(mail_id) + mail = yield self.mail(mail_id) + if mail.mailbox_name.upper() == u'TRASH': + yield self.mail_store.delete_mail(mail_id) else: - trashed_mail = self.mailboxes.move_to_trash(mail_id) - self.search_engine.index_mail(trashed_mail) + yield self.mail_store.move_mail_to_mailbox(mail_id, 'TRASH') + @defer.inlineCallbacks def recover_mail(self, mail_id): - recovered_mail = self.mailboxes.move_to_inbox(mail_id) - self.search_engine.index_mail(recovered_mail) + yield self.mail_store.move_mail_to_mailbox(mail_id, 'INBOX') + @defer.inlineCallbacks def delete_permanent(self, mail_id): - mail = self.mail(mail_id) - self.search_engine.remove_from_index(mail_id) - self.querier.remove_mail(mail) + yield self.mail_store.delete_mail(mail_id) diff --git a/service/pixelated/adapter/services/mailbox.py b/service/pixelated/adapter/services/mailbox.py deleted file mode 100644 index a4029d78..00000000 --- a/service/pixelated/adapter/services/mailbox.py +++ /dev/null @@ -1,46 +0,0 @@ -# -# Copyright (c) 2014 ThoughtWorks, Inc. -# -# Pixelated is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# Pixelated is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. - - -class Mailbox(object): - - def __init__(self, mailbox_name, querier, search_engine): - self.mailbox_name = mailbox_name - self.mailbox_tag = mailbox_name.lower() - self.search_engine = search_engine - self.querier = querier - - @property - def fresh(self): - return self.querier.get_lastuid(self.mailbox_name) == 0 - - def mail(self, mail_id): - return self.querier.mail(mail_id) - - def add(self, mail): - added_mail = self.querier.create_mail(mail, self.mailbox_name) - self.search_engine.index_mail(added_mail) - return added_mail - - def remove(self, ident): - mail = self.querier.mail(ident) - self.search_engine.remove_from_index(mail.ident) - mail.remove_all_tags() - self.querier.remove_mail(mail) - - @classmethod - def create(cls, mailbox_name, soledad_querier, search_engine): - return Mailbox(mailbox_name, soledad_querier, search_engine) diff --git a/service/pixelated/adapter/services/mailboxes.py b/service/pixelated/adapter/services/mailboxes.py deleted file mode 100644 index c2b61ca8..00000000 --- a/service/pixelated/adapter/services/mailboxes.py +++ /dev/null @@ -1,74 +0,0 @@ -# -# Copyright (c) 2014 ThoughtWorks, Inc. -# -# Pixelated is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# Pixelated is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. -from pixelated.adapter.services.mailbox import Mailbox -from pixelated.adapter.listeners.mailbox_indexer_listener import MailboxIndexerListener -from pixelated.adapter.model.mail import welcome_mail - - -class Mailboxes(object): - - def __init__(self, account, soledad_querier, search_engine): - self.account = account - self.querier = soledad_querier - self.search_engine = search_engine - for mailbox_name in account.mailboxes: - MailboxIndexerListener.listen(self.account, mailbox_name, soledad_querier) - - def _create_or_get(self, mailbox_name): - mailbox_name = mailbox_name.upper() - if mailbox_name not in self.account.mailboxes: - self.account.addMailbox(mailbox_name) - MailboxIndexerListener.listen(self.account, mailbox_name, self.querier) - return Mailbox.create(mailbox_name, self.querier, self.search_engine) - - @property - def inbox(self): - return self._create_or_get('INBOX') - - @property - def drafts(self): - return self._create_or_get('DRAFTS') - - @property - def trash(self): - return self._create_or_get('TRASH') - - @property - def sent(self): - return self._create_or_get('SENT') - - def mailboxes(self): - return [self._create_or_get(leap_mailbox_name) for leap_mailbox_name in self.account.mailboxes] - - def move_to_trash(self, mail_id): - return self._move_to(mail_id, self.trash) - - def move_to_inbox(self, mail_id): - return self._move_to(mail_id, self.inbox) - - def _move_to(self, mail_id, mailbox): - mail = self.querier.mail(mail_id) - mail.set_mailbox(mailbox.mailbox_name) - mail.save() - return mail - - def mail(self, mail_id): - return self.querier.mail(mail_id) - - def add_welcome_mail_for_fresh_user(self): - if self.inbox.fresh: - mail = welcome_mail() - self.inbox.add(mail) diff --git a/service/pixelated/adapter/soledad/soledad_duplicate_removal_mixin.py b/service/pixelated/adapter/soledad/soledad_duplicate_removal_mixin.py deleted file mode 100644 index 0dd3d497..00000000 --- a/service/pixelated/adapter/soledad/soledad_duplicate_removal_mixin.py +++ /dev/null @@ -1,41 +0,0 @@ -# -# Copyright (c) 2014 ThoughtWorks, Inc. -# -# Pixelated is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# Pixelated is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. -from pixelated.adapter.soledad.soledad_facade_mixin import SoledadDbFacadeMixin - - -class SoledadDuplicateRemovalMixin(SoledadDbFacadeMixin, object): - - def remove_duplicates(self): - for mailbox in ['INBOX', 'DRAFTS', 'SENT', 'TRASH']: - self._remove_dup_inboxes(mailbox) - self._remove_dup_recent(mailbox) - - def _remove_many(self, docs): - [self.delete_doc(doc) for doc in docs] - - def _remove_dup_inboxes(self, mailbox_name): - mailboxes = self.get_mbox(mailbox_name) - if len(mailboxes) == 0: - return - mailboxes_to_remove = sorted(mailboxes, key=lambda x: x.content['created'])[1:len(mailboxes)] - self._remove_many(mailboxes_to_remove) - - def _remove_dup_recent(self, mailbox_name): - rct = self.get_recent_by_mbox(mailbox_name) - if len(rct) == 0: - return - rct_to_remove = sorted(rct, key=lambda x: len(x.content['rct']), reverse=True)[1:len(rct)] - self._remove_many(rct_to_remove) diff --git a/service/pixelated/adapter/soledad/soledad_facade_mixin.py b/service/pixelated/adapter/soledad/soledad_facade_mixin.py deleted file mode 100644 index 2a50b17d..00000000 --- a/service/pixelated/adapter/soledad/soledad_facade_mixin.py +++ /dev/null @@ -1,72 +0,0 @@ -# -# Copyright (c) 2014 ThoughtWorks, Inc. -# -# Pixelated is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# Pixelated is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. - - -class SoledadDbFacadeMixin(object): - - def get_all_flags(self): - return self.soledad.get_from_index('by-type', 'flags') - - def get_all_flags_by_mbox(self, mbox): - return self.soledad.get_from_index('by-type-and-mbox', 'flags', mbox) if mbox else [] - - def get_content_by_phash(self, phash): - content = self.soledad.get_from_index('by-type-and-payloadhash', 'cnt', phash) if phash else [] - if len(content): - return content[0] - - def get_flags_by_chash(self, chash): - flags = self.soledad.get_from_index('by-type-and-contenthash', 'flags', chash) if chash else [] - if len(flags): - return flags[0] - - def get_header_by_chash(self, chash): - header = self.soledad.get_from_index('by-type-and-contenthash', 'head', chash) if chash else [] - if len(header): - return header[0] - - def get_recent_by_mbox(self, mbox): - return self.soledad.get_from_index('by-type-and-mbox', 'rct', mbox) if mbox else [] - - def put_doc(self, doc): - return self.soledad.put_doc(doc) - - def create_doc(self, doc): - return self.soledad.create_doc(doc) - - def create_docs(self, docs): - for doc in docs: - self.create_doc(doc) - - def delete_doc(self, doc): - return self.soledad.delete_doc(doc) - - def idents_by_mailbox(self, mbox): - return set(doc.content['chash'] for doc in self.soledad.get_from_index('by-type-and-mbox-and-deleted', 'flags', mbox, '0')) if mbox else set() - - def get_all_mbox(self): - return self.soledad.get_from_index('by-type', 'mbox') - - def get_mbox(self, mbox): - return self.soledad.get_from_index('by-type-and-mbox', 'mbox', mbox) if mbox else [] - - def get_lastuid(self, mbox): - if isinstance(mbox, str): - mbox = self.get_mbox(mbox)[0] - return mbox.content['lastuid'] - - def get_search_index_masterkey(self): - return self.soledad.get_from_index('by-type', 'index_key') diff --git a/service/pixelated/adapter/soledad/soledad_reader_mixin.py b/service/pixelated/adapter/soledad/soledad_reader_mixin.py deleted file mode 100644 index 347938ed..00000000 --- a/service/pixelated/adapter/soledad/soledad_reader_mixin.py +++ /dev/null @@ -1,121 +0,0 @@ -# -# Copyright (c) 2014 ThoughtWorks, Inc. -# -# Pixelated is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# Pixelated is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. -import base64 -import logging -import quopri -import re - -from pixelated.adapter.model.mail import PixelatedMail -from pixelated.adapter.soledad.soledad_facade_mixin import SoledadDbFacadeMixin - - -logger = logging.getLogger(__name__) - - -class SoledadReaderMixin(SoledadDbFacadeMixin, object): - - def all_mails(self): - fdocs_chash = [(fdoc, fdoc.content['chash']) for fdoc in self.get_all_flags()] - if len(fdocs_chash) == 0: - return [] - return self._build_mails_from_fdocs(fdocs_chash) - - def _build_mails_from_fdocs(self, fdocs_chash): - if len(fdocs_chash) == 0: - return [] - - fdocs_hdocs = [] - for fdoc, chash in fdocs_chash: - hdoc = self.get_header_by_chash(chash) - if not hdoc: - continue - fdocs_hdocs.append((fdoc, hdoc)) - - fdocs_hdocs_bodyphash = [(f[0], f[1], f[1].content.get('body')) for f in fdocs_hdocs] - fdocs_hdocs_bdocs_parts = [] - for fdoc, hdoc, body_phash in fdocs_hdocs_bodyphash: - bdoc = self.get_content_by_phash(body_phash) - if not bdoc: - continue - parts = self._extract_parts(hdoc.content) - fdocs_hdocs_bdocs_parts.append((fdoc, hdoc, bdoc, parts)) - - return [PixelatedMail.from_soledad(*raw_mail, soledad_querier=self) for raw_mail in fdocs_hdocs_bdocs_parts] - - def mail_exists(self, ident): - return self.get_flags_by_chash(ident) - - def mail(self, ident): - fdoc = self.get_flags_by_chash(ident) - hdoc = self.get_header_by_chash(ident) - bdoc = self.get_content_by_phash(hdoc.content['body']) - parts = self._extract_parts(hdoc.content) - - return PixelatedMail.from_soledad(fdoc, hdoc, bdoc, parts=parts, soledad_querier=self) - - def mails(self, idents): - fdocs_chash = [(self.get_flags_by_chash(ident), ident) for ident in - idents] - fdocs_chash = [(result, ident) for result, ident in fdocs_chash if result] - return self._build_mails_from_fdocs(fdocs_chash) - - def attachment(self, attachment_ident, encoding): - bdoc = self.get_content_by_phash(attachment_ident) - return {'content': self._try_decode(bdoc.content['raw'], encoding), - 'content-type': bdoc.content['content-type']} - - def _try_decode(self, raw, encoding): - encoding = encoding.lower() - if encoding == 'base64': - return base64.decodestring(raw) - elif encoding == 'quoted-printable': - return quopri.decodestring(raw) - else: - return str(raw) - - def _extract_parts(self, hdoc, parts=None): - if not parts: - parts = {'alternatives': [], 'attachments': []} - - if hdoc['multi']: - for part_key in hdoc.get('part_map', {}).keys(): - self._extract_parts(hdoc['part_map'][part_key], parts) - else: - headers_dict = {elem[0]: elem[1] for elem in hdoc.get('headers', [])} - if 'attachment' in headers_dict.get('Content-Disposition', ''): - parts['attachments'].append(self._extract_attachment(hdoc, headers_dict)) - else: - parts['alternatives'].append(self._extract_alternative(hdoc, headers_dict)) - return parts - - def _extract_alternative(self, hdoc, headers_dict): - bdoc = self.get_content_by_phash(hdoc['phash']) - - if bdoc is None: - logger.warning("No BDOC content found for message!!!") - raw_content = "" - else: - raw_content = bdoc.content['raw'] - - return {'headers': headers_dict, 'content': raw_content} - - def _extract_attachment(self, hdoc, headers_dict): - content_disposition = headers_dict['Content-Disposition'] - match = re.compile('.*name=\"(.*)\".*').search(content_disposition) - filename = '' - if match: - filename = match.group(1) - return {'headers': headers_dict, 'ident': hdoc['phash'], 'name': filename} diff --git a/service/pixelated/adapter/soledad/soledad_writer_mixin.py b/service/pixelated/adapter/soledad/soledad_writer_mixin.py deleted file mode 100644 index b0d21b93..00000000 --- a/service/pixelated/adapter/soledad/soledad_writer_mixin.py +++ /dev/null @@ -1,47 +0,0 @@ -# -# Copyright (c) 2014 ThoughtWorks, Inc. -# -# Pixelated is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# Pixelated is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. -from pixelated.adapter.soledad.soledad_facade_mixin import SoledadDbFacadeMixin - - -class SoledadWriterMixin(SoledadDbFacadeMixin, object): - - def mark_all_as_not_recent(self): - for mailbox in ['INBOX', 'DRAFTS', 'SENT', 'TRASH']: - rct = self.get_recent_by_mbox(mailbox) - if not rct or not rct[0].content['rct']: - return - rct = rct[0] - rct.content['rct'] = [] - self.put_doc(rct) - - def save_mail(self, mail): - self.put_doc(mail.fdoc) - - def create_mail(self, mail, mailbox_name): - mbox_doc = self.get_mbox(mailbox_name)[0] - uid = self.get_lastuid(mbox_doc) - self.create_docs(mail.get_for_save(next_uid=uid, mailbox=mailbox_name)) - - mbox_doc.content['lastuid'] = uid + 1 - self.put_doc(mbox_doc) - - return self.mail(mail.ident) - - def remove_mail(self, mail): - # FIX-ME: Must go through all the part_map phash to delete all the cdocs - self.delete_doc(mail.fdoc) - self.delete_doc(mail.hdoc) - self.delete_doc(mail.bdoc) diff --git a/service/pixelated/application.py b/service/pixelated/application.py index 6d83c6f7..dfeb8d82 100644 --- a/service/pixelated/application.py +++ b/service/pixelated/application.py @@ -15,9 +15,7 @@ # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. from twisted.internet import reactor -from twisted.internet.threads import deferToThread from twisted.internet import defer -from twisted.web.server import Site from twisted.internet import ssl from OpenSSL import SSL from OpenSSL import crypto @@ -26,15 +24,22 @@ from pixelated.config import arguments from pixelated.config.services import Services from pixelated.config.leap import initialize_leap from pixelated.config import logger +from pixelated.config.site import PixelatedSite from pixelated.resources.loading_page import LoadingResource from pixelated.resources.root_resource import RootResource +from leap.common.events import ( + register, + catalog as events +) + @defer.inlineCallbacks def start_user_agent(loading_app, host, port, sslkey, sslcert, leap_home, leap_session): yield loading_app.stopListening() services = Services(leap_home, leap_session) + yield services.setup(leap_home, leap_session) resource = RootResource() @@ -42,12 +47,13 @@ def start_user_agent(loading_app, host, port, sslkey, sslcert, leap_home, leap_s services.keymanager, services.search_engine, services.mail_service, - services.draft_service) + services.draft_service, + services.feedback_service) if sslkey and sslcert: - reactor.listenSSL(port, Site(resource), _ssl_options(sslkey, sslcert), interface=host) + reactor.listenSSL(port, PixelatedSite(resource), _ssl_options(sslkey, sslcert), interface=host) else: - reactor.listenTCP(port, Site(resource), interface=host) + reactor.listenTCP(port, PixelatedSite(resource), interface=host) # soledad needs lots of threads reactor.threadpool.adjustPoolsize(5, 15) @@ -71,16 +77,13 @@ def _ssl_options(sslkey, sslcert): def initialize(): args = arguments.parse_user_agent_args() logger.init(debug=args.debug) + loading_app = reactor.listenTCP(args.port, PixelatedSite(LoadingResource()), interface=args.host) - loading_app = reactor.listenTCP(args.port, Site(LoadingResource()), interface=args.host) - - deferred = deferToThread( - lambda: initialize_leap( - args.leap_provider_cert, - args.leap_provider_cert_fingerprint, - args.credentials_file, - args.organization_mode, - args.leap_home)) + deferred = initialize_leap(args.leap_provider_cert, + args.leap_provider_cert_fingerprint, + args.credentials_file, + args.organization_mode, + args.leap_home) deferred.addCallback( lambda leap_session: start_user_agent( @@ -96,6 +99,11 @@ def initialize(): failure.printTraceback() reactor.stop() + def _register_shutdown_on_token_expire(leap_session): + register(events.SOLEDAD_INVALID_AUTH_TOKEN, lambda _: reactor.stop()) + return leap_session + + deferred.addCallback(_register_shutdown_on_token_expire) deferred.addErrback(_quit_on_error) reactor.run() diff --git a/service/pixelated/assets/Interstitial.js b/service/pixelated/assets/Interstitial.js index a4c689b9..cf9ef8e4 100644 --- a/service/pixelated/assets/Interstitial.js +++ b/service/pixelated/assets/Interstitial.js @@ -4,13 +4,14 @@ if ($('#hive').length) { var left_pos = img_width * .5; var pixelated = hive.path("M12.4,20.3v31.8l28,15.8l28-15.8V20.3l-28-15.8L12.4,20.3z M39.2,56.4l-16.3-9V27.9l16.3,9.3L39.2,56.4z M57.7,47.4l-16.1,9l0-19.2l16.1-9.4V47.4z M57.7,25.2L40.4,35.5L22.9,25.2l17.5-9.4L57.7,25.2z").transform("translate(319, 50)").attr("fill", "#908e8e"); - var all = hive.group().transform("matrix(2, 0, 0, 2, "+(left_pos - 950)+", -40)"); + var all = hive.group().transform("matrix(2, 0, 0, 2, -100, -100)"); var height = 50; var width = 58; - var rows = $(window).height() / height; + var rows = (($(window).height() / height) / 2) + 1; var cols = (($(window).width() / width) / 2) + 1; + for (var j = 0; j < rows; j++) { for (var i = 0; i < cols; i++) { x = i * width + (j%2*width/2); diff --git a/service/pixelated/assets/welcome.mail b/service/pixelated/assets/welcome.mail index e85694f1..3f233143 100644 --- a/service/pixelated/assets/welcome.mail +++ b/service/pixelated/assets/welcome.mail @@ -5,9 +5,7 @@ To: Replace <will.be@the.user> Content-Type: multipart/alternative; boundary=000boundary000 --000boundary000 -Content-Type: text/plain; charset=UTF-8 - -Welcome to Pixelated Mail, a modern email with encryption. +Welcome to Pixelated Mail, a modern email with encryption. Pixelated Mail is an open source project that aims to provide secure email on the browser with all the functionality we've come to expect of a modern email client. diff --git a/service/pixelated/bitmask_libraries/config.py b/service/pixelated/bitmask_libraries/config.py index efb43411..c521a093 100644 --- a/service/pixelated/bitmask_libraries/config.py +++ b/service/pixelated/bitmask_libraries/config.py @@ -36,7 +36,6 @@ class LeapConfig(object): def __init__(self, leap_home=None, - fetch_interval_in_s=30, timeout_in_s=15, start_background_jobs=False, gpg_binary=discover_gpg_binary()): @@ -45,4 +44,3 @@ class LeapConfig(object): self.timeout_in_s = timeout_in_s self.start_background_jobs = start_background_jobs self.gpg_binary = gpg_binary - self.fetch_interval_in_s = fetch_interval_in_s diff --git a/service/pixelated/bitmask_libraries/nicknym.py b/service/pixelated/bitmask_libraries/nicknym.py index 220d75e5..826ecb58 100644 --- a/service/pixelated/bitmask_libraries/nicknym.py +++ b/service/pixelated/bitmask_libraries/nicknym.py @@ -15,6 +15,7 @@ # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. from leap.keymanager import KeyManager, openpgp, KeyNotFound from .certs import LeapCertificate +from twisted.internet import defer class NickNym(object): @@ -23,28 +24,35 @@ class NickNym(object): self._email = email_address self.keymanager = KeyManager(self._email, nicknym_url, soledad_session.soledad, - token, LeapCertificate(provider).provider_api_cert, provider.api_uri, - provider.api_version, - uuid, config.gpg_binary) + token=token, ca_cert_path=LeapCertificate(provider).provider_api_cert, api_uri=provider.api_uri, + api_version=provider.api_version, + uid=uuid, gpgbinary=config.gpg_binary) + @defer.inlineCallbacks def generate_openpgp_key(self): - if not self._key_exists(self._email): + key_present = yield self._key_exists(self._email) + if not key_present: print "Generating keys - this could take a while..." - self._gen_key() - self._send_key_to_leap() + yield self._gen_key() + # Sending it anyway for now. TODO: This can be better with real checking (downloading pubkey from nicknym) + yield self._send_key_to_leap() + @defer.inlineCallbacks def _key_exists(self, email): try: - self.keymanager.get_key(email, openpgp.OpenPGPKey, private=True, fetch_remote=False) - return True + yield self.fetch_key(email, private=True, fetch_remote=False) + defer.returnValue(True) except KeyNotFound: - return False + defer.returnValue(False) + + def fetch_key(self, email, private=False, fetch_remote=True): + return self.keymanager.get_key(email, openpgp.OpenPGPKey, private=private, fetch_remote=fetch_remote) def _gen_key(self): - self.keymanager.gen_key(openpgp.OpenPGPKey) + return self.keymanager.gen_key(openpgp.OpenPGPKey) def _send_key_to_leap(self): - self.keymanager.send_key(openpgp.OpenPGPKey) + return self.keymanager.send_key(openpgp.OpenPGPKey) def _discover_nicknym_server(provider): diff --git a/service/pixelated/bitmask_libraries/provider.py b/service/pixelated/bitmask_libraries/provider.py index 315ea7f1..b7f82f8a 100644 --- a/service/pixelated/bitmask_libraries/provider.py +++ b/service/pixelated/bitmask_libraries/provider.py @@ -93,7 +93,7 @@ class LeapProvider(object): digest = get_digest(cert_data, method) if fingerprint.strip() != digest: - raise Exception('Certificate fingerprints don\'t match') + raise Exception('Certificate fingerprints don\'t match! Expected [%s] but got [%s]' % (fingerprint.strip(), digest)) def _validated_get(self, url): session = requests.session() diff --git a/service/pixelated/bitmask_libraries/session.py b/service/pixelated/bitmask_libraries/session.py index a9cb15f2..7abe2a63 100644 --- a/service/pixelated/bitmask_libraries/session.py +++ b/service/pixelated/bitmask_libraries/session.py @@ -18,16 +18,15 @@ import traceback import sys import os -from leap.mail.imap.fetch import LeapIncomingMail -from leap.mail.imap.account import SoledadBackedAccount -from leap.mail.imap.memorystore import MemoryStore -from leap.mail.imap.soledadstore import SoledadStore +from leap.mail.incoming.service import IncomingMail from twisted.internet import reactor from .nicknym import NickNym from leap.auth import SRPAuth +from pixelated.adapter.mailstore import LeapMailStore from .soledad import SoledadSessionFactory from .smtp import LeapSmtp - +from leap.mail.imap.account import IMAPAccount +from twisted.internet import defer SESSIONS = {} @@ -47,48 +46,66 @@ class LeapSession(object): - ``user_auth`` the secure remote password session data after authenticating with LEAP. See http://en.wikipedia.org/wiki/Secure_Remote_Password_protocol (SRPSession) + - ``mail_store`` the MailStore to access the users mails + - ``soledad_session`` the soledad session. See https://leap.se/soledad (LeapSecureRemotePassword) - ``nicknym`` the nicknym instance. See https://leap.se/nicknym (NickNym) - - ``account`` the actual leap mail account. Implements Twisted imap4.IAccount and imap4.INamespacePresenter (SoledadBackedAccount) - - ``incoming_mail_fetcher`` Background job for fetching incoming mails from LEAP server (LeapIncomingMail) """ - def __init__(self, provider, user_auth, soledad_session, nicknym, soledad_account, incoming_mail_fetcher, smtp): + def __init__(self, provider, user_auth, mail_store, soledad_session, nicknym, smtp): self.smtp = smtp self.config = provider.config self.provider = provider self.user_auth = user_auth + self.mail_store = mail_store self.soledad_session = soledad_session self.nicknym = nicknym - self.account = soledad_account - self.incoming_mail_fetcher = incoming_mail_fetcher - self.soledad_session.soledad.sync(defer_decryption=False) - self.nicknym.generate_openpgp_key() - if self.config.start_background_jobs: - self.start_background_jobs() + @defer.inlineCallbacks + def initial_sync(self): + yield self.sync() + yield self.after_first_sync() + defer.returnValue(self) + + @defer.inlineCallbacks + def after_first_sync(self): + yield self.nicknym.generate_openpgp_key() + self.account = self._create_account(self.account_email, self.soledad_session) + self.incoming_mail_fetcher = yield self._create_incoming_mail_fetcher( + self.nicknym, + self.soledad_session, + self.account, + self.account_email()) + reactor.callFromThread(self.incoming_mail_fetcher.startService) + + def _create_account(self, user_mail, soledad_session): + account = IMAPAccount(user_mail, soledad_session.soledad) + return account def account_email(self): name = self.user_auth.username return self.provider.address_for(name) def close(self): - self.stop_background_jobs() + self.stop_background_jobs - def start_background_jobs(self): - self.smtp.ensure_running() - reactor.callFromThread(self.incoming_mail_fetcher.start_loop) + @defer.inlineCallbacks + def _create_incoming_mail_fetcher(self, nicknym, soledad_session, account, user_mail): + inbox = yield account.callWhenReady(lambda _: account.getMailbox('INBOX')) + defer.returnValue(IncomingMail(nicknym.keymanager, + soledad_session.soledad, + inbox.collection, + user_mail)) def stop_background_jobs(self): - self.smtp.stop() - reactor.callFromThread(self.incoming_mail_fetcher.stop) + reactor.callFromThread(self.incoming_mail_fetcher.stopService) def sync(self): try: - self.soledad_session.sync() + return self.soledad_session.sync() except: traceback.print_exc(file=sys.stderr) raise @@ -117,14 +134,13 @@ class LeapSessionFactory(object): account_email = self._provider.address_for(username) soledad = SoledadSessionFactory.create(self._provider, auth.token, auth.uuid, password) + mail_store = LeapMailStore(soledad.soledad) nicknym = self._create_nicknym(account_email, auth.token, auth.uuid, soledad) - account = self._create_account(auth.uuid, soledad) - incoming_mail_fetcher = self._create_incoming_mail_fetcher(nicknym, soledad, account, account_email) smtp = LeapSmtp(self._provider, auth, nicknym.keymanager) - return LeapSession(self._provider, auth, soledad, nicknym, account, incoming_mail_fetcher, smtp) + return LeapSession(self._provider, auth, mail_store, soledad, nicknym, smtp) def _lookup_session(self, key): global SESSIONS @@ -152,10 +168,5 @@ class LeapSessionFactory(object): def _create_nicknym(self, email_address, token, uuid, soledad_session): return NickNym(self._provider, self._config, soledad_session, email_address, token, uuid) - def _create_account(self, uuid, soledad_session): - memstore = MemoryStore(permanent_store=SoledadStore(soledad_session.soledad)) - return SoledadBackedAccount(uuid, soledad_session.soledad, memstore) - - def _create_incoming_mail_fetcher(self, nicknym, soledad_session, account, email_address): - return LeapIncomingMail(nicknym.keymanager, soledad_session.soledad, account, - self._config.fetch_interval_in_s, email_address) + # memstore = MemoryStore(permanent_store=SoledadStore(soledad_session.soledad)) + # return SoledadBackedAccount(uuid, soledad_session.soledad, memstore) diff --git a/service/pixelated/bitmask_libraries/smtp.py b/service/pixelated/bitmask_libraries/smtp.py index 31e56995..ff2792fb 100644 --- a/service/pixelated/bitmask_libraries/smtp.py +++ b/service/pixelated/bitmask_libraries/smtp.py @@ -88,7 +88,7 @@ class LeapSmtp(object): self._local_smtp_service, self._local_smtp_service_socket = setup_smtp_gateway( port=self.local_smtp_port_number, - userid=email, + userid=str(email), keymanager=self._keymanager, smtp_host=self._remote_hostname.encode('UTF-8'), smtp_port=self._remote_port, diff --git a/service/pixelated/bitmask_libraries/soledad.py b/service/pixelated/bitmask_libraries/soledad.py index f0cd9f2f..0546a158 100644 --- a/service/pixelated/bitmask_libraries/soledad.py +++ b/service/pixelated/bitmask_libraries/soledad.py @@ -17,7 +17,7 @@ import errno import os from leap.soledad.client import Soledad -from leap.soledad.common.crypto import WrongMac, UnknownMacMethod +from leap.soledad.common.crypto import WrongMacError, UnknownMacMethodError from pixelated.bitmask_libraries.certs import LeapCertificate SOLEDAD_TIMEOUT = 120 @@ -57,10 +57,16 @@ class SoledadSession(object): secrets = self._secrets_path() local_db = self._local_db_path() - return Soledad(self.user_uuid, unicode(encryption_passphrase), secrets, - local_db, server_url, LeapCertificate(self.provider).provider_api_cert, self.user_token, defer_encryption=False) + return Soledad(self.user_uuid, + passphrase=unicode(encryption_passphrase), + secrets_path=secrets, + local_db_path=local_db, server_url=server_url, + cert_file=LeapCertificate(self.provider).provider_api_cert, + shared_db=None, + auth_token=self.user_token, + defer_encryption=False) - except (WrongMac, UnknownMacMethod), e: + except (WrongMacError, UnknownMacMethodError), e: raise SoledadWrongPassphraseException(e) def _leap_path(self): @@ -82,8 +88,7 @@ class SoledadSession(object): raise def sync(self): - if self.soledad.need_sync(self.soledad.server_url): - self.soledad.sync() + return self.soledad.sync() def _discover_soledad_server(self): try: diff --git a/service/pixelated/config/arguments.py b/service/pixelated/config/arguments.py index fa7fdae4..7a7abe49 100644 --- a/service/pixelated/config/arguments.py +++ b/service/pixelated/config/arguments.py @@ -43,6 +43,7 @@ def parse_maintenance_args(): subparsers.add_parser('dump-soledad', help='dump the soledad database') subparsers.add_parser('sync', help='sync the soledad database') + subparsers.add_parser('repair', help='repair database if possible') return parser.parse_args() diff --git a/service/pixelated/config/leap.py b/service/pixelated/config/leap.py index 52cd4c8f..0409e54f 100644 --- a/service/pixelated/config/leap.py +++ b/service/pixelated/config/leap.py @@ -1,37 +1,69 @@ from __future__ import absolute_import -import random +from leap.common.events import (server as events_server, + register, catalog as events) +from email import message_from_file from pixelated.config import credentials -from leap.common.events import server as events_server from pixelated.bitmask_libraries.config import LeapConfig from pixelated.bitmask_libraries.certs import LeapCertificate from pixelated.bitmask_libraries.provider import LeapProvider from pixelated.bitmask_libraries.session import LeapSessionFactory +from pixelated.adapter.model.mail import InputMail +from twisted.internet import defer +import os +import logging +fresh_account = False + + +@defer.inlineCallbacks def initialize_leap(leap_provider_cert, leap_provider_cert_fingerprint, credentials_file, organization_mode, - leap_home): + leap_home, + initial_sync=True): init_monkeypatches() - events_server.ensure_server(random.randrange(8000, 11999)) - provider, username, password = credentials.read(organization_mode, credentials_file) - LeapCertificate.set_cert_and_fingerprint(leap_provider_cert, leap_provider_cert_fingerprint) + events_server.ensure_server() + register(events.KEYMANAGER_FINISHED_KEY_GENERATION, + set_fresh_account) + provider, username, password = credentials.read(organization_mode, + credentials_file) + LeapCertificate.set_cert_and_fingerprint(leap_provider_cert, + leap_provider_cert_fingerprint) config = LeapConfig(leap_home=leap_home, start_background_jobs=True) provider = LeapProvider(provider, config) LeapCertificate(provider).setup_ca_bundle() leap_session = LeapSessionFactory(provider).create(username, password) - return leap_session + if initial_sync: + leap_session = yield leap_session.initial_sync() + + global fresh_account + if fresh_account: + add_welcome_mail(leap_session.mail_store) + + defer.returnValue(leap_session) + + +def add_welcome_mail(mail_store): + current_path = os.path.dirname(os.path.abspath(__file__)) + with open(os.path.join(current_path, + '..', + 'assets', + 'welcome.mail')) as mail_template_file: + mail_template = message_from_file(mail_template_file) + + input_mail = InputMail.from_python_mail(mail_template) + logging.getLogger('pixelated.config.leap').info('Adding the welcome mail') + mail_store.add_mail('INBOX', input_mail.raw) def init_monkeypatches(): - import pixelated.extensions.protobuf_socket - import pixelated.extensions.sqlcipher_wal - import pixelated.extensions.esmtp_sender_factory - import pixelated.extensions.incoming_decrypt_header - import pixelated.extensions.soledad_sync_exception - import pixelated.extensions.keymanager_fetch_key import pixelated.extensions.requests_urllib3 - import pixelated.extensions.shared_db + + +def set_fresh_account(_, x): + global fresh_account + fresh_account = True diff --git a/service/pixelated/config/logger.py b/service/pixelated/config/logger.py index 52f3f3a5..5c711981 100644 --- a/service/pixelated/config/logger.py +++ b/service/pixelated/config/logger.py @@ -21,7 +21,7 @@ from twisted.python import log def init(debug=False): debug_enabled = debug or os.environ.get('DEBUG', False) - logging_level = logging.DEBUG if debug_enabled else logging.INFO + logging_level = logging.DEBUG if debug_enabled else logging.WARN log_format = "%(asctime)s [%(name)s] %(levelname)s %(message)s" date_format = '%Y-%m-%d %H:%M:%S' @@ -31,4 +31,5 @@ def init(debug=False): filemode='a') observer = log.PythonLoggingObserver() + logging.getLogger('gnupg').setLevel(logging.WARN) observer.start() diff --git a/service/pixelated/config/services.py b/service/pixelated/config/services.py index f1c7a540..41a357dc 100644 --- a/service/pixelated/config/services.py +++ b/service/pixelated/config/services.py @@ -1,65 +1,75 @@ +from pixelated.adapter.mailstore.searchable_mailstore import SearchableMailStore from pixelated.adapter.services.mail_service import MailService from pixelated.adapter.model.mail import InputMail from pixelated.adapter.services.mail_sender import MailSender -from pixelated.adapter.services.mailboxes import Mailboxes -from pixelated.adapter.soledad.soledad_querier import SoledadQuerier from pixelated.adapter.search import SearchEngine from pixelated.adapter.services.draft_service import DraftService -from pixelated.adapter.listeners.mailbox_indexer_listener import MailboxIndexerListener +from pixelated.adapter.listeners.mailbox_indexer_listener import listen_all_mailboxes +from twisted.internet import defer +from pixelated.adapter.search.index_storage_key import SearchIndexStorageKey +from pixelated.adapter.services.feedback_service import FeedbackService class Services(object): def __init__(self, leap_home, leap_session): + pass - soledad_querier = SoledadQuerier(soledad=leap_session.soledad_session.soledad) + @defer.inlineCallbacks + def setup(self, leap_home, leap_session): + InputMail.FROM_EMAIL_ADDRESS = leap_session.account_email() - self.search_engine = self.setup_search_engine( + search_index_storage_key = self.setup_search_index_storage_key(leap_session.soledad_session.soledad) + yield self.setup_search_engine( leap_home, - soledad_querier) + search_index_storage_key) - pixelated_mailboxes = Mailboxes( - leap_session.account, - soledad_querier, - self.search_engine) + self.wrap_mail_store_with_indexing_mail_store(leap_session) + + yield listen_all_mailboxes(leap_session.account, self.search_engine, leap_session.mail_store) self.mail_service = self.setup_mail_service( leap_session, - soledad_querier, - self.search_engine, - pixelated_mailboxes) + self.search_engine) - self.keymanager = self.setup_keymanager(leap_session) - self.draft_service = self.setup_draft_service(pixelated_mailboxes) + self.keymanager = leap_session.nicknym + self.draft_service = self.setup_draft_service(leap_session.mail_store) + self.feedback_service = self.setup_feedback_service(leap_session) - self.post_setup(soledad_querier, leap_session) + yield self.index_all_mails() - def post_setup(self, soledad_querier, leap_session): - self.search_engine.index_mails( - mails=self.mail_service.all_mails(), - callback=soledad_querier.mark_all_as_not_recent) - soledad_querier.remove_duplicates() - InputMail.FROM_EMAIL_ADDRESS = leap_session.account_email() + def wrap_mail_store_with_indexing_mail_store(self, leap_session): + leap_session.mail_store = SearchableMailStore(leap_session.mail_store, self.search_engine) - def setup_keymanager(self, leap_session): - return leap_session.nicknym.keymanager + @defer.inlineCallbacks + def index_all_mails(self): + all_mails = yield self.mail_service.all_mails() + self.search_engine.index_mails(all_mails) - def setup_search_engine(self, leap_home, soledad_querier): - key = soledad_querier.get_index_masterkey() + @defer.inlineCallbacks + def setup_search_engine(self, leap_home, search_index_storage_key): + key_unicode = yield search_index_storage_key.get_or_create_key() + key = str(key_unicode) + print 'The key len is: %s' % len(key) search_engine = SearchEngine(key, agent_home=leap_home) - MailboxIndexerListener.SEARCH_ENGINE = search_engine - return search_engine + self.search_engine = search_engine - def setup_mail_service(self, leap_session, soledad_querier, search_engine, pixelated_mailboxes): - pixelated_mailboxes.add_welcome_mail_for_fresh_user() + def setup_mail_service(self, leap_session, search_engine): + # if False: FIXME + # yield pixelated_mailboxes.add_welcome_mail_for_fresh_user() pixelated_mail_sender = MailSender( leap_session.account_email(), leap_session.smtp) return MailService( - pixelated_mailboxes, pixelated_mail_sender, - soledad_querier, + leap_session.mail_store, search_engine) - def setup_draft_service(self, pixelated_mailboxes): - return DraftService(pixelated_mailboxes) + def setup_draft_service(self, mail_store): + return DraftService(mail_store) + + def setup_search_index_storage_key(self, soledad): + return SearchIndexStorageKey(soledad) + + def setup_feedback_service(self, leap_session): + return FeedbackService(leap_session) diff --git a/service/pixelated/config/site.py b/service/pixelated/config/site.py new file mode 100644 index 00000000..bd149914 --- /dev/null +++ b/service/pixelated/config/site.py @@ -0,0 +1,15 @@ +from twisted.web.server import Site, Request + + +class AddCSPHeaderRequest(Request): + HEADER_VALUES = "default-src 'self'; style-src 'self' 'unsafe-inline'" + + def process(self): + self.setHeader("Content-Security-Policy", self.HEADER_VALUES) + self.setHeader("X-Content-Security-Policy", self.HEADER_VALUES) + self.setHeader("X-Webkit-CSP", self.HEADER_VALUES) + Request.process(self) + + +class PixelatedSite(Site): + requestFactory = AddCSPHeaderRequest diff --git a/service/pixelated/extensions/incoming_decrypt_header.py b/service/pixelated/extensions/incoming_decrypt_header.py deleted file mode 100644 index 2db5dd1d..00000000 --- a/service/pixelated/extensions/incoming_decrypt_header.py +++ /dev/null @@ -1,35 +0,0 @@ -import leap.mail.imap.fetch as fetch - - -def mark_as_encrypted_inline(f): - - def w(*args, **kwargs): - msg, valid_sign = f(*args) - is_encrypted = fetch.PGP_BEGIN in args[1].as_string() and fetch.PGP_END in args[1].as_string() - decrypted_successfully = fetch.PGP_BEGIN not in msg.as_string() and fetch.PGP_END not in msg.as_string() - - if not is_encrypted: - encrypted = 'false' - else: - if decrypted_successfully: - encrypted = 'true' - else: - encrypted = 'fail' - - msg.add_header('X-Pixelated-encryption-status', encrypted) - return msg, valid_sign - - return w - - -def mark_as_encrypted_multipart(f): - - def w(*args, **kwargs): - msg, valid_sign = f(*args) - msg.add_header('X-Pixelated-encryption-status', 'true') - return msg, valid_sign - return w - - -fetch.LeapIncomingMail._maybe_decrypt_inline_encrypted_msg = mark_as_encrypted_inline(fetch.LeapIncomingMail._maybe_decrypt_inline_encrypted_msg) -fetch.LeapIncomingMail._decrypt_multipart_encrypted_msg = mark_as_encrypted_multipart(fetch.LeapIncomingMail._decrypt_multipart_encrypted_msg) diff --git a/service/pixelated/extensions/keymanager_fetch_key.py b/service/pixelated/extensions/keymanager_fetch_key.py index d39d1f96..114e852e 100644 --- a/service/pixelated/extensions/keymanager_fetch_key.py +++ b/service/pixelated/extensions/keymanager_fetch_key.py @@ -57,4 +57,4 @@ def patched_fetch_keys_from_server(self, address): raise KeyNotFound(address) -leap.keymanager.KeyManager._fetch_keys_from_server = patched_fetch_keys_from_server +# leap.keymanager.KeyManager._fetch_keys_from_server = patched_fetch_keys_from_server diff --git a/service/pixelated/extensions/shared_db.py b/service/pixelated/extensions/shared_db.py index 3e8a978e..b433dd50 100644 --- a/service/pixelated/extensions/shared_db.py +++ b/service/pixelated/extensions/shared_db.py @@ -13,4 +13,4 @@ def patched_sign_request(self, method, url_query, params): 'Wrong credentials: %s' % self._creds) -TokenBasedAuth._sign_request = patched_sign_request +# TokenBasedAuth._sign_request = patched_sign_request diff --git a/service/pixelated/extensions/soledad_sync_exception.py b/service/pixelated/extensions/soledad_sync_exception.py index cb3204ad..c3ef5176 100644 --- a/service/pixelated/extensions/soledad_sync_exception.py +++ b/service/pixelated/extensions/soledad_sync_exception.py @@ -19,4 +19,4 @@ def patched_sync(self, defer_decryption=True): client.logger.error("Soledad exception when syncing: %s - %s" % (e.__class__.__name__, e.message)) -client.Soledad.sync = patched_sync +# client.Soledad.sync = patched_sync diff --git a/service/pixelated/maintenance.py b/service/pixelated/maintenance.py index 7170055c..f011658d 100644 --- a/service/pixelated/maintenance.py +++ b/service/pixelated/maintenance.py @@ -14,64 +14,81 @@ # You should have received a copy of the GNU Affero General Public License # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +import logging from mailbox import Maildir from twisted.internet import reactor, defer from twisted.internet.threads import deferToThread +from pixelated.adapter.mailstore.maintenance import SoledadMaintenance from pixelated.config.leap import initialize_leap from pixelated.config import logger, arguments -from leap.mail.imap.fields import WithMsgFields -import time +from leap.mail.constants import MessageFlags + + +REPAIR_COMMAND = 'repair' def initialize(): - import time args = arguments.parse_maintenance_args() logger.init(debug=args.debug) - leap_session = initialize_leap( - args.leap_provider_cert, - args.leap_provider_cert_fingerprint, - args.credentials_file, - organization_mode=False, - leap_home=args.leap_home) + @defer.inlineCallbacks + def _run(): + leap_session = yield initialize_leap( + args.leap_provider_cert, + args.leap_provider_cert_fingerprint, + args.credentials_file, + organization_mode=False, + leap_home=args.leap_home, + initial_sync=_do_initial_sync(args)) - execute_command = create_execute_command(args, leap_session) + execute_command(args, leap_session) - reactor.callWhenRunning(execute_command) + reactor.callWhenRunning(_run) reactor.run() -def create_execute_command(args, leap_session): - def execute_command(): +def _do_initial_sync(args): + return not _is_repair_command(args) + + +def _is_repair_command(args): + return args.command == REPAIR_COMMAND + + +def execute_command(args, leap_session): - def init_soledad(): - return leap_session + def init_soledad(): + return leap_session - def get_soledad_handle(leap_session): - soledad = leap_session.soledad_session.soledad + def get_soledad_handle(leap_session): + soledad = leap_session.soledad_session.soledad - return leap_session, soledad + return leap_session, soledad - def soledad_sync(args): - leap_session, soledad = args + @defer.inlineCallbacks + def soledad_sync(args): + leap_session, soledad = args + log = logging.getLogger('some logger') - soledad.sync() + log.warn('Before sync') - return args + yield soledad.sync() - tearDown = defer.Deferred() + log.warn('after sync') - prepare = deferToThread(init_soledad) - prepare.addCallback(get_soledad_handle) - prepare.addCallback(soledad_sync) - add_command_callback(args, prepare, tearDown) - tearDown.addCallback(soledad_sync) - tearDown.addCallback(shutdown) - tearDown.addErrback(shutdown_on_error) + defer.returnValue(args) - return execute_command + tearDown = defer.Deferred() + + prepare = deferToThread(init_soledad) + prepare.addCallback(get_soledad_handle) + prepare.addCallback(soledad_sync) + add_command_callback(args, prepare, tearDown) + tearDown.addCallback(soledad_sync) + tearDown.addCallback(shutdown) + tearDown.addErrback(shutdown_on_error) def add_command_callback(args, prepareDeferred, finalizeDeferred): @@ -87,6 +104,9 @@ def add_command_callback(args, prepareDeferred, finalizeDeferred): elif args.command == 'sync': # nothing to do here, sync is already part of the chain prepareDeferred.chainDeferred(finalizeDeferred) + elif args.command == REPAIR_COMMAND: + prepareDeferred.addCallback(repair) + prepareDeferred.chainDeferred(finalizeDeferred) else: print 'Unsupported command: %s' % args.command prepareDeferred.chainDeferred(finalizeDeferred) @@ -94,90 +114,95 @@ def add_command_callback(args, prepareDeferred, finalizeDeferred): return finalizeDeferred +@defer.inlineCallbacks def delete_all_mails(args): leap_session, soledad = args - generation, docs = soledad.get_all_docs() + generation, docs = yield soledad.get_all_docs() for doc in docs: if doc.content.get('type', None) in ['head', 'cnt', 'flags']: soledad.delete_doc(doc) - return args + defer.returnValue(args) def is_keep_file(mail): return mail['subject'] is None -def add_mail_folder(account, maildir, folder_name, deferreds): - if folder_name not in account.mailboxes: - account.addMailbox(folder_name) +@defer.inlineCallbacks +def add_mail_folder(store, maildir, folder_name, deferreds): + yield store.add_mailbox(folder_name) - mbx = account.getMailbox(folder_name) for mail in maildir: if is_keep_file(mail): continue - flags = (WithMsgFields.RECENT_FLAG,) if mail.get_subdir() == 'new' else () + flags = (MessageFlags.RECENT_FLAG,) if mail.get_subdir() == 'new' else () if 'S' in mail.get_flags(): - flags = (WithMsgFields.SEEN_FLAG,) + flags + flags = (MessageFlags.SEEN_FLAG,) + flags if 'R' in mail.get_flags(): - flags = (WithMsgFields.ANSWERED_FLAG,) + flags + flags = (MessageFlags.ANSWERED_FLAG,) + flags - deferreds.append(mbx.addMessage(mail.as_string(), flags=flags, notify_on_disk=False)) + deferreds.append(store.add_mail(folder_name, mail.as_string())) + # FIXME support flags @defer.inlineCallbacks def load_mails(args, mail_paths): leap_session, soledad = args - account = leap_session.account + store = leap_session.mail_store deferreds = [] for path in mail_paths: maildir = Maildir(path, factory=None) - add_mail_folder(account, maildir, 'INBOX', deferreds) + yield add_mail_folder(store, maildir, 'INBOX', deferreds) for mail_folder_name in maildir.list_folders(): mail_folder = maildir.get_folder(mail_folder_name) - add_mail_folder(account, mail_folder, mail_folder_name, deferreds) + yield add_mail_folder(store, mail_folder, mail_folder_name, deferreds) + + yield defer.gatherResults(deferreds, consumeErrors=True) - yield defer.DeferredList(deferreds) defer.returnValue(args) def flush_to_soledad(args, finalize): leap_session, soledad = args - account = leap_session.account - memstore = account._memstore - permanent_store = memstore._permanent_store - - d = memstore.write_messages(permanent_store) - def check_flushed(args): - if memstore.is_writing: - reactor.callLater(1, check_flushed, args) - else: - finalize.callback((leap_session, soledad)) + def after_sync(_): + finalize.callback((leap_session, soledad)) - d.addCallback(check_flushed) + d = soledad.sync() + d.addCallback(after_sync) return args +@defer.inlineCallbacks def dump_soledad(args): leap_session, soledad = args - generation, docs = soledad.get_all_docs() + generation, docs = yield soledad.get_all_docs() for doc in docs: print doc print '\n' - return args + defer.returnValue(args) + + +@defer.inlineCallbacks +def repair(args): + leap_session, soledad = args + + yield SoledadMaintenance(soledad).repair() + + defer.returnValue(args) def shutdown(args): - time.sleep(30) + # time.sleep(30) reactor.stop() diff --git a/service/pixelated/register.py b/service/pixelated/register.py index 97f19d2e..2bdbb27b 100644 --- a/service/pixelated/register.py +++ b/service/pixelated/register.py @@ -16,6 +16,7 @@ import re import getpass import logging +import sys from pixelated.config import arguments from pixelated.config import logger as logger_config @@ -24,6 +25,7 @@ from pixelated.bitmask_libraries.config import LeapConfig from pixelated.bitmask_libraries.provider import LeapProvider from pixelated.bitmask_libraries.session import LeapSessionFactory from leap.auth import SRPAuth +from leap.common.events import server as events_server import pixelated.extensions.shared_db @@ -38,14 +40,17 @@ def register( provider_cert, provider_cert_fingerprint): - try: - validate_username(username) - except ValueError: - print('Only lowercase letters, digits, . - and _ allowed.') - if not password: password = getpass.getpass('Please enter password for %s: ' % username) + try: + validate_username(username) + validate_password(password) + except ValueError, e: + print(e.message) + sys.exit(1) + + events_server.ensure_server() LeapCertificate.set_cert_and_fingerprint(provider_cert, provider_cert_fingerprint) config = LeapConfig(leap_home=leap_home) provider = LeapProvider(server_name, config) @@ -60,8 +65,13 @@ def register( def validate_username(username): accepted_characters = '^[a-z0-9\-\_\.]*$' - if not re.match(accepted_characters, username): - raise ValueError + if (not re.match(accepted_characters, username)): + raise ValueError('Only lowercase letters, digits, . - and _ allowed.') + + +def validate_password(password): + if len(password) < 8: + raise ValueError('The password must have at least 8 characters') def initialize(): diff --git a/service/pixelated/resources/__init__.py b/service/pixelated/resources/__init__.py index b244900a..c65e19f3 100644 --- a/service/pixelated/resources/__init__.py +++ b/service/pixelated/resources/__init__.py @@ -17,15 +17,22 @@ import json +class SetEncoder(json.JSONEncoder): + def default(self, obj): + if isinstance(obj, set): + return list(obj) + return super(SetEncoder, self).default(obj) + + def respond_json(entity, request, status_code=200): - json_response = json.dumps(entity) + json_response = json.dumps(entity, cls=SetEncoder) request.responseHeaders.addRawHeader(b"content-type", b"application/json") request.code = status_code return json_response def respond_json_deferred(entity, request, status_code=200): - json_response = json.dumps(entity) + json_response = json.dumps(entity, cls=SetEncoder) request.responseHeaders.addRawHeader(b"content-type", b"application/json") request.code = status_code request.write(json_response) diff --git a/service/pixelated/resources/attachments_resource.py b/service/pixelated/resources/attachments_resource.py index 83c7156d..a78022ec 100644 --- a/service/pixelated/resources/attachments_resource.py +++ b/service/pixelated/resources/attachments_resource.py @@ -18,9 +18,10 @@ import io import re from twisted.protocols.basic import FileSender -from twisted.python.log import err +from twisted.python.log import msg from twisted.web import server from twisted.web.resource import Resource +from twisted.internet import defer class AttachmentResource(Resource): @@ -33,23 +34,33 @@ class AttachmentResource(Resource): self.mail_service = mail_service def render_GET(self, request): + def error_handler(failure): + msg(failure, 'attachment not found') + request.code = 404 + request.finish() encoding = request.args.get('encoding', [None])[0] filename = request.args.get('filename', [self.attachment_id])[0] - attachment = self.mail_service.attachment(self.attachment_id, encoding) - request.setHeader(b'Content-Type', b'application/force-download') request.setHeader(b'Content-Disposition', bytes('attachment; filename=' + filename)) + + d = self._send_attachment(encoding, filename, request) + d.addErrback(error_handler) + + return server.NOT_DONE_YET + + @defer.inlineCallbacks + def _send_attachment(self, encoding, filename, request): + attachment = yield self.mail_service.attachment(self.attachment_id) + bytes_io = io.BytesIO(attachment['content']) - d = FileSender().beginFileTransfer(bytes_io, request) - def cb_finished(_): + try: + request.code = 200 + yield FileSender().beginFileTransfer(bytes_io, request) + finally: bytes_io.close() request.finish() - d.addErrback(err).addCallback(cb_finished) - - return server.NOT_DONE_YET - def _extract_mimetype(self, content_type): match = re.compile('([A-Za-z-]+\/[A-Za-z-]+)').search(content_type) return match.group(1) diff --git a/service/pixelated/resources/contacts_resource.py b/service/pixelated/resources/contacts_resource.py index 5ec39761..c9b81f54 100644 --- a/service/pixelated/resources/contacts_resource.py +++ b/service/pixelated/resources/contacts_resource.py @@ -29,8 +29,16 @@ class ContactsResource(Resource): self._search_engine = search_engine def render_GET(self, request): - query = request.args.get('q', [''])[0] + query = request.args.get('q', ['']) d = deferToThread(lambda: self._search_engine.contacts(query)) d.addCallback(lambda tags: respond_json_deferred(tags, request)) + def handle_error(error): + print 'Something went wrong' + import traceback + traceback.print_exc() + print error + + d.addErrback(handle_error) + return server.NOT_DONE_YET diff --git a/service/pixelated/resources/features_resource.py b/service/pixelated/resources/features_resource.py index 6a1a49ca..927cd9e9 100644 --- a/service/pixelated/resources/features_resource.py +++ b/service/pixelated/resources/features_resource.py @@ -21,12 +21,16 @@ from twisted.web.resource import Resource class FeaturesResource(Resource): DISABLED_FEATURES = ['draftReply'] - isLeaf = True def render_GET(self, request): - try: - disabled_features = {'logout': os.environ['DISPATCHER_LOGOUT_URL']} - except KeyError: - disabled_features = {} - return respond_json({'disabled_features': self.DISABLED_FEATURES, 'dispatcher_features': disabled_features}, request) + dispatcher_features = {} + + if os.environ.get('DISPATCHER_LOGOUT_URL'): + dispatcher_features['logout'] = os.environ.get('DISPATCHER_LOGOUT_URL') + + if os.environ.get('FEEDBACK_URL') is None: + self.DISABLED_FEATURES.append('feedback') + + return respond_json( + {'disabled_features': self.DISABLED_FEATURES, 'dispatcher_features': dispatcher_features}, request) diff --git a/service/pixelated/resources/feedback_resource.py b/service/pixelated/resources/feedback_resource.py new file mode 100644 index 00000000..b989b273 --- /dev/null +++ b/service/pixelated/resources/feedback_resource.py @@ -0,0 +1,32 @@ +# +# Copyright (c) 2015 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +import json + +from twisted.web.resource import Resource +from pixelated.resources import respond_json + + +class FeedbackResource(Resource): + isLeaf = True + + def __init__(self, feedback_service): + Resource.__init__(self) + self.feedback_service = feedback_service + + def render_POST(self, request): + feedback = json.loads(request.content.read()).get('feedback') + self.feedback_service.open_ticket(feedback) + return respond_json({}, request) diff --git a/service/pixelated/resources/keys_resource.py b/service/pixelated/resources/keys_resource.py index 8afb2bf6..6df95b28 100644 --- a/service/pixelated/resources/keys_resource.py +++ b/service/pixelated/resources/keys_resource.py @@ -1,7 +1,5 @@ from email.utils import parseaddr -from leap.keymanager import OpenPGPKey from pixelated.resources import respond_json_deferred -from twisted.internet.threads import deferToThread from twisted.web import server from twisted.web.resource import Resource @@ -25,7 +23,7 @@ class KeysResource(Resource): respond_json_deferred(None, request, status_code=404) _, key_to_find = parseaddr(request.args.get('search')[0]) - d = deferToThread(lambda: self._keymanager.get_key_from_cache(key_to_find, OpenPGPKey)) + d = self._keymanager.fetch_key(key_to_find) d.addCallback(finish_request) d.addErrback(key_not_found) diff --git a/service/pixelated/resources/mail_resource.py b/service/pixelated/resources/mail_resource.py index dff594b0..436842fb 100644 --- a/service/pixelated/resources/mail_resource.py +++ b/service/pixelated/resources/mail_resource.py @@ -1,6 +1,8 @@ import json -from pixelated.resources import respond_json +from pixelated.resources import respond_json, respond_json_deferred from twisted.web.resource import Resource +from twisted.web.server import NOT_DONE_YET +from twisted.python.log import err class MailTags(Resource): @@ -15,11 +17,14 @@ class MailTags(Resource): def render_POST(self, request): new_tags = json.loads(request.content.read()).get('newtags') - try: - mail = self._mail_service.update_tags(self._mail_id, new_tags) - except ValueError as ve: - return respond_json(ve.message, request, 403) - return respond_json(mail.as_dict(), request) + d = self._mail_service.update_tags(self._mail_id, new_tags) + d.addCallback(lambda mail: respond_json_deferred(mail.as_dict(), request)) + + def handle403(failure): + failure.trap(ValueError) + return respond_json_deferred(failure.getErrorMessage(), request, 403) + d.addErrback(handle403) + return NOT_DONE_YET class Mail(Resource): @@ -31,12 +36,21 @@ class Mail(Resource): self._mail_service = mail_service def render_GET(self, request): - mail = self._mail_service.mail(self._mail_id) - return respond_json(mail.as_dict(), request) + d = self._mail_service.mail(self._mail_id) + + d.addCallback(lambda mail: respond_json_deferred(mail.as_dict(), request)) + + return NOT_DONE_YET def render_DELETE(self, request): - self._mail_service.delete_mail(self._mail_id) - return respond_json(None, request) + def response_failed(failure): + err(failure, 'something failed') + request.finish() + + d = self._mail_service.delete_mail(self._mail_id) + d.addCallback(lambda _: respond_json_deferred(None, request)) + d.addErrback(response_failed) + return NOT_DONE_YET class MailResource(Resource): diff --git a/service/pixelated/resources/mails_resource.py b/service/pixelated/resources/mails_resource.py index c4b578ba..93a19a9b 100644 --- a/service/pixelated/resources/mails_resource.py +++ b/service/pixelated/resources/mails_resource.py @@ -1,12 +1,15 @@ import json from pixelated.adapter.services.mail_sender import SMTPDownException from pixelated.adapter.model.mail import InputMail -from pixelated.resources import respond_json, respond_json_deferred +from twisted.web.server import NOT_DONE_YET +from pixelated.resources import respond_json_deferred from twisted.web.resource import Resource from twisted.web import server +from twisted.internet import defer +from twisted.python.log import err from leap.common.events import ( register, - events_pb2 as proto + catalog as events ) @@ -19,9 +22,15 @@ class MailsUnreadResource(Resource): def render_POST(self, request): idents = json.load(request.content).get('idents') + deferreds = [] for ident in idents: - self._mail_service.mark_as_unread(ident) - return respond_json(None, request) + deferreds.append(self._mail_service.mark_as_unread(ident)) + + d = defer.gatherResults(deferreds, consumeErrors=True) + d.addCallback(lambda _: respond_json_deferred(None, request)) + d.addErrback(lambda _: respond_json_deferred(None, request, status_code=500)) + + return NOT_DONE_YET class MailsReadResource(Resource): @@ -33,10 +42,15 @@ class MailsReadResource(Resource): def render_POST(self, request): idents = json.load(request.content).get('idents') + deferreds = [] for ident in idents: - self._mail_service.mark_as_read(ident) + deferreds.append(self._mail_service.mark_as_read(ident)) + + d = defer.gatherResults(deferreds, consumeErrors=True) + d.addCallback(lambda _: respond_json_deferred(None, request)) + d.addErrback(lambda _: respond_json_deferred(None, request, status_code=500)) - return respond_json(None, request) + return NOT_DONE_YET class MailsDeleteResource(Resource): @@ -47,10 +61,19 @@ class MailsDeleteResource(Resource): self._mail_service = mail_service def render_POST(self, request): + def response_failed(failure): + err(failure, 'something failed') + request.finish() + idents = json.loads(request.content.read())['idents'] + deferreds = [] for ident in idents: - self._mail_service.delete_mail(ident) - return respond_json(None, request) + deferreds.append(self._mail_service.delete_mail(ident)) + + d = defer.gatherResults(deferreds, consumeErrors=True) + d.addCallback(lambda _: respond_json_deferred(None, request)) + d.addErrback(response_failed) + return NOT_DONE_YET class MailsRecoverResource(Resource): @@ -62,9 +85,13 @@ class MailsRecoverResource(Resource): def render_POST(self, request): idents = json.loads(request.content.read())['idents'] + deferreds = [] for ident in idents: - self._mail_service.recover_mail(ident) - return respond_json(None, request) + deferreds.append(self._mail_service.recover_mail(ident)) + d = defer.gatherResults(deferreds, consumeErrors=True) + d.addCallback(lambda _: respond_json_deferred(None, request)) + d.addErrback(lambda _: respond_json_deferred(None, request, status_code=500)) + return NOT_DONE_YET class MailsResource(Resource): @@ -75,7 +102,7 @@ class MailsResource(Resource): delivery_error_mail = InputMail.delivery_error_template(delivery_address=event.content) self._mail_service.mailboxes.inbox.add(delivery_error_mail) - register(signal=proto.SMTP_SEND_MESSAGE_ERROR, callback=on_error) + register(events.SMTP_SEND_MESSAGE_ERROR, callback=on_error) def __init__(self, mail_service, draft_service): Resource.__init__(self) @@ -90,16 +117,22 @@ class MailsResource(Resource): def render_GET(self, request): query, window_size, page = request.args.get('q')[0], request.args.get('w')[0], request.args.get('p')[0] - mails, total = self._mail_service.mails(query, window_size, page) + d = self._mail_service.mails(query, window_size, page) - response = { + d.addCallback(lambda (mails, total): { "stats": { "total": total, }, "mails": [mail.as_dict() for mail in mails] - } + }) + d.addCallback(lambda res: respond_json_deferred(res, request)) + + def error_handler(error): + print error - return respond_json(response, request) + d.addErrback(error_handler) + + return NOT_DONE_YET def render_POST(self, request): content_dict = json.loads(request.content.read()) @@ -114,7 +147,8 @@ class MailsResource(Resource): if isinstance(error.value, SMTPDownException): respond_json_deferred({'message': str(error.value)}, request, status_code=503) else: - respond_json_deferred({'message': str(error)}, request, status_code=422) + err(error, 'something failed') + respond_json_deferred({'message': 'an error occurred while sending'}, request, status_code=422) deferred.addCallback(onSuccess) deferred.addErrback(onError) @@ -126,11 +160,25 @@ class MailsResource(Resource): _mail = InputMail.from_dict(content_dict) draft_id = content_dict.get('ident') + def defer_response(deferred): + deferred.addCallback(lambda pixelated_mail: respond_json_deferred({'ident': pixelated_mail.ident}, request)) + if draft_id: - if not self._mail_service.mail_exists(draft_id): - return respond_json("", request, status_code=422) - pixelated_mail = self._draft_service.update_draft(draft_id, _mail) + deferred_check = self._mail_service.mail_exists(draft_id) + + def handleDuplicatedDraftException(error): + respond_json_deferred("", request, status_code=422) + + def return422otherwise(mail_exists): + if not mail_exists: + respond_json_deferred("", request, status_code=422) + else: + new_draft = self._draft_service.update_draft(draft_id, _mail) + new_draft.addErrback(handleDuplicatedDraftException) + defer_response(new_draft) + + deferred_check.addCallback(return422otherwise) else: - pixelated_mail = self._draft_service.create_draft(_mail) + defer_response(self._draft_service.create_draft(_mail)) - return respond_json({'ident': pixelated_mail.ident}, request) + return server.NOT_DONE_YET diff --git a/service/pixelated/resources/root_resource.py b/service/pixelated/resources/root_resource.py index c1111269..8b536450 100644 --- a/service/pixelated/resources/root_resource.py +++ b/service/pixelated/resources/root_resource.py @@ -1,7 +1,9 @@ import os +import requests from pixelated.resources.attachments_resource import AttachmentsResource from pixelated.resources.contacts_resource import ContactsResource from pixelated.resources.features_resource import FeaturesResource +from pixelated.resources.feedback_resource import FeedbackResource from pixelated.resources.mail_resource import MailResource from pixelated.resources.mails_resource import MailsResource from pixelated.resources.tags_resource import TagsResource @@ -21,7 +23,7 @@ class RootResource(Resource): return self return Resource.getChild(self, path, request) - def initialize(self, keymanager, search_engine, mail_service, draft_service): + def initialize(self, keymanager, search_engine, mail_service, draft_service, feedback_service): self.putChild('assets', File(self._static_folder)) self.putChild('keys', KeysResource(keymanager)) self.putChild('attachment', AttachmentsResource(mail_service)) @@ -30,6 +32,7 @@ class RootResource(Resource): self.putChild('tags', TagsResource(search_engine)) self.putChild('mails', MailsResource(mail_service, draft_service)) self.putChild('mail', MailResource(mail_service)) + self.putChild('feedback', FeedbackResource(feedback_service)) def _get_static_folder(self): static_folder = os.path.abspath(os.path.join(os.path.abspath(__file__), "..", "..", "..", "web-ui", "app")) diff --git a/service/pixelated/support/date.py b/service/pixelated/support/date.py index e7cdbb30..0012aeea 100644 --- a/service/pixelated/support/date.py +++ b/service/pixelated/support/date.py @@ -16,6 +16,7 @@ import datetime import dateutil.parser +from email.utils import formatdate from dateutil.tz import tzlocal @@ -23,9 +24,6 @@ def iso_now(): return datetime.datetime.now(tzlocal()).isoformat() -def milliseconds(date): - date = dateutil.parser.parse(date) - date = date.replace(tzinfo=None) - epoch = datetime.datetime.utcfromtimestamp(0) - delta = date - epoch - return int(delta.total_seconds() * 1000) +def mail_date_now(): + date = dateutil.parser.parse(iso_now()) + return formatdate(float(date.strftime('%s'))) diff --git a/service/pixelated/support/encrypted_file_storage.py b/service/pixelated/support/encrypted_file_storage.py index 67036054..567a348a 100644 --- a/service/pixelated/support/encrypted_file_storage.py +++ b/service/pixelated/support/encrypted_file_storage.py @@ -23,7 +23,7 @@ from whoosh.filedb.filestore import FileStorage from whoosh.filedb.structfile import StructFile, BufferFile from leap.soledad.client.crypto import encrypt_sym from leap.soledad.client.crypto import decrypt_sym -from leap.soledad.client.crypto import EncryptionMethods +from leap.soledad.common.crypto import EncryptionMethods from whoosh.util import random_name @@ -56,16 +56,16 @@ class EncryptedFileStorage(FileStorage): return hmac.new(self.signkey, verifiable_payload, sha256).digest() def encrypt(self, content): - iv, ciphertext = encrypt_sym(content, self.masterkey, EncryptionMethods.XSALSA20) + iv, ciphertext = encrypt_sym(content, self.masterkey) mac = self.gen_mac(iv, ciphertext) return ''.join((mac, iv, ciphertext)) def decrypt(self, payload): - payload_mac, iv, ciphertext = payload[:32], payload[32:65], payload[65:] + payload_mac, iv, ciphertext = payload[:32], payload[32:57], payload[57:] generated_mac = self.gen_mac(iv, ciphertext) if sha256(payload_mac).digest() != sha256(generated_mac).digest(): raise Exception("EncryptedFileStorage - Error opening file. Wrong MAC") - return decrypt_sym(ciphertext, self.masterkey, EncryptionMethods.XSALSA20, iv=iv) + return decrypt_sym(ciphertext, self.masterkey, iv) def _encrypt_index_on_close(self, name): def wrapper(struct_file): diff --git a/service/requirements.txt b/service/requirements.txt new file mode 100644 index 00000000..e4c2dce3 --- /dev/null +++ b/service/requirements.txt @@ -0,0 +1,10 @@ +--allow-external u1db --allow-unverified u1db +--allow-external dirspec --allow-unverified dirspec +-e 'git+https://github.com/pixelated-project/leap_pycommon.git@develop#egg=leap.common' +-e 'git+https://github.com/pixelated-project/leap_auth.git#egg=leap.auth' +-e 'git+https://github.com/pixelated-project/soledad.git@develop#egg=leap.soledad.common&subdirectory=common/' +-e 'git+https://github.com/pixelated-project/soledad.git@develop#egg=leap.soledad.client&subdirectory=client/' +-e 'git+https://github.com/pixelated-project/soledad.git@develop#egg=leap.soledad.server&subdirectory=server/' +-e 'git+https://github.com/pixelated-project/keymanager.git@develop#egg=leap.keymanager' +-e 'git+https://github.com/pixelated-project/leap_mail.git@develop#egg=leap.mail' + diff --git a/service/setup.py b/service/setup.py index d0cb967f..f59d10cc 100644 --- a/service/setup.py +++ b/service/setup.py @@ -33,11 +33,13 @@ setup(name='pixelated-user-agent', packages=[ 'pixelated', 'pixelated.adapter', + 'pixelated.adapter.errors', 'pixelated.adapter.listeners', 'pixelated.adapter.model', 'pixelated.adapter.search', 'pixelated.adapter.services', - 'pixelated.adapter.soledad', + 'pixelated.adapter.mailstore', + 'pixelated.adapter.mailstore.maintenance', 'pixelated.bitmask_libraries', 'pixelated.config', 'pixelated.assets', @@ -47,19 +49,14 @@ setup(name='pixelated-user-agent', 'pixelated.extensions' ], install_requires=[ - 'pyasn1==0.1.3', - 'gnupg==1.2.5', - 'Twisted==13.2.0', - 'requests==2.0.0', - 'srp==1.0.4', - 'dirspec==4.2.0', - 'u1db==13.09', - 'leap.auth==0.1.2', - 'leap.keymanager==0.3.8', - 'leap.soledad.common==0.6.3', - 'leap.soledad.client==0.6.3', - 'leap.mail==0.3.9-1-gc1f9c92', - 'whoosh==2.5.7' + 'pyasn1==0.1.8', + 'requests==2.0.0', + 'srp==1.0.4', + 'dirspec==4.2.0', + 'u1db==13.09', + # 'leap.mail>=0.4.0', # not in a stable repo yet, see requirements.txt + 'leap.auth==0.1.2', + 'whoosh==2.5.7' ], entry_points={ 'console_scripts': [ @@ -68,5 +65,4 @@ setup(name='pixelated-user-agent', 'pixelated-register = pixelated.register:initialize' ] }, - include_package_data=True -) + include_package_data=True) diff --git a/service/test/functional/features/attachments.feature b/service/test/functional/features/attachments.feature new file mode 100644 index 00000000..19834a9d --- /dev/null +++ b/service/test/functional/features/attachments.feature @@ -0,0 +1,27 @@ +# +# Copyright (c) 2014 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. + +Feature: Attachments + As a user of Pixelated + I want to download attachments of mails I received + So that my peers are able to send me any kind of content, not just text + + Scenario: User opens a mail attachment + Given I have a mail with an attachment in my inbox + When I open the first mail in the 'inbox' + Then I see the mail has an attachment + #When I open click on the first attachment + #Then the browser downloaded a file diff --git a/service/test/functional/features/checkboxes_and_mailboxes.feature b/service/test/functional/features/checkboxes_and_mailboxes.feature index 47ea806d..09710040 100644 --- a/service/test/functional/features/checkboxes_and_mailboxes.feature +++ b/service/test/functional/features/checkboxes_and_mailboxes.feature @@ -21,7 +21,8 @@ Feature: Checkboxes Scenario: User has a list of emails in each mailboxes that needs to be managed Given I have a mail in my inbox - When I mark the first unread email as read + When I select the tag 'inbox' + And I mark the first unread email as read And I delete the email When I select the tag 'trash' Then the deleted mail is there diff --git a/service/test/functional/features/environment.py b/service/test/functional/features/environment.py index 437529b8..53b13047 100644 --- a/service/test/functional/features/environment.py +++ b/service/test/functional/features/environment.py @@ -16,25 +16,37 @@ import logging import uuid +from crochet import setup, wait_for +from leap.common.events.server import ensure_server +from twisted.internet import defer from test.support.dispatcher.proxy import Proxy from test.support.integration import AppTestClient from selenium import webdriver from pixelated.resources.features_resource import FeaturesResource +from steps.common import * + +setup() + + +@wait_for(timeout=5.0) +def start_app_test_client(client): + ensure_server() + return client.start_client() def before_all(context): logging.disable('INFO') client = AppTestClient() + start_app_test_client(client) + client.listenTCP() proxy = Proxy(proxy_port='8889', app_port='4567') FeaturesResource.DISABLED_FEATURES.append('autoRefresh') context.client = client context.call_to_terminate_proxy = proxy.run_on_a_thread() - context.call_to_terminate = client.run_on_a_thread(logfile='/tmp/behave-tests.log') def after_all(context): - context.call_to_terminate() context.call_to_terminate_proxy() @@ -42,7 +54,7 @@ def before_feature(context, feature): # context.browser = webdriver.Firefox() context.browser = webdriver.PhantomJS() context.browser.set_window_size(1280, 1024) - context.browser.implicitly_wait(10) + context.browser.implicitly_wait(DEFAULT_IMPLICIT_WAIT_TIMEOUT_IN_S) context.browser.set_page_load_timeout(60) # wait for data context.browser.get('http://localhost:8889/') @@ -57,6 +69,20 @@ def after_step(context, step): def after_feature(context, feature): context.browser.quit() + cleanup_all_mails(context) + context.last_mail = None + + +@wait_for(timeout=10.0) +def cleanup_all_mails(context): + @defer.inlineCallbacks + def _delete_all_mails(): + mails = yield context.client.mail_store.all_mails() + for mail in mails: + yield context.client.mail_store.delete_mail(mail.ident) + + return _delete_all_mails() + def save_source(context, filename='/tmp/source.html'): with open(filename, 'w') as out: diff --git a/service/test/functional/features/steps/attachments.py b/service/test/functional/features/steps/attachments.py new file mode 100644 index 00000000..066683bf --- /dev/null +++ b/service/test/functional/features/steps/attachments.py @@ -0,0 +1,55 @@ +# +# Copyright (c) 2014 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +from email.mime.application import MIMEApplication +from time import sleep +from leap.mail.mail import Message +from common import * +from test.support.integration import MailBuilder +from behave import given +from crochet import wait_for +from uuid import uuid4 +from email.MIMEMultipart import MIMEMultipart +from email.mime.text import MIMEText + + +@given(u'I have a mail with an attachment in my inbox') +def add_mail_with_attachment_impl(context): + subject = 'Hi! This the subject %s' % uuid4() + mail = build_mail_with_attachment(subject) + load_mail_into_soledad(context, mail) + context.last_subject = subject + + +def build_mail_with_attachment(subject): + mail = MIMEMultipart() + mail['Subject'] = subject + mail.attach(MIMEText(u'a utf8 message', _charset='utf-8')) + attachment = MIMEApplication('pretend to be binary attachment data') + attachment.add_header('Content-Disposition', 'attachment', filename='filename.txt') + mail.attach(attachment) + + return mail + + +@wait_for(timeout=10.0) +def load_mail_into_soledad(context, mail): + return context.client.mail_store.add_mail('INBOX', mail.as_string()) + + +@then(u'I see the mail has an attachment') +def step_impl(context): + attachments_list = find_elements_by_css_selector(context, '.attachmentsArea li') + assert len(attachments_list) == 1 diff --git a/service/test/functional/features/steps/common.py b/service/test/functional/features/steps/common.py index 93f4cb1f..9a547375 100644 --- a/service/test/functional/features/steps/common.py +++ b/service/test/functional/features/steps/common.py @@ -16,50 +16,77 @@ from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC from selenium.webdriver.support.wait import WebDriverWait -from selenium.common.exceptions import TimeoutException - +from selenium.common.exceptions import TimeoutException, StaleElementReferenceException +import time from test.support.integration import MailBuilder +LOADING = 'loading' + +TIMEOUT_IN_S = 20 + +DEFAULT_IMPLICIT_WAIT_TIMEOUT_IN_S = 10.0 + + +class ImplicitWait(object): + def __init__(self, context, timeout=5.0): + self._context = context + self._timeout = timeout -def wait_until_element_is_invisible_by_locator(context, locator_tuple, timeout=10): + def __enter__(self): + self._context.browser.implicitly_wait(self._timeout) + + def __exit__(self, exc_type, exc_val, exc_tb): + self._context.browser.implicitly_wait(DEFAULT_IMPLICIT_WAIT_TIMEOUT_IN_S) + + +def wait_until_element_is_invisible_by_locator(context, locator_tuple, timeout=TIMEOUT_IN_S): wait = WebDriverWait(context.browser, timeout) wait.until(EC.invisibility_of_element_located(locator_tuple)) -def wait_until_element_is_deleted(context, locator_tuple, timeout=10): +def wait_until_element_is_deleted(context, locator_tuple, timeout=TIMEOUT_IN_S): wait = WebDriverWait(context.browser, timeout) wait.until(lambda s: len(s.find_elements(locator_tuple[0], locator_tuple[1])) == 0) -def wait_for_user_alert_to_disapear(context, timeout=10): +def wait_for_loading_to_finish(context, timeout=TIMEOUT_IN_S): + wait_until_element_is_invisible_by_locator(context, (By.ID, 'loading'), timeout) + + +def wait_for_user_alert_to_disapear(context, timeout=TIMEOUT_IN_S): wait_until_element_is_invisible_by_locator(context, (By.ID, 'user-alerts'), timeout) -def wait_until_elements_are_visible_by_locator(context, locator_tuple, timeout=10): +def wait_until_elements_are_visible_by_locator(context, locator_tuple, timeout=TIMEOUT_IN_S): wait = WebDriverWait(context.browser, timeout) wait.until(EC.presence_of_all_elements_located(locator_tuple)) return context.browser.find_elements(locator_tuple[0], locator_tuple[1]) -def wait_until_elements_are_visible_by_xpath(context, locator_tuple, timeout=10): +def wait_until_elements_are_visible_by_xpath(context, locator_tuple, timeout=TIMEOUT_IN_S): wait = WebDriverWait(context.browser, timeout) wait.until(EC.presence_of_all_elements_located(locator_tuple)) return context.browser.find_elements(locator_tuple[0], locator_tuple[1]) -def wait_until_element_is_visible_by_locator(context, locator_tuple, timeout=10): +def wait_until_element_is_visible_by_locator(context, locator_tuple, timeout=TIMEOUT_IN_S): wait = WebDriverWait(context.browser, timeout) wait.until(EC.visibility_of_element_located(locator_tuple)) return context.browser.find_element(locator_tuple[0], locator_tuple[1]) +def wait_for_condition(context, predicate_func, timeout=TIMEOUT_IN_S, poll_frequency=0.1): + wait = WebDriverWait(context.browser, timeout, poll_frequency=poll_frequency) + wait.until(predicate_func) + + def fill_by_xpath(context, xpath, text): field = context.browser.find_element_by_xpath(xpath) field.send_keys(text) def fill_by_css_selector(context, css_selector, text): - field = context.browser.find_element_by_css_selector(css_selector) + field = find_element_by_css_selector(context, css_selector) field.send_keys(text) @@ -92,8 +119,12 @@ def find_element_by_css_selector(context, css_selector): return wait_until_element_is_visible_by_locator(context, (By.CSS_SELECTOR, css_selector)) -def find_elements_by_css_selector(context, css_selector): - return wait_until_elements_are_visible_by_locator(context, (By.CSS_SELECTOR, css_selector)) +def find_element_by_class_name(context, class_name): + return wait_until_element_is_visible_by_locator(context, (By.CLASS_NAME, class_name)) + + +def find_elements_by_css_selector(context, css_selector, timeout=TIMEOUT_IN_S): + return wait_until_elements_are_visible_by_locator(context, (By.CSS_SELECTOR, css_selector), timeout=timeout) def find_elements_by_xpath(context, xpath): @@ -109,17 +140,31 @@ def element_should_have_content(context, css_selector, content): assert e.text == content -def wait_until_button_is_visible(context, title, timeout=10): +def wait_until_button_is_visible(context, title, timeout=TIMEOUT_IN_S): wait = WebDriverWait(context.browser, timeout) locator_tuple = (By.XPATH, ("//%s[contains(.,'%s')]" % ('button', title))) wait.until(EC.visibility_of_element_located(locator_tuple)) +def execute_ignoring_staleness(func, timeout=TIMEOUT_IN_S): + end_time = time.time() + timeout + while time.time() <= end_time: + try: + return func() + except StaleElementReferenceException: + pass + raise TimeoutException('did not solve stale state until timeout %f' % timeout) + + def click_button(context, title, element='button'): button = find_element_containing_text(context, title, element_type=element) button.click() +def mail_list_with_subject_exists(context, subject): + return find_element_by_xpath(context, "//*[@class='subject-and-tags' and contains(.,'%s')]" % subject) + + def mail_subject(context): e = find_element_by_css_selector(context, '#mail-view .subject') return e.text diff --git a/service/test/functional/features/steps/compose.py b/service/test/functional/features/steps/compose.py index 668579b1..e93468a2 100644 --- a/service/test/functional/features/steps/compose.py +++ b/service/test/functional/features/steps/compose.py @@ -49,9 +49,8 @@ def save_impl(context): @when('I send it') def send_impl(context): - assert page_has_css(context, '#send-button[disabled]') is False - context.browser.find_element(By.ID, 'send-button').click() - # wait_until_element_is_deleted(context, (By.ID, 'send-button'), timeout=120) + send_button = wait_until_element_is_visible_by_locator(context, (By.CSS_SELECTOR, '#send-button:enabled')) + send_button.click() def _enter_recipient(context, recipients_field, to_type): diff --git a/service/test/functional/features/steps/data_setup.py b/service/test/functional/features/steps/data_setup.py index 2a3876fc..fb825aba 100644 --- a/service/test/functional/features/steps/data_setup.py +++ b/service/test/functional/features/steps/data_setup.py @@ -13,11 +13,19 @@ # # You should have received a copy of the GNU Affero General Public License # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +from uuid import uuid4 from test.support.integration import MailBuilder from behave import given +from common import wait_for_condition @given('I have a mail in my inbox') def add_mail_impl(context): - input_mail = MailBuilder().build_input_mail() + subject = 'Hi! This the subject %s' % uuid4() + + input_mail = MailBuilder().with_subject(subject).build_input_mail() context.client.add_mail_to_inbox(input_mail) + + wait_for_condition(context, lambda _: context.client.search_engine.search(subject)[1] > 0, poll_frequency=0.1) + + context.last_subject = subject diff --git a/service/test/functional/features/steps/mail_list.py b/service/test/functional/features/steps/mail_list.py index 5f0a0116..1b850578 100644 --- a/service/test/functional/features/steps/mail_list.py +++ b/service/test/functional/features/steps/mail_list.py @@ -15,10 +15,10 @@ # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. from common import * from selenium.common.exceptions import NoSuchElementException -from time import sleep def find_current_mail(context): + print 'searching for mail [%s]' % context.current_mail_id return find_element_by_id(context, '%s' % context.current_mail_id) @@ -27,11 +27,14 @@ def check_current_mail_is_visible(context): def open_current_mail(context): - sleep(2) e = find_current_mail(context) e.click() +def get_first_email(context): + return wait_until_elements_are_visible_by_locator(context, (By.CSS_SELECTOR, '#mail-list li span a'))[0] + + @then('I see that mail under the \'{tag}\' tag') def impl(context, tag): context.execute_steps("when I select the tag '%s'" % tag) @@ -40,16 +43,14 @@ def impl(context, tag): @when('I open that mail') def impl(context): - sleep(3) find_current_mail(context).click() @when('I open the first mail in the mail list') def impl(context): - first_email = wait_until_elements_are_visible_by_locator(context, (By.CSS_SELECTOR, '#mail-list li span a'))[0] - context.current_mail_id = 'mail-' + first_email.get_attribute('href').split('/')[-1] - first_email.click() - sleep(5) + # it seems page is often still loading so staleness exceptions happen often + context.current_mail_id = 'mail-' + execute_ignoring_staleness(lambda: get_first_email(context).get_attribute('href').split('/')[-1]) + execute_ignoring_staleness(lambda: get_first_email(context).click()) @when('I open the first mail in the \'{tag}\'') @@ -71,7 +72,7 @@ def impl(context): @then('the deleted mail is there') def impl(context): - find_current_mail(context) + mail_list_with_subject_exists(context, context.last_subject) @given('I have mails') @@ -86,22 +87,35 @@ def impl(context): for email in emails: if 'status-read' not in email.get_attribute('class'): + context.current_mail_id = email.get_attribute('id') # we need to get the mail id before manipulating the page email.find_element_by_tag_name('input').click() find_element_by_id(context, 'mark-selected-as-read').click() - context.current_mail_id = email.get_attribute('id') break - sleep(2) - assert 'status-read' in context.browser.find_element_by_id(context.current_mail_id).get_attribute('class') + wait_until_elements_are_visible_by_locator(context, (By.CSS_SELECTOR, '#%s.status-read' % context.current_mail_id)) @when('I delete the email') def impl(context): def last_email(): - return wait_until_elements_are_visible_by_locator(context, (By.CSS_SELECTOR, '#mail-list li'))[0] - context.current_mail_id = last_email().get_attribute('id') - last_email().find_element_by_tag_name('input').click() + return wait_until_element_is_visible_by_locator(context, (By.CSS_SELECTOR, '#mail-list li')) + mail = last_email() + context.current_mail_id = mail.get_attribute('id') + mail.find_element_by_tag_name('input').click() find_element_by_id(context, 'delete-selected').click() - assert context.current_mail_id != find_elements_by_css_selector(context, '#mail-list li span a')[0] + _wait_for_mail_list_to_be_empty(context) + + +def _wait_for_mail_list_to_be_empty(context): + wait_for_loading_to_finish(context) + + def mail_list_is_empty(_): + with ImplicitWait(context, timeout=0.1): + try: + return 0 == len(context.browser.find_elements_by_css_selector('#mail-list li')) + except TimeoutException: + return False + + wait_for_condition(context, mail_list_is_empty) @when('I check all emails') @@ -116,9 +130,4 @@ def impl(context): @then('I should not see any email') def impl(context): - try: - context.browser.find_element(By.CSS_SELECTOR, '#mail-list li span a') - except NoSuchElementException: - assert True - except: - assert False + _wait_for_mail_list_to_be_empty(context) diff --git a/service/test/functional/features/steps/search.py b/service/test/functional/features/steps/search.py index e653c3ed..879e834d 100644 --- a/service/test/functional/features/steps/search.py +++ b/service/test/functional/features/steps/search.py @@ -13,7 +13,7 @@ # # You should have received a copy of the GNU Affero General Public License # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. -from time import sleep +from selenium.webdriver import ActionChains from selenium.webdriver.common.keys import Keys from common import * @@ -22,9 +22,10 @@ from common import * @when('I search for a mail with the words "{search_term}"') def impl(context, search_term): search_field = find_element_by_css_selector(context, '#search-trigger input[type="search"]') - search_field.send_keys(search_term) - search_field.send_keys(Keys.ENTER) - sleep(1) + ActionChains(context.browser)\ + .send_keys_to_element(search_field, search_term)\ + .send_keys_to_element(search_field, Keys.ENTER)\ + .perform() @then('I see one or more mails in the search results') diff --git a/service/test/functional/features/steps/tag_list.py b/service/test/functional/features/steps/tag_list.py index 443c5173..a3315835 100644 --- a/service/test/functional/features/steps/tag_list.py +++ b/service/test/functional/features/steps/tag_list.py @@ -17,20 +17,20 @@ from common import * def click_first_element_with_class(context, classname): - elements = context.browser.find_elements_by_class_name(classname) - elements[0].click() + element = find_element_by_class_name(context, classname) + element.click() def is_side_nav_expanded(context): - e = context.browser.find_elements_by_class_name('content')[0].get_attribute('class').count(u'move-right') == 1 - return e + e = find_element_by_class_name(context, 'content') + return u'move-right' in e.get_attribute("class") def expand_side_nav(context): if is_side_nav_expanded(context): return - toggle = context.browser.find_elements_by_class_name('side-nav-toggle')[0] + toggle = find_element_by_class_name(context, 'side-nav-toggle') toggle.click() @@ -39,11 +39,24 @@ def impl(context, tag): wait_for_user_alert_to_disapear(context) expand_side_nav(context) - wait_until_element_is_visible_by_locator(context, (By.ID, 'tag-%s' % tag), 20) + # try this multiple times as there are some race conditions + try_again = 2 + success = False + while (not success) and (try_again > 0): + try: + wait_until_element_is_visible_by_locator(context, (By.ID, 'tag-%s' % tag), timeout=20) - e = find_element_by_id(context, 'tag-%s' % tag) - e.click() - wait_until_elements_are_visible_by_locator(context, (By.CSS_SELECTOR, "#mail-list li span a[href*='%s']" % tag)) + e = find_element_by_id(context, 'tag-%s' % tag) + e.click() + + wait_until_element_is_visible_by_locator(context, (By.CSS_SELECTOR, "#mail-list li span a[href*='%s']" % tag), timeout=20) + success = True + except TimeoutException: + pass + finally: + try_again -= 1 + + assert success @when('I am in \'{tag}\'') diff --git a/service/test/integration/test_contacts.py b/service/test/integration/test_contacts.py index 1d82b0d7..4a0957a8 100644 --- a/service/test/integration/test_contacts.py +++ b/service/test/integration/test_contacts.py @@ -14,52 +14,48 @@ # You should have received a copy of the GNU Affero General Public License # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. from test.support.integration import SoledadTestBase, MailBuilder -import os +from twisted.internet import defer import json +import pkg_resources class ContactsTest(SoledadTestBase): + @defer.inlineCallbacks def test_TO_CC_and_BCC_fields_are_being_searched(self): input_mail = MailBuilder().with_tags(['important']).build_input_mail() - self.add_mail_to_inbox(input_mail) + yield self.add_mail_to_inbox(input_mail) - d = self.get_contacts(query='recipient') + contacts = yield self.get_contacts(query='recipient') - def _assert(contacts): - self.assertTrue('recipient@to.com' in contacts) - self.assertTrue('recipient@cc.com' in contacts) - self.assertTrue('recipient@bcc.com' in contacts) - d.addCallback(_assert) - return d + self.assertTrue('recipient@to.com' in contacts) + self.assertTrue('recipient@cc.com' in contacts) + self.assertTrue('recipient@bcc.com' in contacts) + @defer.inlineCallbacks def test_FROM_address_is_being_searched(self): - input_mail = MailBuilder().with_tags(['important']).build_input_mail() - self.add_mail_to_inbox(input_mail) + input_mail = MailBuilder().with_tags(['important']).with_from('Formatted Sender <sender@from.com>').build_input_mail() + yield self.add_mail_to_inbox(input_mail) - d = self.get_contacts(query='Sender') + contacts = yield self.get_contacts(query='Sender') - def _assert(contacts): - self.assertIn('Formatted Sender <sender@from.com>', contacts) - d.addCallback(_assert) - return d + self.assertIn('Formatted Sender <sender@from.com>', contacts) + @defer.inlineCallbacks def test_trash_and_drafts_mailboxes_are_being_ignored(self): - self.add_multiple_to_mailbox(1, mailbox='INBOX', to='recipient@inbox.com') - self.add_multiple_to_mailbox(1, mailbox='DRAFTS', to='recipient@drafts.com') - self.add_multiple_to_mailbox(1, mailbox='SENT', to='recipient@sent.com') - self.add_multiple_to_mailbox(1, mailbox='TRASH', to='recipient@trash.com') + yield self.add_multiple_to_mailbox(1, mailbox='INBOX', to='recipient@inbox.com') + yield self.add_multiple_to_mailbox(1, mailbox='DRAFTS', to='recipient@drafts.com') + yield self.add_multiple_to_mailbox(1, mailbox='SENT', to='recipient@sent.com') + yield self.add_multiple_to_mailbox(1, mailbox='TRASH', to='recipient@trash.com') - d = self.get_contacts(query='recipient') + contacts = yield self.get_contacts(query='recipient') - def _assert(contacts): - self.assertTrue('recipient@inbox.com' in contacts) - self.assertTrue('recipient@sent.com' in contacts) - self.assertFalse('recipient@drafts.com' in contacts) - self.assertFalse('recipient@trash.com' in contacts) - d.addCallback(_assert) - return d + self.assertTrue('recipient@inbox.com' in contacts) + self.assertTrue('recipient@sent.com' in contacts) + self.assertFalse('recipient@drafts.com' in contacts) + self.assertFalse('recipient@trash.com' in contacts) + @defer.inlineCallbacks def test_deduplication_on_same_mail_address_using_largest(self): input_mail = MailBuilder().with_tags(['important']).build_input_mail() @@ -69,44 +65,12 @@ class ContactsTest(SoledadTestBase): formatted_input_mail.with_bcc('Recipient Carbon <recipient@bcc.com>') formatted_input_mail = formatted_input_mail.build_input_mail() - self.add_mail_to_inbox(input_mail) - self.add_mail_to_inbox(formatted_input_mail) - - d = self.get_contacts(query='Recipient') - - def _assert(contacts): - self.assertEquals(3, len(contacts)) - self.assertTrue('Recipient Principal <recipient@to.com>' in contacts) - self.assertTrue('Recipient Copied <recipient@cc.com>' in contacts) - self.assertTrue('Recipient Carbon <recipient@bcc.com>' in contacts) - d.addCallback(_assert) - return d - - def test_bounced_addresses_are_ignored(self): - to_be_bounced = MailBuilder().with_to('this_mail_was_bounced@domain.com').build_input_mail() - self.add_mail_to_inbox(to_be_bounced) - - bounced_mail_template = MailBuilder().build_input_mail() - bounced_mail = self.mailboxes.inbox.add(bounced_mail_template) - bounced_mail.hdoc.content = self._bounced_mail_hdoc_content() - bounced_mail.save() - self.search_engine.index_mail(bounced_mail) - - not_bounced_mail = MailBuilder( - ).with_tags(['important']).with_to('this_mail_was_not@bounced.com').build_input_mail() - self.add_mail_to_inbox(not_bounced_mail) - - d = self.get_contacts(query='this') - - def _assert(contacts): - self.assertNotIn('this_mail_was_bounced@domain.com', contacts) - self.assertNotIn("MAILER-DAEMON@domain.org (Mail Delivery System)", contacts) - self.assertIn('this_mail_was_not@bounced.com', contacts) - d.addCallback(_assert) - return d - - def _bounced_mail_hdoc_content(self): - hdoc_file = os.path.join(os.path.dirname(__file__), '..', 'unit', 'fixtures', 'bounced_mail_hdoc.json') - with open(hdoc_file) as f: - hdoc = json.loads(f.read()) - return hdoc + yield self.add_mail_to_inbox(input_mail) + yield self.add_mail_to_inbox(formatted_input_mail) + + contacts = yield self.get_contacts(query='Recipient') + + self.assertEquals(3, len(contacts)) + self.assertTrue('Recipient Principal <recipient@to.com>' in contacts) + self.assertTrue('Recipient Copied <recipient@cc.com>' in contacts) + self.assertTrue('Recipient Carbon <recipient@bcc.com>' in contacts) diff --git a/service/test/integration/test_delete_mail.py b/service/test/integration/test_delete_mail.py index 987cf307..9e5143e1 100644 --- a/service/test/integration/test_delete_mail.py +++ b/service/test/integration/test_delete_mail.py @@ -13,47 +13,53 @@ # # You should have received a copy of the GNU Affero General Public License # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. - +from twisted.internet import defer from test.support.integration import SoledadTestBase, MailBuilder class DeleteMailTest(SoledadTestBase): + @defer.inlineCallbacks def test_move_mail_to_trash_when_deleting(self): input_mail = MailBuilder().with_subject('Mail with tags').build_input_mail() - self.add_mail_to_inbox(input_mail) + mail = yield self.add_mail_to_inbox(input_mail) - inbox_mails = self.get_mails_by_tag('inbox') + inbox_mails = yield self.get_mails_by_tag('inbox') self.assertEquals(1, len(inbox_mails)) - self.delete_mail(input_mail.ident) + yield self.delete_mail(mail.mail_id) - inbox_mails = self.get_mails_by_tag('inbox') + inbox_mails = yield self.get_mails_by_tag('inbox') self.assertEquals(0, len(inbox_mails)) - trash_mails = self.get_mails_by_tag('trash') + trash_mails = yield self.get_mails_by_tag('trash') self.assertEquals(1, len(trash_mails)) + @defer.inlineCallbacks def test_delete_mail_when_trashing_mail_from_trash_mailbox(self): - mails = self.add_multiple_to_mailbox(1, 'trash') - self.delete_mails([mails[0].ident]) + mails = yield self.add_multiple_to_mailbox(1, 'trash') + yield self.delete_mails([mails[0].ident]) - trash_mails = self.get_mails_by_tag('trash') + trash_mails = yield self.get_mails_by_tag('trash') self.assertEqual(0, len(trash_mails)) + @defer.inlineCallbacks def test_move_mail_to_trash_when_delete_multiple(self): - mails = self.add_multiple_to_mailbox(5, 'inbox') + yield self.add_multiple_to_mailbox(1, 'trash') + mails = yield self.add_multiple_to_mailbox(5, 'inbox') mail_idents = [m.ident for m in mails] - self.delete_mails(mail_idents) + yield self.delete_mails(mail_idents) - inbox = self.get_mails_by_tag('inbox') + inbox = yield self.get_mails_by_tag('inbox') self.assertEquals(0, len(inbox)) + @defer.inlineCallbacks def test_delete_permanently_when_mails_are_in_trash(self): - mails = self.add_multiple_to_mailbox(5, 'trash') - self.delete_mails([m.ident for m in mails]) + mails = yield self.add_multiple_to_mailbox(5, 'trash') + mail_idents = [m.ident for m in mails] - trash = self.get_mails_by_tag('trash') + yield self.delete_mails(mail_idents) + trash = yield self.get_mails_by_tag('trash') self.assertEquals(0, len(trash)) diff --git a/service/pixelated/adapter/soledad/soledad_querier.py b/service/test/integration/test_draft_service.py index e0b215d3..00b1fcfe 100644 --- a/service/pixelated/adapter/soledad/soledad_querier.py +++ b/service/test/integration/test_draft_service.py @@ -13,17 +13,20 @@ # # You should have received a copy of the GNU Affero General Public License # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. -from pixelated.adapter.soledad.soledad_duplicate_removal_mixin import SoledadDuplicateRemovalMixin -from pixelated.adapter.soledad.soledad_reader_mixin import SoledadReaderMixin -from pixelated.adapter.soledad.soledad_search_key_masterkey_retrieval_mixin import SoledadSearchIndexMasterkeyRetrievalMixin -from pixelated.adapter.soledad.soledad_writer_mixin import SoledadWriterMixin +from twisted.internet import defer -class SoledadQuerier(SoledadWriterMixin, - SoledadReaderMixin, - SoledadDuplicateRemovalMixin, - SoledadSearchIndexMasterkeyRetrievalMixin, - object): +from test.support.integration import SoledadTestBase, MailBuilder - def __init__(self, soledad): - self.soledad = soledad + +class DraftServiceTest(SoledadTestBase): + + @defer.inlineCallbacks + def test_store_and_load_draft(self): + input_mail = MailBuilder().with_body('some test text').build_input_mail() + + stored_draft = yield self.draft_service.create_draft(input_mail) + + draft = yield self.mail_store.get_mail(stored_draft.ident, include_body=True) + + self.assertEqual('some test text', draft.body) diff --git a/service/test/integration/test_drafts.py b/service/test/integration/test_drafts.py index 3a0f120b..d0505d75 100644 --- a/service/test/integration/test_drafts.py +++ b/service/test/integration/test_drafts.py @@ -16,7 +16,7 @@ from test.support.integration import SoledadTestBase, MailBuilder from mockito import unstub, when, any -from twisted.internet.defer import Deferred +from twisted.internet import defer class DraftsTest(SoledadTestBase): @@ -24,14 +24,15 @@ class DraftsTest(SoledadTestBase): def tearDown(self): unstub() + @defer.inlineCallbacks def test_post_sends_mail_and_deletes_previous_draft_if_it_exists(self): - # act is if sending the mail by SMTP succeeded - sendmail_deferred = Deferred() + # act as if sending the mail by SMTP succeeded + sendmail_deferred = defer.Deferred() when(self.mail_sender).sendmail(any()).thenReturn(sendmail_deferred) # creates one draft first_draft = MailBuilder().with_subject('First draft').build_json() - first_draft_ident = self.put_mail(first_draft)[0]['ident'] + first_draft_ident = (yield self.put_mail(first_draft)[0])['ident'] # sends an updated version of the draft second_draft = MailBuilder().with_subject('Second draft').with_ident(first_draft_ident).build_json() @@ -39,69 +40,68 @@ class DraftsTest(SoledadTestBase): sendmail_deferred.callback(None) # SMTP succeeded - def onSuccess(mail): - sent_mails = self.get_mails_by_tag('sent') - drafts = self.get_mails_by_tag('drafts') + yield deferred_res - # make sure there is one email in the sent mailbox and it is the second draft - self.assertEquals(1, len(sent_mails)) - self.assertEquals('Second draft', sent_mails[0].subject) + sent_mails = yield self.get_mails_by_tag('sent') + drafts = yield self.get_mails_by_tag('drafts') - # make sure that there are no drafts in the draft mailbox - self.assertEquals(0, len(drafts)) + # make sure there is one email in the sent mailbox and it is the second draft + self.assertEquals(1, len(sent_mails)) + self.assertEquals('Second draft', sent_mails[0].subject) - deferred_res.addCallback(onSuccess) - return deferred_res + # make sure that there are no drafts in the draft mailbox + self.assertEquals(0, len(drafts)) + @defer.inlineCallbacks def test_post_sends_mail_even_when_draft_does_not_exist(self): - # act is if sending the mail by SMTP succeeded - sendmail_deferred = Deferred() + # act as if sending the mail by SMTP succeeded + sendmail_deferred = defer.Deferred() when(self.mail_sender).sendmail(any()).thenReturn(sendmail_deferred) first_draft = MailBuilder().with_subject('First draft').build_json() - deferred_res = self.post_mail(first_draft) + res = self.post_mail(first_draft) sendmail_deferred.callback(True) + yield res - def onSuccess(result): - sent_mails = self.get_mails_by_tag('sent') - drafts = self.get_mails_by_tag('drafts') - - self.assertEquals(1, len(sent_mails)) - self.assertEquals('First draft', sent_mails[0].subject) - self.assertEquals(0, len(drafts)) + sent_mails = yield self.get_mails_by_tag('sent') + drafts = yield self.get_mails_by_tag('drafts') - deferred_res.addCallback(onSuccess) - return deferred_res + self.assertEquals(1, len(sent_mails)) + self.assertEquals('First draft', sent_mails[0].subject) + self.assertEquals(0, len(drafts)) def post_mail(self, data): deferred_res, req = self.post('/mails', data) - deferred_res.callback(None) return deferred_res + @defer.inlineCallbacks def test_put_creates_a_draft_if_it_does_not_exist(self): mail = MailBuilder().with_subject('A new draft').build_json() - self.put_mail(mail) - mails = self.get_mails_by_tag('drafts') + yield self.put_mail(mail)[0] + mails = yield self.get_mails_by_tag('drafts') self.assertEquals('A new draft', mails[0].subject) + @defer.inlineCallbacks def test_put_updates_draft_if_it_already_exists(self): draft = MailBuilder().with_subject('First draft').build_json() - draft_ident = self.put_mail(draft)[0]['ident'] + draft_ident = (yield self.put_mail(draft)[0])['ident'] updated_draft = MailBuilder().with_subject('First draft edited').with_ident(draft_ident).build_json() - self.put_mail(updated_draft) + yield self.put_mail(updated_draft)[0] - drafts = self.get_mails_by_tag('drafts') + drafts = yield self.get_mails_by_tag('drafts') self.assertEquals(1, len(drafts)) self.assertEquals('First draft edited', drafts[0].subject) + @defer.inlineCallbacks def test_respond_unprocessable_entity_if_draft_to_remove_doesnt_exist(self): draft = MailBuilder().with_subject('First draft').build_json() - self.put_mail(draft) + yield self.put_mail(draft)[0] updated_draft = MailBuilder().with_subject('First draft edited').with_ident('NOTFOUND').build_json() - _, request = self.put_mail(updated_draft) + response, request = self.put_mail(updated_draft) + yield response self.assertEquals(422, request.code) diff --git a/service/test/integration/test_feedback_service.py b/service/test/integration/test_feedback_service.py new file mode 100644 index 00000000..dd32374e --- /dev/null +++ b/service/test/integration/test_feedback_service.py @@ -0,0 +1,16 @@ +import os +import unittest +from mockito import when +from twisted.internet import defer +from test.support.integration import AppTestClient + + +class TestFeedbackService(unittest.TestCase, AppTestClient): + @defer.inlineCallbacks + def test_open_ticket(self): + yield self.start_client() + self.feedback_service.FEEDBACK_URL = "https://dev.pixelated-project.org/tickets" + when(self.leap_session).account_email().thenReturn("text@pixelated-project.org") + response = self.feedback_service.open_ticket("Pixelated is awesome!") + + self.assertEquals(response.status_code, 200) diff --git a/service/test/integration/test_incoming_mail.py b/service/test/integration/test_incoming_mail.py new file mode 100644 index 00000000..8a5540dc --- /dev/null +++ b/service/test/integration/test_incoming_mail.py @@ -0,0 +1,50 @@ +# +# Copyright (c) 2014 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +from pixelated.adapter.listeners.mailbox_indexer_listener import MailboxIndexerListener +from test.support.integration import SoledadTestBase, MailBuilder +from twisted.internet import defer, reactor + +IGNORED = None + + +class IncomingMailTest(SoledadTestBase): + + @defer.inlineCallbacks + def test_message_collection(self): + # given + MailboxIndexerListener.SEARCH_ENGINE = self.search_engine + mbx = yield self.account.getMailbox('INBOX') + input_mail = MailBuilder().build_input_mail() + + # when + yield MailboxIndexerListener.listen(self.account, 'INBOX', self.mail_store) + yield mbx.addMessage(input_mail.raw, [], notify_just_mdoc=False) + + # then + yield self.wait_in_reactor() # event handlers are called async, wait for it + + mails, mail_count = self.search_engine.search('in:all') + self.assertEqual(1, mail_count) + self.assertEqual(1, len(mails)) + + def wait_in_reactor(self): + d = defer.Deferred() + + def done_waiting(): + d.callback(None) + + reactor.callLater(1, done_waiting) + return d diff --git a/service/test/integration/test_leap_mailstore.py b/service/test/integration/test_leap_mailstore.py new file mode 100644 index 00000000..8f401bdd --- /dev/null +++ b/service/test/integration/test_leap_mailstore.py @@ -0,0 +1,116 @@ +# +# Copyright (c) 2015 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +from email.mime.application import MIMEApplication +from email.mime.multipart import MIMEMultipart +from email.mime.text import MIMEText +from test.support.integration import SoledadTestBase, load_mail_from_file +from twisted.internet import defer + + +class LeapMailStoreTest(SoledadTestBase): + + @defer.inlineCallbacks + def setUp(self): + yield super(LeapMailStoreTest, self).setUp() + + @defer.inlineCallbacks + def test_get_mail_with_body(self): + self.maxDiff = None + mail = load_mail_from_file('mbox00000000') + mail_id = yield self._create_mail_in_soledad(mail) + expected_mail_dict = {'body': u'Dignissimos ducimus veritatis. Est tenetur consequatur quia occaecati. Vel sit sit voluptas.\n\nEarum distinctio eos. Accusantium qui sint ut quia assumenda. Facere dignissimos inventore autem sit amet. Pariatur voluptatem sint est.\n\nUt recusandae praesentium aspernatur. Exercitationem amet placeat deserunt quae consequatur eum. Unde doloremque suscipit quia.\n\n', 'header': {u'date': u'Tue, 21 Apr 2015 08:43:27 +0000 (UTC)', u'to': [u'carmel@murazikortiz.name'], u'x-tw-pixelated-tags': u'nite, macro, trash', u'from': u'darby.senger@zemlak.biz', u'subject': u'Itaque consequatur repellendus provident sunt quia.'}, 'ident': mail_id, 'status': [], 'tags': set([]), 'replying': {'all': {'cc-field': [], 'to-field': [u'carmel@murazikortiz.name', u'darby.senger@zemlak.biz']}, 'single': u'darby.senger@zemlak.biz'}, 'textPlainBody': u'Dignissimos ducimus veritatis. Est tenetur consequatur quia occaecati. Vel sit sit voluptas.\n\nEarum distinctio eos. Accusantium qui sint ut quia assumenda. Facere dignissimos inventore autem sit amet. Pariatur voluptatem sint est.\n\nUt recusandae praesentium aspernatur. Exercitationem amet placeat deserunt quae consequatur eum. Unde doloremque suscipit quia.\n\n', 'mailbox': u'inbox', 'attachments': [], 'security_casing': {'imprints': [{'state': 'no_signature_information'}], 'locks': []}} + + result = yield self.mail_store.get_mail(mail_id, include_body=True) + self.assertIsNotNone(result) + self.assertEqual(expected_mail_dict, result.as_dict()) + + @defer.inlineCallbacks + def test_round_trip_through_soledad_does_not_modify_content(self): + mail = load_mail_from_file('mbox00000000') + mail_id = yield self._create_mail_in_soledad(mail) + expected_mail_dict = {'body': u'Dignissimos ducimus veritatis. Est tenetur consequatur quia occaecati. Vel sit sit voluptas.\n\nEarum distinctio eos. Accusantium qui sint ut quia assumenda. Facere dignissimos inventore autem sit amet. Pariatur voluptatem sint est.\n\nUt recusandae praesentium aspernatur. Exercitationem amet placeat deserunt quae consequatur eum. Unde doloremque suscipit quia.\n\n', 'header': {u'date': u'Tue, 21 Apr 2015 08:43:27 +0000 (UTC)', u'to': [u'carmel@murazikortiz.name'], u'x-tw-pixelated-tags': u'nite, macro, trash', u'from': u'darby.senger@zemlak.biz', u'subject': u'Itaque consequatur repellendus provident sunt quia.'}, 'ident': mail_id, 'status': [], 'tags': set([])} + + mail = yield self.mail_store.add_mail('INBOX', mail.as_string()) + fetched_mail = yield self.mail_store.get_mail(mail_id, include_body=True) + self.assertEqual(expected_mail_dict['header'], mail.as_dict()['header']) + self.assertEqual(expected_mail_dict['header'], fetched_mail.as_dict()['header']) + + @defer.inlineCallbacks + def test_round_trip_through_soledad_keeps_attachment(self): + input_mail = MIMEMultipart() + input_mail.attach(MIMEText(u'a utf8 message', _charset='utf-8')) + attachment = MIMEApplication('pretend to be binary attachment data') + attachment.add_header('Content-Disposition', 'attachment', filename='filename.txt') + input_mail.attach(attachment) + + mail = yield self.mail_store.add_mail('INBOX', input_mail.as_string()) + fetched_mail = yield self.mail_store.get_mail(mail.ident, include_body=True) + + # _, docs = yield self.soledad.get_all_docs() + # for doc in docs: + # print '\n%s\n' % doc + + # self.assertEqual(1, len(mail.as_dict()['attachments'])) + # print fetched_mail.as_dict() + # self.assertEqual(1, len(fetched_mail.as_dict()['attachments'])) + + @defer.inlineCallbacks + def test_all_mails(self): + mail = load_mail_from_file('mbox00000000') + yield self._create_mail_in_soledad(mail) + + mails = yield self.mail_store.all_mails() + + self.assertEqual(1, len(mails)) + self.assertEqual('Itaque consequatur repellendus provident sunt quia.', mails[0].subject) + + @defer.inlineCallbacks + def test_add_and_remove_mail(self): + yield self.adaptor.initialize_store(self.soledad) + mail = load_mail_from_file('mbox00000000') + yield self.mail_store.add_mailbox('INBOX') + + msg = yield self.mail_store.add_mail('INBOX', mail.as_string()) + + yield self.mail_store.delete_mail(msg.mail_id) + + deleted_msg = yield self.mail_store.get_mail(msg.mail_id) + + self.assertIsNone(deleted_msg) + + @defer.inlineCallbacks + def test_add_add_mail_twice(self): + yield self.adaptor.initialize_store(self.soledad) + mail = load_mail_from_file('mbox00000000', enforceUniqueMessageId=True) + mail2 = load_mail_from_file('mbox00000000', enforceUniqueMessageId=True) + yield self.mail_store.add_mailbox('INBOX') + + msg1 = yield self.mail_store.add_mail('INBOX', mail.as_string()) + msg2 = yield self.mail_store.add_mail('INBOX', mail2.as_string()) + + self.assertIsNotNone(msg1.ident) + self.assertIsNotNone(msg2.ident) + + @defer.inlineCallbacks + def test_get_mailbox_mail_ids(self): + mail = load_mail_from_file('mbox00000000') + yield self.mail_store.add_mailbox('INBOX') + mail = yield self.mail_store.add_mail('INBOX', mail.as_string()) + + mails = yield self.mail_store.get_mailbox_mail_ids('INBOX') + + self.assertEqual(1, len(mails)) + self.assertEqual(mail.mail_id, mails[0]) diff --git a/service/test/integration/test_mark_as_read_unread.py b/service/test/integration/test_mark_as_read_unread.py index 6119f121..48879e4a 100644 --- a/service/test/integration/test_mark_as_read_unread.py +++ b/service/test/integration/test_mark_as_read_unread.py @@ -14,85 +14,93 @@ # You should have received a copy of the GNU Affero General Public License # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +from twisted.internet import defer + from test.support.integration import SoledadTestBase, MailBuilder from pixelated.adapter.model.status import Status class MarkAsReadUnreadTest(SoledadTestBase): + @defer.inlineCallbacks def test_mark_single_as_read(self): input_mail = MailBuilder().build_input_mail() - self.add_mail_to_inbox(input_mail) + mail = yield self.add_mail_to_inbox(input_mail) - mails = self.get_mails_by_tag('inbox') + mails = yield self.get_mails_by_tag('inbox') self.assertNotIn('read', mails[0].status) - self.mark_many_as_read([input_mail.ident]) + yield self.mark_many_as_read([mail.ident]) - mails = self.get_mails_by_tag('inbox') + mails = yield self.get_mails_by_tag('inbox') self.assertIn('read', mails[0].status) + @defer.inlineCallbacks def test_mark_single_as_unread(self): - input_mail = MailBuilder().with_status([Status.SEEN]).build_input_mail() - self.add_mail_to_inbox(input_mail) + input_mail = MailBuilder().build_input_mail() + mail = yield self.add_mail_to_inbox(input_mail) + yield self.mark_many_as_read([mail.ident]) - self.mark_many_as_unread([input_mail.ident]) - mail = self.get_mails_by_tag('inbox')[0] + yield self.mark_many_as_unread([mail.ident]) + result = (yield self.get_mails_by_tag('inbox'))[0] - self.assertNotIn('read', mail.status) + self.assertNotIn('read', result.status) + @defer.inlineCallbacks def test_mark_many_mails_as_unread(self): input_mail = MailBuilder().with_status([Status.SEEN]).build_input_mail() input_mail2 = MailBuilder().with_status([Status.SEEN]).build_input_mail() - self.add_mail_to_inbox(input_mail) - self.add_mail_to_inbox(input_mail2) + mail1 = yield self.add_mail_to_inbox(input_mail) + mail2 = yield self.add_mail_to_inbox(input_mail2) + yield self.mark_many_as_read([mail1.ident, mail2.ident]) - self.mark_many_as_unread([input_mail.ident, input_mail2.ident]) + yield self.mark_many_as_unread([mail1.ident, mail2.ident]) - mails = self.get_mails_by_tag('inbox') + mails = yield self.get_mails_by_tag('inbox') self.assertNotIn('read', mails[0].status) self.assertNotIn('read', mails[1].status) + @defer.inlineCallbacks def test_mark_many_mails_as_read(self): input_mail = MailBuilder().build_input_mail() input_mail2 = MailBuilder().build_input_mail() - self.add_mail_to_inbox(input_mail) - self.add_mail_to_inbox(input_mail2) + yield self.add_mail_to_inbox(input_mail) + yield self.add_mail_to_inbox(input_mail2) - mails = self.get_mails_by_tag('inbox') + mails = yield self.get_mails_by_tag('inbox') self.assertNotIn('read', mails[0].status) self.assertNotIn('read', mails[1].status) - response = self.mark_many_as_read([input_mail.ident, input_mail2.ident]) - self.assertEquals(200, response.code) + yield self.mark_many_as_read([mails[0].ident, mails[1].ident]) - mails = self.get_mails_by_tag('inbox') + mails = yield self.get_mails_by_tag('inbox') self.assertIn('read', mails[0].status) self.assertIn('read', mails[1].status) + @defer.inlineCallbacks def test_mark_mixed_status_as_read(self): - input_mail = MailBuilder().build_input_mail() - input_mail2 = MailBuilder().with_status([Status.SEEN]).build_input_mail() + input_mail = MailBuilder().with_subject('first').build_input_mail() + input_mail2 = MailBuilder().with_subject('second').build_input_mail() - self.add_mail_to_inbox(input_mail) - self.add_mail_to_inbox(input_mail2) + yield self.add_mail_to_inbox(input_mail) + mail2 = yield self.add_mail_to_inbox(input_mail2) + yield self.mark_many_as_read([mail2.ident]) - mails = self.get_mails_by_tag('inbox') + mails = yield self.get_mails_by_tag('inbox') read_mails = filter(lambda x: 'read' in x.status, mails) unread_mails = filter(lambda x: 'read' not in x.status, mails) self.assertEquals(1, len(unread_mails)) self.assertEquals(1, len(read_mails)) - response = self.mark_many_as_read([input_mail.ident, input_mail2.ident]) - self.assertEquals(200, response.code) + yield self.mark_many_as_read([mails[0].ident, mails[1].ident]) - mails = self.get_mails_by_tag('inbox') + mails = yield self.get_mails_by_tag('inbox') self.assertIn('read', mails[0].status) self.assertIn('read', mails[1].status) diff --git a/service/test/integration/test_retrieve_attachment.py b/service/test/integration/test_retrieve_attachment.py index 2c446b42..e7e8670d 100644 --- a/service/test/integration/test_retrieve_attachment.py +++ b/service/test/integration/test_retrieve_attachment.py @@ -13,27 +13,38 @@ # # You should have received a copy of the GNU Affero General Public License # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +from email.mime.application import MIMEApplication +from email.mime.multipart import MIMEMultipart +from email.mime.text import MIMEText from test.support.integration.soledad_test_base import SoledadTestBase +from twisted.internet import defer class RetrieveAttachmentTest(SoledadTestBase): + @defer.inlineCallbacks def test_attachment_content_is_retrieved(self): - ident = 'F4E99C1CEC4D300A4223A96CCABBE0304BDBC31C550A5A03E207A5E4C3C71A22' - attachment_dict = {'content-disposition': 'attachment', - 'content-transfer-encoding': '', - 'type': 'cnt', - 'raw': 'cGVxdWVubyBhbmV4byA6RAo=', - 'phash': ident, - 'content-type': 'text/plain; charset=US-ASCII; name="attachment_pequeno.txt"'} + attachment_id, input_mail = self._create_mail_with_attachment() + yield self.mail_store.add_mail('INBOX', input_mail.as_string()) - self.add_document_to_soledad(attachment_dict) + attachment, req = yield self.get_attachment(attachment_id, 'base64') - d = self.get_attachment(ident, 'base64') + self.assertEqual(200, req.code) + self.assertEquals('pretend to be binary attachment data', attachment) - def _assert(attachment): - self.assertEquals('pequeno anexo :D\n', attachment) - d.addCallback(_assert) + def _create_mail_with_attachment(self): + input_mail = MIMEMultipart() + input_mail.attach(MIMEText(u'a utf8 message', _charset='utf-8')) + attachment = MIMEApplication('pretend to be binary attachment data') + attachment.add_header('Content-Disposition', 'attachment', filename='filename.txt') + input_mail.attach(attachment) + attachment_id = 'B5B4ED80AC3B894523D72E375DACAA2FC6606C18EDF680FE95903086C8B5E14A' + return attachment_id, input_mail - return d + @defer.inlineCallbacks + def test_attachment_error_returned_if_id_not_found(self): + attachment, req = yield self.get_attachment('invalid attachment id', 'base64') + + self.assertEqual(404, req.code) + self.assertIsNone(attachment) diff --git a/service/test/integration/test_search.py b/service/test/integration/test_search.py index f90ed80f..aafcb4fc 100644 --- a/service/test/integration/test_search.py +++ b/service/test/integration/test_search.py @@ -15,130 +15,133 @@ # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. from test.support.integration import SoledadTestBase, MailBuilder +from twisted.internet import defer class SearchTest(SoledadTestBase): + @defer.inlineCallbacks def test_that_tags_returns_all_tags(self): - input_mail = MailBuilder().with_tags(['important']).build_input_mail() - self.add_mail_to_inbox(input_mail) + input_mail = MailBuilder().build_input_mail() + mail = yield self.add_mail_to_inbox(input_mail) + yield self.mail_service.update_tags(mail.ident, ['important']) - d = self.get_tags() + all_tags = yield self.get_tags() - def _assert(all_tags): - all_tag_names = [t['name'] for t in all_tags] - self.assertTrue('inbox' in all_tag_names) - self.assertTrue('sent' in all_tag_names) - self.assertTrue('trash' in all_tag_names) - self.assertTrue('drafts' in all_tag_names) - self.assertTrue('important' in all_tag_names) - d.addCallback(_assert) - return d + all_tag_names = [t['name'] for t in all_tags] + self.assertTrue('inbox' in all_tag_names) + self.assertTrue('sent' in all_tag_names) + self.assertTrue('trash' in all_tag_names) + self.assertTrue('drafts' in all_tag_names) + self.assertTrue('important' in all_tag_names) + @defer.inlineCallbacks def test_that_tags_are_filtered_by_query(self): - input_mail = MailBuilder().with_tags(['ateu', 'catoa', 'luat', 'zuado']).build_input_mail() - self.add_mail_to_inbox(input_mail) - - d = self.get_tags(q=["at"], skipDefaultTags=["true"]) + input_mail = MailBuilder().build_input_mail() + mail = yield self.add_mail_to_inbox(input_mail) + yield self.mail_service.update_tags(mail.ident, ['ateu', 'catoa', 'luat', 'zuado']) - def _assert(all_tags): - all_tag_names = [t['name'] for t in all_tags] - self.assertEqual(3, len(all_tag_names)) - self.assertTrue('ateu' in all_tag_names) - self.assertTrue('catoa' in all_tag_names) - self.assertTrue('luat' in all_tag_names) + all_tags = yield self.get_tags(q=["at"], skipDefaultTags=["true"]) - d.addCallback(_assert) - return d + all_tag_names = [t['name'] for t in all_tags] + self.assertEqual(3, len(all_tag_names)) + self.assertTrue('ateu' in all_tag_names) + self.assertTrue('catoa' in all_tag_names) + self.assertTrue('luat' in all_tag_names) + @defer.inlineCallbacks def test_tags_with_multiple_words_are_searchable(self): - input_mail = MailBuilder().with_tags(['one tag four words']).build_input_mail() - self.add_mail_to_inbox(input_mail) + input_mail = MailBuilder().build_input_mail() + mail = yield self.add_mail_to_inbox(input_mail) + yield self.mail_service.update_tags(mail.ident, ['one tag four words']) - first_page = self.get_mails_by_tag('"one tag four words"', page=1, window=1) + first_page = yield self.get_mails_by_tag('"one tag four words"', page=1, window=1) self.assertEqual(len(first_page), 1) + @defer.inlineCallbacks def test_that_default_tags_are_ignorable(self): - input_mail = MailBuilder().with_tags(['sometag']).build_input_mail() - self.add_mail_to_inbox(input_mail) + input_mail = MailBuilder().build_input_mail() + mail = yield self.add_mail_to_inbox(input_mail) + yield self.mail_service.update_tags(mail.ident, ['sometag']) - d = self.get_tags(skipDefaultTags=["true"]) + all_tags = yield self.get_tags(skipDefaultTags=["true"]) - def _assert(all_tags): - all_tag_names = [t['name'] for t in all_tags] - self.assertEqual(1, len(all_tag_names)) - self.assertTrue('sometag' in all_tag_names) - d.addCallback(_assert) - return d + all_tag_names = [t['name'] for t in all_tags] + self.assertEqual(1, len(all_tag_names)) + self.assertTrue('sometag' in all_tag_names) + @defer.inlineCallbacks def test_tags_count(self): - self.add_multiple_to_mailbox(num=10, mailbox='inbox', flags=['\\Recent']) - self.add_multiple_to_mailbox(num=5, mailbox='inbox', flags=['\\Seen']) - self.add_multiple_to_mailbox(num=3, mailbox='inbox', flags=['\\Recent'], tags=['important', 'later']) - self.add_multiple_to_mailbox(num=1, mailbox='inbox', flags=['\\Seen'], tags=['important']) + yield self.add_multiple_to_mailbox(num=10, mailbox='inbox', flags=['\\Recent']) + yield self.add_multiple_to_mailbox(num=5, mailbox='inbox', flags=['\\Seen']) + yield self.add_multiple_to_mailbox(num=3, mailbox='inbox', flags=['\\Recent'], tags=['important', 'later']) + yield self.add_multiple_to_mailbox(num=1, mailbox='inbox', flags=['\\Seen'], tags=['important']) - d = self.get_tags() + tags_count = yield self.get_tags() - def _assert(tags_count): - self.assertEqual(self.get_count(tags_count, 'inbox')['total'], 19) - self.assertEqual(self.get_count(tags_count, 'inbox')['read'], 6) - self.assertEqual(self.get_count(tags_count, 'important')['total'], 4) - self.assertEqual(self.get_count(tags_count, 'important')['read'], 1) - d.addCallback(_assert) - return d + self.assertEqual(self.get_count(tags_count, 'inbox')['total'], 19) + self.assertEqual(self.get_count(tags_count, 'inbox')['read'], 6) + self.assertEqual(self.get_count(tags_count, 'important')['total'], 4) + self.assertEqual(self.get_count(tags_count, 'important')['read'], 1) + @defer.inlineCallbacks def test_search_mails_different_window(self): input_mail = MailBuilder().build_input_mail() input_mail2 = MailBuilder().build_input_mail() - self.add_mail_to_inbox(input_mail) - self.add_mail_to_inbox(input_mail2) + yield self.add_mail_to_inbox(input_mail) + yield self.add_mail_to_inbox(input_mail2) - first_page = self.get_mails_by_tag('inbox', page=1, window=1) + first_page = yield self.get_mails_by_tag('inbox', page=1, window=1) self.assertEqual(len(first_page), 1) + @defer.inlineCallbacks def test_search_mails_with_multiple_pages(self): input_mail = MailBuilder().build_input_mail() input_mail2 = MailBuilder().build_input_mail() - self.add_mail_to_inbox(input_mail) - self.add_mail_to_inbox(input_mail2) + mail1 = yield self.add_mail_to_inbox(input_mail) + mail2 = yield self.add_mail_to_inbox(input_mail2) - first_page = self.get_mails_by_tag('inbox', page=1, window=1) - second_page = self.get_mails_by_tag('inbox', page=2, window=1) + first_page = yield self.get_mails_by_tag('inbox', page=1, window=1) + second_page = yield self.get_mails_by_tag('inbox', page=2, window=1) - idents = [input_mail.ident, input_mail2.ident] + idents = [mail1.ident, mail2.ident] self.assertIn(first_page[0].ident, idents) self.assertIn(second_page[0].ident, idents) + @defer.inlineCallbacks def test_page_zero_fetches_first_page(self): input_mail = MailBuilder().build_input_mail() - self.add_mail_to_inbox(input_mail) - page = self.get_mails_by_tag('inbox', page=0, window=1) - self.assertEqual(page[0].ident, input_mail.ident) + mail = yield self.add_mail_to_inbox(input_mail) + page = yield self.get_mails_by_tag('inbox', page=0, window=1) + self.assertEqual(page[0].ident, mail.ident) def get_count(self, tags_count, mailbox): for tag in tags_count: if tag['name'] == mailbox: return tag['counts'] + @defer.inlineCallbacks def test_order_by_date(self): input_mail = MailBuilder().with_date('2014-10-15T15:15').build_input_mail() input_mail2 = MailBuilder().with_date('2014-10-15T15:16').build_input_mail() - self.add_mail_to_inbox(input_mail) - self.add_mail_to_inbox(input_mail2) + mail1 = yield self.add_mail_to_inbox(input_mail) + mail2 = yield self.add_mail_to_inbox(input_mail2) - results = self.get_mails_by_tag('inbox') - self.assertEqual(results[0].ident, input_mail2.ident) - self.assertEqual(results[1].ident, input_mail.ident) + results = yield self.get_mails_by_tag('inbox') + self.assertEqual(results[0].ident, mail2.ident) + self.assertEqual(results[1].ident, mail1.ident) + @defer.inlineCallbacks def test_search_base64_body(self): body = u'bl\xe1' input_mail = MailBuilder().with_body(body.encode('utf-8')).build_input_mail() - self.add_mail_to_inbox(input_mail) - results = self.search(body) + + mail = yield self.add_mail_to_inbox(input_mail) + results = yield self.search(body) self.assertGreater(len(results), 0, 'No results returned from search') - self.assertEquals(results[0].ident, input_mail.ident) + self.assertEquals(results[0].ident, mail.ident) diff --git a/service/test/integration/test_soledad_querier.py b/service/test/integration/test_soledad_querier.py deleted file mode 100644 index f4c23961..00000000 --- a/service/test/integration/test_soledad_querier.py +++ /dev/null @@ -1,83 +0,0 @@ -# -# Copyright (c) 2014 ThoughtWorks, Inc. -# -# Pixelated is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# Pixelated is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. - -import copy -import time - -from test.support.integration import SoledadTestBase, MailBuilder -from leap.mail.imap.fields import WithMsgFields - - -class SoledadQuerierTest(SoledadTestBase, WithMsgFields): - - def setUp(self): - SoledadTestBase.setUp(self) - self.maxDiff = None - - def _get_empty_mailbox(self): - return copy.deepcopy(self.EMPTY_MBOX) - - def _create_mailbox(self, mailbox_name): - new_mailbox = self._get_empty_mailbox() - new_mailbox['mbox'] = mailbox_name - new_mailbox['created'] = int(time.time() * 10E2) - return self.soledad.create_doc(new_mailbox) - - def _get_mailboxes_from_soledad(self, mailbox_name): - return [m for m in self.soledad.get_from_index('by-type', 'mbox') if m.content['mbox'] == mailbox_name] - - def test_remove_dup_mailboxes_keeps_the_one_with_the_highest_last_uid(self): - self.add_multiple_to_mailbox(3, 'INBOX') # by now we already have one inbox with 3 mails - self._create_mailbox('INBOX') # now we have a duplicate - - # make sure we have two - inboxes = self._get_mailboxes_from_soledad('INBOX') - self.assertEqual(2, len(inboxes)) - - self.soledad_querier.remove_duplicates() - - # make sure we only have one, and the one with the right lastuid - inboxes = self._get_mailboxes_from_soledad('INBOX') - self.assertEqual(1, len(inboxes)) - self.assertEqual(3, inboxes[0].content['lastuid']) - - def test_all_mails_skips_incomplete_mails(self): - # creating incomplete mail, we will only save the fdoc - fdoc, hdoc, bdoc = MailBuilder().build_input_mail().get_for_save(1, 'INBOX') - self.soledad.create_doc(fdoc) - - mails = self.soledad_querier.all_mails() - self.assertEqual(0, len(mails)) # mail is incomplete since it only has fdoc - - # adding the hdoc still doesn't complete the mail - self.soledad.create_doc(hdoc) - - mails = self.soledad_querier.all_mails() - self.assertEqual(0, len(mails)) - - # now the mail is complete - self.soledad.create_doc(bdoc) - - mails = self.soledad_querier.all_mails() - self.assertEqual(1, len(mails)) - - def test_get_mails_by_chash(self): - mails = self.add_multiple_to_mailbox(3, 'INBOX') - chashes = [mail.ident for mail in mails] - - fetched_mails = self.soledad_querier.mails(chashes) - - self.assertEquals([m.as_dict() for m in fetched_mails], [m.as_dict() for m in mails]) diff --git a/service/test/integration/test_tags.py b/service/test/integration/test_tags.py index 168e035f..0e0fe66c 100644 --- a/service/test/integration/test_tags.py +++ b/service/test/integration/test_tags.py @@ -15,6 +15,8 @@ # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. import json +from twisted.internet import defer + from test.support.integration import SoledadTestBase, MailBuilder from pixelated.adapter.services.tag_service import SPECIAL_TAGS @@ -24,67 +26,72 @@ class TagsTest(SoledadTestBase): def _tags_json(self, tags): return json.dumps({'newtags': tags}) + @defer.inlineCallbacks def test_add_tag_to_an_inbox_mail_and_query(self): - mail = MailBuilder().with_subject('Mail with tags').build_input_mail() - self.add_mail_to_inbox(mail) + input_mail = MailBuilder().with_subject('Mail with tags').build_input_mail() + mail = yield self.add_mail_to_inbox(input_mail) - self.post_tags(mail.ident, self._tags_json(['IMPORTANT'])) + yield self.post_tags(mail.ident, self._tags_json(['IMPORTANT'])) - mails = self.get_mails_by_tag('inbox') + mails = yield self.get_mails_by_tag('inbox') self.assertEquals({'IMPORTANT'}, set(mails[0].tags)) - mails = self.get_mails_by_tag('IMPORTANT') + mails = yield self.get_mails_by_tag('IMPORTANT') self.assertEquals('Mail with tags', mails[0].subject) + @defer.inlineCallbacks def test_use_old_casing_when_same_tag_with_different_casing_is_posted(self): - mail = MailBuilder().with_subject('Mail with tags').build_input_mail() - self.add_mail_to_inbox(mail) - self.post_tags(mail.ident, self._tags_json(['ImPoRtAnT'])) - mails = self.get_mails_by_tag('ImPoRtAnT') + input_mail = MailBuilder().with_subject('Mail with tags').build_input_mail() + mail = yield self.add_mail_to_inbox(input_mail) + yield self.post_tags(mail.ident, self._tags_json(['ImPoRtAnT'])) + mails = yield self.get_mails_by_tag('ImPoRtAnT') self.assertEquals({'ImPoRtAnT'}, set(mails[0].tags)) - another_mail = MailBuilder().with_subject('Mail with tags').build_input_mail() - self.add_mail_to_inbox(another_mail) - self.post_tags(another_mail.ident, self._tags_json(['IMPORTANT'])) - mails = self.get_mails_by_tag('IMPORTANT') + another_input_mail = MailBuilder().with_subject('Mail with tags').build_input_mail() + another_mail = yield self.add_mail_to_inbox(another_input_mail) + yield self.post_tags(another_mail.ident, self._tags_json(['IMPORTANT'])) + mails = yield self.get_mails_by_tag('IMPORTANT') self.assertEquals(0, len(mails)) - mails = self.get_mails_by_tag('ImPoRtAnT') + mails = yield self.get_mails_by_tag('ImPoRtAnT') self.assertEquals(2, len(mails)) self.assertEquals({'ImPoRtAnT'}, set(mails[0].tags)) self.assertEquals({'ImPoRtAnT'}, set(mails[1].tags)) + @defer.inlineCallbacks def test_tags_are_case_sensitive(self): - mail = MailBuilder().with_subject('Mail with tags').build_input_mail() - self.add_mail_to_inbox(mail) + input_mail = MailBuilder().with_subject('Mail with tags').build_input_mail() + mail = yield self.add_mail_to_inbox(input_mail) - self.post_tags(mail.ident, self._tags_json(['ImPoRtAnT'])) + yield self.post_tags(mail.ident, self._tags_json(['ImPoRtAnT'])) - mails = self.get_mails_by_tag('important') + mails = yield self.get_mails_by_tag('important') self.assertEquals(0, len(mails)) - mails = self.get_mails_by_tag('IMPORTANT') + mails = yield self.get_mails_by_tag('IMPORTANT') self.assertEquals(0, len(mails)) - mails = self.get_mails_by_tag('ImPoRtAnT') + mails = yield self.get_mails_by_tag('ImPoRtAnT') self.assertEquals({'ImPoRtAnT'}, set(mails[0].tags)) + @defer.inlineCallbacks def test_empty_tags_are_not_allowed(self): - mail = MailBuilder().with_subject('Mail with tags').build_input_mail() - self.add_mail_to_inbox(mail) + input_mail = MailBuilder().with_subject('Mail with tags').build_input_mail() + mail = yield self.add_mail_to_inbox(input_mail) - self.post_tags(mail.ident, self._tags_json(['tag1', ' '])) + yield self.post_tags(mail.ident, self._tags_json(['tag1', ' '])) - mail = self.get_mail(mail.ident) + mail = yield self.get_mail(mail.ident) self.assertEquals(mail['tags'], ['tag1']) + @defer.inlineCallbacks def test_addition_of_reserved_tags_is_not_allowed(self): - mail = MailBuilder().with_subject('Mail with tags').build_input_mail() - self.add_mail_to_inbox(mail) + input_mail = MailBuilder().with_subject('Mail with tags').build_input_mail() + mail = yield self.add_mail_to_inbox(input_mail) for tag in SPECIAL_TAGS: - response = self.post_tags(mail.ident, self._tags_json([tag.name.upper()])) + response = yield self.post_tags(mail.ident, self._tags_json([tag.name.upper()])) self.assertEquals("None of the following words can be used as tags: %s" % tag.name, response) - mail = self.mailboxes.inbox.mail(mail.ident) + mail = yield self.mail_store.get_mail(mail.ident) self.assertNotIn('drafts', mail.tags) diff --git a/service/test/integration/test_welcome_mail.py b/service/test/integration/test_welcome_mail.py deleted file mode 100644 index a5ca555a..00000000 --- a/service/test/integration/test_welcome_mail.py +++ /dev/null @@ -1,34 +0,0 @@ -# -# Copyright (c) 2014 ThoughtWorks, Inc. -# -# Pixelated is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# Pixelated is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. - -from test.support.integration import SoledadTestBase - - -class TestWelcomeMail(SoledadTestBase): - - def test_welcome_mail_is_added_only_once(self): - self.mailboxes.add_welcome_mail_for_fresh_user() - self.mailboxes.add_welcome_mail_for_fresh_user() - inbox_mails = self.get_mails_by_tag('inbox') - self.assertEquals(1, len(inbox_mails)) - - def test_empty_mailbox_doesnt_mean_fresh_mailbox(self): - self.mailboxes.add_welcome_mail_for_fresh_user() - inbox_mails = self.get_mails_by_tag('inbox') - self.delete_mail(inbox_mails[0].ident) - self.mailboxes.add_welcome_mail_for_fresh_user() - inbox_mails = self.get_mails_by_tag('inbox') - self.assertEquals(0, len(inbox_mails)) diff --git a/service/test/perf/contacts/test_Contacts.py b/service/test/perf/contacts/test_Contacts.py index 8bfb898d..967e9eb1 100644 --- a/service/test/perf/contacts/test_Contacts.py +++ b/service/test/perf/contacts/test_Contacts.py @@ -14,22 +14,37 @@ # You should have received a copy of the GNU Affero General Public License # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. import unittest +import logging from funkload.FunkLoadTestCase import FunkLoadTestCase from test.support.integration import AppTestClient +from test.support.dispatcher.proxy import Proxy +from crochet import setup, wait_for +from leap.common.events.server import ensure_server +setup() + + +@wait_for(timeout=5.0) +def start_app_test_client(client): + ensure_server() + return client.start_client() class Contacts(FunkLoadTestCase): def setUpBench(self): + logging.disable('INFO') client = AppTestClient() + start_app_test_client(client) + client.listenTCP() + proxy = Proxy(proxy_port='8889', app_port='4567') # setup data client.add_multiple_to_mailbox(10, 'INBOX', to='to@inbox.com', cc='cc@inbox.com', bcc='bcc@inbox.com') client.add_multiple_to_mailbox(10, 'TRASH', to='to@trash.com', cc='cc@trash.com', bcc='bcc@trash.com') client.add_multiple_to_mailbox(10, 'DRAFTS', to='to@drafts.com', cc='cc@drafts.com', bcc='bcc@drafts.com') - self.call_to_terminate = client.run_on_a_thread(logfile='results/app.log') + self.call_to_terminate = proxy.run_on_a_thread() def tearDownBench(self): self.call_to_terminate() diff --git a/service/test/support/integration/__init__.py b/service/test/support/integration/__init__.py index 8d0bbb7a..1c94eab8 100644 --- a/service/test/support/integration/__init__.py +++ b/service/test/support/integration/__init__.py @@ -16,3 +16,4 @@ from .app_test_client import AppTestClient from .model import MailBuilder, ResponseMail from .soledad_test_base import SoledadTestBase +from .util import load_mail_from_file diff --git a/service/test/support/integration/app_test_client.py b/service/test/support/integration/app_test_client.py index 52372507..369a393d 100644 --- a/service/test/support/integration/app_test_client.py +++ b/service/test/support/integration/app_test_client.py @@ -15,26 +15,31 @@ # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. import json import multiprocessing +from leap.mail.adaptors.soledad import SoledadMailAdaptor from mockito import mock import os import shutil import time import uuid +import random -from leap.mail.imap.account import SoledadBackedAccount + +from leap.mail.imap.account import IMAPAccount from leap.soledad.client import Soledad -from mock import MagicMock, Mock -from twisted.internet import reactor +from mock import Mock +from twisted.internet import reactor, defer from twisted.internet.defer import succeed from twisted.web.resource import getChildForRequest -from twisted.web.server import Site +# from twisted.web.server import Site as PixelatedSite +from pixelated.adapter.services.feedback_service import FeedbackService +from pixelated.config.site import PixelatedSite + +from pixelated.adapter.mailstore import LeapMailStore +from pixelated.adapter.mailstore.searchable_mailstore import SearchableMailStore -from pixelated.adapter.model.mail import PixelatedMail from pixelated.adapter.search import SearchEngine from pixelated.adapter.services.draft_service import DraftService from pixelated.adapter.services.mail_service import MailService -from pixelated.adapter.services.mailboxes import Mailboxes -from pixelated.adapter.soledad.soledad_querier import SoledadQuerier from pixelated.resources.root_resource import RootResource from test.support.integration.model import MailBuilder from test.support.test_helper import request_mock @@ -47,33 +52,39 @@ class AppTestClient(object): ACCOUNT = 'test' MAIL_ADDRESS = 'test@pixelated.org' - def __init__(self): - self.start_client() + # def __init__(self): + # self.start_client() + @defer.inlineCallbacks def start_client(self): soledad_test_folder = self._generate_soledad_test_folder_name() SearchEngine.DEFAULT_INDEX_HOME = soledad_test_folder self.cleanup = lambda: shutil.rmtree(soledad_test_folder) - PixelatedMail.from_email_address = self.MAIL_ADDRESS + self.soledad = yield initialize_soledad(tempdir=soledad_test_folder) - self.soledad = initialize_soledad(tempdir=soledad_test_folder) - self.soledad_querier = self._create_soledad_querier(self.soledad, self.INDEX_KEY) self.keymanager = mock() self.search_engine = SearchEngine(self.INDEX_KEY, agent_home=soledad_test_folder) self.mail_sender = self._create_mail_sender() - self.account = SoledadBackedAccount(self.ACCOUNT, self.soledad, MagicMock()) - self.mailboxes = Mailboxes(self.account, self.soledad_querier, self.search_engine) - self.draft_service = DraftService(self.mailboxes) + self.mail_store = SearchableMailStore(LeapMailStore(self.soledad), self.search_engine) - self.mail_service = self._create_mail_service(self.mailboxes, self.mail_sender, self.soledad_querier, self.search_engine) - self.search_engine.index_mails(self.mail_service.all_mails()) + account_ready_cb = defer.Deferred() + self.account = IMAPAccount(self.ACCOUNT, self.soledad, account_ready_cb) + yield account_ready_cb + self.draft_service = DraftService(self.mail_store) + self.leap_session = mock() + self.feedback_service = FeedbackService(self.leap_session) + + self.mail_service = self._create_mail_service(self.mail_sender, self.mail_store, self.search_engine) + mails = yield self.mail_service.all_mails() + self.search_engine.index_mails(mails) self.resource = RootResource() - self.resource.initialize(self.keymanager, self.search_engine, self.mail_service, self.draft_service) + self.resource.initialize( + self.keymanager, self.search_engine, self.mail_service, self.draft_service, self.feedback_service) def _render(self, request, as_json=True): def get_str(_str): @@ -95,9 +106,12 @@ class AppTestClient(object): d.addCallback(get_request_written_data) return d, request - def run_on_a_thread(self, logfile='/tmp/app_test_client.log', port=4567, host='0.0.0.0'): + def listenTCP(self, port=4567, host='127.0.0.1'): + reactor.listenTCP(port, PixelatedSite(self.resource), interface=host) + + def run_on_a_thread(self, logfile='/tmp/app_test_client.log', port=4567, host='127.0.0.1'): def _start(): - reactor.listenTCP(port, Site(self.resource), interface=host) + self.listenTCP(port, host) reactor.run() process = multiprocessing.Process(target=_start) process.start() @@ -121,37 +135,38 @@ class AppTestClient(object): request = request_mock(path=path, body=body, headers={'Content-Type': ['application/json']}, method="DELETE") return self._render(request) - def add_document_to_soledad(self, _dict): - self.soledad_querier.soledad.create_doc(_dict) - + @defer.inlineCallbacks def add_mail_to_inbox(self, input_mail): - mail = self.mailboxes.inbox.add(input_mail) - if input_mail.tags: - mail.update_tags(input_mail.tags) - self.search_engine.index_mail(mail) + mail = yield self.mail_store.add_mail('INBOX', input_mail.raw) + defer.returnValue(mail) + @defer.inlineCallbacks def add_multiple_to_mailbox(self, num, mailbox='', flags=[], tags=[], to='recipient@to.com', cc='recipient@cc.com', bcc='recipient@bcc.com'): mails = [] + yield self.mail_store.add_mailbox(mailbox) for _ in range(num): - input_mail = MailBuilder().with_status(flags).with_tags(tags).with_to(to).with_cc(cc).with_bcc(bcc).build_input_mail() - mail = self.mailboxes._create_or_get(mailbox).add(input_mail) + builder = MailBuilder().with_status(flags).with_tags(tags).with_to(to).with_cc(cc).with_bcc(bcc) + builder.with_body(str(random.random())) + input_mail = builder.build_input_mail() + mail = yield self.mail_store.add_mail(mailbox, input_mail.raw) + if tags: + mail.tags |= set(tags) + if flags: + for flag in flags: + mail.flags.add(flag) + if tags or flags: + yield self.mail_store.update_mail(mail) mails.append(mail) - mail.update_tags(input_mail.tags) if tags else None - self.search_engine.index_mails(mails) if tags else None - return mails - def _create_soledad_querier(self, soledad, index_key): - soledad_querier = SoledadQuerier(soledad) - soledad_querier.get_index_masterkey = lambda: index_key - return soledad_querier + defer.returnValue(mails) def _create_mail_sender(self): mail_sender = Mock() mail_sender.sendmail.side_effect = lambda mail: succeed(mail) return mail_sender - def _create_mail_service(self, mailboxes, mail_sender, soledad_querier, search_engine): - mail_service = MailService(mailboxes, mail_sender, soledad_querier, search_engine) + def _create_mail_service(self, mail_sender, mail_store, search_engine): + mail_service = MailService(mail_sender, mail_store, search_engine) return mail_service def _generate_soledad_test_folder_name(self, soledad_test_folder='/tmp/soledad-test/test'): @@ -161,17 +176,27 @@ class AppTestClient(object): tags = 'tag:%s' % tag return self.search(tags, page, window) + @defer.inlineCallbacks def search(self, query, page=1, window=100): - res, req = self.get("/mails", { + res, _ = self.get("/mails", { 'q': [query], 'w': [str(window)], 'p': [str(page)] }) - return [ResponseMail(m) for m in res['mails']] + res = yield res + defer.returnValue([ResponseMail(m) for m in res['mails']]) + + @defer.inlineCallbacks + def get_mails_by_mailbox_name(self, mbox_name): + mail_ids = yield self.mail_store.get_mailbox_mail_ids(mbox_name) + mails = yield self.mail_store.get_mails(mail_ids) + defer.returnValue(mails) + @defer.inlineCallbacks def get_attachment(self, ident, encoding): - res, req = self.get("/attachment/%s" % ident, {'encoding': [encoding]}, as_json=False) - return res + deferred_result, req = self.get("/attachment/%s" % ident, {'encoding': [encoding]}, as_json=False) + res = yield deferred_result + defer.returnValue((res, req)) def put_mail(self, data): res, req = self.put('/mails', data) @@ -191,25 +216,26 @@ class AppTestClient(object): def delete_mail(self, mail_ident): res, req = self.delete("/mail/%s" % mail_ident) - return req + return res def delete_mails(self, idents): res, req = self.post("/mails/delete", json.dumps({'idents': idents})) - return req + return res def mark_many_as_unread(self, idents): res, req = self.post('/mails/unread', json.dumps({'idents': idents})) - return req + return res def mark_many_as_read(self, idents): res, req = self.post('/mails/read', json.dumps({'idents': idents})) - return req + return res def get_contacts(self, query): res, req = self.get('/contacts', get_args={'q': query}) return res +@defer.inlineCallbacks def initialize_soledad(tempdir): if os.path.isdir(tempdir): shutil.rmtree(tempdir) @@ -240,5 +266,9 @@ def initialize_soledad(tempdir): local_db_path, server_url, cert_file, - defer_encryption=False) - return _soledad + defer_encryption=False, + syncable=False) + + yield SoledadMailAdaptor().initialize_store(_soledad) + + defer.returnValue(_soledad) diff --git a/service/test/support/integration/model.py b/service/test/support/integration/model.py index e90a3ec5..c6f6a754 100644 --- a/service/test/support/integration/model.py +++ b/service/test/support/integration/model.py @@ -15,6 +15,7 @@ # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. import json +from pixelated.support import date from pixelated.adapter.model.mail import InputMail from pixelated.adapter.model.status import Status @@ -26,7 +27,8 @@ class MailBuilder: 'to': ['recipient@to.com'], 'cc': ['recipient@cc.com'], 'bcc': ['recipient@bcc.com'], - 'subject': 'Hi! This the subject' + 'subject': 'Hi! This the subject', + 'date': date.mail_date_now() }, 'body': "Hello,\nThis is the body of this message\n\nRegards,\n\n--\nPixelated.\n", 'status': [] @@ -45,6 +47,10 @@ class MailBuilder: self.mail['header']['subject'] = subject return self + def with_from(self, sender): + self.mail['header']['from'] = sender + return self + def with_to(self, to): self.mail['header']['to'] = to return self @@ -76,6 +82,9 @@ class MailBuilder: def build_input_mail(self): return InputMail.from_dict(self.mail) + def build_leap_mail(self): + return LeapMail.from_dict(self.mail) + class ResponseMail: def __init__(self, mail_dict): diff --git a/service/test/support/integration/soledad_test_base.py b/service/test/support/integration/soledad_test_base.py index c49de00a..e3e582d2 100644 --- a/service/test/support/integration/soledad_test_base.py +++ b/service/test/support/integration/soledad_test_base.py @@ -13,8 +13,14 @@ # # You should have received a copy of the GNU Affero General Public License # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +from uuid import uuid4 +from leap.mail.adaptors.soledad import SoledadMailAdaptor +from leap.mail.mail import Message +from twisted.internet import defer from twisted.trial import unittest +from pixelated.adapter.mailstore import LeapMailStore from test.support.integration.app_test_client import AppTestClient +from leap.common.events.flags import set_events_enabled class SoledadTestBase(unittest.TestCase, AppTestClient): @@ -22,8 +28,28 @@ class SoledadTestBase(unittest.TestCase, AppTestClient): DEFERRED_TIMEOUT = 120 DEFERRED_TIMEOUT_LONG = 300 + @defer.inlineCallbacks def setUp(self): - self.start_client() + set_events_enabled(False) + super(SoledadTestBase, self).setUp() + self.adaptor = SoledadMailAdaptor() + self.mbox_uuid = str(uuid4()) + yield self.start_client() def tearDown(self): + set_events_enabled(True) self.cleanup() + + @defer.inlineCallbacks + def _create_mail_in_soledad(self, mail): + yield self.adaptor.initialize_store(self.soledad) + mbox = yield self.adaptor.get_or_create_mbox(self.soledad, 'INBOX') + message = self._convert_mail_to_leap_message(mail, mbox.uuid) + yield self.adaptor.create_msg(self.soledad, message) + + defer.returnValue(message.get_wrapper().mdoc.doc_id) + + def _convert_mail_to_leap_message(self, mail, mbox_uuid): + message = self.adaptor.get_msg_from_string(Message, mail.as_string()) + message.get_wrapper().set_mbox_uuid(mbox_uuid) + return message diff --git a/service/pixelated/adapter/soledad/soledad_search_key_masterkey_retrieval_mixin.py b/service/test/support/integration/util.py index 05d32779..302edeaa 100644 --- a/service/pixelated/adapter/soledad/soledad_search_key_masterkey_retrieval_mixin.py +++ b/service/test/support/integration/util.py @@ -13,19 +13,19 @@ # # You should have received a copy of the GNU Affero General Public License # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. -from pixelated.adapter.soledad.soledad_facade_mixin import SoledadDbFacadeMixin +from email.parser import Parser +from email.utils import make_msgid import os -import base64 +import pkg_resources -class SoledadSearchIndexMasterkeyRetrievalMixin(SoledadDbFacadeMixin, object): +def load_mail_from_file(mail_file, enforceUniqueMessageId=False): + mailset_dir = pkg_resources.resource_filename('test.unit.fixtures', 'mailset') + mail_file = os.path.join(mailset_dir, 'new', mail_file) + with open(mail_file) as f: + mail = Parser().parse(f) - def get_index_masterkey(self): - result = self.get_search_index_masterkey() - index_key_doc = result[0] if result else None + if enforceUniqueMessageId: + mail.add_header('Message-Id', make_msgid()) - if not index_key_doc: - new_index_key = os.urandom(64) # 32 for encryption, 32 for hmac - self.create_doc(dict(type='index_key', value=base64.encodestring(new_index_key))) - return new_index_key - return base64.decodestring(index_key_doc.content['value']) + return mail diff --git a/service/test/support/mockito/__init__.py b/service/test/support/mockito/__init__.py new file mode 100644 index 00000000..c8ffc55e --- /dev/null +++ b/service/test/support/mockito/__init__.py @@ -0,0 +1,40 @@ +# +# Copyright (c) 2015 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +from mockito.invocation import AnswerSelector, CompositeAnswer + + +class FunctionReturn(object): + """ + Instead of returning a constant value a function is called + """ + def __init__(self, function_answer): + self.function_answer = function_answer + + def answer(self): + return self.function_answer() + + +def thenAnswer(self, answer_function): + """mockito does not support the thenAnswer style. This method monkey patches it into the library""" + if not self.answer: + self.answer = CompositeAnswer(FunctionReturn(answer_function)) + self.invocation.stub_with(self.answer) + else: + self.answer.add(FunctionReturn(answer_function)) + + return self + +AnswerSelector.thenAnswer = thenAnswer diff --git a/service/test/support/test_helper.py b/service/test/support/test_helper.py index c37c1408..21f59d8f 100644 --- a/service/test/support/test_helper.py +++ b/service/test/support/test_helper.py @@ -17,7 +17,7 @@ from datetime import datetime import io from twisted.web.test.test_web import DummyRequest -from pixelated.adapter.model.mail import InputMail, PixelatedMail +from pixelated.adapter.model.mail import InputMail LEAP_FLAGS = ['\\Seen', @@ -53,9 +53,9 @@ class TestDoc(object): return self.content[key] -def leap_mail(uid=0, flags=LEAP_FLAGS, headers=None, extra_headers={}, mbox='INBOX', body='body', +def leap_mail(uid=0, flags=LEAP_FLAGS, headers=None, extra_headers={}, mbox_uuid='INBOX', body='body', chash='chash'): - fdoc = TestDoc({'flags': flags, 'mbox': mbox, 'type': 'flags', 'uid': uid, 'chash': chash}) + fdoc = TestDoc({'flags': flags, 'mbox_uuid': mbox_uuid, 'type': 'flags', 'uid': uid, 'chash': chash}) if headers is None: headers = {} @@ -69,12 +69,6 @@ def leap_mail(uid=0, flags=LEAP_FLAGS, headers=None, extra_headers={}, mbox='INB return (fdoc, hdoc, bdoc) -def pixelated_mail(uid=0, flags=LEAP_FLAGS, headers=None, extra_headers={}, mbox='INBOX', body='body', chash='chash'): - fdoc, hdoc, bdoc = leap_mail(uid, flags, headers, extra_headers, mbox, body, chash) - - return PixelatedMail.from_soledad(fdoc, hdoc, bdoc) - - def input_mail(): mail = InputMail() mail.fdoc = TestDoc({}) diff --git a/service/pixelated/adapter/soledad/__init__.py b/service/test/unit/adapter/mailstore/__init__.py index 2756a319..c5c30cde 100644 --- a/service/pixelated/adapter/soledad/__init__.py +++ b/service/test/unit/adapter/mailstore/__init__.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2014 ThoughtWorks, Inc. +# Copyright (c) 2015 ThoughtWorks, Inc. # # Pixelated is free software: you can redistribute it and/or modify # it under the terms of the GNU Affero General Public License as published by diff --git a/service/test/unit/adapter/mailstore/maintenance/__init__.py b/service/test/unit/adapter/mailstore/maintenance/__init__.py new file mode 100644 index 00000000..c5c30cde --- /dev/null +++ b/service/test/unit/adapter/mailstore/maintenance/__init__.py @@ -0,0 +1,15 @@ +# +# Copyright (c) 2015 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. diff --git a/service/test/unit/adapter/mailstore/maintenance/test_soledad_maintenance.py b/service/test/unit/adapter/mailstore/maintenance/test_soledad_maintenance.py new file mode 100644 index 00000000..9a89d62b --- /dev/null +++ b/service/test/unit/adapter/mailstore/maintenance/test_soledad_maintenance.py @@ -0,0 +1,109 @@ +# +# Copyright (c) 2015 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +from leap.soledad.common.document import SoledadDocument +from twisted.internet import defer + +from twisted.trial import unittest +from mockito import mock, when, verify, never +from pixelated.adapter.mailstore.maintenance import SoledadMaintenance +from leap.keymanager.openpgp import OpenPGPKey + +SOME_EMAIL_ADDRESS = 'foo@example.tld' +SOME_KEY_ID = '4914254E384E264C' + + +class TestSoledadMaintenance(unittest.TestCase): + + def test_repair_is_deferred(self): + soledad = mock() + when(soledad).get_all_docs().thenReturn(defer.succeed((1, []))) + + d = SoledadMaintenance(soledad).repair() + + self.assertIsInstance(d, defer.Deferred) + + @defer.inlineCallbacks + def test_repair_delete_public_key_active_docs(self): + soledad = mock() + key = self._public_key(SOME_EMAIL_ADDRESS, SOME_KEY_ID) + active_doc = SoledadDocument(doc_id='some_doc', json=key.get_active_json(SOME_EMAIL_ADDRESS)) + when(soledad).get_all_docs().thenReturn(defer.succeed((1, [active_doc]))) + + yield SoledadMaintenance(soledad).repair() + + verify(soledad).delete_doc(active_doc) + + @defer.inlineCallbacks + def test_repair_delete_public_key_docs(self): + soledad = mock() + key = self._public_key(SOME_EMAIL_ADDRESS, SOME_KEY_ID) + active_doc = SoledadDocument(doc_id='some_doc', json=key.get_active_json(SOME_EMAIL_ADDRESS)) + key_doc = SoledadDocument(doc_id='some_doc', json=key.get_json()) + when(soledad).get_all_docs().thenReturn(defer.succeed((1, [key_doc, active_doc]))) + + yield SoledadMaintenance(soledad).repair() + + verify(soledad).delete_doc(active_doc) + verify(soledad).delete_doc(key_doc) + + @defer.inlineCallbacks + def test_repair_keeps_active_and_key_doc_if_private_key_exists(self): + soledad = mock() + key = self._public_key(SOME_EMAIL_ADDRESS, SOME_KEY_ID) + private_key = self._private_key(SOME_EMAIL_ADDRESS, SOME_KEY_ID) + active_doc = SoledadDocument(doc_id='some_doc', json=key.get_active_json(SOME_EMAIL_ADDRESS)) + key_doc = SoledadDocument(doc_id='some_doc', json=key.get_json()) + private_key_doc = SoledadDocument(doc_id='some_doc', json=private_key.get_json()) + when(soledad).get_all_docs().thenReturn(defer.succeed((1, [key_doc, active_doc, private_key_doc]))) + + yield SoledadMaintenance(soledad).repair() + + verify(soledad, never).delete_doc(key_doc) + verify(soledad, never).delete_doc(active_doc) + verify(soledad, never).delete_doc(private_key_doc) + + @defer.inlineCallbacks + def test_repair_only_deletes_key_docs(self): + soledad = mock() + key = self._public_key(SOME_EMAIL_ADDRESS, SOME_KEY_ID) + key_doc = SoledadDocument(doc_id='some_doc', json=key.get_active_json(SOME_EMAIL_ADDRESS)) + other_doc = SoledadDocument(doc_id='something', json='{}') + when(soledad).get_all_docs().thenReturn(defer.succeed((1, [key_doc, other_doc]))) + + yield SoledadMaintenance(soledad).repair() + + verify(soledad, never).delete_doc(other_doc) + + @defer.inlineCallbacks + def test_repair_recreates_public_key_active_doc_if_necessary(self): + soledad = mock() + + private_key = self._private_key(SOME_EMAIL_ADDRESS, SOME_KEY_ID) + private_key_doc = SoledadDocument(doc_id='some_doc', json=private_key.get_json()) + when(soledad).get_all_docs().thenReturn(defer.succeed((1, [private_key_doc]))) + + yield SoledadMaintenance(soledad).repair() + + verify(soledad).create_doc_from_json('{"key_id": "4914254E384E264C", "tags": ["keymanager-active"], "type": "OpenPGPKey-active", "private": false, "address": "foo@example.tld"}') + + def _public_key(self, address, keyid): + return self._gpgkey(address, keyid, private=False) + + def _private_key(self, address, keyid): + return self._gpgkey(address, keyid, private=True) + + def _gpgkey(self, address, keyid, private=False): + return OpenPGPKey(address, key_id=keyid, private=private) diff --git a/service/test/unit/adapter/mailstore/test_body_parser.py b/service/test/unit/adapter/mailstore/test_body_parser.py new file mode 100644 index 00000000..9d58637c --- /dev/null +++ b/service/test/unit/adapter/mailstore/test_body_parser.py @@ -0,0 +1,57 @@ +# -*- coding: utf-8 -*- +# +# Copyright (c) 2015 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +import unittest +from mock import patch +from pixelated.adapter.mailstore.body_parser import BodyParser + + +class BodyParserTest(unittest.TestCase): + + def test_simple_text(self): + parser = BodyParser('simple text') + + self.assertEqual('simple text', parser.parsed_content()) + + def test_base64_text(self): + parser = BodyParser('dGVzdCB0ZXh0\n', content_type='text/plain; charset="utf-8"', content_transfer_encoding='base64') + + self.assertEqual('test text', parser.parsed_content()) + + def test_8bit_transfer_encoding_with_iso_8859_1_str_input(self): + data = 'Hmm, here are \xdcml\xe4\xfcts again!' + parser = BodyParser(data, content_type='text/plain; charset=iso-8859-1', content_transfer_encoding='8bit') + + self.assertEqual(u'Hmm, here are Ümläüts again!', parser.parsed_content()) + + def test_8bit_transfer_encoding_with_iso_8859_1_unicode_input(self): + data = u'Hmm, here are \xdcml\xe4\xfcts again!' + parser = BodyParser(data, content_type='text/plain; charset=iso-8859-1', content_transfer_encoding='8bit') + + self.assertEqual(u'Hmm, here are Ümläüts again!', parser.parsed_content()) + + def test_base64_with_default_us_ascii_encoding(self): + parser = BodyParser('dGVzdCB0ZXh0\n', content_type='text/plain', content_transfer_encoding='base64') + + self.assertEqual('test text', parser.parsed_content()) + + @patch('pixelated.adapter.mailstore.body_parser.logger') + def test_body_parser_logs_problems_and_then_ignores_invalid_chars(self, logger_mock): + data = u'unkown char: \ufffd' + parser = BodyParser(data, content_type='text/plain; charset=iso-8859-1', content_transfer_encoding='8bit') + + self.assertEqual(u'unkown char: ', parser.parsed_content()) + logger_mock.warn.assert_called_with(u'Failed to encode content for charset iso-8859-1. Ignoring invalid chars: \'latin-1\' codec can\'t encode character u\'\\ufffd\' in position 13: ordinal not in range(256)') diff --git a/service/test/unit/adapter/mailstore/test_leap_mail.py b/service/test/unit/adapter/mailstore/test_leap_mail.py new file mode 100644 index 00000000..ef585654 --- /dev/null +++ b/service/test/unit/adapter/mailstore/test_leap_mail.py @@ -0,0 +1,230 @@ +# -*- coding: utf-8 -*- +# +# Copyright (c) 2015 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +from mock import patch +from twisted.trial.unittest import TestCase + +from pixelated.adapter.mailstore.leap_mailstore import LeapMail, AttachmentInfo + + +class TestLeapMail(TestCase): + def test_leap_mail(self): + mail = LeapMail('', 'INBOX', {'From': 'test@example.test', 'Subject': 'A test Mail', 'To': 'receiver@example.test'}) + + self.assertEqual('test@example.test', mail.from_sender) + self.assertEqual(['receiver@example.test'], mail.to) + self.assertEqual('A test Mail', mail.subject) + + def test_email_addresses_in_to_are_split_into_a_list(self): + mail = LeapMail('', 'INBOX', {'To': 'first@example.test,second@example.test'}) + + self.assertEqual(['first@example.test', 'second@example.test'], mail.headers['To']) + + def test_email_addresses_in_cc_are_split_into_a_list(self): + mail = LeapMail('', 'INBOX', {'Cc': 'first@example.test,second@example.test'}) + + self.assertEqual(['first@example.test', 'second@example.test'], mail.headers['Cc']) + + def test_email_addresses_in_bcc_are_split_into_a_list(self): + mail = LeapMail('', 'INBOX', {'Bcc': 'first@example.test,second@example.test'}) + + self.assertEqual(['first@example.test', 'second@example.test'], mail.headers['Bcc']) + + def test_email_addresses_might_be_empty_array(self): + mail = LeapMail('', 'INBOX', {'Cc': None}) + + self.assertEqual([], mail.headers['Cc']) + + def test_as_dict(self): + mail = LeapMail('doc id', 'INBOX', {'From': 'test@example.test', 'Subject': 'A test Mail', 'To': 'receiver@example.test,receiver2@other.test'}, ('foo', 'bar')) + self.maxDiff = None + expected = { + 'header': { + 'from': 'test@example.test', + 'subject': 'A test Mail', + 'to': ['receiver@example.test', 'receiver2@other.test'], + + }, + 'ident': 'doc id', + 'mailbox': 'inbox', + 'tags': {'foo', 'bar'}, + 'status': [], + 'body': None, + 'textPlainBody': None, + 'security_casing': { + 'imprints': [{'state': 'no_signature_information'}], + 'locks': [] + }, + 'replying': {'all': {'cc-field': [], + 'to-field': ['receiver@example.test', + 'test@example.test', + 'receiver2@other.test']}, + 'single': 'test@example.test'}, + 'attachments': [] + } + + self.assertEqual(expected, mail.as_dict()) + + def test_as_dict_with_body(self): + body = 'some body content' + mail = LeapMail('doc id', 'INBOX', {'From': 'test@example.test', 'Subject': 'A test Mail', 'To': 'receiver@example.test'}, ('foo', 'bar'), body=body) + + self.assertEqual(body, mail.as_dict()['body']) + + def test_as_dict_with_attachments(self): + mail = LeapMail('doc id', 'INBOX', attachments=[AttachmentInfo('id', 'name', 'encoding')]) + + self.assertEqual([{'ident': 'id', 'name': 'name', 'encoding': 'encoding'}], + mail.as_dict()['attachments']) + + def test_as_dict_headers_with_special_chars(self): + expected_address = u'"\xc4lbert \xdcbr\xf6" <\xe4\xfc\xf6@example.mail>' + expected_subject = u'H\xe4ll\xf6 W\xf6rld' + mail = LeapMail('', 'INBOX', + {'From': '=?iso-8859-1?q?=22=C4lbert_=DCbr=F6=22_=3C=E4=FC=F6=40example=2Email=3E?=', + 'To': '=?iso-8859-1?q?=22=C4lbert_=DCbr=F6=22_=3C=E4=FC=F6=40example=2Email=3E?=', + 'Cc': '=?iso-8859-1?q?=22=C4lbert_=DCbr=F6=22_=3C=E4=FC=F6=40example=2Email=3E?=', + 'Subject': '=?iso-8859-1?q?H=E4ll=F6_W=F6rld?='}) + + self.assertEqual(expected_address, mail.as_dict()['header']['from']) + self.assertEqual([expected_address], mail.as_dict()['header']['to']) + self.assertEqual([expected_address], mail.as_dict()['header']['cc']) + self.assertEqual(expected_subject, mail.as_dict()['header']['subject']) + + def test_as_dict_replying_with_special_chars(self): + expected_address = u'"\xc4lbert \xdcbr\xf6" <\xe4\xfc\xf6@example.mail>' + mail = LeapMail('', 'INBOX', + {'From': '=?iso-8859-1?q?=22=C4lbert_=DCbr=F6=22_=3C=E4=FC=F6=40example=2Email=3E?=', + 'To': '=?iso-8859-1?q?=22=C4lbert_=DCbr=F6=22_=3C=E4=FC=F6=40example=2Email=3E?=', + 'Cc': '=?iso-8859-1?q?=22=C4lbert_=DCbr=F6=22_=3C=E4=FC=F6=40example=2Email=3E?=', + 'Subject': '=?iso-8859-1?q?H=E4ll=F6_W=F6rld?='}) + + self.assertEqual([expected_address], mail.as_dict()['replying']['all']['to-field']) + self.assertEqual([expected_address], mail.as_dict()['replying']['all']['cc-field']) + self.assertEqual(expected_address, mail.as_dict()['replying']['single']) + + def test_reply_all_result_does_not_contain_own_address_in_to_with_spaces(self): + my_address = 'myaddress@example.test' + + with patch('pixelated.adapter.mailstore.leap_mailstore.InputMail.FROM_EMAIL_ADDRESS', my_address): + mail = LeapMail('', 'INBOX', + {'From': 'test@example.test', + 'To': 'receiver@example.test, %s ' % my_address}) + + self.assertEqual(['receiver@example.test', 'test@example.test'], mail.as_dict()['replying']['all']['to-field']) + + def test_reply_all_result_does_not_contain_own_address_in_to_with_name(self): + my_address = 'myaddress@example.test' + + with patch('pixelated.adapter.mailstore.leap_mailstore.InputMail.FROM_EMAIL_ADDRESS', my_address): + mail = LeapMail('', 'INBOX', + {'From': 'test@example.test', + 'To': 'receiver@example.test, Folker Bernitt <%s>' % my_address}) + + self.assertEqual(['receiver@example.test', 'test@example.test'], mail.as_dict()['replying']['all']['to-field']) + + def test_reply_all_result_does_not_contain_own_address_in_to_with_encoded(self): + my_address = 'myaddress@example.test' + + with patch('pixelated.adapter.mailstore.leap_mailstore.InputMail.FROM_EMAIL_ADDRESS', my_address): + mail = LeapMail('', 'INBOX', + {'From': 'test@example.test', + 'To': 'receiver@example.test, =?iso-8859-1?q?=C4lbert_=3Cmyaddress=40example=2Etest=3E?='}) + + self.assertEqual(['receiver@example.test', 'test@example.test'], mail.as_dict()['replying']['all']['to-field']) + + def test_reply_all_result_does_not_contain_own_address_in_cc(self): + my_address = 'myaddress@example.test' + + with patch('pixelated.adapter.mailstore.leap_mailstore.InputMail.FROM_EMAIL_ADDRESS', my_address): + mail = LeapMail('', 'INBOX', + {'From': 'test@example.test', + 'To': 'receiver@example.test', + 'Cc': my_address}) + + self.assertEqual([], mail.as_dict()['replying']['all']['cc-field']) + + def test_reply_all_result_does_not_contain_own_address_if_sender(self): + my_address = 'myaddress@example.test' + + with patch('pixelated.adapter.mailstore.leap_mailstore.InputMail.FROM_EMAIL_ADDRESS', my_address): + mail = LeapMail('', 'INBOX', + {'From': 'myaddress@example.test', + 'To': 'receiver@example.test'}) + + self.assertEqual(['receiver@example.test'], mail.as_dict()['replying']['all']['to-field']) + + def test_reply_all_result_does_contain_own_address_if_only_recipient(self): + my_address = 'myaddress@example.test' + + with patch('pixelated.adapter.mailstore.leap_mailstore.InputMail.FROM_EMAIL_ADDRESS', my_address): + mail = LeapMail('', 'INBOX', + {'From': 'myaddress@example.test', + 'To': 'myaddress@example.test'}) + + self.assertEqual(['myaddress@example.test'], mail.as_dict()['replying']['all']['to-field']) + + def test_reply_result_swaps_sender_and_recipient_if_i_am_the_sender(self): + my_address = 'myaddress@example.test' + + with patch('pixelated.adapter.mailstore.leap_mailstore.InputMail.FROM_EMAIL_ADDRESS', my_address): + mail = LeapMail('', 'INBOX', + {'From': 'myaddress@example.test', + 'To': 'recipient@example.test'}) + + self.assertEqual('recipient@example.test', mail.as_dict()['replying']['single']) + + def test_as_dict_with_mixed_encodings(self): + subject = 'Another test with =?iso-8859-1?B?3G1s5Px0?= =?iso-8859-1?Q?s?=' + mail = LeapMail('', 'INBOX', + {'Subject': subject}) + + self.assertEqual(u'Another test with Ümläüts', mail.as_dict()['header']['subject']) + + def test_raw_constructed_by_headers_and_body(self): + body = 'some body content' + mail = LeapMail('doc id', 'INBOX', {'From': 'test@example.test', 'Subject': 'A test Mail', 'To': 'receiver@example.test'}, ('foo', 'bar'), body=body) + + result = mail.raw + + expected_raw = 'To: receiver@example.test\nFrom: test@example.test\nSubject: A test Mail\n\nsome body content' + self.assertEqual(expected_raw, result) + + def test_headers_none_recipients_are_converted_to_empty_array(self): + mail = LeapMail('id', 'INBOX', {'To': None, 'Cc': None, 'Bcc': None}) + + self.assertEquals([], mail.headers['To']) + self.assertEquals([], mail.headers['Cc']) + self.assertEquals([], mail.headers['Bcc']) + + def test_security_casing(self): + # No Encryption, no Signature + mail = LeapMail('id', 'INBOX', {}) + self.assertEqual({'locks': [], 'imprints': [{'state': 'no_signature_information'}]}, mail.security_casing) + + # Encryption + mail = LeapMail('id', 'INBOX', {'X-Leap-Encryption': 'decrypted'}) + self.assertEqual([{'state': 'valid'}], mail.security_casing['locks']) + + mail = LeapMail('id', 'INBOX', {'X-Leap-Encryption': 'false'}) + self.assertEqual([], mail.security_casing['locks']) + + # Signature + mail = LeapMail('id', 'INBOX', {'X-Leap-Signature': 'valid'}) + self.assertEqual([{'seal': {'validity': 'valid'}, 'state': 'valid'}], mail.security_casing['imprints']) + + mail = LeapMail('id', 'INBOX', {'X-Leap-Signature': 'invalid'}) + self.assertEqual([], mail.security_casing['imprints']) diff --git a/service/test/unit/adapter/mailstore/test_leap_mailstore.py b/service/test/unit/adapter/mailstore/test_leap_mailstore.py new file mode 100644 index 00000000..4eabc144 --- /dev/null +++ b/service/test/unit/adapter/mailstore/test_leap_mailstore.py @@ -0,0 +1,474 @@ +# -*- coding: utf-8 -*- +# +# Copyright (c) 2015 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +import base64 +from email.header import Header +from email.mime.application import MIMEApplication +from email.mime.multipart import MIMEMultipart +from email.mime.text import MIMEText +import json +import quopri +from uuid import uuid4 +from email.parser import Parser +import os +from leap.soledad.common.document import SoledadDocument +from leap.mail.adaptors.soledad_indexes import MAIL_INDEXES +from twisted.internet.defer import FirstError +from twisted.trial.unittest import TestCase +from leap.mail import constants +from twisted.internet import defer +from mockito import mock, when, verify, any as ANY +import test.support.mockito +from leap.mail.adaptors.soledad import SoledadMailAdaptor, MailboxWrapper, ContentDocWrapper +import pkg_resources +from leap.mail.mail import Message +from pixelated.adapter.mailstore import underscore_uuid + +from pixelated.adapter.mailstore.leap_mailstore import LeapMailStore, LeapMail, AttachmentInfo + + +class TestLeapMailStore(TestCase): + def setUp(self): + self.soledad = mock() + self.mbox_uuid = str(uuid4()) + self.doc_by_id = {} + self.mbox_uuid_by_name = {} + self.mbox_soledad_docs = [] + + when(self.soledad).get_from_index('by-type', 'mbox').thenAnswer(lambda: defer.succeed(self.mbox_soledad_docs)) + self._mock_get_mailbox('INBOX') + + @defer.inlineCallbacks + def test_get_mail_not_exist(self): + when(self.soledad).get_doc(ANY()).thenAnswer(lambda: defer.succeed(None)) + store = LeapMailStore(self.soledad) + + mail = yield store.get_mail(_format_mdoc_id(uuid4(), 1)) + + self.assertIsNone(mail) + + @defer.inlineCallbacks + def test_get_mail(self): + mdoc_id, _ = self._add_mail_fixture_to_soledad_from_file('mbox00000000') + + store = LeapMailStore(self.soledad) + + mail = yield store.get_mail(mdoc_id) + + self.assertIsInstance(mail, LeapMail) + self.assertEqual('darby.senger@zemlak.biz', mail.from_sender) + self.assertEqual(['carmel@murazikortiz.name'], mail.to) + self.assertEqual('Itaque consequatur repellendus provident sunt quia.', mail.subject) + self.assertIsNone(mail.body) + self.assertEqual('INBOX', mail.mailbox_name) + + @defer.inlineCallbacks + def test_get_mail_from_mailbox(self): + other, _ = self._mock_get_mailbox('OTHER', create_new_uuid=True) + mdoc_id, _ = self._add_mail_fixture_to_soledad_from_file('mbox00000000', other.uuid) + + store = LeapMailStore(self.soledad) + + mail = yield store.get_mail(mdoc_id) + + self.assertEqual('OTHER', mail.mailbox_name) + + @defer.inlineCallbacks + def test_get_two_different_mails(self): + first_mdoc_id, _ = self._add_mail_fixture_to_soledad_from_file('mbox00000000') + second_mdoc_id, _ = self._add_mail_fixture_to_soledad_from_file('mbox00000001') + + store = LeapMailStore(self.soledad) + + mail1 = yield store.get_mail(first_mdoc_id) + mail2 = yield store.get_mail(second_mdoc_id) + + self.assertNotEqual(mail1, mail2) + self.assertEqual('Itaque consequatur repellendus provident sunt quia.', mail1.subject) + self.assertEqual('Error illum dignissimos autem eos aspernatur.', mail2.subject) + + @defer.inlineCallbacks + def test_get_mails(self): + first_mdoc_id, _ = self._add_mail_fixture_to_soledad_from_file('mbox00000000') + second_mdoc_id, _ = self._add_mail_fixture_to_soledad_from_file('mbox00000001') + + store = LeapMailStore(self.soledad) + + mails = yield store.get_mails([first_mdoc_id, second_mdoc_id]) + + self.assertEqual(2, len(mails)) + self.assertEqual('Itaque consequatur repellendus provident sunt quia.', mails[0].subject) + self.assertEqual('Error illum dignissimos autem eos aspernatur.', mails[1].subject) + + @defer.inlineCallbacks + def test_get_mails_fails_for_invalid_mail_id(self): + store = LeapMailStore(self.soledad) + + try: + yield store.get_mails(['invalid']) + self.fail('Exception expected') + except FirstError: + pass + + @defer.inlineCallbacks + def test_get_mail_with_body(self): + expeted_body = 'Dignissimos ducimus veritatis. Est tenetur consequatur quia occaecati. Vel sit sit voluptas.\n\nEarum distinctio eos. Accusantium qui sint ut quia assumenda. Facere dignissimos inventore autem sit amet. Pariatur voluptatem sint est.\n\nUt recusandae praesentium aspernatur. Exercitationem amet placeat deserunt quae consequatur eum. Unde doloremque suscipit quia.\n\n' + mdoc_id, _ = self._add_mail_fixture_to_soledad_from_file('mbox00000000') + + store = LeapMailStore(self.soledad) + + mail = yield store.get_mail(mdoc_id, include_body=True) + + self.assertEqual(expeted_body, mail.body) + + @defer.inlineCallbacks + def test_get_mail_attachment(self): + attachment_id = 'AAAA9AAD9E153D24265395203C53884506ABA276394B9FEC02B214BF9E77E48E' + doc = SoledadDocument(json=json.dumps({'content_type': 'foo/bar', 'raw': 'asdf'})) + when(self.soledad).get_from_index('by-type-and-payloadhash', 'cnt', attachment_id).thenReturn(defer.succeed([doc])) + store = LeapMailStore(self.soledad) + + attachment = yield store.get_mail_attachment(attachment_id) + + self.assertEqual({'content-type': 'foo/bar', 'content': bytearray('asdf')}, attachment) + + @defer.inlineCallbacks + def test_get_mail_attachment_different_content_encodings(self): + attachment_id = '1B0A9AAD9E153D24265395203C53884506ABA276394B9FEC02B214BF9E77E48E' + encoding_examples = [('', 'asdf', 'asdf'), + ('base64', 'asdf', 'YXNkZg=='), + ('quoted-printable', 'äsdf', '=C3=A4sdf')] + + for transfer_encoding, data, encoded_data in encoding_examples: + doc = SoledadDocument(json=json.dumps({'content_type': 'foo/bar', 'raw': encoded_data, + 'content_transfer_encoding': transfer_encoding})) + when(self.soledad).get_from_index('by-type-and-payloadhash', 'cnt', attachment_id).thenReturn(defer.succeed([doc])) + store = LeapMailStore(self.soledad) + + attachment = yield store.get_mail_attachment(attachment_id) + + self.assertEqual(bytearray(data), attachment['content']) + + @defer.inlineCallbacks + def test_get_mail_attachment_throws_exception_if_attachment_does_not_exist(self): + attachment_id = '1B0A9AAD9E153D24265395203C53884506ABA276394B9FEC02B214BF9E77E48E' + when(self.soledad).get_from_index('by-type-and-payloadhash', 'cnt', attachment_id).thenReturn(defer.succeed([])) + store = LeapMailStore(self.soledad) + try: + yield store.get_mail_attachment(attachment_id) + self.fail('ValueError exception expected') + except ValueError: + pass + + @defer.inlineCallbacks + def test_update_mail(self): + mdoc_id, fdoc_id = self._add_mail_fixture_to_soledad_from_file('mbox00000000') + soledad_fdoc = self.doc_by_id[fdoc_id] + when(self.soledad).put_doc(soledad_fdoc).thenReturn(defer.succeed(None)) + + store = LeapMailStore(self.soledad) + + mail = yield store.get_mail(mdoc_id) + + mail.tags.add('new_tag') + + yield store.update_mail(mail) + + verify(self.soledad).put_doc(soledad_fdoc) + self.assertTrue('new_tag' in soledad_fdoc.content['tags']) + + @defer.inlineCallbacks + def test_all_mails(self): + first_mdoc_id, _ = self._add_mail_fixture_to_soledad_from_file('mbox00000000') + second_mdoc_id, _ = self._add_mail_fixture_to_soledad_from_file('mbox00000001') + when(self.soledad).get_from_index('by-type', 'meta').thenReturn(defer.succeed([self.doc_by_id[first_mdoc_id], self.doc_by_id[second_mdoc_id]])) + + store = LeapMailStore(self.soledad) + + mails = yield store.all_mails() + + self.assertIsNotNone(mails) + self.assertEqual(2, len(mails)) + self.assertEqual('Itaque consequatur repellendus provident sunt quia.', mails[0].subject) + self.assertEqual('Error illum dignissimos autem eos aspernatur.', mails[1].subject) + + @defer.inlineCallbacks + def test_add_mailbox(self): + when(self.soledad).list_indexes().thenReturn(defer.succeed(MAIL_INDEXES)).thenReturn(defer.succeed(MAIL_INDEXES)) + when(self.soledad).get_from_index('by-type-and-mbox', 'mbox', 'TEST').thenReturn(defer.succeed([])) + self._mock_create_soledad_doc(self.mbox_uuid, MailboxWrapper(mbox='TEST')) + when(self.soledad).get_doc(self.mbox_uuid).thenAnswer(lambda: defer.succeed(self.doc_by_id[self.mbox_uuid])) + when(self.soledad).put_doc(ANY()).thenAnswer(lambda: defer.succeed(None)) + store = LeapMailStore(self.soledad) + + mbox = yield store.add_mailbox('TEST') + + self.assertIsNotNone(mbox) + self.assertEqual(self.mbox_uuid, mbox.doc_id) + self.assertEqual('TEST', mbox.mbox) + self.assertIsNotNone(mbox.uuid) + # assert index got updated + + @defer.inlineCallbacks + def test_get_mailbox_names_always_contains_inbox(self): + store = LeapMailStore(self.soledad) + + names = yield store.get_mailbox_names() + + self.assertEqual({'INBOX'}, names) + + @defer.inlineCallbacks + def test_get_mailbox_names(self): + self._mock_get_mailbox('OTHER', create_new_uuid=True) + store = LeapMailStore(self.soledad) + + names = yield store.get_mailbox_names() + + self.assertEqual({'INBOX', 'OTHER'}, names) + + @defer.inlineCallbacks + def test_handles_unmapped_mailbox_uuid(self): + # given + store = LeapMailStore(self.soledad) + new_uuid = 'UNICORN' + + # if no mailbox doc is created yet (async hell?) + when(self.soledad).get_from_index('by-type', 'mbox').thenReturn(defer.succeed([])) + + # then it should point to empty, which is all mails + name = yield store._mailbox_name_from_uuid(new_uuid) + self.assertEquals('', name) + + @defer.inlineCallbacks + def test_add_mail(self): + expected_message = self._add_create_mail_mocks_to_soledad_from_fixture_file('mbox00000000') + mail = self._load_mail_from_file('mbox00000000') + self._mock_get_mailbox('INBOX') + + store = LeapMailStore(self.soledad) + + message = yield store.add_mail('INBOX', mail.as_string()) + + self.assertIsInstance(message, LeapMail) + self._assert_message_docs_created(expected_message, message) + + @defer.inlineCallbacks + def test_add_mail_with_attachment(self): + input_mail = MIMEMultipart() + input_mail.attach(MIMEText(u'a utf8 message', _charset='utf-8')) + attachment = MIMEApplication('pretend to be binary attachment data') + attachment.add_header('Content-Disposition', 'attachment', filename='filename.txt') + input_mail.attach(attachment) + mocked_message = self._add_create_mail_mocks_to_soledad(input_mail) + store = LeapMailStore(self.soledad) + + message = yield store.add_mail('INBOX', input_mail.as_string()) + + expected = [{'ident': self._cdoc_phash_from_message(mocked_message, 2), 'name': 'filename.txt', 'encoding': 'base64'}] + self.assertEqual(expected, message.as_dict()['attachments']) + + @defer.inlineCallbacks + def test_add_mail_with_nested_attachments(self): + input_mail = MIMEMultipart() + input_mail.attach(MIMEText(u'a utf8 message', _charset='utf-8')) + attachment = MIMEApplication('pretend to be binary attachment data') + attachment.add_header('Content-Disposition', 'attachment', filename='filename.txt') + nested_attachment = MIMEMultipart() + nested_attachment.attach(attachment) + input_mail.attach(nested_attachment) + mocked_message = self._add_create_mail_mocks_to_soledad(input_mail) + store = LeapMailStore(self.soledad) + + message = yield store.add_mail('INBOX', input_mail.as_string()) + + expected = [{'ident': self._cdoc_phash_from_message(mocked_message, 2), 'name': 'filename.txt', 'encoding': 'base64'}] + self.assertEqual(expected, message.as_dict()['attachments']) + + @defer.inlineCallbacks + def test_add_mail_with_special_chars(self): + input_mail = MIMEText(u'a utf8 message', _charset='utf-8') + input_mail['From'] = Header(u'"Älbert Übrö" <äüö@example.mail>', 'iso-8859-1') + input_mail['Subject'] = Header(u'Hällö Wörld', 'iso-8859-1') + self._add_create_mail_mocks_to_soledad(input_mail) + store = LeapMailStore(self.soledad) + + message = yield store.add_mail('INBOX', input_mail.as_string()) + + self.assertEqual(u'"\xc4lbert \xdcbr\xf6" <\xe4\xfc\xf6@example.mail>', message.as_dict()['header']['from']) + + def _cdoc_phash_from_message(self, mocked_message, attachment_nr): + return mocked_message.get_wrapper().cdocs[attachment_nr].future_doc_id[2:] + + @defer.inlineCallbacks + def test_delete_mail(self): + mdoc_id, fdoc_id = self._add_mail_fixture_to_soledad_from_file('mbox00000000') + + store = LeapMailStore(self.soledad) + + yield store.delete_mail(mdoc_id) + + self._assert_mail_got_deleted(fdoc_id, mdoc_id) + + @defer.inlineCallbacks + def test_get_mailbox_mail_ids(self): + mdoc_id, fdoc_id = self._add_mail_fixture_to_soledad_from_file('mbox00000000') + when(self.soledad).get_from_index('by-type-and-mbox-uuid', 'flags', underscore_uuid(self.mbox_uuid)).thenReturn(defer.succeed([self.doc_by_id[fdoc_id]])) + self._mock_get_mailbox('INBOX') + store = LeapMailStore(self.soledad) + + mail_ids = yield store.get_mailbox_mail_ids('INBOX') + + self.assertEqual(1, len(mail_ids)) + self.assertEqual(mdoc_id, mail_ids[0]) + + @defer.inlineCallbacks + def test_delete_mailbox(self): + _, mbox_soledad_doc = self._mock_get_mailbox('INBOX') + store = LeapMailStore(self.soledad) + when(self.soledad).delete_doc(mbox_soledad_doc).thenReturn(defer.succeed(None)) + + yield store.delete_mailbox('INBOX') + + verify(self.soledad).delete_doc(self.doc_by_id[mbox_soledad_doc.doc_id]) + # should also verify index is updated + + @defer.inlineCallbacks + def test_copy_mail_to_mailbox(self): + expected_message = self._add_create_mail_mocks_to_soledad_from_fixture_file('mbox00000000') + mail_id, fdoc_id = self._add_mail_fixture_to_soledad_from_file('mbox00000000') + self._mock_get_mailbox('TRASH') + store = LeapMailStore(self.soledad) + + mail = yield store.copy_mail_to_mailbox(mail_id, 'TRASH') + + self._assert_message_docs_created(expected_message, mail, only_mdoc_and_fdoc=True) + + @defer.inlineCallbacks + def test_move_to_mailbox(self): + expected_message = self._add_create_mail_mocks_to_soledad_from_fixture_file('mbox00000000') + mail_id, fdoc_id = self._add_mail_fixture_to_soledad_from_file('mbox00000000') + self._mock_get_mailbox('TRASH') + store = LeapMailStore(self.soledad) + + mail = yield store.move_mail_to_mailbox(mail_id, 'TRASH') + + self._assert_message_docs_created(expected_message, mail, only_mdoc_and_fdoc=True) + self._assert_mail_got_deleted(fdoc_id, mail_id) + + def _assert_mail_got_deleted(self, fdoc_id, mail_id): + verify(self.soledad).delete_doc(self.doc_by_id[mail_id]) + verify(self.soledad).delete_doc(self.doc_by_id[fdoc_id]) + + def _assert_message_docs_created(self, expected_message, actual_message, only_mdoc_and_fdoc=False): + wrapper = expected_message.get_wrapper() + + verify(self.soledad).create_doc(wrapper.mdoc.serialize(), doc_id=actual_message.mail_id) + verify(self.soledad).create_doc(wrapper.fdoc.serialize(), doc_id=wrapper.fdoc.future_doc_id) + if not only_mdoc_and_fdoc: + verify(self.soledad).create_doc(wrapper.hdoc.serialize(), doc_id=wrapper.hdoc.future_doc_id) + for nr, cdoc in wrapper.cdocs.items(): + verify(self.soledad).create_doc(cdoc.serialize(), doc_id=wrapper.cdocs[nr].future_doc_id) + + def _mock_get_mailbox(self, mailbox_name, create_new_uuid=False): + mbox_uuid = self.mbox_uuid if not create_new_uuid else str(uuid4()) + when(self.soledad).list_indexes().thenReturn(defer.succeed(MAIL_INDEXES)).thenReturn( + defer.succeed(MAIL_INDEXES)) + doc_id = str(uuid4()) + mbox = MailboxWrapper(doc_id=doc_id, mbox=mailbox_name, uuid=mbox_uuid) + soledad_doc = SoledadDocument(doc_id, json=json.dumps(mbox.serialize())) + when(self.soledad).get_from_index('by-type-and-mbox', 'mbox', mailbox_name).thenReturn(defer.succeed([soledad_doc])) + self._mock_get_soledad_doc(doc_id, mbox) + + self.mbox_uuid_by_name[mailbox_name] = mbox_uuid + self.mbox_soledad_docs.append(soledad_doc) + + return mbox, soledad_doc + + def _add_mail_fixture_to_soledad_from_file(self, mail_file, mbox_uuid=None): + mail = self._load_mail_from_file(mail_file) + return self._add_mail_fixture_to_soledad(mail, mbox_uuid) + + def _add_mail_fixture_to_soledad(self, mail, mbox_uuid=None): + msg = self._convert_mail_to_leap_message(mail, mbox_uuid) + wrapper = msg.get_wrapper() + mdoc_id = wrapper.mdoc.future_doc_id + fdoc_id = wrapper.mdoc.fdoc + hdoc_id = wrapper.mdoc.hdoc + cdoc_id = wrapper.mdoc.cdocs[0] + + self._mock_get_soledad_doc(mdoc_id, wrapper.mdoc) + self._mock_get_soledad_doc(fdoc_id, wrapper.fdoc) + self._mock_get_soledad_doc(hdoc_id, wrapper.hdoc) + self._mock_get_soledad_doc(cdoc_id, wrapper.cdocs[1]) + return mdoc_id, fdoc_id + + def _add_create_mail_mocks_to_soledad_from_fixture_file(self, mail_file): + mail = self._load_mail_from_file(mail_file) + return self._add_create_mail_mocks_to_soledad(mail) + + def _add_create_mail_mocks_to_soledad(self, example_mail): + mail = self._convert_mail_to_leap_message(example_mail) + wrapper = mail.get_wrapper() + + mdoc_id = wrapper.mdoc.future_doc_id + fdoc_id = wrapper.mdoc.fdoc + hdoc_id = wrapper.mdoc.hdoc + + self._mock_create_soledad_doc(mdoc_id, wrapper.mdoc) + self._mock_create_soledad_doc(fdoc_id, wrapper.fdoc) + self._mock_create_soledad_doc(hdoc_id, wrapper.hdoc) + + for _, cdoc in wrapper.cdocs.items(): + self._mock_create_soledad_doc(cdoc.future_doc_id, cdoc) + self._mock_get_soledad_doc(cdoc.future_doc_id, cdoc) + + return mail + + def _convert_mail_to_leap_message(self, mail, mbox_uuid=None): + msg = SoledadMailAdaptor().get_msg_from_string(Message, mail.as_string()) + if mbox_uuid is None: + msg.get_wrapper().set_mbox_uuid(self.mbox_uuid) + else: + msg.get_wrapper().set_mbox_uuid(mbox_uuid) + + return msg + + def _mock_get_soledad_doc(self, doc_id, doc): + soledad_doc = SoledadDocument(doc_id, json=json.dumps(doc.serialize())) + + # when(self.soledad).get_doc(doc_id).thenReturn(defer.succeed(soledad_doc)) + when(self.soledad).get_doc(doc_id).thenAnswer(lambda: defer.succeed(soledad_doc)) + + self.doc_by_id[doc_id] = soledad_doc + + def _mock_create_soledad_doc(self, doc_id, doc): + soledad_doc = SoledadDocument(doc_id, json=json.dumps(doc.serialize())) + if doc.future_doc_id: + when(self.soledad).create_doc(doc.serialize(), doc_id=doc_id).thenReturn(defer.succeed(soledad_doc)) + else: + when(self.soledad).create_doc(doc.serialize()).thenReturn(defer.succeed(soledad_doc)) + self.doc_by_id[doc_id] = soledad_doc + + def _load_mail_from_file(self, mail_file): + mailset_dir = pkg_resources.resource_filename('test.unit.fixtures', 'mailset') + mail_file = os.path.join(mailset_dir, 'new', mail_file) + with open(mail_file) as f: + mail = Parser().parse(f) + return mail + + +def _format_mdoc_id(mbox_uuid, chash): + return constants.METAMSGID.format(mbox_uuid=mbox_uuid, chash=chash) diff --git a/service/test/unit/adapter/mailstore/test_searchable_mailstore.py b/service/test/unit/adapter/mailstore/test_searchable_mailstore.py new file mode 100644 index 00000000..8c571201 --- /dev/null +++ b/service/test/unit/adapter/mailstore/test_searchable_mailstore.py @@ -0,0 +1,112 @@ +# +# Copyright (c) 2015 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +from email.parser import Parser +import os +from mockito import verify, mock, when +import pkg_resources +from twisted.internet import defer +from twisted.trial.unittest import TestCase +from pixelated.adapter.mailstore import MailStore +from pixelated.adapter.mailstore.leap_mailstore import LeapMail +from pixelated.adapter.mailstore.searchable_mailstore import SearchableMailStore +from pixelated.adapter.search import SearchEngine + + +ANY_MAILBOX = 'INBOX' + + +class TestSearchableMailStore(TestCase): + + def setUp(self): + super(TestSearchableMailStore, self).setUp() + self.search_index = mock(mocked_obj=SearchEngine) + self.delegate_mail_store = mock(mocked_obj=MailStore) + self.store = SearchableMailStore(self.delegate_mail_store, self.search_index) + + @defer.inlineCallbacks + def test_add_mail_delegates_to_mail_store_and_updates_index(self): + mail = self._load_mail_from_file('mbox00000000') + leap_mail = LeapMail('id', ANY_MAILBOX) + when(self.delegate_mail_store).add_mail(ANY_MAILBOX, mail).thenReturn(defer.succeed(leap_mail)) + + result = yield self.store.add_mail(ANY_MAILBOX, mail) + + verify(self.delegate_mail_store).add_mail(ANY_MAILBOX, mail) + verify(self.search_index).index_mail(leap_mail) + self.assertEqual(leap_mail, result) + + @defer.inlineCallbacks + def test_delete_mail_delegates_to_mail_store_and_updates_index(self): + when(self.delegate_mail_store).delete_mail('mail id').thenReturn(defer.succeed(None)) + when(self.search_index).remove_from_index('mail id').thenReturn(defer.succeed(None)) + + yield self.store.delete_mail('mail id') + + verify(self.delegate_mail_store).delete_mail('mail id') + verify(self.search_index).remove_from_index('mail id') + + @defer.inlineCallbacks + def test_update_mail_delegates_to_mail_store_and_updates_index(self): + leap_mail = LeapMail('id', ANY_MAILBOX) + + yield self.store.update_mail(leap_mail) + + verify(self.delegate_mail_store).update_mail(leap_mail) + verify(self.search_index).index_mail(leap_mail) + + @defer.inlineCallbacks + def test_copy_mail_delegates_to_mail_store_and_updates_index(self): + copied_mail = LeapMail('new id', ANY_MAILBOX) + when(self.delegate_mail_store).copy_mail_to_mailbox('mail id', ANY_MAILBOX).thenReturn(defer.succeed(copied_mail)) + + result = yield self.store.copy_mail_to_mailbox('mail id', ANY_MAILBOX) + + verify(self.search_index).index_mail(copied_mail) + self.assertEqual(copied_mail, result) + + @defer.inlineCallbacks + def test_move_mail_delegates_to_mail_store_and_updates_index(self): + moved_mail = LeapMail('new id', ANY_MAILBOX) + when(self.delegate_mail_store).move_mail_to_mailbox('mail id', ANY_MAILBOX).thenReturn(defer.succeed(moved_mail)) + + result = yield self.store.move_mail_to_mailbox('mail id', ANY_MAILBOX) + + verify(self.search_index).remove_from_index('mail id') + verify(self.search_index).index_mail(moved_mail) + self.assertEqual(moved_mail, result) + + @defer.inlineCallbacks + def test_other_methods_are_delegated(self): + mail = LeapMail('mail id', ANY_MAILBOX) + when(self.delegate_mail_store).get_mail('mail id').thenReturn(defer.succeed(mail), defer.succeed(mail)) + result = yield self.store.get_mail('mail id') + + self.assertEqual(mail, result) + + @defer.inlineCallbacks + def test_delete_mailbox_is_not_implemented(self): + try: + yield self.store.delete_mailbox(ANY_MAILBOX) + self.fail("Should raise NotImplementedError") + except NotImplementedError: + pass + + def _load_mail_from_file(self, mail_file): + mailset_dir = pkg_resources.resource_filename('test.unit.fixtures', 'mailset') + mail_file = os.path.join(mailset_dir, 'new', mail_file) + with open(mail_file) as f: + mail = Parser().parse(f) + return mail diff --git a/service/test/unit/adapter/search/test_index_storage_key.py b/service/test/unit/adapter/search/test_index_storage_key.py new file mode 100644 index 00000000..e60c69ef --- /dev/null +++ b/service/test/unit/adapter/search/test_index_storage_key.py @@ -0,0 +1,52 @@ +# +# Copyright (c) 2015 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +from leap.soledad.common.document import SoledadDocument +from mockito import mock, when, unstub, verify +from twisted.internet import defer +from twisted.trial import unittest +from pixelated.adapter.search.index_storage_key import SearchIndexStorageKey +import os + + +class TestSearchIndexStorageKey(unittest.TestCase): + + def tearDown(self): + unstub() + + @defer.inlineCallbacks + def test_get_or_create_key_returns_key(self): + soledad = mock() + + when(soledad).get_from_index('by-type', 'index_key').thenReturn([SoledadDocument(json='{"value": "somekey"}')]) + + key = yield SearchIndexStorageKey(soledad).get_or_create_key() + + self.assertEqual('somekey', key) + + @defer.inlineCallbacks + def test_get_or_create_creates_key_if_not_exists(self): + expected_key = '\x8brN\xa3\xe5-\x828 \x95\x8d\n\xc6\x0c\x82\n\xd7!\xa9\xb0.\xcc\\h\xa9\x98\xe9V\xc1*<\xfe\xbb\x8f\xcd\x7f\x8c#\xff\xf9\x840\xdf{}\x97\xebS-*\xe2f\xf9B\xa9\xb1\x0c\x1d-C)\xc5\xa0B' + base64_encoded_key = 'i3JOo+UtgjgglY0KxgyCCtchqbAuzFxoqZjpVsEqPP67j81/jCP/+YQw33t9l+tTLSriZvlCqbEM\nHS1DKcWgQg==\n' + soledad = mock() + + when(soledad).get_from_index('by-type', 'index_key').thenReturn([]) + when(os).urandom(64).thenReturn(expected_key) + + key = yield SearchIndexStorageKey(soledad).get_or_create_key() + + self.assertEqual(expected_key, key) + + verify(soledad).create_doc(dict(type='index_key', value=base64_encoded_key)) diff --git a/service/test/unit/adapter/search/test_search.py b/service/test/unit/adapter/search/test_search.py index 1d9076a2..76e704b6 100644 --- a/service/test/unit/adapter/search/test_search.py +++ b/service/test/unit/adapter/search/test_search.py @@ -16,6 +16,7 @@ import unittest +from pixelated.adapter.mailstore.leap_mailstore import LeapMail from pixelated.adapter.search import SearchEngine from tempdir import TempDir from test.support import test_helper @@ -56,7 +57,7 @@ class SearchEngineTest(unittest.TestCase): } # when - se.index_mail(test_helper.pixelated_mail(extra_headers=headers, chash='mailid')) + se.index_mail(LeapMail('mailid', 'INBOX', headers=headers)) # test_helper.pixelated_mail(extra_headers=headers, chash='mailid')) result = se.search('folker') diff --git a/service/test/unit/adapter/test_draft_service.py b/service/test/unit/adapter/test_draft_service.py index 79eca5f6..c2b7cd93 100644 --- a/service/test/unit/adapter/test_draft_service.py +++ b/service/test/unit/adapter/test_draft_service.py @@ -1,4 +1,6 @@ import unittest +from twisted.internet import defer +from pixelated.adapter.mailstore.leap_mailstore import LeapMail from pixelated.adapter.model.mail import InputMail from pixelated.adapter.services.draft_service import DraftService @@ -10,21 +12,20 @@ class DraftServiceTest(unittest.TestCase): def setUp(self): self.mailboxes = mock() - self.drafts_mailbox = mock() - self.draft_service = DraftService(self.mailboxes) - self.mailboxes.drafts = self.drafts_mailbox + self.mail_store = mock() + self.draft_service = DraftService(self.mail_store) def test_add_draft(self): mail = InputMail() self.draft_service.create_draft(mail) - verify(self.drafts_mailbox).add(mail) + verify(self.mail_store).add_mail('DRAFTS', mail.raw) def test_update_draft(self): mail = InputMail.from_dict(test_helper.mail_dict()) - when(self.drafts_mailbox).add(mail).thenReturn(mail) + when(self.mail_store).add_mail('DRAFTS', mail.raw).thenReturn(defer.succeed(LeapMail('id', 'DRAFTS'))) self.draft_service.update_draft(mail.ident, mail) - inorder.verify(self.drafts_mailbox).add(mail) - inorder.verify(self.drafts_mailbox).remove(mail.ident) + inorder.verify(self.mail_store).add_mail('DRAFTS', mail.raw) + inorder.verify(self.mail_store).delete_mail(mail.ident) diff --git a/service/test/unit/adapter/test_mail.py b/service/test/unit/adapter/test_mail.py index 1a9280ff..dc344992 100644 --- a/service/test/unit/adapter/test_mail.py +++ b/service/test/unit/adapter/test_mail.py @@ -14,378 +14,20 @@ # # You should have received a copy of the GNU Affero General Public License # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. -import unittest +from twisted.trial import unittest import pixelated.support.date -from pixelated.adapter.model.mail import PixelatedMail, InputMail +from pixelated.adapter.model.mail import InputMail, HEADERS_KEY from mockito import mock, unstub, when from test.support import test_helper import dateutil.parser as dateparser import base64 -from leap.mail.imap.fields import fields +from leap.mail.adaptors import soledad_indexes as fields from datetime import datetime import os import json - - -class TestPixelatedMail(unittest.TestCase): - def setUp(self): - self.querier = mock() - - def tearDown(self): - unstub() - - def test_parse_date_from_soledad_uses_date_header_if_available(self): - leap_mail_date = 'Wed, 3 Sep 2014 12:36:17 -0300' - leap_mail_date_in_iso_format = "2014-09-03T12:36:17-03:00" - - leap_mail = test_helper.leap_mail(headers={'date': leap_mail_date}) - - mail = PixelatedMail.from_soledad(*leap_mail, soledad_querier=self.querier) - - self.assertEqual(str(mail.headers['Date']), leap_mail_date_in_iso_format) - - def test_parse_date_from_soledad_fallback_to_received_header_if_date_header_isnt_available(self): - leap_mail_date = "Wed, 03 Sep 2014 13:11:15 -0300" - leap_mail_date_in_iso_format = "2014-09-03T13:11:15-03:00" - leap_mail_received_header = "by bitmask.local from 127.0.0.1 with ESMTP ;\n " + leap_mail_date - - leap_mail = test_helper.leap_mail(headers={'received': leap_mail_received_header}) - - mail = PixelatedMail.from_soledad(*leap_mail, soledad_querier=self.querier) - - self.assertEqual(str(mail.headers['Date']), leap_mail_date_in_iso_format) - - def test_parse_date_from_soledad_fallback_to_now_if_neither_date_nor_received_header(self): - leap_mail_date_in_iso_format = "2014-09-03T13:11:15-03:00" - - when(pixelated.support.date).iso_now().thenReturn(leap_mail_date_in_iso_format) - fdoc, hdoc, bdoc = test_helper.leap_mail() - del hdoc.content['date'] - - mail = PixelatedMail.from_soledad(fdoc, hdoc, bdoc, soledad_querier=self.querier) - - self.assertEqual(str(mail.headers['Date']), leap_mail_date_in_iso_format) - - def test_use_datetime_now_as_fallback_for_invalid_date(self): - leap_mail_date = u'söme däte' - date_expected = "2014-09-03T13:11:15-03:00" - - when(pixelated.support.date).iso_now().thenReturn(date_expected) - leap_mail = test_helper.leap_mail(headers={'date': leap_mail_date}) - - mail = PixelatedMail.from_soledad(*leap_mail, soledad_querier=self.querier) - - self.assertEqual(str(mail.headers['Date']), date_expected) - - def test_fall_back_to_ascii_if_invalid_received_header(self): - leap_mail_received_header = u"söme invalid received heäder\n" - date_expected = "2014-09-03T13:11:15-03:00" - - when(pixelated.support.date).iso_now().thenReturn(date_expected) - leap_mail = test_helper.leap_mail(headers={'received': leap_mail_received_header}) - - mail = PixelatedMail.from_soledad(*leap_mail, soledad_querier=self.querier) - - self.assertEqual(mail.headers['Date'], date_expected) - - def test_update_tags_return_a_set_with_the_current_tags(self): - soledad_docs = test_helper.leap_mail(extra_headers={'X-tags': '["custom_1", "custom_2"]'}) - pixelated_mail = PixelatedMail.from_soledad(*soledad_docs, soledad_querier=self.querier) - - current_tags = pixelated_mail.update_tags({'custom_1', 'custom_3'}) - self.assertEquals({'custom_3', 'custom_1'}, current_tags) - - def test_mark_as_read(self): - mail = PixelatedMail.from_soledad(*test_helper.leap_mail(flags=[]), soledad_querier=self.querier) - - mail.mark_as_read() - - self.assertEquals(mail.fdoc.content['flags'], ['\\Seen']) - - def test_mark_as_not_recent(self): - mail = PixelatedMail.from_soledad(*test_helper.leap_mail(flags=['\\Recent']), soledad_querier=self.querier) - - mail.mark_as_not_recent() - - self.assertEquals(mail.fdoc.content['flags'], []) - - def test_get_for_save_adds_from(self): - InputMail.FROM_EMAIL_ADDRESS = 'me@pixelated.org' - headers = {'Subject': 'The subject', - 'Date': str(datetime.now()), - 'To': 'me@pixelated.org'} - - input_mail = InputMail() - input_mail.headers = headers - - self.assertEqual('me@pixelated.org', input_mail.get_for_save(1, 'SENT')[1][fields.HEADERS_KEY]['From']) - - def test_as_dict(self): - headers = {'Subject': 'The subject', - 'From': 'someone@pixelated.org', - 'To': 'me@pixelated.org'} - fdoc, hdoc, bdoc = test_helper.leap_mail(flags=['\\Recent'], - extra_headers=headers) - - InputMail.FROM_EMAIL_ADDRESS = 'me@pixelated.org' - - mail = PixelatedMail.from_soledad(fdoc, hdoc, bdoc, soledad_querier=self.querier) - - _dict = mail.as_dict() - - self.maxDiff = None - - self.assertEquals(_dict, {'htmlBody': None, - 'textPlainBody': 'body', - 'header': { - 'date': dateparser.parse(hdoc.content['date']).isoformat(), - 'from': 'someone@pixelated.org', - 'subject': 'The subject', - 'to': ['me@pixelated.org'], - 'cc': [], - 'bcc': [] - }, - 'ident': 'chash', - 'mailbox': 'inbox', - 'security_casing': {'imprints': [{'state': 'no_signature_information'}], 'locks': []}, - 'status': ['recent'], - 'tags': [], - 'attachments': [], - 'replying': { - 'single': 'someone@pixelated.org', - 'all': { - 'to-field': ['someone@pixelated.org'], - 'cc-field': [] - } - }}) - - def test_use_reply_to_address_for_replying(self): - headers = {'Subject': 'The subject', - 'From': 'someone@pixelated.org', - 'Reply-To': 'reply-to-this-address@pixelated.org', - 'To': 'me@pixelated.org, \nalice@pixelated.org'} - fdoc, hdoc, bdoc = test_helper.leap_mail(flags=['\\Recent'], - extra_headers=headers) - - InputMail.FROM_EMAIL_ADDRESS = 'me@pixelated.org' - - mail = PixelatedMail.from_soledad(fdoc, hdoc, bdoc, soledad_querier=self.querier) - - _dict = mail.as_dict() - - self.assertEquals(_dict['replying'], {'single': 'reply-to-this-address@pixelated.org', - 'all': { - 'to-field': ['alice@pixelated.org', 'reply-to-this-address@pixelated.org'], - 'cc-field': [] - }}) - - def test_alternatives_body(self): - parts = {'alternatives': [], 'attachments': []} - parts['alternatives'].append({'content': 'blablabla', 'headers': {'Content-Type': 'text/plain'}}) - parts['alternatives'].append({'content': '<p>blablabla</p>', 'headers': {'Content-Type': 'text/html'}}) - - mail = PixelatedMail.from_soledad(None, None, self._create_bdoc(raw='blablabla'), parts=parts, soledad_querier=None) - - self.assertRegexpMatches(mail.html_body, '^<p>blablabla</p>$') - self.assertRegexpMatches(mail.text_plain_body, '^blablabla$') - - def test_html_is_none_if_multiple_alternatives_have_no_html_part(self): - parts = { - 'attachments': [], - 'alternatives': [ - {'content': u'content', 'headers': {u'Content-Type': u'text/plain; charset=us-ascii'}}, - {'content': u'', 'headers': {u'Some info': u'info'}}]} - - mail = PixelatedMail.from_soledad(None, None, None, parts=parts, soledad_querier=None) - self.assertIsNone(mail.html_body) - - def test_percent_character_is_allowed_on_body(self): - parts = {'alternatives': [], 'attachments': []} - parts['alternatives'].append({'content': '100% happy with percentage symbol', 'headers': {'Content-Type': 'text/plain'}}) - parts['alternatives'].append({'content': '<p>100% happy with percentage symbol</p>', 'headers': {'Content-Type': 'text/html'}}) - - mail = PixelatedMail.from_soledad(None, None, self._create_bdoc(raw="100% happy with percentage symbol"), parts=parts, soledad_querier=None) - - self.assertRegexpMatches(mail.text_plain_body, '([\s\S]*100%)') - self.assertRegexpMatches(mail.html_body, '([\s\S]*100%)') - - def test_content_type_header_of_mail_part_is_used(self): - plain_headers = {'Content-Type': 'text/plain; charset=iso-8859-1', 'Content-Transfer-Encoding': 'quoted-printable'} - html_headers = {'Content-Type': 'text/html; charset=utf-8', 'Content-Transfer-Encoding': 'quoted-printable'} - parts = {'alternatives': [{'content': 'H=E4llo', 'headers': plain_headers}, {'content': '<p>H=C3=A4llo</p>', 'headers': html_headers}]} - - mail = PixelatedMail.from_soledad(None, None, self._create_bdoc(raw='some raw body'), parts=parts, soledad_querier=None) - - self.assertEqual(2, len(mail.alternatives)) - self.assertEquals(u'H\xe4llo', mail.text_plain_body) - self.assertEquals(u'<p>H\xe4llo</p>', mail.html_body) - - def test_multi_line_content_type_header_is_supported(self): - plain_headers = {'Content-Type': 'text/plain;\ncharset=iso-8859-1', 'Content-Transfer-Encoding': 'quoted-printable'} - html_headers = {'Content-Type': 'text/html;\ncharset=utf-8', 'Content-Transfer-Encoding': 'quoted-printable'} - parts = {'alternatives': [{'content': 'H=E4llo', 'headers': plain_headers}, {'content': '<p>H=C3=A4llo</p>', 'headers': html_headers}]} - - mail = PixelatedMail.from_soledad(None, None, self._create_bdoc(raw='some raw body'), parts=parts, soledad_querier=None) - - self.assertEqual(2, len(mail.alternatives)) - self.assertEquals(u'H\xe4llo', mail.text_plain_body) - self.assertEquals(u'<p>H\xe4llo</p>', mail.html_body) - - def test_broken_content_type_defaults_to_usascii(self): - plain_headers = {'Content-Type': 'I lie to you', 'Content-Transfer-Encoding': 'quoted-printable'} - html_headers = {'Content-Type': 'text/html;\ncharset=utf-8', 'Content-Transfer-Encoding': 'quoted-printable'} - parts = {'alternatives': [{'content': 'H=E4llo', 'headers': plain_headers}, {'content': '<p>H=C3=A4llo</p>', 'headers': html_headers}]} - - mail = PixelatedMail.from_soledad(None, None, self._create_bdoc(raw='some raw body'), parts=parts, soledad_querier=None) - - self.assertEquals(u'H=E4llo', mail.text_plain_body) - - def test_broken_encoding_defaults_to_8bit(self): - plain_headers = {'Content-Type': 'text/plain;\ncharset=iso-8859-1', 'Content-Transfer-Encoding': 'I lie to you!'} - html_headers = {'Content-Type': 'text/html;\ncharset=utf-8', 'Content-Transfer-Encoding': 'quoted-printable'} - parts = {'alternatives': [{'content': 'H=E4llo', 'headers': plain_headers}, {'content': '<p>H=C3=A4llo</p>', 'headers': html_headers}]} - - mail = PixelatedMail.from_soledad(None, None, self._create_bdoc(raw='some raw body'), parts=parts, soledad_querier=None) - - self.assertEquals(u'H=E4llo', mail.text_plain_body) - self.assertEquals(u'<p>H\xe4llo</p>', mail.html_body) - - def test_clean_line_breaks_on_address_headers(self): - many_recipients = 'One <one@mail.com>,\nTwo <two@mail.com>, Normal <normal@mail.com>,\nalone@mail.com' - headers = {'Cc': many_recipients, - 'Bcc': many_recipients, - 'To': many_recipients} - fdoc, hdoc, bdoc = test_helper.leap_mail(flags=['\\Recent'], - extra_headers=headers) - - mail = PixelatedMail.from_soledad(fdoc, hdoc, bdoc, soledad_querier=self.querier) - - for header_label in ['To', 'Cc', 'Bcc']: - for address in mail.headers[header_label]: - self.assertNotIn('\n', address) - self.assertNotIn(',', address) - self.assertEquals(4, len(mail.headers[header_label])) - - def test_that_body_understands_base64(self): - body = u'bl\xe1' - encoded_body = unicode(body.encode('utf-8').encode('base64')) - - fdoc, hdoc, bdoc = test_helper.leap_mail() - parts = {'alternatives': []} - parts['alternatives'].append({'content': encoded_body, 'headers': {'Content-Transfer-Encoding': 'base64'}}) - mail = PixelatedMail.from_soledad(fdoc, hdoc, bdoc, soledad_querier=self.querier, parts=parts) - - self.assertEquals(body, mail.text_plain_body) - - def test_that_body_understands_7bit(self): - body = u'testtext' - encoded_body = body - - fdoc, hdoc, bdoc = test_helper.leap_mail() - parts = {'alternatives': []} - parts['alternatives'].append({'content': encoded_body, 'headers': {'Content-Transfer-Encoding': '7bit'}}) - mail = PixelatedMail.from_soledad(fdoc, hdoc, bdoc, soledad_querier=self.querier, parts=parts) - - self.assertEquals(body, mail.text_plain_body) - - def test_that_body_understands_8bit(self): - body = u'testtext' - encoded_body = body - - fdoc, hdoc, bdoc = test_helper.leap_mail() - parts = {'alternatives': []} - parts['alternatives'].append({'content': encoded_body, 'headers': {'Content-Transfer-Encoding': '8bit'}}) - mail = PixelatedMail.from_soledad(fdoc, hdoc, bdoc, soledad_querier=self.querier, parts=parts) - - self.assertEquals(body, mail.text_plain_body) - - def test_bounced_mails_are_recognized(self): - bounced_mail_hdoc = os.path.join(os.path.dirname(__file__), '..', 'fixtures', 'bounced_mail_hdoc.json') - with open(bounced_mail_hdoc) as f: - hdoc = json.loads(f.read()) - - bounced_leap_mail = test_helper.leap_mail() - bounced_leap_mail[1].content = hdoc - bounced_mail = PixelatedMail.from_soledad(*bounced_leap_mail, soledad_querier=self.querier) - - not_bounced_leap_mail = test_helper.leap_mail() - not_bounced_mail = PixelatedMail.from_soledad(*not_bounced_leap_mail, soledad_querier=self.querier) - - self.assertTrue(bounced_mail.bounced) - self.assertIn('this_mail_was_bounced@domain.com', bounced_mail.bounced) - self.assertIn("MAILER-DAEMON@domain.org (Mail Delivery System)", bounced_mail.bounced) - self.assertFalse(not_bounced_mail.bounced) - - def test_ignore_transient_failures(self): - """ - Persistent errors should start with 5. - See: http://www.iana.org/assignments/smtp-enhanced-status-codes/smtp-enhanced-status-codes.xhtml - """ - bounced_mail_hdoc = os.path.join(os.path.dirname(__file__), '..', 'fixtures', 'bounced_mail_hdoc.json') - with open(bounced_mail_hdoc) as f: - content = f.read() - # Change status to 4.XXX.YYY (only the first number is relevant here) - content = content.replace("5.1.1", "4.X.Y") - hdoc = json.loads(content) - - temporary_bounced_leap_mail = test_helper.leap_mail() - temporary_bounced_leap_mail[1].content = hdoc - temporary_bounced_mail = PixelatedMail.from_soledad(*temporary_bounced_leap_mail, soledad_querier=self.querier) - - not_bounced_leap_mail = test_helper.leap_mail() - not_bounced_mail = PixelatedMail.from_soledad(*not_bounced_leap_mail, soledad_querier=self.querier) - - self.assertFalse(temporary_bounced_mail.bounced) - self.assertFalse(not_bounced_mail.bounced) - - def _create_bdoc(self, raw): - class FakeBDoc: - def __init__(self, raw): - self.content = {'raw': raw} - return FakeBDoc(raw) - - def test_encoding_special_character_on_header(self): - subject = "=?UTF-8?Q?test_encoding_St=C3=A4ch?=" - email_from = "=?UTF-8?Q?St=C3=A4ch_<stach@pixelated-project.org>?=" - email_to = "=?utf-8?b?IsOEw7zDtiDDlsO8w6QiIDxmb2xrZXJAcGl4ZWxhdGVkLXByb2plY3Qub3Jn?=\n =?utf-8?b?PiwgRsO2bGtlciA8Zm9sa2VyQHBpeGVsYXRlZC1wcm9qZWN0Lm9yZz4=?=" - - pixel_mail = PixelatedMail() - - self.assertEqual(pixel_mail._decode_header(subject), 'test encoding St\xc3\xa4ch') - self.assertEqual(pixel_mail._decode_header(email_from), 'St\xc3\xa4ch <stach@pixelated-project.org>') - self.assertEqual(pixel_mail._decode_header(email_to), '"\xc3\x84\xc3\xbc\xc3\xb6 \xc3\x96\xc3\xbc\xc3\xa4" <folker@pixelated-project.org>, F\xc3\xb6lker <folker@pixelated-project.org>') - self.assertEqual(pixel_mail._decode_header(None), None) - - def test_headers_are_encoded_right(self): - subject = "=?UTF-8?Q?test_encoding_St=C3=A4ch?=" - email_from = "=?UTF-8?Q?St=C3=A4ch_<stach@pixelated-project.org>?=" - email_to = "=?utf-8?b?IsOEw7zDtiDDlsO8w6QiIDxmb2xrZXJAcGl4ZWxhdGVkLXByb2plY3Qub3Jn?=\n =?utf-8?b?PiwgRsO2bGtlciA8Zm9sa2VyQHBpeGVsYXRlZC1wcm9qZWN0Lm9yZz4=?=" - email_cc = "=?UTF-8?Q?St=C3=A4ch_<stach@pixelated-project.org>?=" - email_bcc = "=?UTF-8?Q?St=C3=A4ch_<stach@pixelated-project.org>?=" - - leap_mail = test_helper.leap_mail(extra_headers={'Subject': subject, 'From': email_from, 'To': email_to, 'Cc': email_cc, 'Bcc': email_bcc}) - - mail = PixelatedMail.from_soledad(*leap_mail, soledad_querier=self.querier) - - self.assertEqual(str(mail.headers['Subject']), 'test encoding St\xc3\xa4ch') - self.assertEqual(str(mail.headers['From']), 'St\xc3\xa4ch <stach@pixelated-project.org>') - self.assertEqual(mail.headers['To'], ['"\xc3\x84\xc3\xbc\xc3\xb6 \xc3\x96\xc3\xbc\xc3\xa4" <folker@pixelated-project.org>', 'F\xc3\xb6lker <folker@pixelated-project.org>']) - self.assertEqual(mail.headers['Cc'], ['St\xc3\xa4ch <stach@pixelated-project.org>']) - self.assertEqual(mail.headers['Bcc'], ['St\xc3\xa4ch <stach@pixelated-project.org>']) - - mail.as_dict() - - def test_parse_UTF8_headers_with_CharsetAscii(self): - leap_mail_from = u'"söme ümläuds" <lisa5@dev.pixelated-project.org>' - leap_mail_to = u'"söme ümläuds" <lisa5@dev.pixelated-project.org>,\n"söme ümläuds" <lisa5@dev.pixelated-project.org>' - - leap_mail = test_helper.leap_mail(extra_headers={'From': leap_mail_from, 'Subject': "some subject", 'To': leap_mail_to}) - - mail = PixelatedMail.from_soledad(*leap_mail, soledad_querier=self.querier) - - mail.headers['From'].encode('ascii') - self.assertEqual(mail.headers['To'], ['"sme mluds" <lisa5@dev.pixelated-project.org>', '"sme mluds" <lisa5@dev.pixelated-project.org>']) +import pkg_resources +from twisted.internet import defer def simple_mail_dict(): @@ -420,7 +62,7 @@ def multipart_mail_dict(): class InputMailTest(unittest.TestCase): def test_to_mime_multipart_should_add_blank_fields(self): - pixelated.support.date.iso_now = lambda: 'date now' + pixelated.support.date.mail_date_now = lambda: 'date now' mail_dict = simple_mail_dict() mail_dict['header']['to'] = '' @@ -435,8 +77,23 @@ class InputMailTest(unittest.TestCase): self.assertNotRegexpMatches(mime_multipart.as_string(), "\nCc: \n") self.assertNotRegexpMatches(mime_multipart.as_string(), "\nSubject: \n") + def test_single_recipient(self): + mail_single_recipient = { + 'body': '', + 'header': { + 'to': ['to@pixelated.org'], + 'cc': [''], + 'bcc': [''], + 'subject': 'Oi' + } + } + + result = InputMail.from_dict(mail_single_recipient).raw + + self.assertRegexpMatches(result, 'To: to@pixelated.org') + def test_to_mime_multipart(self): - pixelated.support.date.iso_now = lambda: 'date now' + pixelated.support.date.mail_date_now = lambda: 'date now' mime_multipart = InputMail.from_dict(simple_mail_dict()).to_mime_multipart() @@ -447,6 +104,16 @@ class InputMailTest(unittest.TestCase): self.assertRegexpMatches(mime_multipart.as_string(), "\nSubject: Oi\n") self.assertRegexpMatches(mime_multipart.as_string(), base64.b64encode(simple_mail_dict()['body'])) + def test_to_mime_multipart_with_special_chars(self): + mail_dict = simple_mail_dict() + mail_dict['header']['to'] = u'"Älbert Übrö \xF0\x9F\x92\xA9" <äüö@example.mail>' + pixelated.support.date.mail_date_now = lambda: 'date now' + + mime_multipart = InputMail.from_dict(mail_dict).to_mime_multipart() + + expected_part_of_encoded_to = 'Iiwgw4QsIGwsIGIsIGUsIHIsIHQsICAsIMOcLCBiLCByLCDDtiwgICwgw7As' + self.assertRegexpMatches(mime_multipart.as_string(), expected_part_of_encoded_to) + def test_smtp_format(self): InputMail.FROM_EMAIL_ADDRESS = 'pixelated@org' diff --git a/service/test/unit/adapter/test_mail_service.py b/service/test/unit/adapter/test_mail_service.py index f5e29b0c..6faf5140 100644 --- a/service/test/unit/adapter/test_mail_service.py +++ b/service/test/unit/adapter/test_mail_service.py @@ -14,33 +14,37 @@ # You should have received a copy of the GNU Affero General Public License # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. from twisted.trial import unittest -from pixelated.adapter.model.mail import InputMail, PixelatedMail +from pixelated.adapter.mailstore.leap_mailstore import LeapMail +from pixelated.adapter.model.mail import InputMail +from pixelated.adapter.model.status import Status from pixelated.adapter.services.mail_service import MailService from test.support.test_helper import mail_dict, leap_mail -from mockito import mock, unstub, when, verify, verifyNoMoreInteractions, any -from twisted.internet.defer import Deferred +from mockito import mock, unstub, when, verify, verifyNoMoreInteractions, any as ANY +from twisted.internet import defer class TestMailService(unittest.TestCase): def setUp(self): self.drafts = mock() - self.querier = mock() + self.mail_store = mock() self.mailboxes = mock() - self.mailboxes.drafts = self.drafts + + self.mailboxes.drafts = defer.succeed(self.drafts) + self.mailboxes.trash = mock() self.mailboxes.sent = mock() self.mail_sender = mock() self.search_engine = mock() - self.mail_service = MailService(self.mailboxes, self.mail_sender, self.querier, self.search_engine) + self.mail_service = MailService(self.mail_sender, self.mail_store, self.search_engine) def tearDown(self): unstub() def test_send_mail(self): - when(InputMail).from_dict(any()).thenReturn('inputmail') - when(self.mail_sender).sendmail(any()).thenReturn(Deferred()) + when(InputMail).from_dict(ANY()).thenReturn('inputmail') + when(self.mail_sender).sendmail(ANY()).thenReturn(defer.Deferred()) sent_deferred = self.mail_service.send_mail(mail_dict()) @@ -50,64 +54,110 @@ class TestMailService(unittest.TestCase): return sent_deferred + @defer.inlineCallbacks def test_send_mail_removes_draft(self): - mail_ident = 'Some ident' - mail = mail_dict() - mail['ident'] = mail_ident - when(InputMail).from_dict(any()).thenReturn('inputmail') - deferred = Deferred() - when(self.mail_sender).sendmail(any()).thenReturn(deferred) + mail = LeapMail('id', 'INBOX') + when(mail).raw = 'raw mail' + when(InputMail).from_dict(ANY()).thenReturn(mail) + when(self.mail_store).delete_mail('12').thenReturn(defer.succeed(None)) + when(self.mail_store).add_mail('SENT', ANY()).thenReturn(mail) - sent_deferred = self.mail_service.send_mail(mail) + deferred_success = defer.succeed(None) + when(self.mail_sender).sendmail(ANY()).thenReturn(deferred_success) - verify(self.mail_sender).sendmail("inputmail") + yield self.mail_service.send_mail({'ident': '12'}) - def assert_removed_from_drafts(_): - verify(self.drafts).remove(any()) + verify(self.mail_sender).sendmail(mail) + verify(self.mail_store).add_mail('SENT', mail.raw) + verify(self.mail_store).delete_mail('12') - sent_deferred.addCallback(assert_removed_from_drafts) - sent_deferred.callback('Assume sending mail succeeded') + @defer.inlineCallbacks + def test_send_mail_marks_as_read(self): + mail = LeapMail('id', 'INBOX') + when(mail).raw = 'raw mail' + when(InputMail).from_dict(ANY()).thenReturn(mail) + when(self.mail_store).delete_mail('12').thenReturn(defer.succeed(None)) + when(self.mail_sender).sendmail(ANY()).thenReturn(defer.succeed(None)) - return sent_deferred + sent_mail = LeapMail('id', 'INBOX') + add_mail_deferral = defer.succeed(sent_mail) + when(self.mail_store).add_mail('SENT', ANY()).thenReturn(add_mail_deferral) - def test_send_mail_does_not_delete_draft_on_error(self): - when(InputMail).from_dict(any()).thenReturn('inputmail') - when(self.mail_sender).sendmail(any()).thenReturn(Deferred()) + yield self.mail_service.send_mail({'ident': '12'}) - send_deferred = self.mail_service.send_mail(mail_dict()) + self.assertIn(Status.SEEN, sent_mail.flags) + verify(self.mail_store).update_mail(sent_mail) - verify(self.mail_sender).sendmail("inputmail") + @defer.inlineCallbacks + def test_send_mail_does_not_delete_draft_on_error(self): + when(InputMail).from_dict(ANY()).thenReturn('inputmail') + + deferred_failure = defer.fail(Exception("Assume sending mail failed")) + when(self.mail_sender).sendmail(ANY()).thenReturn(deferred_failure) - def assert_not_removed_from_drafts(_): + try: + yield self.mail_service.send_mail({'ident': '12'}) + self.fail("send_mail is expected to raise if underlying call fails") + except: + verify(self.mail_sender).sendmail("inputmail") verifyNoMoreInteractions(self.drafts) - send_deferred.addErrback(assert_not_removed_from_drafts) + @defer.inlineCallbacks + def test_mark_as_read(self): + mail = LeapMail(1, 'INBOX') + when(self.mail_store).get_mail(1, include_body=True).thenReturn(mail) + yield self.mail_service.mark_as_read(1) - send_deferred.errback(Exception('Assume sending mail failed')) + self.assertIn(Status.SEEN, mail.flags) + verify(self.mail_store).update_mail(mail) - return send_deferred + @defer.inlineCallbacks + def test_mark_as_unread(self): + mail = LeapMail(1, 'INBOX') + mail.flags.add(Status.SEEN) - def test_mark_as_read(self): - mail = mock() - when(self.mail_service).mail(any()).thenReturn(mail) - self.mail_service.mark_as_read(1) + when(self.mail_store).get_mail(1, include_body=True).thenReturn(mail) + yield self.mail_service.mark_as_unread(1) + + verify(self.mail_store).update_mail(mail) - verify(mail).mark_as_read() + self.assertNotEqual(mail.status, Status.SEEN) + @defer.inlineCallbacks def test_delete_mail(self): - mail_to_delete = PixelatedMail.from_soledad(*leap_mail(), soledad_querier=None) - when(self.mail_service).mail(1).thenReturn(mail_to_delete) + mail_to_delete = LeapMail(1, 'INBOX') + when(self.mail_store).get_mail(1, include_body=True).thenReturn(defer.succeed(mail_to_delete)) - self.mail_service.delete_mail(1) + yield self.mail_service.delete_mail(1) - verify(self.mailboxes).move_to_trash(1) + verify(self.mail_store).move_mail_to_mailbox(1, 'TRASH') + @defer.inlineCallbacks def test_recover_mail(self): - mail_to_recover = PixelatedMail.from_soledad(*leap_mail(), soledad_querier=None) + mail_to_recover = LeapMail(1, 'TRASH') when(self.mail_service).mail(1).thenReturn(mail_to_recover) - when(self.mailboxes).move_to_inbox(1).thenReturn(mail_to_recover) + when(self.mail_store).move_mail_to_mailbox(1, 'INBOX').thenReturn(mail_to_recover) + + yield self.mail_service.recover_mail(1) + + verify(self.mail_store).move_mail_to_mailbox(1, 'INBOX') + + @defer.inlineCallbacks + def test_get_attachment(self): + attachment_dict = {'content': bytearray('data'), 'content-type': 'text/plain'} + when(self.mail_store).get_mail_attachment('some attachment id').thenReturn(defer.succeed(attachment_dict)) + + attachment = yield self.mail_service.attachment('some attachment id') + + self.assertEqual(attachment_dict, attachment) + + @defer.inlineCallbacks + def test_update_tags_return_a_set_with_the_current_tags(self): + mail = LeapMail(1, 'INBOX', tags={'custom_1', 'custom_2'}) + when(self.mail_store).get_mail(1, include_body=True).thenReturn(mail) + when(self.search_engine).tags(query='', skip_default_tags=True).thenReturn([]) - self.mail_service.recover_mail(1) + updated_mail = yield self.mail_service.update_tags(1, {'custom_1', 'custom_3'}) - verify(self.mailboxes).move_to_inbox(1) - verify(self.search_engine).index_mail(mail_to_recover) + verify(self.mail_store).update_mail(mail) + self.assertEqual({'custom_1', 'custom_3'}, updated_mail.tags) diff --git a/service/test/unit/adapter/test_mailbox.py b/service/test/unit/adapter/test_mailbox.py deleted file mode 100644 index ed634648..00000000 --- a/service/test/unit/adapter/test_mailbox.py +++ /dev/null @@ -1,42 +0,0 @@ -# -# Copyright (c) 2014 ThoughtWorks, Inc. -# -# Pixelated is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# Pixelated is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. -import unittest - -from pixelated.adapter.model.mail import PixelatedMail -from pixelated.adapter.services.mailbox import Mailbox -from mockito import mock, when, verify -from test.support import test_helper - - -class PixelatedMailboxTest(unittest.TestCase): - def setUp(self): - self.querier = mock() - self.search_engine = mock() - self.mailbox = Mailbox('INBOX', self.querier, self.search_engine) - - def test_remove_message_from_mailbox(self): - mail = PixelatedMail.from_soledad(*test_helper.leap_mail(), soledad_querier=self.querier) - when(self.querier).mail(1).thenReturn(mail) - - self.mailbox.remove(1) - - verify(self.querier).remove_mail(mail) - - def test_fresh_mailbox_checking_lastuid(self): - when(self.querier).get_lastuid('INBOX').thenReturn(0) - self.assertTrue(self.mailbox.fresh) - when(self.querier).get_lastuid('INBOX').thenReturn(1) - self.assertFalse(self.mailbox.fresh) diff --git a/service/test/unit/adapter/test_mailbox_indexer_listener.py b/service/test/unit/adapter/test_mailbox_indexer_listener.py index 71c9cd15..9ad3c94d 100644 --- a/service/test/unit/adapter/test_mailbox_indexer_listener.py +++ b/service/test/unit/adapter/test_mailbox_indexer_listener.py @@ -13,15 +13,18 @@ # # You should have received a copy of the GNU Affero General Public License # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. -import unittest +from twisted.trial import unittest -from mockito import mock, when, verify +from mockito import mock, when, verify, any as ANY from pixelated.adapter.listeners.mailbox_indexer_listener import MailboxIndexerListener +from twisted.internet import defer + +from pixelated.adapter.listeners.mailbox_indexer_listener import logger class MailboxListenerTest(unittest.TestCase): def setUp(self): - self.querier = mock() + self.mail_store = mock() self.account = mock() self.account.mailboxes = [] @@ -32,11 +35,11 @@ class MailboxListenerTest(unittest.TestCase): mailbox.listeners = set() when(mailbox).addListener = lambda x: mailbox.listeners.add(x) - self.assertNotIn(MailboxIndexerListener('INBOX', self.querier), mailbox.listeners) + self.assertNotIn(MailboxIndexerListener('INBOX', self.mail_store), mailbox.listeners) - MailboxIndexerListener.listen(self.account, 'INBOX', self.querier) + MailboxIndexerListener.listen(self.account, 'INBOX', self.mail_store) - self.assertIn(MailboxIndexerListener('INBOX', self.querier), mailbox.listeners) + self.assertIn(MailboxIndexerListener('INBOX', self.mail_store), mailbox.listeners) def test_reindex_missing_idents(self): search_engine = mock() @@ -44,11 +47,20 @@ class MailboxListenerTest(unittest.TestCase): MailboxIndexerListener.SEARCH_ENGINE = search_engine - listener = MailboxIndexerListener('INBOX', self.querier) - when(self.querier).idents_by_mailbox('INBOX').thenReturn({'ident1', 'ident2', 'missing_ident'}) - self.querier.used_arguments = [] - self.querier.mails = lambda x: self.querier.used_arguments.append(x) + listener = MailboxIndexerListener('INBOX', self.mail_store) + when(self.mail_store).get_mailbox_mail_ids('INBOX').thenReturn({'ident1', 'ident2', 'missing_ident'}) + self.mail_store.used_arguments = [] + self.mail_store.get_mails = lambda x: self.mail_store.used_arguments.append(x) listener.newMessages(10, 5) - verify(self.querier, times=1).idents_by_mailbox('INBOX') - self.assertIn({'missing_ident'}, self.querier.used_arguments) + verify(self.mail_store, times=1).get_mails('INBOX') + self.assertIn({'missing_ident'}, self.mail_store.used_arguments) + + @defer.inlineCallbacks + def test_catches_exceptions_to_not_break_other_listeners(self): + when(logger).error(ANY()).thenReturn(None) + listener = MailboxIndexerListener('INBOX', self.mail_store) + + yield listener.newMessages(1, 1) + + verify(logger).error(ANY()) diff --git a/service/test/unit/adapter/test_mailboxes.py b/service/test/unit/adapter/test_mailboxes.py deleted file mode 100644 index 6ff3849b..00000000 --- a/service/test/unit/adapter/test_mailboxes.py +++ /dev/null @@ -1,41 +0,0 @@ -# -# Copyright (c) 2014 ThoughtWorks, Inc. -# -# Pixelated is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# Pixelated is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. -import unittest - -from pixelated.adapter.model.mail import PixelatedMail -from pixelated.adapter.services.mailboxes import Mailboxes -from mockito import mock, when, verify -from test.support import test_helper -from mock import MagicMock - - -class PixelatedMailboxesTest(unittest.TestCase): - - def setUp(self): - self.querier = mock() - self.search_engine = mock() - self.account = MagicMock() - self.mailboxes = Mailboxes(self.account, self.querier, self.search_engine) - - def test_move_to_inbox(self): - mail = PixelatedMail.from_soledad(*test_helper.leap_mail(), soledad_querier=self.querier) - when(self.querier).mail(1).thenReturn(mail) - when(mail).save().thenReturn(None) - - mail.set_mailbox('TRASH') - recovered_mail = self.mailboxes.move_to_inbox(1) - self.assertEquals('INBOX', recovered_mail.mailbox_name) - verify(mail).save() diff --git a/service/test/unit/adapter/test_soledad_querier.py b/service/test/unit/adapter/test_soledad_querier.py deleted file mode 100644 index e5ea457d..00000000 --- a/service/test/unit/adapter/test_soledad_querier.py +++ /dev/null @@ -1,150 +0,0 @@ -# -# Copyright (c) 2014 ThoughtWorks, Inc. -# -# Pixelated is free software: you can redistribute it and/or modify -# it under the terms of the GNU Affero General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. -# -# Pixelated is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU Affero General Public License for more details. -# -# You should have received a copy of the GNU Affero General Public License -# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. -import unittest -import json -import base64 -import quopri - -from pixelated.adapter.soledad.soledad_querier import SoledadQuerier -from mockito import mock, when, any -import os - - -class SoledadQuerierTest(unittest.TestCase): - - def test_extract_parts(self): - soledad = mock() - bdoc = mock() - bdoc.content = {'raw': 'esse papo seu ta qualquer coisa'} - when(soledad).get_from_index('by-type-and-payloadhash', 'cnt', any(unicode)).thenReturn([bdoc]) - multipart_attachment_file = os.path.join(os.path.dirname(__file__), '..', 'fixtures', 'multipart_attachment.json') - with open(multipart_attachment_file) as f: - hdoc = json.loads(f.read()) - querier = SoledadQuerier(soledad) - - parts = querier._extract_parts(hdoc) - - self.assertIn('alternatives', parts.keys()) - self.assertIn('attachments', parts.keys()) - self.assertEquals(2, len(parts['alternatives'])) - self.assertEquals(1, len(parts['attachments'])) - - self.check_alternatives(parts) - self.check_attachments(parts) - - def check_alternatives(self, parts): - for alternative in parts['alternatives']: - self.assertIn('headers', alternative) - self.assertIn('content', alternative) - - def check_attachments(self, parts): - for attachment in parts['attachments']: - self.assertIn('headers', attachment) - self.assertIn('ident', attachment) - self.assertIn('name', attachment) - - def test_extract_part_without_headers(self): - soledad = mock() - bdoc = mock() - bdoc.content = {'raw': 'esse papo seu ta qualquer coisa'} - when(soledad).get_from_index('by-type-and-payloadhash', 'cnt', any(unicode)).thenReturn([bdoc]) - hdoc = {'multi': True, 'part_map': {'1': {'multi': False, 'phash': u'0400BEBACAFE'}}} - querier = SoledadQuerier(soledad) - - parts = querier._extract_parts(hdoc) - - self.assertEquals(bdoc.content['raw'], parts['alternatives'][0]['content']) - - def test_extract_handles_missing_part_map(self): - soledad = mock() - hdoc = {u'multi': True, - u'ctype': u'message/delivery-status', - u'headers': [[u'Content-Description', u'Delivery report'], [u'Content-Type', u'message/delivery-status']], - u'parts': 2, - u'phash': None, - u'size': 554} - querier = SoledadQuerier(soledad) - - parts = querier._extract_parts(hdoc) - - self.assertEquals(0, len(parts['alternatives'])) - self.assertEquals(0, len(parts['attachments'])) - - def test_attachment_base64(self): - soledad = mock() - bdoc = mock() - bdoc.content = {'raw': base64.encodestring('esse papo seu ta qualquer coisa'), 'content-type': 'text/plain'} - when(soledad).get_from_index('by-type-and-payloadhash', 'cnt', any(unicode)).thenReturn([bdoc]) - querier = SoledadQuerier(soledad) - - attachment = querier.attachment(u'0400BEBACAFE', 'base64') - - self.assertEquals('esse papo seu ta qualquer coisa', attachment['content']) - - def test_attachment_quoted_printable(self): - soledad = mock() - bdoc = mock() - bdoc.content = {'raw': quopri.encodestring('esse papo seu ta qualquer coisa'), 'content-type': 'text/plain'} - when(soledad).get_from_index('by-type-and-payloadhash', 'cnt', any(unicode)).thenReturn([bdoc]) - querier = SoledadQuerier(soledad) - - attachment = querier.attachment(u'0400BEBACAFE', 'quoted-printable') - - self.assertEquals('esse papo seu ta qualquer coisa', attachment['content']) - - def test_empty_or_null_queries_are_ignored(self): - soledad = mock() - when(soledad).get_from_index(any(), any(), any()).thenReturn(['nonempty', 'list']) - querier = SoledadQuerier(soledad) - - test_parameters = ['', None] - - def call_with_bad_parameters(funct): - for param in test_parameters: - self.assertFalse(funct(param)) - - call_with_bad_parameters(querier.get_all_flags_by_mbox) - call_with_bad_parameters(querier.get_content_by_phash) - call_with_bad_parameters(querier.get_flags_by_chash) - call_with_bad_parameters(querier.get_header_by_chash) - call_with_bad_parameters(querier.get_recent_by_mbox) - call_with_bad_parameters(querier.idents_by_mailbox) - call_with_bad_parameters(querier.get_mbox) - - def test_get_lastuid(self): - soledad = mock() - mbox = mock() - mbox.content = {'lastuid': 0} - when(soledad).get_from_index('by-type-and-mbox', 'mbox', 'INBOX').thenReturn([mbox]) - querier = SoledadQuerier(soledad) - - self.assertEquals(querier.get_lastuid(querier.get_mbox('INBOX')[0]), 0) - mbox.content = {'lastuid': 1} - self.assertEquals(querier.get_lastuid(querier.get_mbox('INBOX')[0]), 1) - - def test_create_mail_increments_uid(self): - soledad = mock() - mbox = mock() - mail = mock() - when(mail).get_for_save(next_uid=any(), mailbox='INBOX').thenReturn([]) - mbox.content = {'lastuid': 0} - when(soledad).get_from_index('by-type-and-mbox', 'mbox', 'INBOX').thenReturn([mbox]) - querier = SoledadQuerier(soledad) - when(querier).mail(any()).thenReturn([]) - - self.assertEquals(querier.get_lastuid(querier.get_mbox('INBOX')[0]), 0) - querier.create_mail(mail, 'INBOX') - self.assertEquals(querier.get_lastuid(querier.get_mbox('INBOX')[0]), 1) diff --git a/service/test/unit/bitmask_libraries/test_abstract_leap.py b/service/test/unit/bitmask_libraries/test_abstract_leap.py index 64de09bc..521d9cd4 100644 --- a/service/test/unit/bitmask_libraries/test_abstract_leap.py +++ b/service/test/unit/bitmask_libraries/test_abstract_leap.py @@ -19,28 +19,33 @@ from uuid import uuid4 import os from mock import Mock, MagicMock +from pixelated.adapter.mailstore import MailStore class AbstractLeapTest(unittest.TestCase): - _uuid = str(uuid4()) - _session_id = str(uuid4()) - _token = str(uuid4()) - leap_home = os.path.join(tempfile.mkdtemp(), 'leap') + def setUp(self): + self._uuid = str(uuid4()) + self._session_id = str(uuid4()) + self._token = str(uuid4()) - config = Mock(leap_home=leap_home, bootstrap_ca_cert_bundle='/some/path/to/ca_cert', ca_cert_bundle='/some/path/to/provider_ca_cert', gpg_binary='/path/to/gpg') - provider = Mock(config=config, server_name='some-server.test', domain='some-server.test', - api_uri='https://api.some-server.test:4430', api_version='1') - soledad = Mock() - soledad_session = Mock(soledad=soledad) - auth = Mock(username='test_user', - api_server_name='some-server.test', - uuid=_uuid, - session_id=_session_id, - token=_token) + self.leap_home = os.path.join(tempfile.mkdtemp(), 'leap') - nicknym = MagicMock() + self.config = Mock(leap_home=self.leap_home, bootstrap_ca_cert_bundle='/some/path/to/ca_cert', ca_cert_bundle='/some/path/to/provider_ca_cert', gpg_binary='/path/to/gpg') + self.provider = Mock(config=self.config, server_name='some-server.test', domain='some-server.test', + api_uri='https://api.some-server.test:4430', api_version='1') + self.soledad = Mock() + self.soledad_session = Mock(soledad=self.soledad) + self.auth = Mock(username='test_user', + api_server_name='some-server.test', + uuid=self._uuid, + session_id=self._session_id, + token=self._token) - soledad_account = MagicMock() + self.nicknym = MagicMock() - mail_fetcher_mock = MagicMock() + self.soledad_account = MagicMock() + + self.mail_fetcher_mock = MagicMock() + + self.mail_store = MagicMock(spec=MailStore) diff --git a/service/test/unit/bitmask_libraries/test_nicknym.py b/service/test/unit/bitmask_libraries/test_nicknym.py index ca3b348d..dc4845d1 100644 --- a/service/test/unit/bitmask_libraries/test_nicknym.py +++ b/service/test/unit/bitmask_libraries/test_nicknym.py @@ -39,12 +39,12 @@ class NickNymTest(AbstractLeapTest): 'test_user@some-server.test', 'https://nicknym.some-server.test:6425/', self.soledad, - self.auth.token, - '/some/path/to/provider_ca_cert', - 'https://api.some-server.test:4430', - '1', - self.auth.uuid, - '/path/to/gpg') + token=self.auth.token, + ca_cert_path='/some/path/to/provider_ca_cert', + api_uri='https://api.some-server.test:4430', + api_version='1', + uid=self.auth.uuid, + gpgbinary='/path/to/gpg') @patch('pixelated.bitmask_libraries.nicknym.KeyManager') def test_gen_key(self, keymanager_mock): @@ -61,5 +61,5 @@ class NickNymTest(AbstractLeapTest): # when/then nicknym.generate_openpgp_key() - keyman.get_key.assert_called_with('test_user@some-server.test', openpgp.OpenPGPKey, fetch_remote=False, private=True) + keyman.get_key.assert_called_with('test_user@some-server.test', openpgp.OpenPGPKey, private=True, fetch_remote=False) keyman.gen_key.assert_called_with(openpgp.OpenPGPKey) diff --git a/service/test/unit/bitmask_libraries/test_provider.py b/service/test/unit/bitmask_libraries/test_provider.py index 1fe5a66d..df851203 100644 --- a/service/test/unit/bitmask_libraries/test_provider.py +++ b/service/test/unit/bitmask_libraries/test_provider.py @@ -188,9 +188,13 @@ class LeapProviderTest(AbstractLeapTest): provider.fetch_valid_certificate() def test_throw_exception_for_invalid_certificate(self): + expected_exception_message = 'Certificate fingerprints don\'t match! Expected [0123456789012345678901234567890123456789012345678901234567890123] but got [06e2300bdbc118c290eda0dc977c24080718f4eeca68c8b0ad431872a2baa22d]' + with HTTMock(provider_json_invalid_fingerprint_mock, ca_cert_mock, not_found_mock): provider = LeapProvider('some-provider.test', self.config) - self.assertRaises(Exception, provider.fetch_valid_certificate) + with self.assertRaises(Exception) as cm: + provider.fetch_valid_certificate() + self.assertEqual(expected_exception_message, cm.exception.message) def test_that_bootstrap_cert_is_used_to_fetch_certificate(self): session = MagicMock(wraps=requests.session()) diff --git a/service/test/unit/bitmask_libraries/test_session.py b/service/test/unit/bitmask_libraries/test_session.py index 0c662ecb..e20f96f9 100644 --- a/service/test/unit/bitmask_libraries/test_session.py +++ b/service/test/unit/bitmask_libraries/test_session.py @@ -18,40 +18,29 @@ from mock import MagicMock from pixelated.bitmask_libraries.session import LeapSession from test_abstract_leap import AbstractLeapTest +from twisted.internet import defer class SessionTest(AbstractLeapTest): def setUp(self): - self.mail_fetcher_mock = MagicMock() + super(SessionTest, self).setUp() self.smtp_mock = MagicMock() - def tearDown(self): - self.mail_fetcher_mock = MagicMock() - - def test_background_jobs_are_started(self): - self.config.start_background_jobs = True - + def test_background_jobs_are_started_during_initial_sync(self): with patch('pixelated.bitmask_libraries.session.reactor.callFromThread', new=_execute_func) as _: - self._create_session() - - self.mail_fetcher_mock.start_loop.assert_called_once_with() - - def test_background_jobs_are_not_started(self): - self.config.start_background_jobs = False - - with patch('pixelated.bitmask_libraries.session.reactor.callFromThread', new=_execute_func) as _: - self._create_session() - - self.assertFalse(self.mail_fetcher_mock.start_loop.called) + with patch('pixelated.bitmask_libraries.session.LeapSession._create_incoming_mail_fetcher') as mail_fetcher_mock: + session = self._create_session() + yield session.initial_sync() + mail_fetcher_mock.startService.assert_called_once_with() def test_that_close_stops_background_jobs(self): with patch('pixelated.bitmask_libraries.session.reactor.callFromThread', new=_execute_func) as _: - session = self._create_session() - - session.close() - - self.mail_fetcher_mock.stop.assert_called_once_with() + with patch('pixelated.bitmask_libraries.session.LeapSession._create_incoming_mail_fetcher') as mail_fetcher_mock: + session = self._create_session() + yield session.initial_sync() + session.close() + mail_fetcher_mock.stopService.assert_called_once_with() def test_that_sync_deferes_to_soledad(self): session = self._create_session() @@ -61,8 +50,7 @@ class SessionTest(AbstractLeapTest): self.soledad_session.sync.assert_called_once_with() def _create_session(self): - return LeapSession(self.provider, self.auth, self.soledad_session, self.nicknym, self.soledad_account, - self.mail_fetcher_mock, self.smtp_mock) + return LeapSession(self.provider, self.auth, self.mail_store, self.soledad_session, self.nicknym, self.smtp_mock) def _execute_func(func): diff --git a/service/test/unit/bitmask_libraries/test_smtp.py b/service/test/unit/bitmask_libraries/test_smtp.py index ec51c56b..9481c488 100644 --- a/service/test/unit/bitmask_libraries/test_smtp.py +++ b/service/test/unit/bitmask_libraries/test_smtp.py @@ -42,6 +42,7 @@ class LeapSmtpTest(AbstractLeapTest): keymanager = MagicMock() def setUp(self): + super(LeapSmtpTest, self).setUp() self.provider.fetch_smtp_json.return_value = { 'hosts': { 'leap-mx': { diff --git a/service/test/unit/bitmask_libraries/test_soledad.py b/service/test/unit/bitmask_libraries/test_soledad.py index a3a1094a..af2cfd0a 100644 --- a/service/test/unit/bitmask_libraries/test_soledad.py +++ b/service/test/unit/bitmask_libraries/test_soledad.py @@ -15,13 +15,15 @@ # along with Pixelated. If not, see <http://www.gnu.org/licenses/>. from mock import patch from pixelated.bitmask_libraries.soledad import SoledadSession +from pixelated.bitmask_libraries.certs import LeapCertificate from test_abstract_leap import AbstractLeapTest -@patch('pixelated.bitmask_libraries.soledad.Soledad') class SoledadSessionTest(AbstractLeapTest): def setUp(self): + super(SoledadSessionTest, self).setUp() + # given self.provider.fetch_soledad_json.return_value = {'hosts': { 'couch1': { @@ -31,39 +33,29 @@ class SoledadSessionTest(AbstractLeapTest): } }} - @patch('pixelated.bitmask_libraries.soledad.Soledad.__init__') - def test_that_soledad_is_created_with_required_params(self, soledad_mock, init_mock): + @patch('pixelated.bitmask_libraries.soledad.Soledad') + def test_that_soledad_is_created_with_required_params(self, soledad_mock): + soledad_mock.return_value = None # when SoledadSession(self.provider, 'any-passphrase', self.auth.token, self.auth.uuid) # then - init_mock.assert_called_with(self.auth.uuid, 'any-passphrase', '%s/soledad/%s.secret' % (self.leap_home, self.auth.uuid), - '%s/soledad/%s.db' % (self.leap_home, self.auth.uuid), - 'https://couch1.some-server.test:1234/user-%s' % self.auth.uuid, - '/some/path/to/ca_cert', self.token, defer_encryption=False) - + soledad_mock.assert_called_with(self.auth.uuid, passphrase=u'any-passphrase', + secrets_path='%s/soledad/%s.secret' % (self.leap_home, self.auth.uuid), + local_db_path='%s/soledad/%s.db' % (self.leap_home, self.auth.uuid), + server_url='https://couch1.some-server.test:1234/user-%s' % self.auth.uuid, + cert_file=LeapCertificate(self.provider).provider_api_cert, + shared_db=None, + auth_token=self.auth.token, defer_encryption=False) + + @patch('pixelated.bitmask_libraries.soledad.Soledad') def test_that_sync_is_called(self, soledad_mock): - instance = soledad_mock.return_value - instance.server_url = '/foo/bar' - instance.need_sync.return_value = True - soledad_session = SoledadSession(self.provider, 'any-passphrase', self.auth.token, self.auth.uuid) - - # when - soledad_session.sync() - - # then - instance.need_sync.assert_called_with('/foo/bar') - instance.sync.assert_called_with() + instance = soledad_mock.return_value + instance.server_url = '/foo/bar' + soledad_session = SoledadSession(self.provider, 'any-passphrase', self.auth.token, self.auth.uuid) - def test_that_sync_not_called_if_not_needed(self, mock): - instance = mock.return_value - instance.server_url = '/foo/bar' - instance.need_sync.return_value = False - soledad_session = SoledadSession(self.provider, 'any-passphrase', self.auth.token, self.auth.uuid) - - # when - soledad_session.sync() + # when + soledad_session.sync() - # then - instance.need_sync.assert_called_with('/foo/bar') - self.assertFalse(instance.sync.called) + # then + instance.sync.assert_called_with() diff --git a/service/test/unit/config/test_register.py b/service/test/unit/config/test_register.py index 8e1a71a4..08cf56f0 100644 --- a/service/test/unit/config/test_register.py +++ b/service/test/unit/config/test_register.py @@ -1,6 +1,6 @@ import unittest -from pixelated.register import validate_username +from pixelated.register import validate_username, validate_password class TestRegister(unittest.TestCase): @@ -13,6 +13,10 @@ class TestRegister(unittest.TestCase): with self.assertRaises(ValueError): validate_username('invalid@username') + def test_password_raises_error_if_shorter_than_8_characters(self): + with self.assertRaises(ValueError): + validate_password('short') + def test_username_pass_when_valid(self): try: validate_username('a.valid_username-123') diff --git a/service/test/unit/config/test_site.py b/service/test/unit/config/test_site.py new file mode 100644 index 00000000..1858bfaf --- /dev/null +++ b/service/test/unit/config/test_site.py @@ -0,0 +1,28 @@ +import unittest +from mockito import mock +from pixelated.config.site import PixelatedSite +from twisted.protocols.basic import LineReceiver + + +class TestPixelatedSite(unittest.TestCase): + def test_add_csp_header_request(self): + request = self.create_request() + request.process() + headers = request.headers + + header_value = "default-src 'self'; style-src 'self' 'unsafe-inline'" + self.assertEqual(headers.get("Content-Security-Policy"), header_value) + self.assertEqual(headers.get("X-Content-Security-Policy"), header_value) + self.assertEqual(headers.get("X-Webkit-CSP"), header_value) + + def create_request(self): + channel = LineReceiver() + channel.site = PixelatedSite(mock()) + request = PixelatedSite.requestFactory(channel=channel, queued=True) + request.method = "GET" + request.uri = "localhost" + request.clientproto = 'HTTP/1.1' + request.prepath = [] + request.postpath = request.uri.split('/')[1:] + request.path = "/" + return request diff --git a/service/test/unit/fixtures/__init__.py b/service/test/unit/fixtures/__init__.py new file mode 100644 index 00000000..e69de29b --- /dev/null +++ b/service/test/unit/fixtures/__init__.py diff --git a/service/test/unit/fixtures/bounced_mail_hdoc.json b/service/test/unit/fixtures/bounced_mail_hdoc.json deleted file mode 100644 index 2cc5997c..00000000 --- a/service/test/unit/fixtures/bounced_mail_hdoc.json +++ /dev/null @@ -1,218 +0,0 @@ -{ - "body": "3583150D422268B15A27553279767CD26D7AC20937E8BDA8DC55BA53CFF0E9B2", - "chash": "E374570706F4263C533103893344FD6CBE62B9807ECCC082C159FECF60277656", - "date": "Wed, 11 Feb 2015 20:49:14 +0100 (CET)", - "headers": { - "Auto-Submitted": "auto-replied", - "Content-Type": "multipart/report; report-type=delivery-status;\n boundary=\"499F57FF7C.1423684154/domain.org\"", - "Date": "Wed, 11 Feb 2015 20:49:14 +0100 (CET)", - "Delivered-To": "412cab846cd0d6327f4505e9c4112c64@domain.org", - "From": "MAILER-DAEMON@domain.org (Mail Delivery System)", - "MIME-Version": "1.0", - "Message-Id": "<20150211194914.12CB3804C7@domain.org>", - "Received": "by domain.org (Postfix)\n id 12CB3804C7; Wed, 11 Feb 2015 20:49:14 +0100 (CET)", - "Return-Path": "<>", - "Subject": "Undelivered Mail Returned to Sender", - "To": "cuzcuz2@domain.org", - "X-Leap-Provenance": "Wed, 11 Feb 2015 19:49:14 -0000; pubkey=\"E2F104EE8B01F675\"", - "X-Leap-Signature": "could not verify", - "X-Original-To": "cuzcuz2@domain.org" - }, - "msgid": null, - "multi": true, - "part_map": { - "1": { - "ctype": "text/plain", - "headers": [ - [ - "Content-Description", - "Notification" - ], - [ - "Content-Type", - "text/plain; charset=us-ascii" - ] - ], - "multi": false, - "parts": 1, - "phash": "3583150D422268B15A27553279767CD26D7AC20937E8BDA8DC55BA53CFF0E9B2", - "size": 896 - }, - "2": { - "ctype": "message/delivery-status", - "headers": [ - [ - "Content-Description", - "Delivery report" - ], - [ - "Content-Type", - "message/delivery-status" - ] - ], - "multi": true, - "parts": 2, - "phash": null, - "size": 772 - }, - "3": { - "ctype": "text/plain", - "headers": [ - [ - "Reporting-MTA", - "dns; domain.org" - ], - [ - "X-Postfix-Queue-ID", - "499F57FF7C" - ], - [ - "X-Postfix-Sender", - "rfc822; cuzcuz2@domain.org" - ], - [ - "Arrival-Date", - "Wed, 11 Feb 2015 20:49:10 +0100 (CET)" - ] - ], - "multi": false, - "part_map": { - "0": { - "ctype": "text/plain", - "headers": [ - [ - "Final-Recipient", - "rfc822; this_mail_was_bounced@domain.com" - ], - [ - "Original-Recipient", - "rfc822;this_mail_was_bounced@domain.com" - ], - [ - "Action", - "failed" - ], - [ - "Status", - "5.1.1" - ], - [ - "Remote-MTA", - "dns; ASPMX.L.DOMAIN.com" - ], - [ - "Diagnostic-Code", - "smtp; 550-5.1.1 The email account that you tried to reach does\n not exist. Please try 550-5.1.1 double-checking the\n recipient's email\n address for typos or 550-5.1.1 unnecessary spaces." - ] - ], - "multi": false, - "parts": 1, - "phash": "E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855", - "size": 497 - }, - "1": { - "ctype": "message/rfc822", - "headers": [ - [ - "Content-Description", - "Undelivered Message" - ], - [ - "Content-Type", - "message/rfc822" - ] - ], - "multi": true, - "parts": 1, - "phash": null, - "size": 2397 - }, - "2": { - "headers": { - "Content-Type": "multipart/signed; protocol=\"application/pgp-signature\";\n micalg=\"pgp-sha512\"; boundary=\"===============5178971458699783746==\"", - "Date": "2015-02-11T16:48:54.245828-03:00", - "From": "cuzcuz2@domain.org", - "MIME-Version": "1.0", - "Message-Id": "<20150211194857.19888.1605637474.0@host>", - "OpenPGP": "id=E2F104EE8B01F675;\n url=\"https://domain.org/key/cuzcuz2\"; preference=\"signencrypt\"", - "Received": "from 0.3.9-1-gc1f9c92 (unknown [127.0.0.1])\n (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits))\n (Client CN \"UNLIMITEDdbe63unx9tfxa286ol3che4vx\",\n Issuer \"LEAP_Example Root CA (client certificates only!)\" (verified OK))\n by domain.org (Postfix) with ESMTPS id 499F57FF7C\n for <this_mail_was_bounced@domain.com>; Wed, 11 Feb 2015 20:49:10 +0100 (CET)", - "Return-Path": "<cuzcuz2@domain.org>", - "Subject": "volte", - "To": "this_mail_was_bounced@domain.com" - }, - "multi": true, - "part_map": { - "1": { - "ctype": "multipart/mixed", - "headers": [ - [ - "Received", - "by bitmask.local from 127.0.0.1 with ESMTP ;\n Wed, 11 Feb 2015 16:48:55 -0300" - ], - [ - "Content-Type", - "multipart/mixed; boundary=\"===============4044974129166450777==\"" - ], - [ - "MIME-Version", - "1.0" - ] - ], - "multi": true, - "parts": 1, - "phash": null, - "size": 370 - }, - "2": { - "ctype": "text/plain", - "headers": [ - [ - "Content-Type", - "text/plain; charset=\"utf-8\"" - ], - [ - "MIME-Version", - "1.0" - ], - [ - "Content-Transfer-Encoding", - "base64" - ] - ], - "multi": false, - "parts": 1, - "phash": "231B2728DE0EF4968B1183F0DD7FA9C963A90996E64B9B7A9AB6936F0B1EADB7", - "size": 100 - } - } - }, - "3": { - "ctype": "application/pgp-signature", - "headers": [ - [ - "Content-Type", - "application/pgp-signature; name=\"signature.asc\"" - ], - [ - "MIME-Version", - "1.0" - ], - [ - "Content-Description", - "OpenPGP Digital Signature" - ] - ], - "multi": false, - "parts": 1, - "phash": "90A694453A671BA3144BA9147ACA4C4B41DF7E714377C584E802ED5469AB5365", - "size": 929 - } - }, - "parts": 1, - "phash": "E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855", - "size": 200 - } - }, - "subject": "Undelivered Mail Returned to Sender", - "type": "head" -} diff --git a/service/test/unit/fixtures/mailset/new/mbox00000000 b/service/test/unit/fixtures/mailset/new/mbox00000000 index 3d01c203..8c80257d 100644 --- a/service/test/unit/fixtures/mailset/new/mbox00000000 +++ b/service/test/unit/fixtures/mailset/new/mbox00000000 @@ -1,4 +1,4 @@ -From darby.senger@zemlak.biz +From: darby.senger@zemlak.biz Subject: Itaque consequatur repellendus provident sunt quia. To: carmel@murazikortiz.name X-TW-Pixelated-Tags: nite, macro, trash diff --git a/service/test/unit/fixtures/mailset/new/mbox00000001 b/service/test/unit/fixtures/mailset/new/mbox00000001 index fc76bba2..ba563430 100644 --- a/service/test/unit/fixtures/mailset/new/mbox00000001 +++ b/service/test/unit/fixtures/mailset/new/mbox00000001 @@ -1,4 +1,4 @@ -From madeline.littel@sanfordruel.com +From: madeline.littel@sanfordruel.com Subject: Error illum dignissimos autem eos aspernatur. To: phyllis@stiedemann.net X-TW-Pixelated-Tags: instadaily, inspiration diff --git a/service/test/unit/maintenance/test_commands.py b/service/test/unit/maintenance/test_commands.py index f1bf6e45..52fe6ca2 100644 --- a/service/test/unit/maintenance/test_commands.py +++ b/service/test/unit/maintenance/test_commands.py @@ -18,13 +18,13 @@ import email from pixelated.maintenance import delete_all_mails, load_mails from pixelated.bitmask_libraries.session import LeapSession -from leap.mail.imap.account import SoledadBackedAccount -from leap.mail.imap.fields import WithMsgFields +from pixelated.adapter.mailstore import MailStore from leap.soledad.client import Soledad from leap.soledad.common.document import SoledadDocument from mock import MagicMock -from os.path import join, dirname -from twisted.internet import defer, reactor +from os.path import join +from twisted.internet import defer +import pkg_resources class TestCommands(unittest.TestCase): @@ -32,10 +32,8 @@ class TestCommands(unittest.TestCase): def setUp(self): self.leap_session = MagicMock(spec=LeapSession) self.soledad = MagicMock(spec=Soledad) - self.account = MagicMock(spec=SoledadBackedAccount) - self.mailbox = MagicMock() - self.leap_session.account = self.account - self.account.getMailbox.return_value = self.mailbox + self.mail_store = MagicMock(spec=MailStore) + self.leap_session.mail_store = self.mail_store self.args = (self.leap_session, self.soledad) @@ -77,23 +75,25 @@ class TestCommands(unittest.TestCase): def test_load_mails_empty_path_list(self): load_mails(self.args, []) - self.assertFalse(self.mailbox.called) + self.assertFalse(self.mail_store.add_mailbox.called) def test_load_mails_adds_mails(self): # given - mail_root = join(dirname(__file__), '..', 'fixtures', 'mailset') - firstMailDeferred = defer.Deferred() - secondMailDeferred = defer.Deferred() - self.mailbox.addMessage.side_effect = [firstMailDeferred, secondMailDeferred] + mail_root = pkg_resources.resource_filename('test.unit.fixtures', 'mailset') + firstMailDeferred = defer.succeed(None) + secondMailDeferred = defer.succeed(None) + self.mail_store.add_mail.side_effect = [firstMailDeferred, secondMailDeferred] + self.mail_store.add_mailbox.return_value = defer.succeed(None) # when d = load_mails(self.args, [mail_root]) # then def assert_mails_added(_): - self.assertTrue(self.mailbox.addMessage.called) - self.mailbox.addMessage.assert_any_call(self._mail_content(join(mail_root, 'new', 'mbox00000000')), flags=(WithMsgFields.RECENT_FLAG,), notify_on_disk=False) - self.mailbox.addMessage.assert_any_call(self._mail_content(join(mail_root, 'new', 'mbox00000001')), flags=(WithMsgFields.RECENT_FLAG,), notify_on_disk=False) + self.assertTrue(self.mail_store.add_mail.called) + self.mail_store.add_mail.assert_any_call('INBOX', self._mail_content(join(mail_root, 'new', 'mbox00000000'))) + self.mail_store.add_mail.assert_any_call('INBOX', self._mail_content(join(mail_root, 'new', 'mbox00000001'))) + # TODO Should we check for flags? def error_callack(err): print err @@ -102,10 +102,6 @@ class TestCommands(unittest.TestCase): d.addCallback(assert_mails_added) d.addErrback(error_callack) - # trigger callbacks for both mails - reactor.callLater(0, firstMailDeferred.callback, None) - reactor.callLater(0, secondMailDeferred.callback, None) - return d def _mail_content(self, mail_file): diff --git a/service/test/unit/resources/test_feedback_resource.py b/service/test/unit/resources/test_feedback_resource.py new file mode 100644 index 00000000..63e6efc4 --- /dev/null +++ b/service/test/unit/resources/test_feedback_resource.py @@ -0,0 +1,27 @@ +import json +from mockito import verify, mock, when +from twisted.trial import unittest +from twisted.web.test.requesthelper import DummyRequest +from pixelated.resources.feedback_resource import FeedbackResource +from test.unit.resources import DummySite + + +class TestFeedbackResource(unittest.TestCase): + def setUp(self): + self.feedback_service = mock() + self.web = DummySite(FeedbackResource(self.feedback_service)) + + def test_sends_feedback_to_leap_web(self): + request = DummyRequest(['/feedback']) + request.method = 'POST' + content = mock() + when(content).read().thenReturn(json.dumps({'feedback': 'Pixelated is awesome!'})) + request.content = content + + d = self.web.get(request) + + def assert_posted_feedback_to_leap_web(_): + verify(self.feedback_service).open_ticket('Pixelated is awesome!') + + d.addCallback(assert_posted_feedback_to_leap_web) + return d diff --git a/service/test/unit/resources/test_keys_resources.py b/service/test/unit/resources/test_keys_resources.py index be79424b..1990efe8 100644 --- a/service/test/unit/resources/test_keys_resources.py +++ b/service/test/unit/resources/test_keys_resources.py @@ -1,8 +1,11 @@ +import json +import ast from mockito import mock, when from leap.keymanager import OpenPGPKey, KeyNotFound from pixelated.resources.keys_resource import KeysResource import twisted.trial.unittest as unittest from twisted.web.test.requesthelper import DummyRequest +from twisted.internet import defer from test.unit.resources import DummySite @@ -15,7 +18,7 @@ class TestKeysResource(unittest.TestCase): def test_returns_404_if_key_not_found(self): request = DummyRequest(['/keys']) request.addArg('search', 'some@inexistent.key') - when(self.keymanager).get_key_from_cache('some@inexistent.key', OpenPGPKey).thenRaise(KeyNotFound()) + when(self.keymanager).fetch_key('some@inexistent.key').thenReturn(defer.fail(KeyNotFound())) d = self.web.get(request) @@ -28,17 +31,30 @@ class TestKeysResource(unittest.TestCase): def test_returns_the_key_as_json_if_found(self): request = DummyRequest(['/keys']) request.addArg('search', 'some@key') - when(self.keymanager).get_key_from_cache('some@key', OpenPGPKey).thenReturn(OpenPGPKey('some@key')) + when(self.keymanager).fetch_key('some@key').thenReturn(defer.succeed(OpenPGPKey('some@key'))) d = self.web.get(request) + expected = { + "tags": ["keymanager-key"], + "fingerprint": '', + "private": False, + 'sign_used': False, + 'refreshed_at': 0, + "expiry_date": 0, + "address": 'some@key', + 'encr_used': False, + 'last_audited_at': 0, + 'key_data': '', + 'length': 0, + 'key_id': '', + 'validation': 'Weak_Chain', + 'type': 'OpenPGPKey', + } + def assert_response(_): - self.assertEquals('"{\\"tags\\": [\\"keymanager-key\\"], \\"fingerprint\\": null, ' - '\\"private\\": null, \\"expiry_date\\": null, \\"address\\": ' - '\\"some@key\\", \\"last_audited_at\\": null, \\"key_data\\": null, ' - '\\"length\\": null, \\"key_id\\": null, \\"validation\\": null, ' - '\\"type\\": \\"<class \'leap.keymanager.openpgp.OpenPGPKey\'>\\", ' - '\\"first_seen_at\\": null}"', request.written[0]) + actual = json.loads(ast.literal_eval(request.written[0])) + self.assertEquals(expected, actual) d.addCallback(assert_response) return d @@ -46,7 +62,7 @@ class TestKeysResource(unittest.TestCase): def test_returns_unauthorized_if_key_is_private(self): request = DummyRequest(['/keys']) request.addArg('search', 'some@key') - when(self.keymanager).get_key_from_cache('some@key', OpenPGPKey).thenReturn(OpenPGPKey('some@key', private=True)) + when(self.keymanager).fetch_key('some@key').thenReturn(defer.succeed(OpenPGPKey('some@key', private=True))) d = self.web.get(request) diff --git a/service/test/unit/support/test_encrypted_file_storage.py b/service/test/unit/support/test_encrypted_file_storage.py index 2a6735c3..69b82f3d 100644 --- a/service/test/unit/support/test_encrypted_file_storage.py +++ b/service/test/unit/support/test_encrypted_file_storage.py @@ -25,9 +25,13 @@ class EncryptedFileStorageTest(unittest.TestCase): self.key = '2\x06\xf87F:\xd2\xe2]w\xc9\x0c\xb8\x9b\x8e\xd3\x92\t\xabHu\xa6\xa3\x9a\x8d\xec\x0c\xab<8\xbb\x12\xfbP\xf2\x83"\xa1\xcf7\x92\xb0!\xfe\xebM\x80\x8a\x14\xe6\xf9xr\xf5#\x8f\x1bs\xb3#\x0e)a\xd8' self.msg = 'this is a very, very secret binary message: \xbe\xba\xca\xfe' self.path = os.path.join('tmp', 'search_test') + self._cleanup_path() self.storage = EncryptedFileStorage(self.path, self.key) def tearDown(self): + self._cleanup_path() + + def _cleanup_path(self): if os.path.exists(self.path): shutil.rmtree(self.path) diff --git a/service/test/unit/test_application.py b/service/test/unit/test_application.py index b2799d4c..16317ee5 100644 --- a/service/test/unit/test_application.py +++ b/service/test/unit/test_application.py @@ -14,9 +14,11 @@ class ApplicationTest(unittest.TestCase): self.sslcert = sslcert self.home = 'leap_home' + @patch('leap.common.events.client') @patch('pixelated.application.reactor') @patch('pixelated.application.Services') - def test_that_create_app_binds_to_tcp_port_if_no_ssl_options(self, services_mock, reactor_mock): + def test_that_create_app_binds_to_tcp_port_if_no_ssl_options(self, services_mock, reactor_mock, _): + # FIXME patch something closer, instead of leap.common app_mock = MagicMock() leap_session = MagicMock() config = ApplicationTest.MockConfig(12345, '127.0.0.1', leap_session) @@ -32,9 +34,11 @@ class ApplicationTest(unittest.TestCase): d.addCallback(_assert) return d + @patch('leap.common.events.client') @patch('pixelated.application.reactor') @patch('pixelated.application.Services') - def test_that_create_app_binds_to_ssl_if_ssl_options(self, services_mock, reactor_mock): + def test_that_create_app_binds_to_ssl_if_ssl_options(self, services_mock, reactor_mock, _): + # FIXME patch something closer, instead of leap.common app_mock = MagicMock() leap_session = MagicMock() pixelated.application._ssl_options = lambda x, y: 'options' diff --git a/service/test/unit/test_welcome_mail.py b/service/test/unit/test_welcome_mail.py new file mode 100644 index 00000000..7e9ab0c9 --- /dev/null +++ b/service/test/unit/test_welcome_mail.py @@ -0,0 +1,73 @@ +# +# Copyright (c) 2014 ThoughtWorks, Inc. +# +# Pixelated is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# Pixelated is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with Pixelated. If not, see <http://www.gnu.org/licenses/>. + +import os +import unittest +from mockito import verify, mock +from mockito.matchers import Matcher +from email import message_from_file +from pixelated.config.leap import add_welcome_mail +from pixelated.adapter.model.mail import InputMail + + +class TestWelcomeMail(unittest.TestCase): + + def test_add_welcome_mail(self): + mail_store = mock() + input_mail = self._get_welcome_mail() + + add_welcome_mail(mail_store) + capture = WelcomeMailCapture() + + verify(mail_store).add_mail('INBOX', capture) + capture.assert_mail(input_mail.raw) + + def _get_welcome_mail(self): + current_path = os.path.dirname(os.path.abspath(__file__)) + with open(os.path.join(current_path, + '..', + '..', + 'pixelated', + 'assets', + 'welcome.mail')) as mail_template_file: + mail_template = message_from_file(mail_template_file) + + return InputMail.from_python_mail(mail_template) + + +class WelcomeMailCapture(Matcher): + + def matches(self, arg): + self.value = arg + return True + + def assert_mail(self, mail): + captured_mail = self._format(self.value) + expected_mail = self._format(mail) + assert captured_mail == expected_mail + + def _format(self, mail): + splitter = '\n' + arr = mail.split(splitter) + arr = self._remove_variable_value(arr) + + return splitter.join(arr) + + def _remove_variable_value(self, arr): + arr.pop(0) + arr.pop(6) + arr.pop(44) + return arr diff --git a/service/test_requirements.txt b/service/test_requirements.txt index 526c8495..95e68f21 100644 --- a/service/test_requirements.txt +++ b/service/test_requirements.txt @@ -8,4 +8,4 @@ tornado==4.0.2 pep8==1.5.7 tempdir==0.6 coverage - +crochet==1.4.0 diff --git a/web-ui/Makefile b/web-ui/Makefile index 383fecb7..0bc9d5d4 100644 --- a/web-ui/Makefile +++ b/web-ui/Makefile @@ -18,7 +18,7 @@ DESTDIR=target compile: npm install - node_modules/bower/bin/bower install + node_modules/bower/bin/bower --allow-root install ./go package clean: diff --git a/web-ui/app/index.html b/web-ui/app/index.html index cbdae267..281d661e 100644 --- a/web-ui/app/index.html +++ b/web-ui/app/index.html @@ -16,7 +16,6 @@ </head> <body> - <div class="off-canvas-wrap move-right menu" data-offcanvas> <div class="inner-wrap"> <section id="left-pane" class="left-off-canvas-menu"> @@ -49,6 +48,9 @@ </a> <nav id="tag-list"></nav> <div class="side-nav-bottom"> + <div class="version">0.3.1-beta</div> + + <nav id="feedback"></nav> <nav id="logout"></nav> </div> </section> @@ -58,7 +60,6 @@ <div class="off-canvas-wrap content" data-offcanvas> <header id="main" > <div id="user-alerts" class="message-panel"></div> - <div id="loading" class="message-panel"><span>Loading...</span></div> </header> <div class="inner-wrap"> @@ -97,11 +98,9 @@ <script src="assets/bower_components/foundation/js/foundation.js" ></script> <script src="assets/bower_components/foundation/js/foundation/foundation.reveal.js" ></script> <script src="assets/bower_components/foundation/js/foundation/foundation.offcanvas.js"></script> +<script src="assets/js/foundation/initialize_foundation.js"></script> <script src="assets/bower_components/requirejs/require.js" data-main="assets/js/main.js"></script> <!--usemin_end--> -<script> -$(document).foundation(); -</script> </body> </html> diff --git a/web-ui/app/js/dispatchers/right_pane_dispatcher.js b/web-ui/app/js/dispatchers/right_pane_dispatcher.js index 8de89858..870bcd92 100644 --- a/web-ui/app/js/dispatchers/right_pane_dispatcher.js +++ b/web-ui/app/js/dispatchers/right_pane_dispatcher.js @@ -23,10 +23,11 @@ define( 'mail_view/ui/reply_section', 'mail_view/ui/draft_box', 'mail_view/ui/no_message_selected_pane', + 'mail_view/ui/feedback_box', 'page/events' ], - function(defineComponent, ComposeBox, MailView, ReplySection, DraftBox, NoMessageSelectedPane, events) { + function(defineComponent, ComposeBox, MailView, ReplySection, DraftBox, NoMessageSelectedPane, FeedbackBox, events) { 'use strict'; return defineComponent(rightPaneDispatcher); @@ -35,6 +36,7 @@ define( this.defaultAttrs({ rightPane: '#right-pane', composeBox: 'compose-box', + feedbackBox: 'feedback-box', mailView: 'mail-view', noMessageSelectedPane: 'no-message-selected-pane', replySection: 'reply-section', @@ -60,6 +62,11 @@ define( ComposeBox.attachTo(stage, {currentTag: this.attr.currentTag}); }; + this.openFeedbackBox = function() { + var stage = this.reset(this.attr.feedbackBox); + FeedbackBox.attachTo(stage); + }; + this.openMail = function(ev, data) { var stage = this.reset(this.attr.mailView); MailView.attachTo(stage, data); @@ -97,6 +104,7 @@ define( this.on(document, events.dispatchers.rightPane.openComposeBox, this.openComposeBox); this.on(document, events.dispatchers.rightPane.openDraft, this.openDraft); this.on(document, events.ui.mail.open, this.openMail); + this.on(document, events.dispatchers.rightPane.openFeedbackBox, this.openFeedbackBox); this.on(document, events.dispatchers.rightPane.openNoMessageSelected, this.openNoMessageSelectedPane); this.on(document, events.dispatchers.rightPane.selectTag, this.selectTag); this.on(document, events.ui.tag.selected, this.saveTag); diff --git a/web-ui/app/js/feedback/feedback_trigger.js b/web-ui/app/js/feedback/feedback_trigger.js new file mode 100644 index 00000000..598f9060 --- /dev/null +++ b/web-ui/app/js/feedback/feedback_trigger.js @@ -0,0 +1,39 @@ +/* + * Copyright (c) 2015 ThoughtWorks, Inc. + * + * Pixelated is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Pixelated is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with Pixelated. If not, see <http://www.gnu.org/licenses/>. + */ + +define(['flight/lib/component', 'views/templates', 'page/events', 'features'], + function (defineComponent, templates, events, features) { + 'use strict'; + + return defineComponent(function () { + this.render = function () { + this.$node.html(templates.feedback.feedback()); + }; + + this.onClick = function() { + this.trigger(document, events.dispatchers.rightPane.openFeedbackBox); + }; + + this.after('initialize', function () { + if (features.isEnabled('feedback')) { + this.render(); + this.on('click', this.onClick); + } + }); + + }); +}); diff --git a/web-ui/app/js/foundation/initialize_foundation.js b/web-ui/app/js/foundation/initialize_foundation.js new file mode 100644 index 00000000..b382a168 --- /dev/null +++ b/web-ui/app/js/foundation/initialize_foundation.js @@ -0,0 +1,3 @@ +'use strict'; + +$(document).foundation(); diff --git a/web-ui/app/js/helpers/monitored_ajax.js b/web-ui/app/js/helpers/monitored_ajax.js index 0068e10c..7f9a9beb 100644 --- a/web-ui/app/js/helpers/monitored_ajax.js +++ b/web-ui/app/js/helpers/monitored_ajax.js @@ -31,9 +31,6 @@ define(['page/events', 'views/i18n'], function (events, i18n) { var originalBeforeSend = config.beforeSend; config.beforeSend = function () { - if (!config.skipLoadingWarning) { - $('#loading').show(); - } if (originalBeforeSend) { originalBeforeSend(); } @@ -41,9 +38,6 @@ define(['page/events', 'views/i18n'], function (events, i18n) { var originalComplete = config.complete; config.complete = function () { - if (!config.skipLoadingWarning) { - $('#loading').fadeOut(500); - } if (originalComplete) { originalComplete(); } diff --git a/web-ui/app/js/mail_view/data/feedback_sender.js b/web-ui/app/js/mail_view/data/feedback_sender.js new file mode 100644 index 00000000..2232dbe4 --- /dev/null +++ b/web-ui/app/js/mail_view/data/feedback_sender.js @@ -0,0 +1,49 @@ +/* + * Copyright (c) 2015 ThoughtWorks, Inc. + * + * Pixelated is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Pixelated is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with Pixelated. If not, see <http://www.gnu.org/licenses/>. + */ +define( + [ + 'flight/lib/component', + 'helpers/monitored_ajax', + 'page/events' + ], + function (defineComponent, monitoredAjax, events) { + 'use strict'; + + return defineComponent(function () { + this.defaultAttrs({ + feedbackResource: '/feedback' + }); + + this.successSubmittingFeedback = function() { + this.trigger(document, events.feedback.submitted); + }; + + this.submitFeedback = function(event, data) { + monitoredAjax.call(_, this, this.attr.feedbackResource, { + type: 'POST', + dataType: 'json', + contentType: 'application/json; charset=utf-8', + data: JSON.stringify(data) + }).done(this.successSubmittingFeedback()); + }; + + this.after('initialize', function () { + this.on(document, events.feedback.submit, this.submitFeedback); + }); + + }); +}); diff --git a/web-ui/app/js/mail_view/data/mail_sender.js b/web-ui/app/js/mail_view/data/mail_sender.js index c84a4f97..0d11c636 100644 --- a/web-ui/app/js/mail_view/data/mail_sender.js +++ b/web-ui/app/js/mail_view/data/mail_sender.js @@ -55,7 +55,7 @@ define( type: 'POST', dataType: 'json', contentType: 'application/json; charset=utf-8', - data: JSON.stringify(data), + data: JSON.stringify(data) }).done(successSendingMail(this)).fail(failureSendingMail(this)); }; @@ -66,7 +66,6 @@ define( dataType: 'json', contentType: 'application/json; charset=utf-8', data: JSON.stringify(mail), - skipLoadingWarning: true, skipErrorMessage: true }); }; diff --git a/web-ui/app/js/mail_view/ui/compose_box.js b/web-ui/app/js/mail_view/ui/compose_box.js index 1b209b23..28f6dc83 100644 --- a/web-ui/app/js/mail_view/ui/compose_box.js +++ b/web-ui/app/js/mail_view/ui/compose_box.js @@ -54,6 +54,8 @@ define( this.renderComposeBox = function() { this.render(templates.compose.box, {}); + this.enableFloatlabel('input.floatlabel'); + this.enableFloatlabel('textarea.floatlabel'); this.select('recipientsFields').show(); this.on(this.select('closeButton'), 'click', this.showNoMessageSelected); this.enableAutoSave(); diff --git a/web-ui/app/js/mail_view/ui/draft_box.js b/web-ui/app/js/mail_view/ui/draft_box.js index adad108f..8c2e15c7 100644 --- a/web-ui/app/js/mail_view/ui/draft_box.js +++ b/web-ui/app/js/mail_view/ui/draft_box.js @@ -66,6 +66,8 @@ define( body: body }); + this.enableFloatlabel('input.floatlabel'); + this.enableFloatlabel('textarea.floatlabel'); this.select('recipientsFields').show(); this.select('bodyBox').focus(); this.select('tipMsg').hide(); diff --git a/web-ui/app/js/mail_view/ui/draft_button.js b/web-ui/app/js/mail_view/ui/draft_button.js new file mode 100644 index 00000000..1a89c414 --- /dev/null +++ b/web-ui/app/js/mail_view/ui/draft_button.js @@ -0,0 +1,41 @@ +/* +* Copyright (c) 2014 ThoughtWorks, Inc. +* +* Pixelated is free software: you can redistribute it and/or modify +* it under the terms of the GNU Affero General Public License as published by +* the Free Software Foundation, either version 3 of the License, or +* (at your option) any later version. +* +* Pixelated is distributed in the hope that it will be useful, +* but WITHOUT ANY WARRANTY; without even the implied warranty of +* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +* GNU Affero General Public License for more details. +* +* You should have received a copy of the GNU Affero General Public License +* along with Pixelated. If not, see <http://www.gnu.org/licenses/>. +*/ +'use strict'; + +define([ + 'flight/lib/component', + 'page/events', +], +function (defineComponent, events) { + return defineComponent(draftButton); + + function draftButton() { + this.enableButton = function () { + this.$node.prop('disabled', false); + }; + + this.disableButton = function () { + this.$node.prop('disabled', true); + }; + + this.after('initialize', function(){ + this.disableButton(); + this.on(document, events.mail.saveDraft, this.disableButton); + this.on(document, events.mail.draftSaved, this.enableButton); + }); + } +}); diff --git a/web-ui/app/js/mail_view/ui/feedback_box.js b/web-ui/app/js/mail_view/ui/feedback_box.js new file mode 100644 index 00000000..eb079b5b --- /dev/null +++ b/web-ui/app/js/mail_view/ui/feedback_box.js @@ -0,0 +1,65 @@ +/* + * Copyright (c) 2015 ThoughtWorks, Inc. + * + * Pixelated is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Pixelated is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with Pixelated. If not, see <http://www.gnu.org/licenses/>. + */ + +define(['flight/lib/component', 'views/templates', 'page/events', 'features'], + function (defineComponent, templates, events, features) { + 'use strict'; + + return defineComponent(function () { + this.defaultAttrs({ + 'closeButton': '.close-mail-button', + 'submitButton': '#send-button', + 'textBox': '#text-box', + }); + + this.render = function () { + this.$node.html(templates.compose.feedback()); + }; + + this.openFeedbackBox = function() { + var stage = this.reset('feedback-box'); + this.attachTo(stage); + this.enableFloatlabel('input.floatlabel'); + this.enableFloatlabel('textarea.floatlabel'); + }; + + this.showNoMessageSelected = function() { + this.trigger(document, events.dispatchers.rightPane.openNoMessageSelected); + }; + + this.submitFeedback = function () { + var feedback = this.select('textBox').val(); + this.trigger(document, events.feedback.submit, { feedback: feedback }); + }; + + this.showSuccessMessage = function () { + this.trigger(document, events.ui.userAlerts.displayMessage, { message: 'Thanks for your feedback!' }); + }; + + this.after('initialize', function () { + if (features.isEnabled('feedback')) { + this.render(); + this.on(document, events.dispatchers.rightPane.openFeedbackBox, this.openFeedbackBox); + this.on(document, events.feedback.submitted, this.showNoMessageSelected); + this.on(document, events.feedback.submitted, this.showSuccessMessage); + this.on(this.select('closeButton'), 'click', this.showNoMessageSelected); + this.on(this.select('submitButton'), 'click', this.submitFeedback); + } + }); + + }); +}); diff --git a/web-ui/app/js/mail_view/ui/mail_view.js b/web-ui/app/js/mail_view/ui/mail_view.js index 71a67e5a..3c5b0ccc 100644 --- a/web-ui/app/js/mail_view/ui/mail_view.js +++ b/web-ui/app/js/mail_view/ui/mail_view.js @@ -25,11 +25,10 @@ define( 'mixins/with_hide_and_show', 'mixins/with_mail_tagging', 'page/events', - 'views/i18n', - 'features' + 'views/i18n' ], - function (defineComponent, templates, mailActions, viewHelpers, withHideAndShow, withMailTagging, events, i18n, features) { + function (defineComponent, templates, mailActions, viewHelpers, withHideAndShow, withMailTagging, events, i18n) { return defineComponent(mailView, mailActions, withHideAndShow, withMailTagging); @@ -50,15 +49,15 @@ define( var signed, encrypted; data.mail.security_casing = data.mail.security_casing || {}; - if(features.isEnabled('signatureStatus')) { - signed = this.checkSigned(data.mail); - } - if(features.isEnabled('encryptionStatus')) { - encrypted = this.checkEncrypted(data.mail); + signed = this.checkSigned(data.mail); + encrypted = this.checkEncrypted(data.mail); + + if(data.mail.mailbox === 'sent') { + encrypted = undefined; } var attachments = _.map(data.mail.attachments, function(a){ - return { 'encoding': a.headers['Content-Transfer-Encoding'], 'name': a.name, 'ident': a.ident }; + return { 'encoding': a.encoding, 'name': a.name, 'ident': a.ident }; }); this.$node.html(templates.mails.fullView({ @@ -69,7 +68,6 @@ define( tags: data.mail.tags, encryptionStatus: encrypted, signatureStatus: signed, - features: features, attachments: attachments })); @@ -104,16 +102,29 @@ define( var status = ['encrypted']; - if(_.any(mail.security_casing.locks, function (lock) { return lock.state === 'valid'; })) { status.push('encryption-valid'); } - else { status.push('encryption-error'); } + var hasAnyEncryptionInfo = _.any(mail.security_casing.locks, function (lock) { + return lock.state === 'valid'; + }); + + if(hasAnyEncryptionInfo) { + status.push('encryption-valid'); + } else { + status.push('encryption-error'); + } return status.join(' '); }; this.checkSigned = function(mail) { - if(_.isEmpty(mail.security_casing.imprints)) { return 'not-signed'; } + if(_.isEmpty(mail.security_casing.imprints)) { + return 'not-signed'; + } + + var hasNoSignatureInformation = _.any(mail.security_casing.imprints, function (imprint) { + return imprint.state === 'no_signature_information'; + }); - if(_.any(mail.security_casing.imprints, function(imprint) { return imprint.state === 'no_signature_information'; })) { + if(hasNoSignatureInformation) { return ''; } @@ -130,7 +141,6 @@ define( status.push('signature-not-trusted'); } - return status.join(' '); }; diff --git a/web-ui/app/js/main.js b/web-ui/app/js/main.js index 06eca8dc..5bb1165c 100644 --- a/web-ui/app/js/main.js +++ b/web-ui/app/js/main.js @@ -21,6 +21,7 @@ requirejs.config({ paths: { 'mail_list': 'js/mail_list', 'page': 'js/page', + 'feedback': 'js/feedback', 'flight': 'bower_components/flight', 'hbs': 'js/generated/hbs', 'helpers': 'js/helpers', @@ -35,9 +36,9 @@ requirejs.config({ 'mixins': 'js/mixins', 'search': 'js/search', 'foundation': 'js/foundation', + 'features': 'js/features/features', 'i18next': 'bower_components/i18next/i18next.amd', 'quoted-printable': 'bower_components/quoted-printable', - 'features': 'js/features/features', 'utf8': 'bower_components/utf8' } }); diff --git a/web-ui/app/js/mixins/with_mail_edit_base.js b/web-ui/app/js/mixins/with_mail_edit_base.js index 848fe026..b4a2b0c8 100644 --- a/web-ui/app/js/mixins/with_mail_edit_base.js +++ b/web-ui/app/js/mixins/with_mail_edit_base.js @@ -23,9 +23,10 @@ define( 'page/events', 'views/i18n', 'mail_view/ui/send_button', + 'mail_view/ui/draft_button', 'flight/lib/utils' ], - function(viewHelper, Recipients, DraftSaveStatus, events, i18n, SendButton, utils) { + function(viewHelper, Recipients, DraftSaveStatus, events, i18n, SendButton, DraftButton, utils) { 'use strict'; function withMailEditBase() { @@ -94,6 +95,7 @@ define( this.on(this.select('draftButton'), 'click', this.buildAndSaveDraft); this.on(this.select('trashButton'), 'click', this.trashMail); SendButton.attachTo(this.select('sendButton')); + DraftButton.attachTo(this.select('draftButton')); this.warnSendButtonOfRecipients(); }; @@ -203,6 +205,20 @@ define( this.trigger(document, events.ui.userAlerts.displayMessage, { message: 'Your message was sent!' }); }; + this.enableFloatlabel = function(element) { + var showClass = 'showfloatlabel'; + $(element).bind('keyup', function() { + var label = $(this).prev('label'); + if (this.value !== '') { + label.addClass(showClass); + $(this).addClass(showClass); + } else { + label.removeClass(showClass); + $(this).removeClass(showClass); + } + }); + }; + this.after('initialize', function () { this.on(document, events.dispatchers.rightPane.clear, this.teardown); this.on(document, events.ui.recipients.updated, this.recipientsUpdated); diff --git a/web-ui/app/js/page/default.js b/web-ui/app/js/page/default.js index 1571202e..1189d1ad 100644 --- a/web-ui/app/js/page/default.js +++ b/web-ui/app/js/page/default.js @@ -42,7 +42,11 @@ define( 'views/recipientListFormatter', 'flight/lib/logger', 'page/logout', - 'page/logout_shortcut' + 'page/logout_shortcut', + 'feedback/feedback_trigger', + 'mail_view/ui/feedback_box', + 'mail_view/data/feedback_sender', + 'page/version', ], function ( @@ -72,7 +76,11 @@ define( recipientListFormatter, withLogging, logout, - logoutShortcut) { + logoutShortcut, + feedback, + feedbackBox, + feedbackSender, + version) { 'use strict'; function initialize(path) { @@ -105,6 +113,10 @@ define( offCanvas.attachTo(document); logout.attachTo('#logout'); logoutShortcut.attachTo('#logout-shortcut'); + version.attachTo('.version'); + + feedback.attachTo('#feedback'); + feedbackSender.attachTo(document); } return initialize; diff --git a/web-ui/app/js/page/events.js b/web-ui/app/js/page/events.js index dfc2b852..cfc3d7db 100644 --- a/web-ui/app/js/page/events.js +++ b/web-ui/app/js/page/events.js @@ -100,6 +100,10 @@ define(function () { highlightResults: 'search:highlightResults', resetHighlight: 'search:resetHighlight' }, + feedback: { + submit: 'feedback:submit', + submitted: 'feedback:submitted' + }, mail: { here: 'mail:here', want: 'mail:want', @@ -169,6 +173,7 @@ define(function () { dispatchers: { rightPane: { openComposeBox: 'dispatchers:rightPane:openComposeBox', + openFeedbackBox: 'dispatchers:rightPane:openFeedbackBox', openNoMessageSelected: 'dispatchers:rightPane:openNoMessageSelected', openNoMessageSelectedWithoutPushState: 'dispatchers:rightPane:openNoMessageSelectedWithoutPushState', refreshMailList: 'dispatchers:rightPane:refreshMailList', diff --git a/web-ui/app/js/page/pane_contract_expand.js b/web-ui/app/js/page/pane_contract_expand.js index 153e38e5..aee8c44f 100644 --- a/web-ui/app/js/page/pane_contract_expand.js +++ b/web-ui/app/js/page/pane_contract_expand.js @@ -42,6 +42,7 @@ define(['flight/lib/component', 'page/events'], function (describeComponent, eve this.on(document, events.ui.mail.open, this.contractMiddlePaneExpandRightPane); this.on(document, events.dispatchers.rightPane.openComposeBox, this.contractMiddlePaneExpandRightPane); this.on(document, events.dispatchers.rightPane.openDraft, this.contractMiddlePaneExpandRightPane); + this.on(document, events.dispatchers.rightPane.openFeedbackBox, this.contractMiddlePaneExpandRightPane); this.on(document, events.dispatchers.rightPane.openNoMessageSelected, this.expandMiddlePaneContractRightPane); this.expandMiddlePaneContractRightPane(); }); diff --git a/web-ui/app/js/page/version.js b/web-ui/app/js/page/version.js new file mode 100644 index 00000000..e5299f52 --- /dev/null +++ b/web-ui/app/js/page/version.js @@ -0,0 +1,31 @@ +/* + * Copyright (c) 2015 ThoughtWorks, Inc. + * + * Pixelated is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * Pixelated is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with Pixelated. If not, see <http://www.gnu.org/licenses/>. + */ +define(['flight/lib/component', 'views/templates'], function (defineComponent, templates) { + 'use strict'; + + return defineComponent(function () { + + this.render = function () { + this.$node.html(templates.page.version()); + }; + + this.after('initialize', function () { + this.render(); + }); + + }); +}); diff --git a/web-ui/app/js/views/templates.js b/web-ui/app/js/views/templates.js index 5e991a20..61fb0486 100644 --- a/web-ui/app/js/views/templates.js +++ b/web-ui/app/js/views/templates.js @@ -25,7 +25,8 @@ define(['hbs/templates'], function (templates) { replySection: window.Pixelated['app/templates/compose/reply_section.hbs'], recipientInput: window.Pixelated['app/templates/compose/recipient_input.hbs'], fixedRecipient: window.Pixelated['app/templates/compose/fixed_recipient.hbs'], - recipients: window.Pixelated['app/templates/compose/recipients.hbs'] + recipients: window.Pixelated['app/templates/compose/recipients.hbs'], + feedback: window.Pixelated['app/templates/compose/feedback_box.hbs'] }, tags: { tagList: window.Pixelated['app/templates/tags/tag_list.hbs'], @@ -57,7 +58,11 @@ define(['hbs/templates'], function (templates) { }, page: { logout: window.Pixelated['app/templates/page/logout.hbs'], - logoutShortcut: window.Pixelated['app/templates/page/logout_shortcut.hbs'] + logoutShortcut: window.Pixelated['app/templates/page/logout_shortcut.hbs'], + version: window.Pixelated['app/templates/page/version.hbs'] + }, + feedback: { + feedback: window.Pixelated['app/templates/feedback/feedback_trigger.hbs'] } }; diff --git a/web-ui/app/locales/en/translation.json b/web-ui/app/locales/en/translation.json index a360cc02..5992216a 100644 --- a/web-ui/app/locales/en/translation.json +++ b/web-ui/app/locales/en/translation.json @@ -60,6 +60,8 @@ "Search results for:": "Search results for:", "Tags": "Tags", "Forward": "Forward", + "Submit Feedback": "Submit Feedback", + "feedback-placeholder": "Tell us what you like, didn't like, what is missing and generally what you think about Pixelated.", "tags": { "inbox": "Inbox", diff --git a/web-ui/app/locales/pt/translation.json b/web-ui/app/locales/pt/translation.json index 19b68c55..2885e12b 100644 --- a/web-ui/app/locales/pt/translation.json +++ b/web-ui/app/locales/pt/translation.json @@ -9,7 +9,7 @@ "Could not update mail tags": "Não foi possível atualizar as etiquetas do email", "Invalid tag name": "Nome de etiqueta inválido", "Could not delete email": "Não foi possível deletar o email", - "Could not fetch messages": "Não foi possível buscar as mensagems", + "Could not fetch messages": "Não foi possível buscar as mensagens", "sending-mail": "Enviando...", "tags": { diff --git a/web-ui/app/scss/_compose.scss b/web-ui/app/scss/_compose.scss index acff745d..6f3ae29e 100644 --- a/web-ui/app/scss/_compose.scss +++ b/web-ui/app/scss/_compose.scss @@ -23,29 +23,50 @@ } // COMPOSE PANE -#compose-box, #draft-box, #reply-box { +#compose-box, #draft-box, #reply-box, #feedback-box { + div.floatlabel { + position: relative; + } margin: 5px 0 50px 30px; padding: 0; .input-container { border-bottom: 1px solid #DDD; padding: 1px; } - label { + label, span { color: #AAA; padding: 0.5rem; cursor: text; display: inline-block; padding: 10px; } + label.floatlabel { + padding: 0.4rem !important; + position: absolute; + font-size: 0.6rem; + transition: all 0.1s linear; + opacity: 0; + font-weight: bold; + } + label.showfloatlabel { + color: #64BCD0 !important; + top: -0.3rem; + opacity: 1; + } input, textarea { margin: 0; border: none; + transition: all 0.1s linear; } - input { - &#subject { - font-size: 1.6875rem; - line-height: 1.4; - } + input.showfloatlabel, textarea.showfloatlabel { + padding-top: 1rem !important; + } + input#subject, #feedback-subject { + font-size: 1.6875rem; + line-height: 1.4; + } + #feedback-subject { + color: #333; } textarea { border-bottom: 2px solid #DDD; diff --git a/web-ui/app/scss/_read.scss b/web-ui/app/scss/_read.scss index d621f672..24de425a 100644 --- a/web-ui/app/scss/_read.scss +++ b/web-ui/app/scss/_read.scss @@ -13,7 +13,6 @@ padding: 0px 0; margin: 1px 0 0 0; .recipients { - border-bottom: 1px solid #DDD; padding-bottom: 5px; line-height: 1.5em; i { @@ -39,6 +38,10 @@ width:95%; flex-shrink:1; } + .headline-area { + clear: both; + border-top: 1px solid #DDD; + } } h3 { margin-bottom: 0; diff --git a/web-ui/app/scss/styles.scss b/web-ui/app/scss/styles.scss index 4f2a56ee..7b1a2ec3 100644 --- a/web-ui/app/scss/styles.scss +++ b/web-ui/app/scss/styles.scss @@ -390,17 +390,15 @@ section { } } - ul#logout { + ul#logout, ul#feedback { li { - color: $action_buttons; background-color: $navigation_background; padding: 5px 10px; position: relative; @include searching(4px, 19px, #333, 0.7em); &:hover { - background-color: $action_buttons; - color: $navigation_background; + color: $navigation_background; } div { @@ -417,6 +415,24 @@ section { } } + ul#logout li{ + color: $action_buttons; + &:hover { + background-color: $action_buttons; + } + } + + ul#feedback{ + margin-bottom: 0; + + li { + color: $light_orange; + &:hover { + background-color: $light_orange; + } + } + } + h3 { color: white; text-transform: uppercase; @@ -774,6 +790,11 @@ div.side-nav-bottom { position: fixed; bottom: 0; background-color: $navigation_background; + + .version { + padding-left: 55px; + padding-bottom: 3px; + } } @import "mascot.scss"; diff --git a/web-ui/app/templates/compose/compose_box.hbs b/web-ui/app/templates/compose/compose_box.hbs index d5501e69..2a1d27b0 100644 --- a/web-ui/app/templates/compose/compose_box.hbs +++ b/web-ui/app/templates/compose/compose_box.hbs @@ -1,8 +1,14 @@ <button class="close-mail-button"> <i class="fa fa-times"></i> </button> -<input type="text" id="subject" value="{{subject}}" placeholder="{{t 'Subject'}}" tabindex="1"/> -<textarea id="text-box" placeholder="{{t 'Body'}}" tabindex="2">{{body}}</textarea> +<div class="floatlabel"> + <label class="floatlabel" for="subject">Subject</label> + <input class="floatlabel" name="subject" type="text" id="subject" value="{{subject}}" placeholder="{{t 'Subject'}}" tabindex="1"/> +</div> +<div class="floatlabel"> + <label class="floatlabel" for="body">Body</label> + <textarea class="floatlabel" name="body" id="text-box" placeholder="{{t 'Body'}}" tabindex="2">{{body}}</textarea> +</div> {{> recipients }} diff --git a/web-ui/app/templates/compose/feedback_box.hbs b/web-ui/app/templates/compose/feedback_box.hbs new file mode 100644 index 00000000..ab5b3018 --- /dev/null +++ b/web-ui/app/templates/compose/feedback_box.hbs @@ -0,0 +1,15 @@ +<button class="close-mail-button"> + <i class="fa fa-times"></i> +</button> +<div class="floatlabel"> + <span id="feedback-subject">Feedback</span> +</div> +<div class="floatlabel"> + <label class="floatlabel" for="text-box">Body</label> + <textarea class="floatlabel" name="body" id="text-box" placeholder="{{t 'feedback-placeholder'}}" tabindex="2">{{body}}</textarea> +</div> + + +<div class="buttons-group columns"> + <button id="send-button" tabindex="6">{{t 'Submit Feedback'}} </button> +</div> diff --git a/web-ui/app/templates/feedback/feedback_trigger.hbs b/web-ui/app/templates/feedback/feedback_trigger.hbs new file mode 100644 index 00000000..7f3f8ef1 --- /dev/null +++ b/web-ui/app/templates/feedback/feedback_trigger.hbs @@ -0,0 +1,8 @@ +<ul id="feedback"> + <a title="Feedback" href="#"> + <li> + <div class="fa fa-exclamation-circle"></div> + <i class="shortcut-label"></i> Feedback + </li> + </a> +</ul> diff --git a/web-ui/app/templates/mails/full_view.hbs b/web-ui/app/templates/mails/full_view.hbs index 77994860..0383b821 100644 --- a/web-ui/app/templates/mails/full_view.hbs +++ b/web-ui/app/templates/mails/full_view.hbs @@ -36,7 +36,7 @@ <div class="recipients column large-2 text-right"> <span class="received-date">{{ formatDate header.date }}</span> </div> - <div> + <div class="headline-area"> <h3 class="subjectArea column large-10 no-padding"> <span class="subject">{{ header.subject }}</span> diff --git a/web-ui/app/templates/page/version.hbs b/web-ui/app/templates/page/version.hbs new file mode 100644 index 00000000..40804ff3 --- /dev/null +++ b/web-ui/app/templates/page/version.hbs @@ -0,0 +1 @@ +version: UNKNOWN_VERSION
\ No newline at end of file diff --git a/web-ui/bower.json b/web-ui/bower.json index 3b51fefa..9d7a50c5 100644 --- a/web-ui/bower.json +++ b/web-ui/bower.json @@ -12,7 +12,8 @@ "quoted-printable": "0.2.1", "typeahead.js": "~0.10.5", "jasmine-flight": "~3.0.0", - "utf8": "~2.0.0" + "utf8": "~2.0.0", + "modernizr": "~2.8.3" }, "devDependencies": { "handlebars": "2.0.0", diff --git a/web-ui/config/add_git_version.sh b/web-ui/config/add_git_version.sh new file mode 100755 index 00000000..2732abed --- /dev/null +++ b/web-ui/config/add_git_version.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash + +TEMPLATE_FILE="app/js/generated/hbs/templates.js" + +COMMITISH=$(git rev-parse --short HEAD) + +perl -pi -e "s/UNKNOWN_VERSION/$COMMITISH/" $TEMPLATE_FILE + +if [ ! -f "$TEMPLATE_FILE" ] ; then + echo "file $TEMPLATE_FILE not found" 1>&2 + exit 1 +fi + +exit 0 diff --git a/web-ui/config/buildoptions.js b/web-ui/config/buildoptions.js index 400dffb1..63c91653 100644 --- a/web-ui/config/buildoptions.js +++ b/web-ui/config/buildoptions.js @@ -25,4 +25,3 @@ include: ['js/main'], name: 'bower_components/almond/almond' }) - diff --git a/web-ui/config/package.sh b/web-ui/config/package.sh index f7d7f3aa..2ba5e1a0 100644 --- a/web-ui/config/package.sh +++ b/web-ui/config/package.sh @@ -28,6 +28,7 @@ mkdir -p dist ./go handlebars ./go imagemin ./go minify_html +./go add_git_version ./go buildmain @@ -47,7 +48,7 @@ app/bower_components/typeahead.js/dist/typeahead.bundle.min.js \ app/bower_components/foundation/js/foundation.js \ app/bower_components/foundation/js/foundation/foundation.reveal.js \ app/bower_components/foundation/js/foundation/foundation.offcanvas.js \ +app/js/foundation/initialize_foundation.js \ .tmp/app.concatenated.js > dist/app.js node_modules/.bin/minify dist/app.js > dist/app.min.js rm dist/app.js - diff --git a/web-ui/package.json b/web-ui/package.json index 95d3ae72..f7c46a47 100644 --- a/web-ui/package.json +++ b/web-ui/package.json @@ -23,7 +23,7 @@ "watch": "0.14.0" }, "scripts": { - "test": "npm run build && node_modules/karma/bin/karma start --single-run --browsers PhantomJS $GRUNT_OPTS", + "test": "npm run jshint --silent && npm run build && node_modules/karma/bin/karma start --single-run --browsers PhantomJS $GRUNT_OPTS", "debug": "npm run build && node_modules/karma/bin/karma start --browsers Chrome $GRUNT_OPTS", "watch": "npm run compass-watch & npm run handlebars-watch", "watch-test": "node_modules/karma/bin/karma start", @@ -31,12 +31,13 @@ "handlebars-watch": "node_modules/.bin/watch 'npm run handlebars' app/templates", "compass": "compass compile", "compass-watch": "compass watch", - "build": "npm run clean && npm run handlebars && npm run compass", + "build": "npm run clean && npm run handlebars && npm run add_git_version && npm run compass", "jshint": "jshint --config=.jshintrc app test", "clean": "rm -rf .tmp/ dist/**/* app/js/generated/hbs/* app/css/*", "buildmain": "node_modules/requirejs/bin/r.js -o config/buildoptions.js", "package": "/bin/bash config/package.sh", "imagemin": "node config/imagemin.js", - "minify_html": "node_modules/.bin/html-minifier app/index.html --collapse-whitespace | sed 's|<!--usemin_start-->.*<!--usemin_end-->|<script src=\"assets/app.min.js\" type=\"text/javascript\"></script>|' > dist/index.html" + "minify_html": "node_modules/.bin/html-minifier app/index.html --collapse-whitespace | sed 's|<!--usemin_start-->.*<!--usemin_end-->|<script src=\"assets/app.min.js\" type=\"text/javascript\"></script>|' > dist/index.html", + "add_git_version": "/bin/bash config/add_git_version.sh" } } diff --git a/web-ui/test/spec/dispatchers/right_pane_dispatcher.spec.js b/web-ui/test/spec/dispatchers/right_pane_dispatcher.spec.js index 4187610e..9df1d557 100644 --- a/web-ui/test/spec/dispatchers/right_pane_dispatcher.spec.js +++ b/web-ui/test/spec/dispatchers/right_pane_dispatcher.spec.js @@ -53,6 +53,14 @@ describeComponent('dispatchers/right_pane_dispatcher', function () { expect(pushStateEvent).toHaveBeenTriggeredOnAndWith(document, jasmine.objectContaining({ isDisplayNoMessageSelected: true })); }); + it('listens to open feedback event and open feedback box', function () { + var feedbackBox = require('mail_view/ui/feedback_box'); + spyOn(feedbackBox, 'attachTo'); + + this.component.trigger(document, Pixelated.events.dispatchers.rightPane.openFeedbackBox); + + expect(feedbackBox.attachTo).toHaveBeenCalled(); + }); }); @@ -66,6 +74,7 @@ describeComponent('dispatchers/right_pane_dispatcher', function () { }); }); + describe('on initialization', function () { var noMessageSelectedPane; diff --git a/web-ui/test/spec/feedback/feedback_trigger.spec.js b/web-ui/test/spec/feedback/feedback_trigger.spec.js new file mode 100644 index 00000000..3860fc79 --- /dev/null +++ b/web-ui/test/spec/feedback/feedback_trigger.spec.js @@ -0,0 +1,38 @@ +describeComponent('feedback/feedback_trigger', function () { + 'use strict'; + + describe('Feedback link', function () { + var features; + + beforeEach(function() { + features = require('features'); + }); + + it('Should provide feedback link if logout is enabled', function () { + spyOn(features, 'isEnabled').and.returnValue(true); + this.setupComponent('<nav id="feedback"></nav>', {}); + + var feedback_link = this.component.$node.find('a')[0]; + expect(feedback_link).toExist(); + }); + + it('Should not provide feedback link if disabled', function() { + spyOn(features, 'isEnabled').and.returnValue(false); + this.setupComponent('<nav id="feedback"></nav>', {}); + + var feedback_link = this.component.$node.find('a')[0]; + expect(feedback_link).not.toExist(); + }); + + it('Should trigger ui:feedback:open event on click', function () { + + this.setupComponent('<nav id="feedback"></nav>', {}); + var spy = spyOnEvent(document, Pixelated.events.dispatchers.rightPane.openFeedbackBox); + + this.$node.find('a').click(); + expect(spy).toHaveBeenTriggeredOn(document); + }); + + }); +}); + diff --git a/web-ui/test/spec/mail_view/data/feedback_sender.spec.js b/web-ui/test/spec/mail_view/data/feedback_sender.spec.js new file mode 100644 index 00000000..30952c3f --- /dev/null +++ b/web-ui/test/spec/mail_view/data/feedback_sender.spec.js @@ -0,0 +1,27 @@ +describeComponent('mail_view/data/feedback_sender', function () { + 'use strict'; + + + beforeEach(function () { + this.setupComponent(); + }); + + it('sends feedback with a POST to the server', function() { + var data = {feedback: 'Pixelated is awesome!'}; + var feedbackSubmittedEventSpy = spyOnEvent(document, Pixelated.events.feedback.submitted); + var deferred = $.Deferred(); + + spyOn($, 'ajax').and.returnValue(deferred); + + this.component.trigger(document, Pixelated.events.feedback.submit, data); + + deferred.resolve(); + + expect(feedbackSubmittedEventSpy).toHaveBeenTriggeredOn(document); + + expect($.ajax.calls.mostRecent().args[0]).toEqual('/feedback'); + expect($.ajax.calls.mostRecent().args[1].type).toEqual('POST'); + expect(JSON.parse($.ajax.calls.mostRecent().args[1].data)).toEqual(data); + }); + +}); diff --git a/web-ui/test/spec/mail_view/ui/compose_box.spec.js b/web-ui/test/spec/mail_view/ui/compose_box.spec.js index 414bc022..8e07481a 100644 --- a/web-ui/test/spec/mail_view/ui/compose_box.spec.js +++ b/web-ui/test/spec/mail_view/ui/compose_box.spec.js @@ -99,6 +99,15 @@ describeComponent('mail_view/ui/compose_box', function () { expect(openNoMessageSelectedPaneEvent).not.toHaveBeenTriggeredOn(document); }); + + it('should call the enableFloatlabel method when events.mail.here is trigged', function() { + spyOn(this.component, 'enableFloatlabel'); + + this.component.renderComposeBox(); + + expect(this.component.enableFloatlabel).toHaveBeenCalledWith('input.floatlabel'); + expect(this.component.enableFloatlabel).toHaveBeenCalledWith('textarea.floatlabel'); + }); }); describe('close button behavior', function() { @@ -126,4 +135,60 @@ describeComponent('mail_view/ui/compose_box', function () { })); }); }); + + describe('subject label', function() { + var input; + var label; + + beforeEach(function() { + input = $(this.component.$node).find('input'); + label = input.prev('label'); + + this.component.enableFloatlabel(input); + }); + + it('should show the subject label after the user starts typing', function() { + input.val('test'); + input.trigger('keyup'); + + expect(input.hasClass('showfloatlabel')).toEqual(true); + expect(label.hasClass('showfloatlabel')).toEqual(true); + }); + + it('should not show the subject label if the field is empty', function() { + input.val(''); + input.trigger('keyup'); + + expect(input.hasClass('showfloatlabel')).toEqual(false); + expect(label.hasClass('showfloatlabel')).toEqual(false); + }); + }); + + describe('body label', function() { + var textarea; + var label; + + beforeEach(function() { + textarea = $(this.component.$node).find('textarea'); + label = textarea.prev('label'); + + this.component.enableFloatlabel(textarea); + }); + + it('should show the subject label after the user starts typing', function() { + textarea.text('test'); + textarea.trigger('keyup'); + + expect(textarea.hasClass('showfloatlabel')).toEqual(true); + expect(label.hasClass('showfloatlabel')).toEqual(true); + }); + + it('should not show the subject label if the field is empty', function() { + textarea.text(''); + textarea.trigger('keyup'); + + expect(textarea.hasClass('showfloatlabel')).toEqual(false); + expect(label.hasClass('showfloatlabel')).toEqual(false); + }); + }); }); diff --git a/web-ui/test/spec/mail_view/ui/draft_box.spec.js b/web-ui/test/spec/mail_view/ui/draft_box.spec.js index 0113ca01..be3b4039 100644 --- a/web-ui/test/spec/mail_view/ui/draft_box.spec.js +++ b/web-ui/test/spec/mail_view/ui/draft_box.spec.js @@ -24,11 +24,8 @@ describeComponent('mail_view/ui/draft_box', function () { }); describe('after initialize', function () { - beforeEach(function () { - this.setupComponent({mailIdent: '1'}); - }); - it('renders the compose box when mail is received', function () { + this.setupComponent({mailIdent: '1'}); var templates = require('views/templates'); spyOn(this.component, 'render'); @@ -65,4 +62,14 @@ describeComponent('mail_view/ui/draft_box', function () { expect(openNoMessageSelectedEvent).toHaveBeenTriggeredOn(document); }); + it('should call the enableFloatlabel method when events.mail.here is trigged', function() { + this.setupComponent({mailIdent: mail.ident}); + spyOn(this.component, 'enableFloatlabel'); + + this.component.trigger(this.component, Pixelated.events.mail.here, { mail: mail }); + + expect(this.component.enableFloatlabel).toHaveBeenCalledWith('input.floatlabel'); + expect(this.component.enableFloatlabel).toHaveBeenCalledWith('textarea.floatlabel'); + }); + }); diff --git a/web-ui/test/spec/mail_view/ui/draft_button.spec.js b/web-ui/test/spec/mail_view/ui/draft_button.spec.js new file mode 100644 index 00000000..de607507 --- /dev/null +++ b/web-ui/test/spec/mail_view/ui/draft_button.spec.js @@ -0,0 +1,40 @@ +/* global Pixelated */ + +describeComponent('mail_view/ui/draft_button', function(){ + 'use strict'; + + describe('draft save button', function(){ + beforeEach(function(){ + this.setupComponent('<button></button>'); + }); + + describe('after initialize', function(){ + it('should be disabled', function(){ + expect(this.$node).toBeDisabled(); + }); + }); + + describe('when enabled', function(){ + beforeEach(function(){ + this.$node.prop('disabled', false); + }); + + it('should be disabled when saving draft message', function(){ + $(document).trigger(Pixelated.events.mail.saveDraft, {}); + expect(this.$node).toBeDisabled(); + }); + }); + + describe('when disabled', function(){ + beforeEach(function(){ + this.$node.prop('disabled', true); + }); + + it('should be enabled when draft message has been saved', function(){ + $(document).trigger(Pixelated.events.mail.draftSaved, {}); + expect(this.$node).not.toBeDisabled(); + }); + }); + + }); +}); diff --git a/web-ui/test/spec/mail_view/ui/feedback_box.spec.js b/web-ui/test/spec/mail_view/ui/feedback_box.spec.js new file mode 100644 index 00000000..4702672c --- /dev/null +++ b/web-ui/test/spec/mail_view/ui/feedback_box.spec.js @@ -0,0 +1,45 @@ +describeComponent('mail_view/ui/feedback_box', function () { + 'use strict'; + beforeEach(function () { + Pixelated.mockBloodhound(); + this.setupComponent('<div></div>'); + }); + + + describe('close button behavior', function() { + + it('should fire Show no message selected if the close button is clicked', function() { + var spy = spyOnEvent(document, Pixelated.events.dispatchers.rightPane.openNoMessageSelected); + this.component.select('closeButton').click(); + expect(spy).toHaveBeenTriggeredOn(document); + }); + + }); + + describe('when submit feedback', function () { + + it('should fire submit feedback event', function () { + var spy = spyOnEvent(document, Pixelated.events.feedback.submit); + + this.component.select('textBox').val('Pixelated is Awesome!'); + this.component.select('submitButton').click(); + expect(spy).toHaveBeenTriggeredOnAndWith(document, {feedback: 'Pixelated is Awesome!'}); + }); + + it('should close feedback box after submit', function() { + var spy = spyOnEvent(document, Pixelated.events.dispatchers.rightPane.openNoMessageSelected); + + this.component.trigger(document, Pixelated.events.feedback.submitted); + expect(spy).toHaveBeenTriggeredOn(document); + }); + + it('should shows success message after submit', function () { + var spy = spyOnEvent(document, Pixelated.events.ui.userAlerts.displayMessage); + + this.component.trigger(document, Pixelated.events.feedback.submitted); + expect(spy).toHaveBeenTriggeredOnAndWith(document, {message: 'Thanks for your feedback!'}); + }); + + }); + +}); diff --git a/web-ui/test/spec/mail_view/ui/mail_view.spec.js b/web-ui/test/spec/mail_view/ui/mail_view.spec.js index deb7fb88..fe763919 100644 --- a/web-ui/test/spec/mail_view/ui/mail_view.spec.js +++ b/web-ui/test/spec/mail_view/ui/mail_view.spec.js @@ -265,6 +265,16 @@ describeComponent('mail_view/ui/mail_view', function () { expect(openNoMessageSelectedEvent).toHaveBeenTriggeredOn(document); }); + it('shows a download link for attachments', function() { + var withAttachments = {mail: Pixelated.testData().parsedMail.withAttachments}; + + this.component.displayMail({}, withAttachments); + + var attachmentLink = $(this.component.$node.find('.attachmentsArea li').html()); + var expectedLink = '/attachment/912ec803b2ce49e4a541068d495ab570?encoding=base64&filename=filename.txt'; + expect(attachmentLink.attr('href')) .toBe(expectedLink); + }); + function creatingEvent(event, keyCode) { var e = $.Event(event); e.which = keyCode; diff --git a/web-ui/test/spec/page/pane_contract_expand.spec.js b/web-ui/test/spec/page/pane_contract_expand.spec.js index 4f4ff49b..6c7ebcfc 100644 --- a/web-ui/test/spec/page/pane_contract_expand.spec.js +++ b/web-ui/test/spec/page/pane_contract_expand.spec.js @@ -44,6 +44,13 @@ describeComponent('page/pane_contract_expand', function () { expect($('#right-pane').attr('class')).toEqual(this.component.attr.RIGHT_PANE_EXPAND_CLASSES); }); + it('contracts middle pane and expands right pane on open draft', function () { + $(document).trigger(Pixelated.events.dispatchers.rightPane.openFeedbackBox); + + expect($('#middle-pane-container').attr('class')).toEqual(this.component.attr.MIDDLE_PANE_CONTRACT_CLASSES); + expect($('#right-pane').attr('class')).toEqual(this.component.attr.RIGHT_PANE_EXPAND_CLASSES); + }); + it('expands middle pane and contracts right pane on event on open no message selected pane', function () { $(document).trigger(Pixelated.events.dispatchers.rightPane.openNoMessageSelected); diff --git a/web-ui/test/test-main.js b/web-ui/test/test-main.js index 9cd5b12a..42ff1ba2 100644 --- a/web-ui/test/test-main.js +++ b/web-ui/test/test-main.js @@ -16,6 +16,7 @@ requirejs.config({ 'flight': 'app/bower_components/flight', 'views': 'app/js/views', 'helpers': 'app/js/helpers', + 'feedback': 'app/js/feedback', 'tags': 'app/js/tags', 'mail_list': 'app/js/mail_list', 'mail_list_actions': 'app/js/mail_list_actions', diff --git a/web-ui/test/test_data.js b/web-ui/test/test_data.js index 446fd7c6..62492bbe 100644 --- a/web-ui/test/test_data.js +++ b/web-ui/test/test_data.js @@ -202,6 +202,32 @@ define(function() { getMailPartByContentType: function () { return; } }; + var withAttachments = { + header: { + to:'jed_waelchi@cummerata.info', + from:'laurel@hamill.info', + subject:'Velit aut tempora animi ut nulla esse.', + date:'2014-06-04T14:41:13-03:00' + }, + ident:1, + tags:['textplain'], + mailbox: ['inbox'], + status:[], + textPlainBody: 'Hello Everyone', + isSentMail: function() { return false; }, + isDraftMail: function() { return false; }, + replyToAddress: function() { return { to: ['laurel@hamill.info'], cc: [] }; }, + replyToAllAddress: function() { return { to: ['laurel@hamill.info'], cc: [] }; }, + isMailMultipartAlternative: function() { return false; }, + availableBodyPartsContentType: function() { return []; }, + getMailPartByContentType: function() { return; }, + attachments: [{ + ident: '912ec803b2ce49e4a541068d495ab570', + name: 'filename.txt', + encoding: 'base64' + }] + }; + var testData = { rawMail: { mail: rawMail, @@ -215,7 +241,8 @@ define(function() { parsedMail: { simpleTextPlain: simpleTextPlainMail, html: htmlNoEncodingMail, - draft: draftMail + draft: draftMail, + withAttachments: withAttachments } }; |