From 5a3a2012bb8982ad0884ed659e61e969345e6fde Mon Sep 17 00:00:00 2001 From: "Kali Kaneko (leap communications)" Date: Mon, 29 Aug 2016 23:10:17 -0400 Subject: [pkg] move mail source to leap.bitmask.mail --- docs/bitmask-mail-index.rst | 96 ++++++++++++++++++++++ docs/mail-hacking.rst | 195 ++++++++++++++++++++++++++++++++++++++++++++ docs/mail-journey.rst | 40 +++++++++ 3 files changed, 331 insertions(+) create mode 100644 docs/bitmask-mail-index.rst create mode 100644 docs/mail-hacking.rst create mode 100644 docs/mail-journey.rst (limited to 'docs') diff --git a/docs/bitmask-mail-index.rst b/docs/bitmask-mail-index.rst new file mode 100644 index 0000000..a2133f4 --- /dev/null +++ b/docs/bitmask-mail-index.rst @@ -0,0 +1,96 @@ +.. leap.mail documentation master file, created by + sphinx-quickstart on Mon Aug 25 19:19:48 2014. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +leap.mail +========= + +*decentralized and secure mail delivery and synchronization* + +This is the documentation for the ``leap.mail`` module. It is a `twisted`_ +package that allows to receive, process, send and access existing messages using +the `LEAP`_ platform. + +One way to use this library is to let it launch two standard mail services, +``smtp`` and ``imap``, that run as local proxies and interact with a remote +``LEAP`` provider that offers *a soledad syncronization endpoint* and receives +the outgoing email. This is what `Bitmask`_ client does. + +From the release 0.4.0 on, it's also possible to use a protocol-agnostic email +public API, so that third party mail clients can manipulate the data layer. This +is what the awesome MUA in the `Pixelated`_ project is using. + +.. _`twisted`: https://twistedmatrix.com/trac/ +.. _`LEAP`: https://leap.se/en/docs +.. _`Bitmask`: https://bitmask.net/en/features#email +.. _`Pixelated`: https://pixelated-project.org/ + +How does this all work? +----------------------- + +All the underlying data storage and sync is handled by a library called +`soledad`_, which handles encryption, storage and sync. Based on `u1db`_, +documents are stored locally as local ``sqlcipher`` tables, and syncs against +the soledad sync service in the provider. + +OpenPGP key generation and keyring management are handled by another leap +python library: `keymanager`_. + +See :ref:`the life cycle of a leap email ` for an overview of the life cycle +of an email through ``LEAP`` providers. + +.. _`Soledad`: https://leap.se/en/docs/design/soledad +.. _`u1db`: https://en.wikipedia.org/wiki/U1DB +.. _`keymanager`: https://github.com/leapcode/keymanager/ + + +Data model +---------- +.. TODO clear document types documentation. + +The data model at the present moment consists of several *document types* that split email into +different documents that are stored in ``Soledad``. The idea behind this is to +keep clear the separation between *mutable* and *inmutable* parts, and still being able to +reconstruct arbitrarily nested email structures easily. + +Documentation index +=================== + +.. +.. Contents: +.. toctree:: + :maxdepth: 2 + + hacking + +.. intro +.. tutorial + + +API documentation +----------------- + +If you were looking for the documentation of the ``leap.mail`` module, you will +find it here. + +Of special interest is the `public mail api`_, which should remain relatively +stable across the next few releases. + +.. _`public mail api`: api/mail.html#module-mail + + +.. toctree:: + :maxdepth: 2 + + api/leap.mail + + + +Indices and tables +================== + +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` + diff --git a/docs/mail-hacking.rst b/docs/mail-hacking.rst new file mode 100644 index 0000000..6c49c21 --- /dev/null +++ b/docs/mail-hacking.rst @@ -0,0 +1,195 @@ +.. _hacking: + +======== +Hacking +======== + +Some hints oriented to `leap.mail` hackers. These notes are mostly related to +the imap server, although they probably will be useful for other pieces too. + +Don't panic! Just manhole into it +================================= + +If you want to inspect the objects living in your application memory, in +realtime, you can manhole into it. + +First of all, check that the modules ``PyCrypto`` and ``pyasn1`` are installed +into your system, they are needed for it to work. + +You just have to pass the ``LEAP_MAIL_MANHOLE=1`` enviroment variable while +launching the client:: + + LEAP_MAIL_MANHOLE=1 bitmask --debug + +And then you can ssh into your application! (password is "leap"):: + + ssh boss@localhost -p 2222 + +Did I mention how *awesome* twisted is?? ``:)`` + + +Profiling +========= +If using ``twistd`` to launch the server, you can use twisted profiling +capabities:: + + LEAP_MAIL_CONFIG=~/.leapmailrc twistd --profile=/tmp/mail-profiling -n -y imap-server.tac + +``--profiler`` option allows you to select different profilers (default is +"hotshot"). + +You can also do profiling when using the ``bitmask`` client. Enable the +``LEAP_PROFILE_IMAPCMD`` environment flag to get profiling of certain IMAP +commands:: + + LEAP_PROFILE_IMAPCMD=1 bitmask --debug + +Offline mode +============ + +The client has an ``--offline`` flag that will make the Mail services (imap, +currently) not try to sync with remote replicas. Very useful during development, +although you need to login with the remote server at least once before being +able to use it. + +Mutt config +=========== + +You cannot live without mutt? You're lucky! Use the following minimal config +with the imap service:: + + set folder="imap://user@provider@localhost:1984" + set spoolfile="imap://user@provider@localhost:1984/INBOX" + set ssl_starttls = no + set ssl_force_tls = no + set imap_pass=MAHSIKRET + + +Running the service with twistd +=============================== + +In order to run the mail service (currently, the imap server only), you will +need a config with this info:: + + [leap_mail] + userid = "user@provider" + uuid = "deadbeefdeadabad" + passwd = "foobar" # Optional + +In the ``LEAP_MAIL_CONFIG`` enviroment variable. If you do not specify a password +parameter, you'll be prompted for it. + +In order to get the user uid (uuid), look into the +``~/.config/leap/leap-backend.conf`` file after you have logged in into your +provider at least once. + +Run the twisted service:: + + LEAP_MAIL_CONFIG=~/.leapmailrc twistd -n -y imap-server.tac + +Now you can telnet into your local IMAP server and read your mail like a real +programmer™:: + + % telnet localhost 1984 + Trying 127.0.0.1... + Connected to localhost. + Escape character is '^]'. + * OK [CAPABILITY IMAP4rev1 LITERAL+ IDLE NAMESPACE] Twisted IMAP4rev1 Ready + tag LOGIN me@myprovider.net mahsikret + tag OK LOGIN succeeded + tag SELECT Inbox + * 2 EXISTS + * 1 RECENT + * FLAGS (\Seen \Answered \Flagged \Deleted \Draft \Recent List) + * OK [UIDVALIDITY 1410453885932] UIDs valid + tag OK [READ-WRITE] SELECT successful + ^] + telnet> Connection closed. + + +Although you probably prefer to use ``offlineimap`` for tests:: + + offlineimap -c LEAPofflineimapRC-tests + + +Minimal offlineimap configuration +--------------------------------- + +You can use this as a sample offlineimap config file:: + + [general] + accounts = leap-local + + [Account leap-local] + localrepository = LocalLeap + remoterepository = RemoteLeap + + [Repository LocalLeap] + type = Maildir + localfolders = ~/LEAPMail/Mail + + [Repository RemoteLeap] + type = IMAP + ssl = no + remotehost = localhost + remoteport = 1984 + remoteuser = user + remotepass = pass + +Testing utilities +----------------- +There are a bunch of utilities to test IMAP delivery in ``imap/tests`` folder. +If looking for a quick way of inspecting mailboxes, have a look at ``getmail``:: + + ./getmail me@testprovider.net mahsikret + 1. Drafts + 2. INBOX + 3. Trash + Which mailbox? [1] 2 + 1 Subject: this is the time of the revolution + 2 Subject: ignore me + + Which message? [1] (Q quits) 1 + 1 X-Leap-Provenance: Thu, 11 Sep 2014 16:52:11 -0000; pubkey="C1F8DE10BD151F99" + Received: from mx1.testprovider.net(mx1.testprovider.net [198.197.196.195]) + (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) + (Client CN "*.foobar.net", Issuer "Gandi Standard SSL CA" (not verified)) + by blackhole (Postfix) with ESMTPS id DEADBEEF + for ; Thu, 11 Sep 2014 16:52:10 +0000 (UTC) + Delivered-To: 926d4915cfd42b6d96d38660c04613af@testprovider.net + Message-Id: <20140911165205.GB8054@samsara> + From: Kali + + (snip) + +IMAP Message Rendering Regressions +---------------------------------- + +For testing the IMAP server implementation, there is a litte regressions script +that needs some manual work from your side. + +First of all, you need an already initialized account. Which for now basically +means you have created a new account with a provider that offers the Encrypted +Mail Service, using the Bitmask Client wizard. Then you need to log in with that +account, and let it generate the secrets and sync with the remote for a first +time. After this you can run the twistd server locally and offline. + +From the ``leap.mail.imap.tests`` folder, and with an already initialized server +running:: + + ./regressions_mime_struct user@provider pass path_to_samples/ + +You can find several message samples in the ``leap/mail/tests`` folder. + + +Debugging IMAP commands +======================= + +Use ``ngrep`` to obtain logs of the commands:: + + sudo ngrep -d lo -W byline port 1984 + +To get verbose output from thunderbird/icedove, set the following environment +variable:: + + NSPR_LOG_MODULES="imap:5" icedove diff --git a/docs/mail-journey.rst b/docs/mail-journey.rst new file mode 100644 index 0000000..7e64f18 --- /dev/null +++ b/docs/mail-journey.rst @@ -0,0 +1,40 @@ +.. _mail_journey: + +The life cycle of a LEAP Email +============================== +The following are just some notes to facilitate the understanding of the +leap.mail internals to developers and collaborators. + +Server-side: receiving mail from the outside world +-------------------------------------------------- + +1. the mx server receives an email, it gets through all the postfix validation and it's written into disk +2. ``leap_mx`` gets that write in disk notification and encrypts the incoming mail to its intended recipient's pubkey +3. that encrypted blob gets written into couchdb in a way soledad will catch it in the next sync + + +Client-side: fetching and processing incoming email +--------------------------------------------------- +you have an imap and an smtp local server. For IMAP: + +1. soledad syncs +2. **fetch** module sees if there's new encrypted mail for the current user from leap_mx +3. if there is, the mail is decrypted using the user private key, and splitted + into several parts according to the internal mail data model (separating + mutable and inmutable email parts). Those documents it encrypts it properly + like other soledad documents and deletes the pubkey encrypted doc +4. desktop client is notified that there are N new mails +5. when a MUA connects to the **imap** local server, the different documents are glued + together and presented as response to the different imap commands. + + +Client side: sending email +-------------------------- + +1. you write an email to a recipient and hit send +2. the **smtp** local server gets that mail, it checks from whom it is and to whom it is for +3. it signs the mail with the ``From:``'s privkey +4. it retrieves ``To:``'s pubkey with the keymanager and if it finds it encrypts the mail to him/her +5. if it didn't find it and you don't have your client configured to "always encrypt", it goes out with just the signature +6. else, it fails out complaining about this conflict +7. that mail gets relayed to the provider's **smtp** server -- cgit v1.2.3