[docs] revamp bitmask dev docs

3 files changed, 307 insertions, 0 deletions

diff --git a/docs/mail/hacking.rst b/docs/mail/hacking.rst
new file mode 100644
index 00000000..27087e52
--- /dev/null
+++ b/docs/mail/hacking.rst
@@ -0,0 +1,173 @@
+.. _hacking:
+
+Hacking  on Bitmask Mail
+========================
+
+Some hints oriented to `leap.mail` hackers. These notes are mostly related to
+the imap server, although they probably will be useful for other pieces too.
+
+Don't panic! Just manhole into it
+---------------------------------
+
+
+If you want to inspect the objects living in your application memory, in
+realtime, you can manhole into it.
+
+First of all, check that the modules ``PyCrypto`` and ``pyasn1`` are installed
+into your system, they are needed for it to work.
+
+You just have to pass the ``LEAP_MAIL_MANHOLE=1`` enviroment variable while
+launching the client::
+
+  LEAP_MAIL_MANHOLE=1 bitmask --debug
+
+And then you can ssh into your application! (password is "leap")::
+
+  ssh boss@localhost -p 2222
+
+Did I mention how *awesome* twisted is?? ``:)``
+
+
+Profiling
+----------
+
+If using ``twistd`` to launch the server, you can use twisted profiling
+capabities::
+
+  LEAP_MAIL_CONFIG=~/.leapmailrc twistd --profile=/tmp/mail-profiling -n -y imap-server.tac
+
+``--profiler`` option allows you to select different profilers (default is
+"hotshot").
+
+You can also do profiling when using the ``bitmask`` client. Enable the
+``LEAP_PROFILE_IMAPCMD`` environment flag to get profiling of certain IMAP
+commands::
+
+ LEAP_PROFILE_IMAPCMD=1 bitmask --debug
+
+
+Mutt config
+------------
+
+You cannot live without mutt? You're lucky! Use the following minimal config
+with the imap service::
+
+ set folder="imap://user@provider@localhost:1984"
+ set spoolfile="imap://user@provider@localhost:1984/INBOX"
+ set ssl_starttls = no
+ set ssl_force_tls = no
+ set imap_pass=MAHSIKRET
+
+
+
+Debugging IMAP
+------------------------------
+After IMAP service is running, you can telnet into your local IMAP server and read your mail like a real programmer™::
+
+  % telnet localhost 1984
+  Trying 127.0.0.1...
+  Connected to localhost.
+  Escape character is '^]'.
+  * OK [CAPABILITY IMAP4rev1 LITERAL+ IDLE NAMESPACE] Twisted IMAP4rev1 Ready
+  tag LOGIN me@myprovider.net mahsikret
+  tag OK LOGIN succeeded
+  tag SELECT Inbox
+  * 2 EXISTS
+  * 1 RECENT
+  * FLAGS (\Seen \Answered \Flagged \Deleted \Draft \Recent List)
+  * OK [UIDVALIDITY 1410453885932] UIDs valid
+  tag OK [READ-WRITE] SELECT successful
+  ^]
+  telnet> Connection closed.
+
+
+Although you probably prefer to use ``offlineimap`` for tests:: 
+
+  offlineimap -c LEAPofflineimapRC-tests
+
+
+Use ``ngrep`` to obtain live logs of the commands and responses::
+
+  sudo ngrep -d lo -W byline port 1984
+
+
+Thunderbird
+---------------------------
+
+To get verbose output from thunderbird/icedove, set the following environment
+variable::
+
+  NSPR_LOG_MODULES="imap:5" icedove
+
+
+Minimal offlineimap configuration
+---------------------------------
+
+You can use this as a sample offlineimap config file::
+
+  [general]
+  accounts = leap-local
+
+  [Account leap-local]
+  localrepository = LocalLeap
+  remoterepository = RemoteLeap
+
+  [Repository LocalLeap]
+  type = Maildir
+  localfolders = ~/LEAPMail/Mail
+
+  [Repository RemoteLeap]
+  type = IMAP
+  ssl = no
+  remotehost = localhost
+  remoteport = 1984
+  remoteuser = user
+  remotepass = pass
+
+Testing utilities
+-----------------
+There are a bunch of utilities to test IMAP delivery in ``imap/tests`` folder.
+If looking for a quick way of inspecting mailboxes, have a look at ``getmail``::
+
+ ./getmail me@testprovider.net mahsikret
+ 1. Drafts
+ 2. INBOX
+ 3. Trash
+ Which mailbox? [1] 2
+ 1 Subject: this is the time of the revolution
+ 2 Subject: ignore me
+
+ Which message? [1] (Q quits) 1
+ 1 X-Leap-Provenance: Thu, 11 Sep 2014 16:52:11 -0000; pubkey="C1F8DE10BD151F99"
+ Received: from mx1.testprovider.net(mx1.testprovider.net [198.197.196.195])
+ (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
+ (Client CN "*.foobar.net", Issuer "Gandi Standard SSL CA" (not verified))
+ by blackhole (Postfix) with ESMTPS id DEADBEEF
+ for <me@testprovider.net>; Thu, 11 Sep 2014 16:52:10 +0000 (UTC)
+ Delivered-To: 926d4915cfd42b6d96d38660c04613af@testprovider.net
+ Message-Id: <20140911165205.GB8054@samsara>
+ From: Kali <kali@leap.se>
+ 
+ (snip)
+
+IMAP Message Rendering Regressions
+----------------------------------
+
+For testing the IMAP server implementation, there is a litte regressions script
+that needs some manual work from your side.
+
+First of all, you need an already initialized account. Which for now basically
+means you have created a new account with a provider that offers the Encrypted
+Mail Service, using the Bitmask Client wizard. Then you need to log in with that
+account, and let it generate the secrets and sync with the remote for a first
+time. After this you can run the twistd server locally and offline.
+
+From the ``leap.mail.imap.tests`` folder, and with an already initialized server
+running::
+
+  ./regressions_mime_struct user@provider pass path_to_samples/
+
+You can find several message samples in the ``leap/mail/tests`` folder.
+ 
+
+
diff --git a/docs/mail/index.rst b/docs/mail/index.rst
new file mode 100644
index 00000000..22bdcf4d
--- /dev/null
+++ b/docs/mail/index.rst
@@ -0,0 +1,88 @@
+:LastChangedDate: $LastChangedDate$
+:LastChangedRevision: $LastChangedRevision$
+:LastChangedBy: $LastChangedBy$
+
+.. _mail:
+
+Bitmask Mail
+================================
+*decentralized and secure mail delivery and synchronization*
+
+This is the documentation for the ``leap.mail`` module. It is a `twisted`_
+module, hanging from the ``leap.bitmask`` namespace, that allows to receive, process, send and access existing messages using the `LEAP`_ platform.
+
+One way to use this library is to let it launch two standard mail services,
+``smtp`` and ``imap``, that run as local proxies and interact with a remote
+``LEAP`` provider that offers *a soledad syncronization endpoint* and receives 
+the outgoing email. This is what `Bitmask`_ client does.
+
+From the mail release 0.4.0 on, it's also possible to use a protocol-agnostic email
+public API, so that third party mail clients can manipulate the data layer. This
+is what the awesome MUA in the `Pixelated`_ project is using.
+
+From release 0.10 on, the Bitmask Bundles will also ship a branded version of
+the Pixelated User Agent, that will be served locally. This will be one of the
+recommended ways of accessing the user emails. The other will be Thunderbird, by
+using the `Bitmask Thunderbird Extension`_.
+
+Note that this used to be a standalone python package, under the ``leap.mail``
+namespace. It was merged into bitmask repo, so it now lives in the
+``leap.bitmask.mail`` namespace. The `legacy repo`_ will no longer be updated.
+
+.. _`twisted`: https://twistedmatrix.com/trac/
+.. _`LEAP`: https://leap.se/en/docs
+.. _`Bitmask`: https://bitmask.net/en/features#email
+.. _`Pixelated`: https://pixelated-project.org/
+.. _`legacy repo`: https://github.com/leapcode/leap_mail/
+.. _`Bitmask Thunderbird Extension`: https://addons.mozilla.org/en-US/thunderbird/addon/bitmask/
+
+How does Bitmask Mail work?
+----------------------------
+
+All the underlying data storage and sync is handled by a library called
+`soledad`_, which handles encryption, storage and sync. Based on `u1db`_,
+documents are stored locally as local ``sqlcipher`` tables, and syncs against
+the soledad sync service in the provider.
+
+OpenPGP key generation and keyring management are handled by another leap
+python library: `keymanager`_.
+
+See :ref:`the life cycle of a leap email <mail_journey>` for an overview of the life cycle
+of an email through ``LEAP`` providers.
+
+.. _`Soledad`: https://leap.se/en/docs/design/soledad
+.. _`u1db`: https://en.wikipedia.org/wiki/U1DB
+.. _`keymanager`: https://github.com/leapcode/keymanager/
+
+The life cycle of a LEAP Email
+------------------------------
+
+For a better picture, you are invited to read about :ref:`the whole journey of a mail in the LEAP system <journey>`.
+
+
+Data model
+--------------------
+
+.. TODO clear document types documentation.
+
+The data model at the present moment consists of several *document types* that split email into different documents that are stored in ``Soledad``. The idea behind this is to
+keep clear the separation between *mutable* and *inmutable* parts, and still being able to
+reconstruct arbitrarily nested email structures easily.
+
+Authentication
+---------------------
+Currently, IMAP and SMTP are twisted services that are binded to ``localhost``. These services be initialized by the bitmask.core daemon, but they are not tied to any user session. When an use attempts to log in to those services, a ``twisted.cred`` pluggable authentication plugin will try to lookup a ``mail token`` that is stored inside the soledad encrypted storage.
+
+
+
+
+Mail development resources
+--------------------------
+
+Some old notes that might help you while developing or debugging bitmask mail
+issues.
+
+.. toctree::
+
+   hacking
+
diff --git a/docs/mail/journey.rst b/docs/mail/journey.rst
new file mode 100644
index 00000000..20fdf465
--- /dev/null
+++ b/docs/mail/journey.rst
@@ -0,0 +1,46 @@
+:LastChangedDate: $LastChangedDate$
+:LastChangedRevision: $LastChangedRevision$
+:LastChangedBy: $LastChangedBy$
+
+.. _journey:
+
+
+The life cycle of a LEAP Email
+==============================
+
+The following are just some notes to facilitate the understanding of the
+leap.mail internals to developers and collaborators.
+
+Server-side: receiving mail from the outside world
+--------------------------------------------------
+
+1. the mx server receives an email, it gets through all the postfix validation and it's written into disk
+2. ``leap_mx`` gets that write in disk notification and encrypts the incoming mail to its intended recipient's pubkey
+3. that encrypted blob gets written into couchdb in a way soledad will catch it in the next sync
+
+
+Client-side: fetching and processing incoming email
+---------------------------------------------------
+you have an imap and an smtp local server. For IMAP:
+
+1. soledad syncs
+2. **fetch** module sees if there's new encrypted mail for the current user from leap_mx
+3. if there is, the mail is decrypted using the user private key, and splitted
+   into several parts according to the internal mail data model (separating
+   mutable and inmutable email parts). Those documents it encrypts it properly
+   like other soledad documents and deletes the pubkey encrypted doc
+4. desktop client is notified that there are N new mails 
+5. when a MUA connects to the **imap** local server, the different documents are glued
+   together and presented as response to the different imap commands.
+
+
+Client side: sending email
+--------------------------
+
+1. you write an email to a recipient and hit send
+2. the **smtp** local server gets that mail, it checks from whom it is and to whom it is for
+3. it signs the mail with the ``From:``'s privkey
+4. it retrieves ``To:``'s pubkey with the keymanager and if it finds it encrypts the mail to him/her
+5. if it didn't find it and you don't have your client configured to "always encrypt", it goes out with just the signature
+6. else, it fails out complaining about this conflict
+7. that mail gets relayed to the provider's **smtp** server