+ +
+

Synchronization of Locally Encrypted Data Among Devices.

+
+

What is Soledad?

+

Soledad is a client library and a server daemon that together allow +applications to securely share a common state among devices. It is LEAP’s solution for mail delivery and for synchronizing +client-encrypted data among a user’s devices that access an account in a LEAP +provider.

+

Soledad is cross-platform, open source, scalable, and features a highly +efficient synchronization algorithm. The Client and Server are written in +Python using Twisted. Source code is available +at 0xacab and is licensed under the GPLv3. Client and server are packaged +together and distributed in pypi. Debian packages are also provided for the server-side component.

+
+
+

Soledad documentation

+
+
+

Introduction

+

Soledad consists of a client library and server daemon that allows applications +to securely share a common state among devices. The local application is +presented with a simple, document-centric searchable database API. Any data +saved to the database by the application is client-encrypted, backed up in the +cloud, and synchronized among a user’s devices. Soledad is cross-platform, open +source, scalable, and features a highly efficient synchronization algorithm.

+

Key aspects of Soledad include:

+
    +
  • Client and server: Soledad includes a server daemon and a client application library.
    • +
  • Client-side encrypted sync: Soledad puts very little trust in the server +by encrypting all data before it is +synchronized to the server and by limiting ways in +which the server can modify the user’s data.
    • +
  • Encrypted local storage: All data cached locally is stored in an +encrypted database.
    • +
  • Document database: An application using the Soledad client library is +presented with a document-centric database API +for storage and sync. Documents may be indexed, searched, and versioned.
    • +
  • Encrypted attachments: storage and synchronization of Blobs is +supported.
    • +
+

Soledad is an acronym of “Synchronization of Locally Encrypted Documents Among +Devices” and means “solitude” in Spanish.

+

See also:

+
+
+

The importance of data availability

+

Users today demand high data availability in their applications. As a user +switches from device to device, the expectation is that each application will +reflect the same state information across devices. Additionally, if all devices +are lost or destroyed, the contemporary user expects to be able to restore her +or his application data from the cloud.

+

In many ways, data availability has become a necessary precondition for an +application to be considered “user friendly”. Unfortunately, most applications +attempt to provide high data availability by rolling their own custom solution +or relying on a third party API, such as Dropbox. This approach is has several +drawbacks: the user has no control or access to the data should they wish to +switch applications or data providers; custom data synchronizations schemes are +often an afterthought, poorly designed, and vulnerable to attack and data +breaches; and the user must place total trust in the provider to safeguard her +or his information against requests by repressive governments.

+

Soledad provides secure data availability in a way that is easy for application +developers to incorporate into their code.

+
+
+

Goals

+
+

Security goals

+
    +
  • Client-side encryption: Before any data is synced to the cloud, it should +be encrypted on the client device.
    • +
  • Encrypted local storage: Any data cached in the client should be stored +in an encrypted format.
    • +
  • Resistant to offline attacks: Data stored on the server should be highly +resistant to offline attacks (i.e. an attacker with a static copy of data +stored on the server would have a very hard time discerning much from the +data).
    • +
  • Resistant to online attacks: Analysis of storing and retrieving data +should not leak potentially sensitive information.
    • +
  • Resistance to data tampering: The server should not be able to provide +the client with old or bogus data for a document.
    • +
+
+
+

Synchronization goals

+
    +
  • Consistency: multiple clients should all get sync’ed with the same data.
    • +
  • Selective sync: the ability to partially sync data. For example, so +a mobile device doesn’t need to sync all email attachments.
    • +
  • Multi-platform: supports both desktop and mobile clients.
    • +
  • Quota: the ability to identify how much storage space a user is taking up.
    • +
  • Scalable cloud: distributed master-less storage on the cloud side, with +no single point of failure.
    • +
  • Conflict resolution: conflicts are flagged and handed off to the +application logic to resolve. Usability goals
    • +
  • Availability: the user should always be able to access their data.
    • +
  • Recovery: there should be a mechanism for a user to recover their data +should they forget their password.
    • +
+
+
+

Known limitations

+

These are currently known limitations:

+
    +
  • The server knows when the contents of a document have changed.
    • +
  • There is no facility for sharing documents among multiple users.
    • +
  • Soledad is not able to prevent server from withholding new documents or new +revisions of a document.
    • +
  • Deleted documents are never deleted, just emptied. Useful for security reasons, but could lead to DB bloat.
    • +
+
+
+

Non-goals

+
    +
  • Soledad is not for filesystem synchronization, storage or backup. It provides +an API for application code to synchronize and store arbitrary schema-less +JSON documents in one big flat document database. One could model +a filesystem on top of Soledad, but it would be a bad fit.
    • +
  • Soledad is not intended for decentralized peer-to-peer synchronization, +although the underlying synchronization protocol does not require a server. +Soledad takes a cloud approach in order to ensure that a client has quick +access to an available copy of the data.
    • +
+
+
+ +
+
+
+

Soledad Server

+

Soledad Server is a document store and a blobs server that can synchronize data +with a Soledad Client.

+
+

Configuring

+

Soledad Server looks for a configuration file in +/etc/soledad/soledad-server.conf and will read the following configuration +options from the [soledad-server] section:

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
OptionDescriptionDefault value
couch_urlThe URL of the CouchDB backend storage.http://localhost:5984
create_cmdThe shell command to create user databases.None
admin_netrcThe netrc file to be used for authenticating +with the CouchDB backend storage./etc/couchdb/couchdb.netrc
batchingWhether to use batching capabilities for +synchronization.true
blobsWhether to provide the Blobs functionality or +not.false
blobs_pathThe path for blobs storage in the server’s file +system./var/lib/soledad/blobs
concurrent_blob_writesLimit of concurrent blob writes to the +filesystem.50
services_tokens_fileThe file containing authentication tokens for +services provided through the Services API./etc/soledad/services.tokens
+
+
+

Running

+

This is written as a Twisted application and intended to be run using the +twistd command. To start the soledad server, run:

+
twistd -n --python /path/to/leap/soledad/server/server.tac
+
+
+

An systemd script is included in the Debian packages to make it feasible to start and stop the +Soledad server service using a standard interface.

+
+
+

Migrations

+

Some updates of Soledad need manual intervention for database migration because +of changes to the storage backend. In all such cases, we will document the +steps needed for migration in this page.

+
+

Soledad Server 0.8 to 0.9 - Couch Database schema migration needed

+

Starting with Soledad Server 0.9.0, the CouchDB database schema was changed to +improve speed of the server side storage backend. Because of that, this script +has to be run for all Leap providers that used to provide email using Soledad +Server < 0.9.0.

+

The migration script can be found:

+
    +
  • In the Soledad repository.
    • +
  • In /usr/share/soledad-server/migration/0.9/ when the soledad-server debian package is installed.
    • +
+

Instructions for migration can be found in the README.md file. Make sure to read it carefully and backup your data before starting the migration process.

+
+
+
+
+

Soledad Client

+

The Soledad Client is a Python library aimed to provide access to a document +store that can be synchronized securelly with other deviced through the Soledad +Server. Key aspects of Soledad Client include:

+
    +
  • Encrypted local storage: All data cached locally is stored in an +encrypted database.
    • +
  • Client-side encrypted sync: Soledad puts very little trust in the +server by encrypting all data before it is synchronized to the server and +by limiting ways in which the server can modify the user’s data.
    • +
  • Document database: An application using the Soledad client library is +presented with a document-centric database API for storage and sync. +Documents may be indexed, searched, and versioned.
    • +
  • Blobs storage: The client and server API provide blobs storage, which +can be used both for data delivery in the server side (i.e. email) and +payload storage on the client side.
    • +
+
+

Setting-up

+

The following information is needed in order to instantiate a soledad client:

+
    +
  • uuid: the user’s uuid.
    • +
  • passphrase: the user’s passphrase.
    • +
  • secrets_path: a local path for secrets storage.
    • +
  • local_db_path: a local path for the documents database.
    • +
  • server_url: the Soledad Server’s URL.
    • +
  • cert_file: a local path for the CA certificate.
    • +
  • auth_token: an authentication token obtained after logging into the +provider.
    • +
+

Once all pieces are in place, you can instantiate the client as following:

+
from leap.soledad.client import Soledad
+
+client = Soledad(
+    uuid,
+    passphrase,
+    secrets_path=secrets_path,
+    local_db_path=local_db_path,
+    server_url=server_url,
+    cert_file=cert_file,
+    auth_token=token)
+
+
+
+
+

Usage example

+

Soledad is written in the Twisted asynchronous model, so +you will need to make sure a reactor +is running.

+

An example of usage of Soledad Client for creating a document and Creation of +a document and synchronization is done as follows:

+
from twisted.internet import defer, reactor
+
+@defer.inlineCallbacks
+def client_usage_example():
+
+    # create a document and sync it with the server
+    yield client.create_doc({'my': 'doc'}, doc_id='some-id')
+    doc = yield client.get_doc('some-id')
+    yield client.sync()
+
+    # modify the document and sync again
+    doc.content = {'new': 'content'}
+    yield client.put_doc(doc)
+    yield client.sync()
+
+d = client_usage_example()
+d.addCallback(lambda _: reactor.stop())
+
+reactor.run()
+
+
+
+
+
+

Reference

+

This page gathers reference documentation to understanding the internals of Soledad.

+
+
+

Environment Variables

+

Some environment variables affect the behaviour of Soledad:

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
variableaffectsdescription
SOLEDAD_HTTP_PERSISTclientpersist HTTP connections.
SOLEDAD_USE_PYTHON_LOGGINGclient / serveruse python logging instead of +twisted’s logger.
SOLEDAD_LOG_TO_STDOUTclient / serverlog to standard output.
SOLEDAD_SERVER_CONFIG_FILEserveruse this configuration file +instead of the default one.
LOCAL_SERVICES_PORTserverwhich port to use for local +TCP services.
HTTPS_PORTserverwhich port to use for public +HTTPS services.
+
+
+

Storage secrets

+

Soledad randomly generates secrets that are used to derive encryption keys for +protecting all data that is stored in the server and in the local storage. +These secrets are themselves encrypted using a key derived from the user’s +passphrase, and saved locally on disk.

+

The encrypted secrets are stored in a local file in the user’s in a JSON +structure that looks like this:

+
{
+    'version': 2,
+    'kdf': 'scrypt',
+    'kdf_salt': <base64 encoded salt>,
+    'kdf_length': <the length of the derived key>,
+    'cipher': <a code indicating the cipher used for encryption>,
+    'length': <the length of the plaintext>,
+    'iv': <the initialization vector>,
+    'secrets': <base64 encoding of ciphertext>,
+}
+
+
+

When a client application first wants to use Soledad, it must provide the +user’s password to unlock the storage secrets. Currently, the storage secrets +are shared among all devices with access to a particular user’s Soledad +database.

+

The storage secrets are currently backed up in the provider (encrypted with the +user’s passphrase) for the case where the user looses or resets her device (see +Shared database for more information). There are plans to make this +feature optional, allowing for less trust in the provider while increasing the +responsibility of the user.

+

If the user looses her passphrase, there is currently no way of recovering her +data.

+
+
+

Server-side databases

+

Soledad Server works with one database per user and one shared database in +which user’s encrypted secrets might be stored.

+
+

User databases

+

User databases in the server are named ‘user-<uuid>’ and Soledad Client may +perform synchronization between its local replicas and the user’s database in +the server. Authorization for creating, updating, deleting and retrieving +information about the user database as well as performing synchronization is +handled by the leap.soledad.server.auth module.

+
+
+

Shared database

+

Each user may store password-encrypted recovery data in the shared database.

+

Recovery documents are stored in the database without any information that +may identify the user. In order to achieve this, the doc_id of recovery +documents are obtained as a hash of the user’s uid and the user’s password. +User’s must have a valid token to interact with recovery documents, but the +server does not perform further authentication because it has no way to know +which recovery document belongs to each user.

+

This has some implications:

+
    +
  • The security of the recovery document doc_id, and thus of access to the +recovery document (encrypted) content, as well as tampering with the +stored data, all rely on the difficulty of obtaining the user’s password +(supposing the user’s uid is somewhat public) and the security of the hash +function used to calculate the doc_id.
    • +
  • The security of the content of a recovery document relies on the +difficulty of obtaining the user’s password.
    • +
  • If the user looses his/her password, he/she will not be able to obtain the +recovery document.
    • +
  • Because of the above, it is recommended that recovery documents expire +(not implemented yet) to prevent excess storage.
    • +
+

The authorization for creating, updating, deleting and retrieving recovery +documents on the shared database is handled by leap.soledad.server.auth +module.

+
+
+
+

Client-side databases

+

Soledad Client uses SQLCipher for +storing data. The symmetric key used to unlock databases is chosen randomly and +stored encrypted with the user’s passphrase (see Storage secrets for +more details).

+

Documents and blobs are stored in +different databases protected with the same symmetric secret.

+
+
+

Document encryption

+

Before a JSON document is sent to the server, Soledad Client symmetrically +encrypts it using AES-256 operating in GCM mode. That mode of encryption +automatically calculates a MAC during block encryption, and so gives Soledad +the ability to encrypt on the fly while transmitting data to the server. +Similarly, when downloading a symmetrically encrypted document from the server, +Soledad Client will decrypt it and verify the MAC tag in the end before +accepting the document.

+

Soledad Client will allways do symmetric encryption. Server-side applications +can define their own encryption schemes and Soledad Client will not try to +decrypt in those cases. The symmetric key used to encrypt a document is derived +from the storage secret and the document id, with HMAC using SHA-256 as a hash +function.

+

The calculation of the MAC also takes into account the document revision to +avoid tampering. Soledad Client will refuse to accept a document if it does not +include a higher revision. In this way, the server cannot rollback a document +to an older revision. The server also cannot delete a document, since document +deletion is handled by removing the document contents, marking it as deleted, +and incrementing the revision. However, a server can withhold from the client +new documents and new revisions of a document (including withholding document +deletion).

+
+
+

Document synchronization

+

Soledad follows the U1DB synchronization protocol with some modifications:

+
    +
  • A synchronization always happens between the Soledad Server and one Soledad +Client. Many clients can synchronize with the same server.
    • +
  • Soledad Client always encrypts before sending +data to the server.
    • +
  • Soledad Client refuses to receive a document if it is encrypted and the MAC +is incorrect.
    • +
  • Soledad Server doesn’t try to decide about document convergence based on the +document’s content, because the content is client-encrypted.
    • +
+
+

Synchronization protocol

+

Synchronization between the Soledad Server and one Soledad Client consists of +the following steps:

+
    +
  1. The client asks the server for the information it has stored about the last +time they have synchronized (if ever).
    2. +
  2. The client validates that its information regarding the last synchronization +is consistent with the server’s information, and raises an error if not. +(This could happen for instance if one of the replicas was lost and restored +from backup, or if a user inadvertently tries to synchronize a copied +database.)
    3. +
  3. The client generates a list of changes since the last change the server +knows of.
    4. +
  4. The client checks what the last change is it knows about on the server.
    5. +
  5. If there have been no changes on either side that the other side has not +seen, the synchronization stops here.
    6. +
  6. The client encrypts and sends the changed documents to the server, along +with what the latest change is that it knows about on the server.
    7. +
  7. The server processes the changed documents, and records the client’s latest +change.
    8. +
  8. The server responds with the documents that have changes that the client +does not yet know about.
    9. +
  9. The client decrypts and processes the changed documents, and records the +server’s latest change.
    10. +
  10. If the client has seen no changes unrelated to the synchronization during +this whole process, it now sends the server what its latest change is, so +that the next synchronization does not have to consider changes that were +the result of this one.
    11. +
+
+
+

Synchronization metadata

+

The synchronization information stored on each database replica consists of:

+
    +
  • The replica id of the other replica. (Which should be globally unique +identifier to distinguish database replicas from one another.)
    • +
  • The last known generation and transaction id of the other replica.
    • +
  • The generation and transaction id of this replica at the time of the most +recent succesfully completed synchronization with the other replica.
    • +
+
+
+

Transactions

+

Any change to any document in a database constitutes a transaction. Each +transaction increases the database generation by 1, and is assigned +a transaction id, which is meant to be a unique random string paired with each +generation.

+

The transaction id can be used to detect the case where replica A and replica +B have previously synchronized at generation N, and subsequently replica B is +somehow reverted to an earlier generation (say, a restore from backup, or +somebody made a copy of the database file of replica B at generation < N, and +tries to synchronize that), and then new changes are made to it. It could end +up at generation N again, but with completely different data.

+

Having random unique transaction ids will allow replica A to detect this +situation, and refuse to synchronize to prevent data loss. (Lesson to be +learned from this: do not copy databases around, that is what synchronization +is for.)

+
+
+
+

Authentication

+

Authentication with the Soledad server is made using Twisted’s Pluggable +Authentication system. The +validation of credentials is performed by verifying a token provided by the +client.

+

There are currently two distinct authenticated entry points:

+
    +
  • A public TLS encrypted Users API, providing the Synchronization and +Blobs services, verified against the Leap Platform +tokens database.
    • +
  • A local plaintext Services API, currently providing only the delivery +part of the Incoming service, authenticated against tokens defined in +a file specified on the server configuration file (see the +Services API tokens file section).
    • +
+
+

Authorization header

+

The client has to provide a token encoded in an HTTP auth header, as in:

+
Authorization: Token <base64-encoded uuid:token>
+
+
+

If no token is provided, the request is considered an “anonymous” request. +Anonymous requests can only access GET /, which returns information about the +server (as the version of the server and runtime configuration options).

+
+
+

Services API tokens file

+

Credentials for services accessible through the local Services API entrypoint +can be added into a file, one in each line with the format +servicename:token, like this:

+
incoming:Zm9yYSB0ZW1lciEK
+
+
+

By default, Soledad Server will look for the tokens file in +/etc/soledad/services.tokens but that is configurable (see +Configuring for more information).

+

Currently, the only special credential provided is for the Incoming service.

+
+
+

Implementation

+

Soledad Server package includes a systemd service file that spawns a twistd +daemon that loads a .tac file. +When the server is started, two services are spawned:

+
    +
  • A local entrypoint for services (serving on localhost only).
    • +
  • A public entrypoint for users (serving on public IP).
    • +
  • Localhost and public IP ports are configurable. Default is 2424 for public IP +and 2525 for localhost.
    • +
+
.------------------------------------------------------.
+|                    soledad-server                    |
+|      (twisted.application.service.Application)       |
+'------------------------------------------------------'
+   |                                                |
+.--------------.                      .----------------.
+| 0.0.0.0:2424 |                      | 127.0.0.1:2525 |
+|     (TLS)    |                      |     (TCP)      |
+'--------------'                      '----------------'
+   |                                                |
+.----------------.             .----------------------.
+| Auth for users |             |  Auth for services   |
+|  (UsersRealm)  |             | (LocalServicesRealm) |
+'----------------'             '----------------------'
+   |                                                |
+.------------------.        .-------------------------.
+|    Users API     |        |      Services API       |
+| (PublicResource) |        |     (LocalResource)     |
+'------------------'        '-------------------------'
+   |  .-------.                .-----------------.  |
+   '->| /sync |                |    /incoming    |<-'
+   |  '-------'                | (delivery only) |
+   |  .--------.               '-----------------'
+   '->| /blobs |
+      '--------
+
+
+
+
+
+

Blobs

+

The first versions of Soledad used to store all data as JSON documents, which +means that binary data was also being treated as strings. This has many +drawbacks, as increased memory usage and difficulties to do transfer and crypto +in a proper binary pipeline.

+

Starting with version 0.10.0, Soledad now has a proper blob infrastructure +that decouples payloads from metadata both in storage and in the +synchronization process.

+
+

Server-side

+

Soledad Server provides two different REST APIs for interacting with blobs:

+
    +
  • A public HTTP Blobs API, providing the Blobs service for Soledad Client +(i.e. actual users of the infrastructure).
    • +
  • A local HTTP Incoming Box API, providing the delivery part of the +Incoming Box service, currently used for the MX mail delivery.
    • +
+

Authentication is handled differently for each of the endpoints, see +Authentication for more details.

+
+

HTTP Blobs API

+

The public endpoint provides the following REST API for interacting with the +Blobs service:

+ ++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
pathmethodactionaccepted query string fields
/blobs/{uuid}GETGet a list of blobs. filtered by +a flag.namespace, filter_flag, order_by
/blobs/{uuid}/{blob_id}GETGet the contents of a blob.namespace
/blobs/{uuid}/{blob_id}PUTCreate a blob. The content of the +blob should be sent in the body +of the request.namespace
/blobs/{uuid}/{blob_id}POSTSet the flags for a blob. A list +of flags should be sent in the +body of the request.namespace
/blobs/{uuid}/{blob_id}DELETEDelete a blob.namespace
+

The Blobs service supports namespaces. All requests can be modified by the +namespace query string parameter, and the results will be restricted to +a certain namespace. When no namespace explicitelly given, the default +namespace is used.

+

When listing blobs, the results can be filtered by flag and/or ordered by date +using the filter_flag and order_by query string parameters. The +possible values for order_by are date or +date for increasing +order, or -date for decreasing order.

+
+
+

HTTP Incoming Box API

+

The local endpoint provides the following REST API for interacting with the +Incoming Box service.

+ +++++ + + + + + + + + + + + + +
pathmethodaction
/incoming/{uuid}/{blob_id}PUTCreate an incoming blob. The content of the blob should be sent in the body of the request.
+

All blobs created using this API are inserted under the namespace MX and +flagged as PENDING.

+
+
+

Filesystem backend

+

On the server side, all blobs are currently stored in the filesystem, under +/var/lib/soledad/blobs by default. Blobs are split in subdirectories +according to the user’s uuid, the namespace, and the 3-letter and 6-letter +prefixes of the blobs uuid to prevent too many files in the same directory. +A second file with the extension .flags stores the flags for a blob.

+

As an example, a PUT request to +/incoming/68625dcb68dab741adf29c7159ccff96/c56da69b25a9a11ec2f408a559ccffc6 +would result in the following:

+
/var/lib/soledad/blobs
+└── 68625dcb68dab741adf29c7159ccff96
+    └── MX
+        └── c56
+            └── c56da6
+                ├── c56da69b25a9a11ec2f408a559ccffc6
+                └── c56da69b25a9a11ec2f408a559ccffc6.flags
+
+
+
+
+
+

Client-side

+

On the client-side, blobs can be managed using the BlobManager API. The +BlobManager is responsible for managing storage of blobs both in local and +remote storages

+

All data is stored locally in the blobs table of a SQLCipher database +called {uuid}_blobs.db that lies in the same directory as the Soledad +Client’s JSON documents database. Both databases are encrypted with the same +symmetric secret. All actions performed locally are mirrored remotelly using +the HTTP Blobs API.

+

The BlobManager supports namespaces and flags and can list local and remote +blobs, possibly filtering by flags and ordering by date (increasingly or +decreasingly). It has helper methods to send or fetch all missing blobs, thus +aiding in synchronization of local and remote data.

+

When uploading, the content of the blob is encrypted with a symmetric secret +prior to being sent to the server. When downloading, the content of the blob is +decrypted accordingly.

+
+
+
+

Incoming Box

+

A mechanism for Trusted Applications to write encrypted data for a given user into the Soledad Server, which will sync it to the client to be processed afterwards.

+
    +
  • Version: 0.2.0
    • +
  • Date: 27 jun 2017
    • +
  • Authors: kali, drebs, vshyba
    • +
+ +
+

Overview

+

The Incoming Box is a new feature of Soledad, (from 0.10 forward), that aim at improving the mechanism by which external Trusted Applications can write data that will be delievered to a user’s database in the server side, from where it will be further processed by the Soledad Client. This processing includes decrypting the payload, since the incoming data is expected to be encrypted by the Trusted Application.

+
+
+

Design Goal

+

Use the particular story about MX delivery to guide the design of a general mechanism that makes sense in the context of Soledad as a generic Encrypted Database Solution. +The final solution should still be able to be the backend for different types of applications, without introducing abstractions that are too tied to the encrypted email use case.

+
+
+

Features

+
    +
  1. Deliver data (expected to be encrypted with a public-key mechanism) to the Soledad Server, so that it is written into the data space for a particular user.
    2. +
  2. Provide a mechanism by which Soledad Client, acting on behalf of an user, can download the data, process it in any needed way, and eventually store it again in the conventional main storage that Soledad provides. Since the data is expected to be encrypted by the delivery party, the client-side processing includes any decryption step if needed.
    3. +
  3. Allow to purge a document that has been correctly processed
    4. +
  4. Allow to mark a document that has failed to be decrypted, to avoid trying to decrypt it in every processing loop
    5. +
+
+
+

Conventions

+

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL +NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and +“OPTIONAL” in this document are to be interpreted as described in +RFC 2119.

+
+
+

Terminology

+
    +
  • Blob refers to an encrypted payload that is stored by soledad server as-is. It is assumed that the blob was end-to-end encrypted by an external component before reaching the server. (See the Blobs for more detail)
    • +
  • A BlobsBackend implementation is a particular backend setup by the Soledad Server that stores all the blobs that a given user owns. For now, only a filesystem backend is provided.
    • +
  • An Incoming Message makes reference to the representation of an abstract entity that matches exactly one message item, no matter how it is stored (ie, docs vs. blobs, single blob vs chunked, etc). It can represent one Email Message, one URL, an uploaded File, etc. For the purpose of the email use case, an Incoming Message refers to the encrypted message that MX has delivered to the incoming endpoint, which is pgp-encrypted, and can have been further obfuscated.
    • +
  • By Message Processing we understand the sequence of downloading an incoming message, decrypting it, transform it in any needed way, and deleting the original incoming message.
    • +
  • An Incoming Box is a subset of the Blobs stored for an user, together with the mechanisms that provide semantics to store, list, fetch and process incoming message data chronologically.
    • +
+
+
+

Components

+
    +
  • Soledad Server is the particular implementation that runs in the server (the +twisted implementation is the only one to the moment). This exposes several +endpoints (documents synchronization, blob storage, incoming message box).
    • +
  • BlobsBackend is a particular implementation of the IBlobsBackend interface, +such as FilesystemBlobsBackend, that the server has been configured to use.
    • +
  • Soledad Client is the particular client that runs in different desktop +replicas in the u1db sense (the twisted implementation is the only one to the +moment). It executes a sync periodically, that syncs all metadata docs +(soledad l2db json documents), and then downloads the blobs on demand. The +blobs that need to be synced are both the encrypted blobs that are linked from +metadata docs, and in this case the blobs stored in the space for a given +Incoming Box.
    • +
  • BlobsManager is the component that orchestrates the uploads/downloads and +storage in the client side. The client storage backend currently is SQLCipher, +using the BLOB type.
    • +
  • A Trusted Application is any application that is authorized to write data into +the user incoming box. Initially, LEAP’s Encrypting Remailer Proxy (MX, for +short) is going to be the main trusted application that will drive the +development of the Incoming Box.
    • +
  • Client Trusted Application Consumer is any implementation of Soledad’s client +IIncomingBoxConsumer, which is the interface that the client counterpart of +the Trusted Application needs to implement in order to receive messages. It +provides implementation for processing and saving Incoming Messages. In the +encrypted email case, this component is the Incoming Mail Service in Bitmask +Mail.
    • +
+
+
+

The User Experience

+
    +
  • From the end user perspective (ie, the user of Bitmask Mail in this case), +the behaviour of the Incoming Box client in Soledad is completely +transparent. Periodically, Soledad will call the Client Trusted Application +Consumer with new “messages” of any particular type backed by the Soledad +Client Storage, without any other intervention that introducing the master +passphrase.
    • +
  • From the client API perspective (considering the user is a Client Trusted +Application developer), all it needs to do is to implement +IIncomingBoxConsumer interface and register this new consumer on Soledad +API, telling the desired namespace it wants to consume messages from. +Soledad will then take care of calling the consumer back for processing and +saving of Incoming Messages.
    • +
+
+
+

Writing Data Into The Incoming Box

+
    +
  • Any payload MUST arrive already encrypted to the endpoint of the Incoming Box. +Soledad Server, at version 1 of this spec, will not add any encryption to the +payloads.
    • +
  • The details of the encryption scheme used by the Trusted Application to +encrypt the delivered payload (MX in this case) MUST be shared with the +domain-specific application that processes the incoming message on the client +side (Incoming Mail Service in Bitmask Mail, in this case). This means that +the encryption schema MUST be communicated to the Incoming Box API in the +moment of the delivery.
    • +
  • Incoming Boxes MUST NOT be writeable by any other user or any external +applications.
    • +
+
+
+

Authentication

+
    +
  • The Trusted Application and the Soledad Server exposing the Incoming Box +endpoint MUST share a secret, that is written into the configuration files of +both services.
    • +
  • The Incoming Box MUST NOT be accessible as a public service from the outside.
    • +
+
+
+

Listing All Incoming Messages

+
    +
  • Soledad server will list all the messages in the Incoming Box every time that a client requests it.
    • +
  • The server MUST return the number of pending messages.
    • +
  • The server SHOULD skip messages from the returned set beyond a given size limit, if the client requests it so.
    • +
  • The server MAY allow pagination.
    • +
+
+
+

Processing Incoming Messages

+
    +
  • The Blobs containing the Incoming Messages need the capability to be marked +as in one of the following states: PENDING, PROCESSING, PROCESSED, FAILED.
    • +
  • The default state for a new message in the Incoming Box is PENDING.
    • +
  • Before delivering a Message to a client for processing, the server MUST mark +the blob that contains it as PROCESSING, reserving the message for this +client so other replicas don’t try to repeat the processing.
    • +
  • The server MAY expire the PROCESSING flag if the defined PROCESSING_THRESHOLD +is passed, to avoid data left unusable by stalled clients.
    • +
  • A message marked as PROCESSING MUST only be marked as PROCESSED by the server +when it receives a confirmation by the replica that initiated the download +request. This confirmation signals that the message is ready to be deleted.
    • +
  • A Client MUST request to the server to mark an incoming message as PROCESSED +only when there are guarantees that the incoming message has been processed +without errors, and the parts resulting of its processing are acknowleged to +have been uploaded successfully to the central replica in the server.
    • +
+
+
+

Marking a Message as Failed

+
    +
  • A Soledad Client MUST be able to mark a given message as FAILED. This covers +the case in which a given message failed to be decrypted by a +implementation-related reason (for instance: uncatched exceptions related to +encoding, wrong format in serialization). The rationale is that we don’t want +to increase overhead by retrying decryption on every syncing loop, but we +don’t want to discard a particular payload. Future versions of the client +might implement bugfixes or workarounds to try succeed in the processing.
    • +
  • Therefore, a Soledad Client SHOULD be able to add its own version when it +marks a message as temporarily failed.
    • +
  • After some versions, a message SHOULD be able to be marked as permanently +failed.
    • +
  • Reservation MUST be released, as any other replica MUST be able to pick the +message to retry. This covers the case of a new replica being added with an +updated version, which can be able to handle the failure.
    • +
+
+
+

Deleting Incoming Messagges

+
    +
  • Any message in the Incoming Box marked as PROCESSED MAY be deleted by +the server.
    • +
  • Any message in the Incoming Box marked as PERMANENTLY FAILED MAY be deleted by the server.
    • +
+
+
+

Implementation Details

+
+

Server Blob Backend

+

In the Server Side, the implementation of the Incoming Box MUST be done +exclusively at the level of the BlobStorage. The Blobs implementation in both +Soledad Server and Client have enough knowledge of the incoming box semantics +to allow its processing to be done without resorting to writing documents in +the main soledad json storage.

+

For simplicity, the IncomingBox endpoint is assumed to be running under the +same process space than the rest of the Soledad Server.

+
+

Preffix Namespaces

+

The Trusted Application code responsible for delivering messages into Soledad +Server Incoming Box endpoint MUST specify as a request parameter the +dedicated namespace that it desires to use and be authorized to write into it. +This is done so Soledad can list, filter, flag, fetch and reserve only messages +known to the Trusted Application, avoiding to mix operations with blobs from +the global namespace or messages from another Trusted Application.

+
+
+

LIST commands

+

The server MUST reply to several LIST commands, qualified by namespace and by +other query parameters. Some of these commands are optional, but the server +SHOULD reply to them signaling that they are not supported by the implementation.

+

The Server MUST return a list with the UIDs of the messages.

+

Supported listing features are: +* Filter by namespace, which selects the namespace to do the list operation. +* Filter by flag, which lists only messages/blobs marked with the provided flag. +* Ordering, which changes the order of listing, such as older or newer first. +* Count, which returns the total list size after filtering, instead of the list itself.

+
+
+
+

Client side Processing

+

It is assumed, for simplicity, that the Trusted Application consumer +implementation shares the process memory space with the soledad client, but +this doesn’t have to hold true in the future.

+

The class responsible for client side processing on Soledad Client is named +IncomingBoxProcessingLoop. Its role is to orchestrate processing with the +Incoming Box features on server side, so it can deliver messages to it’s +registered Trusted Application Consumers.

+
    +
  • To begin a processing round, the client starts by asking a list of the +pending messages.
    • +
  • To avoid potentially costly traversals, the client limits the query to the +most recent N blobs flagged as PENDING.
    • +
  • To avoid downloading bulky messages in the incoming queue (for example, +messages with very big attachments), the client MAY limit the query on a +first pass to all pending blobs smaller than X Kb.
    • +
  • After getting the list of Incoming Messages in the PENDING set, the client +MUST start downloading the blobs according to the uuids returned.
    • +
  • Download SHOULD happen in chronological order, from the list. Download may +happen in several modalities: concurrently, or sequentially.
    • +
  • The Soledad Client MUST provide a mechanism so that any clientside +counterpart of the Trusted Application (ie: Bitmask Mail) can register a +consumer for processing and saving each downloaded message.
    • +
  • In the reference implementation, since the consumers that the client +registers are executed in the common event loop of the Soledad Client +process, attention SHOULD be payed to the callbacks not blocking the main +event loop.
    • +
+

Example of a Trusted Application Client Consumer:

+
@implementer(interfaces.IIncomingBoxConsumer)
+class MyConsumer(object):
+    def __init__(self):
+        self.name = 'My Consumer'
+
+    def process(self, item, item_id, encrypted=True):
+        cleartext = my_custom_decrypt(item) if encrypted else item
+        processed_parts = my_custom_processing(item)
+        return defer.succeed(processed_parts)
+
+    def save(self, parts, item_id):
+        return defer.gatherResults([db.save(part) for part in parts])
+
+
+
+
+
+

Future Features

+

Still subject to discussion, but some features that are desired for future iterations are:

+
    +
  • Provide a mechanism to retry documents marked as failed by previous revisions.
    • +
  • Internalizing public key infrastructure (using ECC).
    • +
  • ACLs to allow other users to push documents to an user Incoming Box.
    • +
  • Provide alternative implementations of the Incoming Box endopoint (for example, in Rust)
    • +
+
+

Writing Data When User Quota is Exceeeded

+
    +
  • The server SHOULD move the payload to the permanent storage in the user storage space only after checking that the size of the storage currently occupied by the user data, plus the payload size does not exceed the allowed quota, if any, plus a given tolerance limit.
    • +
  • The Trusted Application SHOULD receive an error message as a response to its storage request, so that it can register the failure to store the data, or inform the sender in the case in which it is acting as a delegate to deliver a message.
    • +
+
+
+

LIST QUALIFIERS

+

In order to improve performance and responsiveness, a list request MAY be +qualified by the following parameters that the server SHOULD satisfy. +The responses are, in any case, a list of the uuids of the Blobs.

+
    +
  • Pagination.
    • +
  • Skip by SIZE THRESHOLD.
    • +
  • Include messages with PROCESSING flag.
    • +
+ +
+

SKIP-BY-SIZE

+
    +
  • SIZE_LIMIT: skips messages bigger than a given size limit, to avoid downloading payloads too big when client is interested in a quick list of incoming messages.
    • +
+

Example:

+
IncomingBox.list('mx', size_limit=10MB)
+
+
+
+
+
+
+
+

Document attachments

+ +
+

Reasoning

+

The type of a Soledad document’s content is JSON, +which is good for efficient lookup and indexing. On the other hand, this is +particularly bad for storing larger amounts of binary data, because:

+
    +
  • the only way to store data in JSON is as unicode string, and this uses more +space than what is actually needed for binary data storage.
    • +
  • upon synchronization, the content of a Soledad document needs to be +completelly transferred and decrypted for the document to be available for +use.
    • +
+

Document attachments were introduced as a means to efficiently store large +payloads of binary data while avoiding the need to wait for their transfer to +have access to the documents’ contents.

+
+
+

Server-side

+

In the server, attachments are stored as Blobs. See +HTTP Blobs API for more information on how to interact with the server +using HTTP.

+

The IBlobsBackend interface is provided, so in the +future there can be different ways to store attachments in the server side +(think of a third-party storage, for example). Currently, the +Filesystem backend is the only one that implements that interface.

+
+
+

Client-side

+

In the client, attachments are relations between JSON documents and blobs.

+

See client-side-attachment-api for reference.

+
+

Usage example

+

The attachments API is currently available in the Document class, and the +document needs to know about the store to be able to manage attachments. When +you create a new document with soledad, that document will already know about +the store that created it, and can put/get/delete an attachment:

+
from twisted.internet.defer import inlineCallbacks
+
+@inlineCallbacks
+def attachment_example(soledad):
+    doc = yield soledad.create_doc({})
+
+    state = yield doc.get_attachment_state()
+    dirty = yield doc.is_dirty()
+
+    assert state == AttachmentStates.NONE
+    assert dirty == False
+
+    yield doc.put_attachment(open('hackers.txt'))
+    state = yield doc.get_attachment_state()
+    dirty = yield doc.is_dirty()
+
+    assert state | AttachmentState.LOCAL
+    assert dirty == True
+
+    yield soledad.put_doc(doc)
+    dirty = yield doc.is_dirty()
+
+    assert dirty == False
+
+    yield doc.upload_attachment()
+    state = yield doc.get_attachment_state()
+
+    assert state | AttachmentState.REMOTE
+    assert state == AttachmentState.SYNCED
+
+    fd = yield doc.get_attachment()
+    assert fd.read() == open('hackers.txt').read()
+
+
+
+
+
+
+
+
+

Development

+

As an open source project, we welcome contributions of all forms. This page +should help you get started with development for Soledad.

+
+
+

Contributing

+

Thank you for your interest in contributing to Soledad!

+
+

Filing bug reports

+

Bug reports are very welcome. Please file them on the 0xacab issue tracker. Please include an extensive +description of the error, the steps to reproduce it, the version of Soledad +used and the provider used (if applicable).

+
+
+

Patches

+

All patches to Soledad should be submitted in the form of pull requests to the +main Soledad repository. These pull +requests should satisfy the following properties:

+
    +
  • The pull request should focus on one particular improvement to Soledad. +Create different pull requests for unrelated features or bugfixes.
    • +
  • Code should follow PEP 8, +especially in the “do what code around you does” sense.
    • +
  • Pull requests that introduce code must test all new behavior they introduce +as well as for previously untested or poorly tested behavior that they touch.
    • +
  • Pull requests are not allowed to break existing tests. We will consider pull +requests that are breaking the CI as work in progress.
    • +
  • When introducing new functionality, please remember to write documentation.
    • +
  • Please sign all of your commits. If you are merging code from other +contributors, you should sign their commits.
    • +
+
+

Review

+

Pull requests must be reviewed before merging. The final responsibility for the +reviewing of merged code lies with the person merging it. Exceptions to this +rule are modifications to documentation, packaging, or tests when the need of +agility of merging without review has little or no impact on the security and +functionality of working code.

+
+
+

Getting help

+

If you need help you can reach us through one of the following means:

+ +
+
+
+
+

Backwards-compatibility and deprecation policy

+

Since Soledad has not reached a stable 1.0 release yet, no guarantees are made +about the stability of its API or the backwards-compatibility of any given +version.

+

Currently, the internal storage representation is experimenting changes that +will take some time to mature and settle up. For the moment, no given SOLEDAD +release is offering any backwards-compatibility guarantees.

+

Although serious efforts are being made to ensure no data is corrupted or lost +while upgrading soledad versions, it’s not advised to use SOLEDAD for any +critical storage at the moment, or to upgrade versions without any external data +backup (for instance, an email application that uses SOLEDAD should allow to +export mail data or PGP keys in a convertible format before upgrading).

+
+

Deprecation Policy

+

The points above standing, the development team behind SOLEDAD will strive to +provide clear migration paths between any two given, consecutive minor +releases, in an automated form wherever possible.

+

This means, for example, that a migration script will be provided with the +0.10 release, to migrate data stored by any of the 0.9.x soledad +versions. Another script will be provided to migrate from 0.10 to 0.11, +etc (but not, for instance, from 0.8 to 0.10).

+

At the same time, there’s a backwards-compatibility policy of deprecating APIs +after 2 minor releases. This means that a feature will start to be marked as +deprecated in 0.10, with a warning being raised for 2 minor releases, and +the API will disappear completely no sooner than in 0.12.

+
+
+
+

Tests

+

We use pytest as a testing framework +and Tox as a test environment manager. +Currently, tests reside in the testing/ folder and some of them need a +couchdb server to be run against.

+

If you do have a couchdb server running on localhost on default port, the +following command should be enough to run tests:

+
tox
+
+
+
+

CouchDB dependency

+

In case you want to use a couchdb on another host or port, use the +–couch-url parameter for pytest:

+
tox -- --couch-url=http://couch_host:5984
+
+
+

If you want to exclude all tests that depend on couchdb, deselect tests marked +with needs_couch:

+
tox -- -m 'not needs_couch'
+
+
+
+
+

Benchmark tests

+

A set of benchmark tests is provided to measure the time and resources taken to +perform some actions. See the documentation for benchmarks.

+
+
+
+

Benchmarks

+

We currently use pytest-benchmark +to write tests to assess the time and resources taken by various tasks.

+

To run benchmark tests, once inside a cloned Soledad repository, do the +following:

+
tox -e benchmark
+
+
+

Results of automated benchmarking for each commit in the repository can be seen +in: https://benchmarks.leap.se/.

+

Benchmark tests also depend on tox and CouchDB. See the Tests page +for more information on how to setup the test environment.

+
+

Test repetition

+

pytest-benchmark runs tests multiple times so it can provide meaningful +statistics for the time taken for a tipical run of a test function. The number +of times that the test is run can be manually or automatically configured.

+

When automatically configured, the number of runs is decided by taking into +account multiple pytest-benchmark configuration parameters. See the the +corresponding documenation for more +details on how automatic calibration works.

+

To achieve a reasonable number of repetitions and a reasonable amount of time +at the same time, we let pytest-benchmark choose the number of repetitions +for faster tests, and manually limit the number of repetitions for slower tests.

+

Currently, tests for synchronization and sqlcipher asynchronous document +creation are fixed to run 4 times each. All the other tests are left for +pytest-benchmark to decide how many times to run each one. With this setup, +the benchmark suite is taking approximatelly 7 minutes to run in our CI server. +As the benchmark suite is run twice (once for time and cpu stats and a second +time for memory stats), the whole benchmarks run takes around 15 minutes.

+

The actual number of times a test is run when calibration is done automatically +by pytest-benchmark depends on many parameters: the time taken for a sample +run and the configuration of the minimum number of rounds and maximum time +allowed for a benchmark. For a snapshot of the number of rounds for each test +function see the soledad benchmarks wiki page.

+
+
+

Sync size statistics

+

Currenly, the main use of Soledad is to synchronize client-encrypted email +data. Because of that, it makes sense to measure the time and resources taken +to synchronize an amount of data that is realistically comparable to a user’s +email box.

+

In order to determine what is a good example of dataset for synchronization +tests, we used the size of messages of one week of incoming and outgoing email +flow of a friendly provider. The statistics that came out from that are (all +sizes are in KB):

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
 outgoingincoming
min0.6750.461
max25531.36125571.748
mean252.411110.626
median5.32014.974
mode1.4041.411
stddev1376.930732.933
+
+
+

Sync test scenarios

+

Ideally, we would want to run tests for a big data set (i.e. a high number of +documents and a big payload size), but that may be infeasible given time and +resource limitations. Because of that, we choose a smaller data set and suppose +that the behaviour is somewhat linear to get an idea for larger sets.

+

Supposing a data set total size of 10MB, some possibilities for number of +documents and document sizes for testing download and upload can be seen below. +Scenarios marked in bold are the ones that are actually run in the current sync +benchmark tests, and you can see the current graphs for each one by following +the corresponding links:

+ +

In each of the above scenarios all the documents are of the same size. If we +want to account for some variability on document sizes, it is sufficient to +come up with a simple scenario where the average, minimum and maximum sizes are +somehow coherent with the above statistics, like the following one:

+
    +
  • 60 x 15KB + 1 x 1MB
    • +
+
+
+
+

Compatibility

+

This page keeps notes about compatibility between different versions of Soledad +and between Soledad and other components of the LEAP Platform.

+
    +
  • Upgrades of Soledad Server < 0.9.0 to >= 0.9.0 need database migration +because older code used to use CouchDB’s design documents, while newer code +got rid of that because it made everything cpu and memory hungry. See the +documentation +for more information.
    • +
  • Soledad Server >= 0.7.0 is incompatible with client < 0.7.0 because of +modifications on encrypted document MAC calculation.
    • +
  • Soledad Server >= 0.7.0 is incompatible with LEAP Platform < 0.6.1 because +that platform version implements ephemeral tokens databases and Soledad +Server needs to act accordingly.
    • +
+
+
+

Changelog

+
+

0.10.4 - master

+
+

Note

+

This version is not yet released and is under active development.

+
+
+

Misc

+
    +
  • Add packages for debian buster.
    • +
  • deb: Make soledad-client depend on soledad-common
    • +
+
+
+
+

0.10.3 - Mon 11 Sep, 2017

+
+

Server

+
    +
  • [feat] Finished adding support for Incoming API
    • +
  • [feat] Get config file name from environment variable.
    • +
  • [bug] Add DELETE method to url mapper.
    • +
  • [bug] Use correct keyword argument for server state initialization.
    • +
  • #8924: [bug] FileBodyProducer consumer usage wasn’t closing the file
    • +
+
+
+

Client

+
    +
  • [feat] Add columns for sync state of blobs inside sqlcipher
    • +
  • [bug] Several bugfixes for BlobManager initialization.
    • +
  • [bug] Fix usage of StringIO class in gzip middleware.
    • +
  • #8924: [bug] FileBodyProducer consumer usage wasn’t closing the file
    • +
+
+
+

Misc

+
    +
  • Use latest version of pytest-benchmark.
    • +
  • Find correct twistd when outside tox envs
    • +
  • Build packages for zesty and stretch.
    • +
  • Add benchmark comparing legacy vs blobs sync.
    • +
  • Add reactor responsiveness tests.
    • +
+
+
+
+

0.10.2 - Mon 21 Aug, 2017

+
+

Server

+
    +
  • Enforce namespace to default on server
    • +
  • Add path partitioning to namespaces
    • +
+
+
+

Client

+
    +
  • Add namespace to local blobs db table
    • +
  • Track namespace information on blobs client
    • +
+
+
+
+

0.10.1 - Mon 07 Aug, 2017

+
+

Server

+
    +
  • Fixes IncomingBox missing preamble separator (space) which causes client to +fail parsing.
    • +
+
+
+

Client

+
    +
  • Adds IncomingBoxProcessLoop and implement the process flow for IncominBox +specification.
    • +
  • Adds IIncomingBoxConsumer interface, which can be used by Soledad apps to +implement consumers for IncomingBox feature.
    • +
+
+
+
+

0.10.0 - 18 July, 2017

+
+

Server

+
    +
  • Add an incoming API for email delivery. In the future, this may be used by +external applications for message delivery.
    • +
  • Add namespace capability.
    • +
  • List incoming blobs in chronological order.
    • +
  • Finish minimal filesystem backend for blobs.
    • +
  • Update BlobManager to support new server features, such as: namespaces, +incoming and listing.
    • +
  • Make the backend configurable for incoming API, so it can use CouchDB now and +Blobs later.
    • +
+
+
+

Client

+
    +
  • Use OpenSSL backend for scrypt if OpenSSL >= 1.1
    • +
+
+
+

Misc

+
    +
  • Refactor preamble to account for PGP encryption scheme
    • +
  • Removes scrypt dependency
    • +
  • Unification of Client, Server and Common in a Single python package.
    • +
  • Build soledad debian package with git-buildpackage.
    • +
  • Document deprecation policy.
    • +
  • Documentation is automatically uploaded to: https://soledad.readthedocs.io/
    • +
  • Launch benchmarks website: https://benchmarks.leap.se/
    • +
+
+
+
+

0.9.6 - 31 May, 2017

+
+

Server

+
    +
  • Minimal Filesystem BlobsBackend implementation, disabled by default.
    • +
+
+
+

Client

+
    +
  • Minimal Blobs manager implementation
    • +
  • Blobs API
    • +
  • Ability to generate recovery code.
    • +
  • Fix deprecated multibackend call (cryptography).
    • +
+
+
+

Misc

+
    +
  • Post benchmark results to elasticsearch
    • +
  • Build docker image and push it to registry every time the dockerfile used for +tests is changed
    • +
  • Fix flaky tests
    • +
  • Cleanup old documentation.
    • +
  • Added dependency on treq.
    • +
  • Improve cpu/memory profiling.
    • +
  • Bumped version to upload wheels to pypi, to workaround for dbschema.sql not +found after installation in virtualenv.
    • +
+
+
+
+

0.9.5 - 17 March, 2017

+
+

Server

+
    +
  • Make database creation appear in logs
    • +
+
+
+

Client

+
    +
  • #8721: Remove offline flag
    • +
  • Fix raising of invalid auth token error
    • +
  • Add default version when decrypting secrets
    • +
  • Secrets version defaults to v1
    • +
+
+
+

Misc

+
    +
  • First steps porting soledad to python3
    • +
+
+
+
+

0.9.3 - 06 March, 2017

+
+

Server

+
    +
  • Refactor authentication code to use twisted credential system.
    • +
  • Announce server blobs capabilities
    • +
  • #8764: Allow unauthenticated users to retrieve the capabilties banner.
    • +
  • #6178: Add robots.txt
    • +
  • #8762: Add a systemd service file
    • +
  • Add script to deploy from git
    • +
+
+
+

Client

+
    +
  • #8758: Add blob size to the crypto preamble
    • +
  • Improve secrets generation and storage code
    • +
  • Add offline status to soledad client api.
    • +
  • Remove syncable property
    • +
+
+
+

Misc

+
    +
  • Improvements in performance benchmarks.
    • +
+
+
+
+

0.9.2 - 22 December, 2016

+
+

Performance improvements

+
    +
  • use AES 256 GCM mode instead of CTR-HMAC.
    • +
  • streaming encryption/decryption and data transfer.
    • +
+
+
+

Server

+
    +
  • move server to a twisted resource entrypoint.
    • +
+
+
+

Client

+
    +
  • use twisted http agent in the client.
    • +
  • maintain backwards compatibility with old crypto scheme (AES 256 CTR-HMAC). +No migration for now, only in 0.10.
    • +
  • remove the encryption/decryption pools, replace for inline streaming crypto.
    • +
  • use sqlcipher transactions on sync.
    • +
+
+
+
+

0.9.1 - 27 November, 2016

+
+

Server side bug fixes

+
    +
  • fix import on create-user-db script
    • +
  • patch twisted logger so it works with twistd –syslog
    • +
  • delay couch state initialization
    • +
  • improve missing couch config doc error logging
    • +
  • separate server application into another file
    • +
+
+
+
+

0.9.0 - 11 November, 2016

+
+

Main features

+
    +
  • Server-side changes in couch backend schema.
    • +
  • Use of tox and pytest to run tests.
    • +
  • Performance tests.
    • +
+
+
+

Server

+

* Attention: Migration needed! *

+

This version of soledad uses a different database schema in the server couch +backend. The difference from the old schema is that the use of design documents +for storing and accessing soledad db metadata was removed because incurred in +too much memory and time overhead for passing data to the javascript +interpreter.

+

Because of that, you need to run a migration script on your database. Check the +scripts/migration/0.9.0/ diretctory for instructions on how to run the +migration script on your database. Don’t forget to backup before running the +script!

+
+
+

Bugfixes

+
    +
  • Fix order of multipart serialization when writing to couch.
    • +
+
+
+

Features

+
    +
  • Log to syslog.
    • +
  • Remove usage of design documents in couch backend.
    • +
  • Use _local couch docs for metadata storage.
    • +
  • Other small improvements in couch backend.
    • +
+
+
+
+

0.8.1 - 14 July, 2016

+
+

Client

+
+

Features

+
    +
  • Add recovery document format version for future migrations.
    • +
  • Use DeferredLock instead of its locking cousin.
    • +
  • Use DeferredSemaphore instead of its locking cousin.
    • +
+
+
+

Bugfixes

+
    +
  • #8180: Initialize OpenSSL context just once.
    • +
  • Remove document content conversion to unicode. Users of API are responsible +for only passing valid JSON to Soledad for storage.
    • +
+
+
+

Misc

+
    +
  • Add ability to get information about sync phases for profiling purposes.
    • +
  • Add script for setting up develop environment.
    • +
  • Refactor bootstrap to remove shared db lock.
    • +
  • Removed multiprocessing from encdecpool with some extra refactoring.
    • +
  • Remove user_id argument from Soledad init.
    • +
+
+
+
+

Common

+
+

Features

+
    +
  • Embed l2db, forking u1db.
    • +
+
+
+

Misc

+
    +
  • Toxify tests.
    • +
+
+
+
+
+

0.8.0 - 18 Apr, 2016

+
+

Client

+
+

Features

+
    +
  • #7656: Emit multi-user aware events.
    • +
  • Client will now send documents at a limited size batch due to changes on SyncTarget. The default limit is 500kB. Disabled by default.
    • +
+
+
+

Bugfixes

+
    +
  • #7503: Do not signal sync completion if sync failed.
    • +
  • Handle missing design doc at GET (get_sync_info). Soledad server can handle this during sync.
    • +
+
+
+

Misc

+
    +
  • #7195: Use cryptography instead of pycryptopp.
    • +
+
+
+

Known Issues

+
    +
  • Upload phase of client syncs is still quite slow. Enabling size limited batching +can help, but you have to make sure that your server is compatible.
    • +
+
+
+
+

Server

+
+

Features

+
    +
  • General performance improvements.
    • +
  • #7509: Moves config directory from /etc/leap to /etc/soledad.
    • +
  • Adds a new config parameter ‘create_cmd’, which allows sysadmin to specify +which command will create a database. That command was added in +pkg/create-user-db and debian package automates steps needed for sudo access.
    • +
  • Read netrc path from configuration file for create-user-db command.
    • +
  • ‘create-user-db’ script now can be configured from soledad-server.conf when +generating the user’s security document.
    • +
  • Migrating a user’s database to newest design documents is now possible by +using a parameter ‘–migrate-all’ on ‘create-user-db’ script.
    • +
  • Remove tsafe monkeypatch from SSL lib, as it was needed for Twisted <12
    • +
  • Added two methods to start and finish a batch on backend. They can be used to +change database behaviour, allowing batch operations to be optimized.
    • +
+
+
+
+

Common

+
+

Features

+
    +
  • Add a sanitized command executor for database creation and re-enable user +database creation on CouchServerState via command line.
    • +
+
+
+

Bugfixes

+
    +
  • #7626: Subclass a leaky leap.common.couch exception to avoid depending on couch.
    • +
+
+
+
+
+
+
+
+

API Reference

+
+
+

Soledad Client API

+
+
+class leap.soledad.client.Soledad(uuid, passphrase, secrets_path, local_db_path, server_url, cert_file, shared_db=None, auth_token=None, with_blobs=False)
+

Bases: object

+

Soledad provides encrypted data storage and sync.

+

A Soledad instance is used to store and retrieve data in a local encrypted +database and synchronize this database with Soledad server.

+

This class is also responsible for bootstrapping users’ account by +creating cryptographic secrets and/or storing/fetching them on Soledad +server.

+
+
+__init__(uuid, passphrase, secrets_path, local_db_path, server_url, cert_file, shared_db=None, auth_token=None, with_blobs=False)
+

Initialize configuration, cryptographic keys and dbs.

+ +++ + + + + + +
Parameters:
    +
  • uuid (str) – User’s uuid.
    • +
  • passphrase (unicode) – The passphrase for locking and unlocking encryption secrets for +local and remote storage.
    • +
  • secrets_path (str) – Path for storing encrypted key used for symmetric encryption.
    • +
  • local_db_path (str) – Path for local encrypted storage db.
    • +
  • server_url (str) – URL for Soledad server. This is used either to sync with the user’s +remote db and to interact with the shared recovery database.
    • +
  • cert_file (str) – Path to the certificate of the ca used to validate the SSL +certificate used by the remote soledad server.
    • +
  • shared_db (HTTPDatabase) – The shared database.
    • +
  • auth_token (str) – Authorization token for accessing remote databases.
    • +
  • with_blobs – A boolean that specifies if this soledad instance should enable +blobs storage when initialized. This will raise if it’s not the +first initialization and the passed value is different from when +the database was first initialized.
    • +
+
Raises:

BootstrapSequenceError – Raised when the secret initialization sequence (i.e. retrieval +from server or generation and storage on server) has failed for +some reason.

+
+
+ +
+
+change_passphrase(new_passphrase)
+

Change the passphrase that encrypts the storage secret.

+ +++ + + + + + +
Parameters:new_passphrase (unicode) – The new passphrase.
Raises:NoStorageSecret – Raised if there’s no storage secret available.
+
+ +
+
+close()
+

Close underlying U1DB database.

+
+ +
+
+create_doc(*args, **kwargs)
+

Create a new document.

+

You can optionally specify the document identifier, but the document +must not already exist. See ‘put_doc’ if you want to override an +existing document. +If the database specifies a maximum document size and the document +exceeds it, create will fail and raise a DocumentTooBig exception.

+ +++ + + + + + + + +
Parameters:
    +
  • content (dict) – A Python dictionary.
    • +
  • doc_id (str) – An optional identifier specifying the document id.
    • +
+
Returns:

A deferred whose callback will be invoked with a document.

+
Return type:

twisted.internet.defer.Deferred

+
+
+ +
+
+create_doc_from_json(json, doc_id=None)
+

Create a new document.

+

You can optionally specify the document identifier, but the document +must not already exist. See ‘put_doc’ if you want to override an +existing document. +If the database specifies a maximum document size and the document +exceeds it, create will fail and raise a DocumentTooBig exception.

+ +++ + + + + + + + +
Parameters:
    +
  • json (dict) – The JSON document string
    • +
  • doc_id (str) – An optional identifier specifying the document id.
    • +
+
Returns:

A deferred whose callback will be invoked with a document.

+
Return type:

twisted.internet.defer.Deferred

+
+
+ +
+
+create_index(index_name, *index_expressions)
+

Create a named index, which can then be queried for future lookups.

+

Creating an index which already exists is not an error, and is cheap. +Creating an index which does not match the index_expressions of the +existing index is an error. +Creating an index will block until the expressions have been evaluated +and the index generated.

+ +++ + + + + + + + +
Parameters:
    +
  • index_name (str) – A unique name which can be used as a key prefix
    • +
  • index_expressions –

    index expressions defining the index +information.

    +

    Examples:

    +

    ”fieldname”, or “fieldname.subfieldname” to index alphabetically +sorted on the contents of a field.

    +

    ”number(fieldname, width)”, “lower(fieldname)”

    +
    • +
+
Returns:

A deferred.

+
Return type:

twisted.internet.defer.Deferred

+
+
+ +
+
+create_recovery_code()
+
+ +
+
+default_prefix = '/home/drebs/.config/leap/soledad'
+

A dictionary that holds locks which avoid multiple sync attempts from the +same database replica. The dictionary indexes are the paths to each local +db, so we guarantee that only one sync happens for a local db at a time.

+
+ +
+
+delete_doc(doc)
+

Mark a document as deleted.

+

Will abort if the current revision doesn’t match doc.rev. +This will also set doc.content to None.

+ +++ + + + + + + + +
Parameters:doc (leap.soledad.common.document.Document) – A document to be deleted.
Returns:A deferred.
Return type:twisted.internet.defer.Deferred
+
+ +
+
+delete_index(index_name)
+

Remove a named index.

+ +++ + + + + + + + +
Parameters:index_name (str) – The name of the index we are removing
Returns:A deferred.
Return type:twisted.internet.defer.Deferred
+
+ +
+
+get_all_docs(include_deleted=False)
+

Get the JSON content for all documents in the database.

+ +++ + + + + + + + +
Parameters:include_deleted (bool) – If set to True, deleted documents will be +returned with empty content. Otherwise deleted documents will not +be included in the results.
Returns:A deferred which, when fired, will pass the a tuple +containing (generation, [Document]) to the callback, with the +current generation of the database, followed by a list of all the +documents in the database.
Return type:twisted.internet.defer.Deferred
+
+ +
+
+get_count_from_index(index_name, *key_values)
+

Return the count for a given combination of index_name +and key values.

+

Extension method made from similar methods in u1db version 13.09

+ +++ + + + + + + + +
Parameters:
    +
  • index_name (str) – The index to query
    • +
  • key_values (tuple) – values to match. eg, if you have +an index with 3 fields then you would have: +get_from_index(index_name, val1, val2, val3)
    • +
+
Returns:

A deferred whose callback will be invoked with the count.

+
Return type:

twisted.internet.defer.Deferred

+
+
+ +
+
+get_doc(doc_id, include_deleted=False)
+

Get the JSON string for the given document.

+ +++ + + + + + + + +
Parameters:
    +
  • doc_id (str) – The unique document identifier
    • +
  • include_deleted (bool) – If set to True, deleted documents will be +returned with empty content. Otherwise asking for a deleted +document will return None.
    • +
+
Returns:

A deferred whose callback will be invoked with a document +object.

+
Return type:

twisted.internet.defer.Deferred

+
+
+ +
+
+get_doc_conflicts(doc_id)
+

Get the list of conflicts for the given document.

+

The order of the conflicts is such that the first entry is the value +that would be returned by “get_doc”.

+ +++ + + + + + + + +
Parameters:doc_id (str) – The unique document identifier
Returns:A deferred whose callback will be invoked with a list of the +Document entries that are conflicted.
Return type:twisted.internet.defer.Deferred
+
+ +
+
+get_docs(doc_ids, check_for_conflicts=True, include_deleted=False)
+

Get the JSON content for many documents.

+ +++ + + + + + + + +
Parameters:
    +
  • doc_ids (list) – A list of document identifiers.
    • +
  • check_for_conflicts (bool) – If set to False, then the conflict check +will be skipped, and ‘None’ will be returned instead of True/False.
    • +
  • include_deleted (bool) – If set to True, deleted documents will be +returned with empty content. Otherwise deleted documents will not +be included in the results.
    • +
+
Returns:

A deferred whose callback will be invoked with an iterable +giving the document object for each document id in matching +doc_ids order.

+
Return type:

twisted.internet.defer.Deferred

+
+
+ +
+
+get_from_index(index_name, *key_values)
+

Return documents that match the keys supplied.

+

You must supply exactly the same number of values as have been defined +in the index. It is possible to do a prefix match by using ‘*’ to +indicate a wildcard match. You can only supply ‘*’ to trailing entries, +(eg ‘val’, ‘*’, ‘*’ is allowed, but ‘*’, ‘val’, ‘val’ is not.) +It is also possible to append a ‘*’ to the last supplied value (eg +‘val*’, ‘*’, ‘*’ or ‘val’, ‘val*’, ‘*’, but not ‘val*’, ‘val’, ‘*’)

+ +++ + + + + + + + +
Parameters:
    +
  • index_name (str) – The index to query
    • +
  • key_values (list) – values to match. eg, if you have +an index with 3 fields then you would have: +get_from_index(index_name, val1, val2, val3)
    • +
+
Returns:

A deferred whose callback will be invoked with a list of +[Document].

+
Return type:

twisted.internet.defer.Deferred

+
+
+ +
+
+get_index_keys(index_name)
+

Return all keys under which documents are indexed in this index.

+ +++ + + + + + + + +
Parameters:index_name (str) – The index to query
Returns:A deferred whose callback will be invoked with a list of +tuples of indexed keys.
Return type:twisted.internet.defer.Deferred
+
+ +
+
+get_or_create_service_token(*args, **kwargs)
+

Return the stored token for a given service, or generates and stores a +random one if it does not exist.

+

These tokens can be used to authenticate services.

+
+ +
+
+get_range_from_index(index_name, start_value, end_value)
+

Return documents that fall within the specified range.

+

Both ends of the range are inclusive. For both start_value and +end_value, one must supply exactly the same number of values as have +been defined in the index, or pass None. In case of a single column +index, a string is accepted as an alternative for a tuple with a single +value. It is possible to do a prefix match by using ‘*’ to indicate +a wildcard match. You can only supply ‘*’ to trailing entries, (eg +‘val’, ‘*’, ‘*’ is allowed, but ‘*’, ‘val’, ‘val’ is not.) It is also +possible to append a ‘*’ to the last supplied value (eg ‘val*’, ‘*’, +‘*’ or ‘val’, ‘val*’, ‘*’, but not ‘val*’, ‘val’, ‘*’)

+ +++ + + + + + + + +
Parameters:
    +
  • index_name (str) – The index to query
    • +
  • start_values (tuple) – tuples of values that define the lower bound of +the range. eg, if you have an index with 3 fields then you would +have: (val1, val2, val3)
    • +
  • end_values (tuple) – tuples of values that define the upper bound of the +range. eg, if you have an index with 3 fields then you would have: +(val1, val2, val3)
    • +
+
Returns:

A deferred whose callback will be invoked with a list of +[Document].

+
Return type:

twisted.internet.defer.Deferred

+
+
+ +
+
+list_indexes()
+

List the definitions of all known indexes.

+ +++ + + + + + +
Returns:A deferred whose callback will be invoked with a list of +[(‘index-name’, [‘field’, ‘field2’])] definitions.
Return type:twisted.internet.defer.Deferred
+
+ +
+
+local_db_file_name = 'soledad.u1db'
+
+ +
+
+local_db_path
+
+ +
+
+put_doc(doc)
+

Update a document.

+

If the document currently has conflicts, put will fail. +If the database specifies a maximum document size and the document +exceeds it, put will fail and raise a DocumentTooBig exception.

+

============================== WARNING ============================== +This method converts the document’s contents to unicode in-place. This +means that after calling put_doc(doc), the contents of the +document, i.e. doc.content, might be different from before the +call. +============================== WARNING ==============================

+ +++ + + + + + + + +
Parameters:doc (leap.soledad.common.document.Document) – A document with new content.
Returns:A deferred whose callback will be invoked with the new +revision identifier for the document. The document object will +also be updated.
Return type:twisted.internet.defer.Deferred
+
+ +
+
+raw_sqlcipher_operation(*args, **kw)
+

Run a raw sqlcipher operation in the local database, and return a +deferred that will be fired with None.

+
+ +
+
+raw_sqlcipher_query(*args, **kw)
+

Run a raw sqlcipher query in the local database, and return a deferred +that will be fired with the result.

+
+ +
+
+resolve_doc(doc, conflicted_doc_revs)
+

Mark a document as no longer conflicted.

+

We take the list of revisions that the client knows about that it is +superseding. This may be a different list from the actual current +conflicts, in which case only those are removed as conflicted. This +may fail if the conflict list is significantly different from the +supplied information. (sync could have happened in the background from +the time you GET_DOC_CONFLICTS until the point where you RESOLVE)

+ +++ + + + + + + + +
Parameters:
    +
  • doc (Document) – A Document with the new content to be inserted.
    • +
  • conflicted_doc_revs (list(str)) – A list of revisions that the new content +supersedes.
    • +
+
Returns:

A deferred.

+
Return type:

twisted.internet.defer.Deferred

+
+
+ +
+
+secrets
+

Return the secrets object.

+ +++ + + + + + +
Returns:The secrets object.
Return type:Secrets
+
+ +
+
+secrets_file_name = 'soledad.json'
+
+ +
+
+sync()
+

Synchronize documents with the server replica.

+

This method uses a lock to prevent multiple concurrent sync processes +over the same local db file.

+ +++ + + + + + +
Returns:A deferred lock that will run the actual sync process when +the lock is acquired, and which will fire with with the local +generation before the synchronization was performed.
Return type:twisted.internet.defer.Deferred
+
+ +
+
+sync_lock
+

Class based lock to prevent concurrent syncs using the same local db +file.

+ +++ + + + + + +
Returns:A shared lock based on this instance’s db file path.
Return type:DeferredLock
+
+ +
+
+sync_stats()
+
+ +
+
+syncing
+

Return wether Soledad is currently synchronizing with the server.

+ +++ + + + + + +
Returns:Wether Soledad is currently synchronizing with the server.
Return type:bool
+
+ +
+
+userid
+
+ +
+ +
+
+

Client-side Blobs API

+

The Soledad Client object has a blobmanager property which is responsible +for handling blobs.

+
+
+class leap.soledad.client._db.blobs.BlobManager(local_path, remote, key, secret, user, token=None, cert_file=None)
+

Bases: object

+

The BlobManager can list, put, get, set flags and synchronize blobs stored +in local and remote storages.

+
+
+__init__(local_path, remote, key, secret, user, token=None, cert_file=None)
+

Initialize the blob manager.

+ +++ + + + +
Parameters:
    +
  • local_path (str) – The path for the local blobs database.
    • +
  • remote (str) – The URL of the remote storage.
    • +
  • secret (str) – The secret used to encrypt/decrypt blobs.
    • +
  • user (str) – The uuid of the user.
    • +
  • token (str) – The access token for interacting with remote storage.
    • +
  • cert_file (str) – The path to the CA certificate file.
    • +
+
+
+ +
+
+close()
+
+ +
+
+concurrent_transfers_limit = 3
+
+ +
+
+concurrent_writes_limit = 100
+
+ +
+
+count(namespace='')
+

Count the number of blobs.

+ +++ + + + + + + + +
Parameters:namespace (str) – Optional parameter to restrict operation to a given namespace.
Returns:A deferred that fires with a dict parsed from the JSON +response, which count key has the number of blobs as value. +Eg.: {“count”: 42}
Return type:twisted.internet.defer.Deferred
+
+ +
+
+delete(blob_id, namespace='')
+

Delete a blob from local and remote storages.

+ +++ + + + + + + + +
Parameters:
    +
  • blob_id (str) – Unique identifier of a blob.
    • +
  • namespace (str) – Optional parameter to restrict operation to a given namespace.
    • +
+
Returns:

A deferred that fires when the operation finishes.

+
Return type:

twisted.internet.defer.Deferred

+
+
+ +
+
+fetch_missing(*args, **kwargs)
+

Compare local and remote blobs and fetch what’s missing in local +storage.

+ +++ + + + +
Parameters:namespace (str) – Optional parameter to restrict operation to a given namespace.
+
+ +
+
+get(*args, **kwargs)
+

Get the blob from local storage or, if not available, from the server.

+ +++ + + + +
Parameters:
    +
  • blob_id (str) – Unique identifier of a blob.
    • +
  • namespace (str) – Optional parameter to restrict operation to a given namespace.
    • +
+
+
+ +
+
+get_flags(*args, **kwargs)
+

Get flags from a given blob_id.

+ +++ + + + + + + + +
Parameters:
    +
  • blob_id (str) – Unique identifier of a blob.
    • +
  • namespace (str) – Optional parameter to restrict operation to a given namespace.
    • +
+
Returns:

A deferred that fires with a list parsed from JSON response. +Eg.: [Flags.PENDING]

+
Return type:

twisted.internet.defer.Deferred

+
+
+ +
+
+local_list(namespace='', sync_status=None)
+
+ +
+
+max_retries = 3
+
+ +
+
+put(doc, size, namespace='')
+

Put a blob in local storage and upload it to server.

+ +++ + + + +
Parameters:
    +
  • doc (leap.soledad.client._document.BlobDoc) – A BlobDoc representing the blob.
    • +
  • size (int) – The size of the blob.
    • +
  • namespace (str) – Optional parameter to restrict operation to a given namespace.
    • +
+
+
+ +
+
+refresh_sync_status_from_server(*args, **kwargs)
+
+ +
+
+remote_list(*args, **kwargs)
+

List blobs from server, with filtering and ordering capabilities.

+ +++ + + + + + + + +
Parameters:
    +
  • namespace (str) – Optional parameter to restrict operation to a given namespace.
    • +
  • order_by (str) – Optional parameter to order results. Possible values are: +date or +date - Ascending order (older first) +-date - Descending order (newer first)
    • +
  • filter_flag (leap.soledad.common.blobs.Flags) – Optional parameter to filter listing to results containing the +specified tag.
    • +
  • only_count (bool) – Optional paramter to return only the number of blobs found.
    • +
+
Returns:

A deferred that fires with a list parsed from the JSON +response, holding the requested list of blobs. +Eg.: [‘blob_id1’, ‘blob_id2’]

+
Return type:

twisted.internet.defer.Deferred

+
+
+ +
+
+send_missing(*args, **kwargs)
+

Compare local and remote blobs and send what’s missing in server.

+ +++ + + + +
Parameters:namespace (str) – Optional parameter to restrict operation to a given namespace.
+
+ +
+
+set_flags(blob_id, flags, namespace='')
+

Set flags for a given blob_id.

+ +++ + + + + + + + +
Parameters:
    +
  • blob_id (str) – Unique identifier of a blob.
    • +
  • flags ([leap.soledad.common.blobs.Flags]) – List of flags to be set.
    • +
  • namespace (str) – Optional parameter to restrict operation to a given namespace.
    • +
+
Returns:

A deferred that fires when the operation finishes.

+
Return type:

twisted.internet.defer.Deferred

+
+
+ +
+
+sync(*args, **kwargs)
+
+ +
+ +
+
+

Client-side Attachments API

+

Soledad Documents implement the IDocumentWithAttachment API that associates +Blobs with documents.

+
+
+class leap.soledad.client._document.IDocumentWithAttachment
+

Bases: zope.interface.Interface

+

A document that can have an attachment.

+
+
+delete_attachment()
+

Delete the attachment of this document.

+

The pointer to the attachment will be removed from the document +content, but the document has to be manually put in the database to +reflect modifications.

+ +++ + + + + + +
Returns:A deferred which fires when the attachment has been deleted +from local storage.
Return type:Deferred
+
+ +
+
+download_attachment()
+

Download this document’s attachment.

+ +++ + + + + + +
Returns:A deferred which fires with the state of the attachment after +it’s been downloaded, or NONE if there’s no attachment for +this document.
Return type:Deferred
+
+ +
+
+get_attachment()
+

Return the data attached to this document.

+

If document content contains a pointer to the attachment, try to get +the attachment from local storage and, if not found, from remote +storage.

+ +++ + + + + + +
Returns:A deferred which fires with a file like-object whose content +is the attachment of this document, or None if nothing is +attached.
Return type:Deferred
+
+ +
+
+get_attachment_state()
+

Return the state of the attachment of this document.

+

The state is a member of AttachmentStates and is of one of NONE, +LOCAL, REMOTE or SYNCED.

+ +++ + + + + + +
Returns:A deferred which fires with The state of the attachment of +this document.
Return type:Deferred
+
+ +
+
+is_dirty()
+

Return whether this document’s content differs from the contents stored +in local database.

+ +++ + + + + + +
Returns:A deferred which fires with True or False, depending on +whether this document is dirty or not.
Return type:Deferred
+
+ +
+
+put_attachment(fd)
+

Attach data to this document.

+

Add the attachment to local storage, enqueue for upload.

+

The document content will be updated with a pointer to the attachment, +but the document has to be manually put in the database to reflect +modifications.

+ +++ + + + + + + + +
Parameters:fd (file-like) – A file-like object whose content will be attached to this +document.
Returns:A deferred which fires when the attachment has been added to +local storage.
Return type:Deferred
+
+ +
+
+set_store(store)
+

Set the store used by this file to manage attachments.

+ +++ + + + +
Parameters:store (Soledad) – The store used to manage attachments.
+
+ +
+
+upload_attachment()
+

Upload this document’s attachment.

+ +++ + + + + + +
Returns:A deferred which fires with the state of the attachment after +it’s been uploaded, or NONE if there’s no attachment for this +document.
Return type:Deferred
+
+ +
+ +
+
+
+
+
+
diff --git a/pages/docs/design/soledad.md b/pages/docs/design/soledad.md deleted file mode 100644 index 8925430..0000000 --- a/pages/docs/design/soledad.md +++ /dev/null @@ -1,432 +0,0 @@ -@title = 'Soledad' -@summary = 'A server daemon and client library to provide client-encrypted application data that is kept synchronized among multiple client devices.' -@toc = true - -Introduction -===================== - -Soledad consists of a client library and server daemon that allows applications to securely share a common state among devices. The local application is presented with a simple, document-centric searchable database API. Any data saved to the database by the application is client-encrypted, backed up in the cloud, and synchronized among a user's devices. Soledad is cross-platform, open source, scalable, and features a highly efficient synchronization algorithm. - -Key aspects of Soledad include: - -* **Client and server:** Soledad includes a server daemon and client application library. -* **Client-side encryption:** Soledad puts very little trust in the server by encrypting all data before it is synchronized to the server and by limiting ways in which the server can modify the user's data. -* **Local storage:** All data cached locally is stored in an encrypted database. -* **Document database:** An application using the Soledad client library is presented with a document-centric database API for storage and sync. Documents may be indexed, searched, and versioned. - -The current reference implementation of Soledad is written in Python and distributed under a GPLv3 license. - -Soledad is an acronym of "Synchronization of Locally Encrypted Documents Among Devices" and means "solitude" in Spanish. - -The importance of data availability -------------------------------------------------- - -Users today demand high data availability in their applications. As a user switches from device to device, the expectation is that each application will reflect the same state information across devices. Additionally, if all devices are lost or destroyed, the contemporary user expects to be able to restore her or his application data from the cloud. - -In many ways, data availability has become a necessary precondition for an application to be considered "user friendly." Unfortunately, most applications attempt to provide high data availability by rolling their own custom solution or relying on a third party API, such as Dropbox. This approach is has several drawbacks: the user has no control or access to the data should they wish to switch applications or data providers; custom data synchronizations schemes are often an afterthought, poorly designed, and vulnerable to attack and data breaches; and the user must place total trust in the provider to safeguard her or his information against requests by repressive governments. - -Soledad provides secure data availability in a way that is easy for application developers to incorporate into their code. - -Goals -====================== - -**Security goals** - -* *Client-side encryption:* Before any data is synced to the cloud, it should be encrypted on the client device. -* *Encrypted local storage:* Any data cached in the client should be stored in an encrypted format. -* *Resistant to offline attacks:* Data stored on the server should be highly resistant to offline attacks (i.e. an attacker with a static copy of data stored on the server would have a very hard time discerning much from the data). -* *Resistant to online attacks:* Analysis of storing and retrieving data should not leak potentially sensitive information. -* *Resistance to data tampering:* The server should not be able to provide the client with old or bogus data for a document. - -**Synchronization goals** - -* *Consistency:* multiple clients should all get sync'ed with the same data. -* *Sync flag:* the ability to partially sync data. For example, so a mobile device doesn't need to sync all email attachments. -* *Multi-platform:* supports both desktop and mobile clients. -* *Quota:* the ability to identify how much storage space a user is taking up. -* *Scalable cloud:* distributed master-less storage on the cloud side, with no single point of failure. -* *Conflict resolution:* conflicts are flagged and handed off to the application logic to resolve. - -**Usability goals** - -* *Availability*: the user should always be able to access their data. -* *Recovery*: there should be a mechanism for a user to recover their data should they forget their password. - -**Known limitations** - -These are currently known limitations: - -* The server knows when the contents of a document have changed. -* There is no facility for sharing documents among multiple users. -* Soledad is not able to prevent server from withholding new documents or new revisions of a document. -* Deleted documents are never deleted, just emptied. Useful for security reasons, but could lead to DB bloat. - -**Non-goals** - -* Soledad is not for filesystem synchronization, storage or backup. It provides an API for application code to synchronize and store arbitrary schema-less JSON documents in one big flat document database. One could model a filesystem on top of Soledad, but it would be a bad fit. -* Soledad is not intended for decentralized peer-to-peer synchronization, although the underlying synchronization protocol does not require a server. Soledad takes a cloud approach in order to ensure that a client has quick access to an available copy of the data. - -Related software -================================== - -[Crypton](https://crypton.io/) - Similar goals to Soledad, but in javascript for HTML5 applications. - -[Mylar](https://github.com/strikeout/mylar) - Like Crypton, Mylar can be used to write secure HTML5 applications in javascript. Uniquely, it includes support for homomorphic encryption to allow server-side searches. - -[Firefox Sync](https://wiki.mozilla.org/Services/Sync) - A client-encrypted data sync from Mozilla, designed to securely synchronize bookmarks and other browser settings. - -[U1DB](https://pythonhosted.org/u1db/) - Similar API as Soledad, without encryption. - -Soledad protocol -=================================== - -Document API ------------------------------------ - -Soledad's document API is similar to the [API used in U1DB](http://pythonhosted.org/u1db/reference-implementation.html). - -* Document storage: `create_doc()`, `put_doc()`, `get_doc()`. -* Synchronization with the server replica: `sync()`. -* Document indexing and searching: `create_index()`, `list_indexes()`, `get_from_index()`, `delete_index()`. -* Document conflict resolution: `get_doc_conflicts()`, `resolve_doc()`. - -For example, create a document, modify it and sync: - - sol.create_doc({'my': 'doc'}, doc_id='mydoc') - doc = sol.get_doc('mydoc') - doc.content = {'new': 'content'} - sol.put_doc(doc) - sol.sync() - -Storage secret ------------------------------------ - -The `storage_secret` is a long, randomly generated key used to derive encryption keys for both the documents stored on the server and the local replica of these documents. The `storage_secret` is block encrypted using a key derived from the user's password and saved locally on disk in a file called `.secret`, which contains a JSON structure that looks like this: - - { - "storage_secrets": { - "": { - "kdf": "scrypt", - "kdf_salt": "", - "kdf_length": , - "cipher": "aes256", - "length": , - "secret": "", - } - } - 'kdf': 'scrypt', - 'kdf_salt': '', - 'kdf_length: - } - -The `storage_secrets` entry is a map that stores information about available storage keys. Currently, Soledad uses only one storage key per provider, but this may change in the future. - -The following fields are stored for one storage key: - -* `secret_id`: a handle used to refer to a particular `storage_secret` and equal to `sha256(storage_secret)`. -* `kdf`: the key derivation function to use. Only scrypt is currently supported. -* `kdf_salt`: the salt used in the kdf. The salt for scrypt is not random, but encodes important parameters like the limits for time and memory. -* `kdf_length`: the length of the derived key resulting from the kdf. -* `cipher`: what cipher to use to encrypt `storage_secret`. It must match `kdf_length` (i.e. the length of the derived_key). -* `length`: the length of `storage_secret`, when not encrypted. -* `secret`: the encrypted `storage_secret`, created by `sym_encrypt(cipher, storage_secret, derived_key)` (base64 encoded). - -Other variables: - -* `derived_key` is equal to `kdf(user_password, kdf_salt, kdf_length)`. -* `storage_secret` is equal to `sym_decrypt(cipher, secret, derived_key)`. - -When a client application first wants to use Soledad, it must provide the user's password to unlock the `storage_secret`: - - from leap.soledad.client import Soledad - sol = Soledad( - uuid='', - passphrase='', - secrets_path='~/.config/leap/soledad/.secret', - local_db_path='~/.config/leap/soledad/.db', - server_url='https://', - cert_file='~/.config/leap/providers//keys/ca/cacert.pem', - auth_token='', - secret_id='') # optional argument - - -Currently, the `storage_secret` is shared among all devices with access to a particular user's Soledad database. See [Recovery and bootstrap](#Recovery.and.bootstrap) for how the `storage_secret` is initially installed on a device. - -We don't use the `derived_key` as the `storage_secret` because we want the user to be able to change their password without needing to re-key. - -Document encryption ------------------------- - -Before a JSON document is synced with the server, it is transformed into a document that looks like this: - - { - "_enc_json": "", - "_enc_scheme": "symkey", - "_enc_method": "aes256ctr", - "_enc_iv": "", - "_mac": "", - "_mac_method": "hmac" - } - -About these fields: - -* `_enc_json`: The original JSON document, encrypted and hex encoded. Calculated as: - * `doc_key = hmac(storage_secret[MAC_KEY_LENGTH:], doc_id)` - * `ciphertext = hex(sym_encrypt(cipher, content, doc_key))` -* `_enc_scheme`: Information about the encryption scheme used to encrypt this document (i.e.`pubkey`, `symkey` or `none`). -* `_enc_method`: Information about the block cipher that is used to encrypt this document. -* `_mac`: A MAC to prevent the server from tampering with stored documents. Calculated as: - * `mac_key = hmac(storage_secret[:MAC_KEY_LENGTH], doc_id)` - * `_mac = hmac(doc_id|rev|ciphertext|_enc_scheme|_enc_method|_enc_iv, mac_key)` -* `_mac_method`: The method used to calculate the mac above (currently hmac). - -Other variables: - -* `doc_key`: This value is unique for every document and only kept in memory. We use `doc_key` instead of simply `storage_secret` in order to hinder possible derivation of `storage_secret` by the server. Every `doc_id` is unique. -* `content`: equal to `sym_decrypt(cipher, ciphertext, doc_key)`. - -When receiving a document with the above structure from the server, Soledad client will first verify that `_mac` is correct, then decrypt the `_enc_json` to find `content`, which it saves as a cleartext document in the local encrypted database replica. - -The document MAC includes the document revision and the client will refuse to download a new document if the document does not include a higher revision. In this way, the server cannot rollback a document to an older revision. The server also cannot delete a document, since document deletion is handled by removing the document contents, marking it as deleted, and incrementing the revision. However, a server can withhold from the client new documents and new revisions of a document (including withholding document deletion). - -The currently supported encryption ciphers are AES256 (CTR mode) and XSalsa20. The currently supported MAC method is HMAC with SHA256. - -Document synchronization ------------------------------------ - -Soledad follows the U1DB synchronization protocol, with some changes: - -* Add the ability to flag some documents so they are not synchronized by default (not fully supported yet). -* Refuse to synchronize a document if it is encrypted and the MAC is incorrect. -* Always use `https:///user-` as the synchronization URL. - - - doc = sol.create_doc({'some': 'data'}) - doc.syncable = False - sol.sync() # will not send the above document to the server! - -Document IDs --------------------- - -Like U1DB, Soledad allows the programmer to use whatever ID they choose for each document. However, it is best practice to let the library choose random IDs for each document so as to ensure you don't leak information. In other words, leave the second argument to `create_doc()` empty. - -Re-keying ------------ - -Sometimes there is a need to change the `storage_secret`. Rather then re-encrypt every document, Soledad implements a system called "lazy revocation" where a new `storage_secret` is generated and used for all subsequent encryption. The old `storage_secret` is still retained and used when decrypting older documents that have not yet been re-encrypted with the new `storage_secret`. - -Authentication ------------------------ - -Unlike U1DB, Soledad only supports token authentication and does not support OAuth. Soledad itself does not handle authentication. Instead, this job is handled by a thin HTTP WSGI middleware layer running in front of the Soledad server daemon, which retrieves valid tokens from a certain shared database and compares with the user-provided token. How the session token is obtained is beyond the scope of Soledad. - -Bootstrap and recovery ------------------------------------------- - -Because documents stored on the server's database replica have their contents encrypted with keys based on the `storage_secret`, initial synchronizations of newly configured provider accounts are only possible if the secret is transferred from one device to another. Thus, installation of Soledad in a new device or account recovery after data loss is only possible if specific recovery data has previously been exported and either stored on the provider or imported on a new device. - -Soledad may export a recovery document containing recovery data, which may be password-encrypted and stored in the server, or stored in a safe environment in order to be later imported into a new Soledad installation. - -**Recovery document** - -An example recovery document: - - { - 'storage_secrets': { - '': { - 'kdf': 'scrypt', - 'kdf_salt': '' - 'kdf_length': - 'cipher': 'aes256', - 'length': , - 'secret': '', - }, - }, - 'kdf': 'scrypt', - 'kdf_salt': '', - 'kdf_length: , - '_mac_method': 'hmac', - '_mac': '' - } - -About these fields: - -* `secret_id`: a handle used to refer to a particular `storage_secret` and equal to `sha256(storage_secret)`. -* `kdf`: the key derivation function to use. Only scrypt is currently supported. -* `kdf_salt`: the salt used in the kdf. The salt for scrypt is not random, but encodes important parameters like the limits for time and memory. -* `kdf_length`: the length of the derived key resulting from the kdf. -* `length`: the length of the secret. -* `secret`: the encrypted `storage_secret`. -* `cipher`: what cipher to use to encrypt `secret`. It must match `kdf_length` (i.e. the length of the `derived_key`). -* `_mac_method`: The method used to calculate the mac above (currently hmac). -* `_mac`: Defined as `hmac(doc_id|rev|ciphertext, doc_key)`. The purpose of this field is to prevent the server from tampering with the stored documents. - -Currently, scrypt parameters are: - - N (CPU/memory cost parameter) = 2^14 = 16384 - p (paralelization parameter) = 1 - r (length of block mixed by SMix()) = 8 - dkLen (length of derived key) = 32 bytes = 256 bits - -Other fields we might want to include in the future: - -* `expires_on`: the month in which this recovery document should be purged from the database. The server may choose to purge documents before their expiration, but it should not let them linger after it. -* `soledad`: the encrypted `soledad.json`, created by `sym_encrypt(cipher, contents(soledad.json), derived_key)` (base64 encoded). -* `reset_token`: an optional encrypted password reset token, if supported by the server, created by `sym_encrypt(cipher, password_reset_token, derived_key)` (base64 encoded). The purpose of the reset token is to allow recovery using the recovery code even if the user has forgotten their password. It is only applicable if using recovery code method. - -**Recovery database** - -In order to support easy recovery, the Soledad client stores a recovery document in a special recovery database. This database is shared among all users. - -The recovery database supports two functions: - -* `get_doc(doc_id)` -* `put_doc(doc_id, recovery_document_content)` - -Anyone may preform an unauthenticated `get_doc` request. To mitigate the potential attacks, the response to queries of the discovery database must have a long delay of X seconds. Also, the `doc_id` is very long (see below). - -Although the database is shared, the user must authenticate via the normal means before they are allowed to put a recovery document. Because of this, a nefarious server might potentially record which user corresponds to which recovery documents. A well behaved server, however, will not retain this information. If the server supports authentication via blind signatures, then this will not be an issue. - - -**Recovery code (yet to be implemented)** - -We intend to offer data recovery by specifying username and a recovery code. The choice of type of recovery (using password or a recovery code) must be made in advance of attempting recovery (e.g. at some point after the user has Soledad successfully running on a device). - -About the optional recovery code: - -* The recovery code should be randomly generated, at least 16 characters in length, and contain all lowercase letters (to make it sane to type into mobile devices). -* The recovery code is not stored by Soledad. When the user needs to bootstrap a new device, a new code is generated. To be used for actual recovery, a user will need to record their recovery code by printing it out or writing it down. -* The recovery code is independent of the password. In other words, if a recovery code is generated, then a user changes their password, the recovery code is still be sufficient to restore a user's account even if the user has lost the password. This feature is dependent on the server supporting a password reset token. Also, generating a new recovery code does not affect the password. -* When a new recovery code is created, and new recovery document must be pushed to the recovery database. A code should not be shown to the user before this happens. -* The recovery code expires when the recovery database record expires (see below). - -The purpose of the recovery code is to prevent a compromised or nefarious Soledad service provider from decrypting a user's storage. The benefit of a recovery code over the user password is that the password has a greater opportunity to be compromised by the server. Even if authentication is performed via Secure Remote Password, the server may still perform a brute force attack to derive the password. - -Reference implementation of client -=================================== - -https://github.com/leapcode/soledad - -Dependencies: - -* [U1DB](https://launchpad.net/u1db) provides an API and protocol for synchronized databases of JSON documents. -* [SQLCipher](http://sqlcipher.net/) provides a block-encrypted SQLite database used for local storage. -* python-gnupg -* scrypt -* pycryptopp - -Local storage --------------------------- - -U1DB reference implementation in Python has an SQLite backend that implements the object store API over a common SQLite database residing in a local file. To allow for encrypted local storage, Soledad adds a SQLCipher backend, built on top of U1DB's SQLite backend, which adds [SQLCipher API](http://sqlcipher.net/sqlcipher-api/) to U1DB. - -**Responsibilities** - -The SQLCipher backend is responsible for: - -* Providing the SQLCipher API for U1DB (`PRAGMA` statements that control encryption parameters). -* Guaranteeing that the local database used for storage is indeed encrypted. -* Guaranteeing secure synchronization: - * All data being sent to a remote replica is encrypted with a symmetric key before being sent. - * Ensure that data received from remote replica is indeed encrypted to a symmetric key when it arrives, and then that it is decrypted before being included in the local database replica. -* Correctly representing and handling new Document properties (e.g. the `sync` flag). - -Part of the Soledad `storage_key` is used directly as the key for the SQLCipher encryption layer. SQLCipher supports the use of a raw 256 bit keys if provided as a 64 character hex string. This will skip the key derivation step (PBKDF2), which is redundant in our case. For example: - - sqlite> PRAGMA key = "x'2DD29CA851E7B56E4697B0E1F08507293D761A05CE4D1B628663F411A8086D99'"; - -**Classes** - -SQLCipher backend classes: - -* `SQLCipherDatabase`: An extension of `SQLitePartialExpandDatabase` used by Soledad Client to store data locally using SQLCipher. It implements the following: - * Need of a password to instantiate the db. - * Verify if the db instance is indeed encrypted. - * Use a LeapSyncTarget for encrypting content before synchronizing over HTTP. - * "Syncable" option for documents (users can mark documents as not syncable, so they do not propagate to the server). - -Encrypted synchronization target --------------------------------------------------- - -To allow for database synchronization among devices, Soledad uses the following conventions: - -* Centralized synchronization scheme: Soledad clients always sync with a server, and never between themselves. -* The server stores its database in a CouchDB database using a REST API over HTTP. -* All data sent to the server is encrypted with a symmetric secret before being sent. Note that this ensures all data received by the server and stored in the CouchDB database has been encrypted by the client. -* All data received from the server is validated as being an encrypted blob, and then is decrypted before being stored in local database. Note that the local database provides a new encryption layer for the data through SQLCipher. - -**Responsibilities** - -Provide sync between local and remote replicas: - -* Encrypt outgoing content. -* Decrypt incoming content. - -**Classes** - -Synchronization-related classes: - -* `SoledadSyncTarget`: an extension of `HTTPSyncTarget` modified to encrypt documents' content before sending them to the network and to have more control of the syncing process. - -Reference implementation of server -====================================================== - -https://github.com/leapcode/soledad - -Dependencies: - -* [CouchDB](https://couchdb.apache.org/] for server storage, via [python client library](https://pypi.python.org/pypi/CouchDB/0.8). -* [Twisted](http://twistedmatrix.com/trac/) to run the WSGI application. -* scrypt -* pycryptopp -* PyOpenSSL - -CouchDB backend -------------------------------- - -In the server side, Soledad stores its database replicas in CouchDB servers. Soledad's CouchDB backend implementation is built on top of U1DB's `CommonBackend`, and stores and fetches data using a remote CouchDB server. It lacks indexing first because we don't need that functionality on server side, but also because if not very well done, it could lack sensitive information about document's contents. - -CouchDB backend is responsible for: - -* Initializing and maintaining the following U1DB replica data in the database: - * Transaction log. - * Conflict log. - * Synchronization log. -* Mapping the U1DB API to CouchDB API. - -**Classes** - -* `CouchDatabase`: A backend used by Soledad Server to store data in CouchDB. -* `CouchSyncTarget`: Just a target for syncing with Couch database. -* `CouchServerState`: Interface of the WSGI server with the CouchDB backend. - -WSGI Server ------------------------------------------ - -The U1DB server reference implementation provides for an HTTP API backed by SQLite databases. Soledad extends this with token-based auth HTTP access to CouchDB databases. - -* Soledad makes use of `twistd` from Twisted API to serve its WSGI application. -* Authentication is done by means of a token. -* Soledad implements a WSGI middleware in server side that: - * Uses the provided token to verify read and write access to each user's private databases and write access to the shared recovery database. - * Allows reading from the shared remote recovery database. - * Uses CouchDB as its backend. - -**Classes** - -* `SoledadAuthMiddleware`: implements the WSGI middleware with token based auth as described before. -* `SoledadApp`: The WSGI application. For now, not different from `u1db.remote.http_app.HTTPApp`. - -**Authentication** - -Soledad Server authentication middleware controls access to user's private databases and to the shared recovery database. Soledad client provides a token for Soledad server that can check the validity of this token for this user's session by querying a certain database. - -A valid token for this user's session is required for: - -* Read and write access to this user's database. -* Read and write access to the shared recovery database. - -Tests -=================== - -To be sure the new implemented backends work correctly, we included in Soledad the U1DB tests that are relevant for the new pieces of code (backends, document, http(s) and sync tests). We also added specific tests to the new functionalities we are building. -- cgit v1.2.3