@title = 'Improved data synchronization'
@author = 'drebs'
@posted_at = '2018-04-27'
@more = true
@preview_image = '/img/pages/mozz/ascii.png'

Last year, LEAP [was awarded a grant from the Mozilla Open Source Support
program](https://leap.se/en/about-us/news/2017/mozilla-thanks) to [improve its
encrypted email
system](https://leap.se/en/about-us/news/2017/perf-improvements-soledad). We
would like to share with you some of the work that has been done as a result of
that, and also thank Mozilla for its initiative.

The general idea was to implement separation of data and metadata for encrypted
email storage and synchronization. That would allow for two fundamental
improvements: a better user experience (because the list of email messages
would then be able to be loaded before the actual contents of the messages),
and a better usage of systems resources (encrypted messages were previously
being treated as plaintext strings and thus consumed lots of unneeded memory
and cpu).

The [roadmap](https://0xacab.org/leap/soledad/wikis/2017-roadmap) included the
creation of a platorm to assess the performance impacts of proposed changes,
the actual implementation of the separation of data and metadata during
synchronization, and some specific performance and scalability improvements of
the sync server implementation.

The live benchmarking platform that was created proved to be very useful to fix
pre-existing performance issues and avoid unexpected performance bugs
introduced during development. A CI environment was used to run performance
test code for every commit made in the repository, and created graphs for
resource consumption (time taken, and amount of memory and cpu used during
execution). The quality of decisions made during development and the resulting
code were improved by having a live performance-wise feedback about proposed
changes.

As for data and metadata separation, changes to
[Soledad](https://leap.se/soledad) were made so that encrypted blobs can now be
delivered by external aplications directly to the user's database. The main use
case for this is for delivery of incoming email. LEAP's architecture is built
so that unencrypted email messages get encrypted before being delivered to the
user's mailbox. Metadata will soon be synchronized to the user's device, and
the user will be able to see the information associated with the message
(origin, date, subject, etc). But the actual content of the message will only
be downloaded and decrypted on demand, when the user clicks to read the
message.

Other than those major developments, the MOSS grant has also alllowed us to
experiment with multiple possibilities of performance improvements, as for
example server parallelization, usage of different asynchronous libraries,
Python 3 comparison, usage of proxies for load balancing, among others.

We very much appreciate the support from Mozilla that has allowed for
concentrated focus on Soledad. The work done within this contract has allowed
for the creation of a system to handle encryption and transfer of binary data,
leading to a much faster and scalable email solution. The critical
implementation of a benchmarking system produced clear data to find
bottlenecks, drive development, and allow for targeted changes to increase
speed and scalability of data encryption and transfer chain.

LEAP looks forward to applying that work to Bitmask Mail where end users will
reap the benefits. Development of this new blobs system and its future
integration with Bitmask Mail will be a critical step in bringing easy to
deploy truly secure email to a mass market.