Improved data synchronization

Last year, LEAP was awarded a grant from the Mozilla Open Source Support program to improve its encrypted email system. We would like to share with you some of the work that has been done as a result of that, and also thank Mozilla for its initiative.

The general idea was to implement separation of data and metadata for encrypted email storage and synchronization. That would allow for two fundamental improvements: a better user experience (because the list of email messages would then be able to be loaded before the actual contents of the messages), and a better usage of systems resources (encrypted messages were previously being treated as plaintext strings and thus consumed lots of unneeded memory and cpu).

The roadmap included the creation of a platorm to assess the performance impacts of proposed changes, the actual implementation of the separation of data and metadata during synchronization, and some specific performance and scalability improvements of the sync server implementation.

The live benchmarking platform that was created proved to be very useful to fix pre-existing performance issues and avoid unexpected performance bugs introduced during development. A CI environment was used to run performance test code for every commit made in the repository, and created graphs for resource consumption (time taken, and amount of memory and cpu used during execution). The quality of decisions made during development and the resulting code were improved by having a live performance-wise feedback about proposed changes.

As for data and metadata separation, changes to Soledad were made so that encrypted blobs can now be delivered by external aplications directly to the user’s database. The main use case for this is for delivery of incoming email. LEAP’s architecture is built so that unencrypted email messages get encrypted before being delivered to the user’s mailbox. Metadata will soon be synchronized to the user’s device, and the user will be able to see the information associated with the message (origin, date, subject, etc). But the actual content of the message will only be downloaded and decrypted on demand, when the user clicks to read the message.

Other than those major developments, the MOSS grant has also alllowed us to experiment with multiple possibilities of performance improvements, as for example server parallelization, usage of different asynchronous libraries, Python 3 comparison, usage of proxies for load balancing, among others.

We very much appreciate the support from Mozilla that has allowed for concentrated focus on Soledad. The work done within this contract has allowed for the creation of a system to handle encryption and transfer of binary data, leading to a much faster and scalable email solution. The critical implementation of a benchmarking system produced clear data to find bottlenecks, drive development, and allow for targeted changes to increase speed and scalability of data encryption and transfer chain.

LEAP looks forward to applying that work to Bitmask Mail where end users will reap the benefits. Development of this new blobs system and its future integration with Bitmask Mail will be a critical step in bringing easy to deploy truly secure email to a mass market.