From 01c79bab6aa9629eabcb5ff8599e257a1af3a9eb Mon Sep 17 00:00:00 2001 From: drebs Date: Fri, 27 Apr 2018 19:32:38 -0300 Subject: Add blog post for improved data sync --- pages/about-us/news/2018/improved-data-sync.md | 64 ++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 pages/about-us/news/2018/improved-data-sync.md (limited to 'pages/about-us/news') diff --git a/pages/about-us/news/2018/improved-data-sync.md b/pages/about-us/news/2018/improved-data-sync.md new file mode 100644 index 0000000..ebf94df --- /dev/null +++ b/pages/about-us/news/2018/improved-data-sync.md @@ -0,0 +1,64 @@ +@title = 'Improved data synchronization' +@author = 'drebs' +@posted_at = '2018-04-27' +@more = true +@preview_image = '/img/pages/mozz/ascii.png' + +Last year, LEAP [was awarded a grant from the Mozilla Open Source Support +program](https://leap.se/en/about-us/news/2017/mozilla-thanks) to [improve its +encrypted email +system](https://leap.se/en/about-us/news/2017/perf-improvements-soledad). We +would like to share with you some of the work that has been done as a result of +that, and also thank Mozilla for its initiative. + +The general idea was to implement separation of data and metadata for encrypted +email storage and synchronization. That would allow for two fundamental +improvements: a better user experience (because the list of email messages +would then be able to be loaded before the actual contents of the messages), +and a better usage of systems resources (encrypted messages were previously +being treated as plaintext strings and thus consumed lots of unneeded memory +and cpu). + +The [roadmap](https://0xacab.org/leap/soledad/wikis/2017-roadmap) included the +creation of a platorm to assess the performance impacts of proposed changes, +the actual implementation of the separation of data and metadata during +synchronization, and some specific performance and scalability improvements of +the sync server implementation. + +The live benchmarking platform that was created proved to be very useful to fix +pre-existing performance issues and avoid unexpected performance bugs +introduced during development. A CI environment was used to run performance +test code for every commit made in the repository, and created graphs for +resource consumption (time taken, and amount of memory and cpu used during +execution). The quality of decisions made during development and the resulting +code were improved by having a live performance-wise feedback about proposed +changes. + +As for data and metadata separation, changes to +[Soledad](https://leap.se/soledad) were made so that encrypted blobs can now be +delivered by external aplications directly to the user's database. The main use +case for this is for delivery of incoming email. LEAP's architecture is built +so that unencrypted email messages get encrypted before being delivered to the +user's mailbox. Metadata will soon be synchronized to the user's device, and +the user will be able to see the information associated with the message +(origin, date, subject, etc). But the actual content of the message will only +be downloaded and decrypted on demand, when the user clicks to read the +message. + +Other than those major developments, the MOSS grant has also alllowed us to +experiment with multiple possibilities of performance improvements, as for +example server parallelization, usage of different asynchronous libraries, +Python 3 comparison, usage of proxies for load balancing, among others. + +We very much appreciate the support from Mozilla that has allowed for +concentrated focus on Soledad. The work done within this contract has allowed +for the creation of a system to handle encryption and transfer of binary data, +leading to a much faster and scalable email solution. The critical +implementation of a benchmarking system produced clear data to find +bottlenecks, drive development, and allow for targeted changes to increase +speed and scalability of data encryption and transfer chain. + +LEAP looks forward to applying that work to Bitmask Mail where end users will +reap the benefits. Development of this new blobs system and its future +integration with Bitmask Mail will be a critical step in bringing easy to +deploy truly secure email to a mass market. -- cgit v1.2.3