summaryrefslogtreecommitdiff
path: root/common/src/leap/soledad
AgeCommit message (Collapse)Author
2016-08-01[bug] retry allocation of gen instead of using a lockdrebs
The use of a lock to allocate the next generation of a change in couch backend suffers from at least 2 problems: 1. all modification to the couch database would have to be made through a soledad server entrypoint, otherwise the lock would have no effect. 2. introducing a lock makes code uglier, harder to debug, and prone to undesired blocks. The solution implemented by this commit is not so elegant, but works for what we need right now. Now, concurrent threads updating the couch database will race for the allocation of a new generation, and retry when they fail to do so. There's no high risk of getting blocked for too much time in the while loop because (1) there's always one thread that wins (what makes the expected number of retries to be N/2 if N is the number of concurrent threads), and (2) the number of concurrent attempts to update the user database is limited by the number of devices syncing at the same time.
2016-08-01[feat] use couch _all_docs for get_docs() and get_all_docs()drebs
The previous solution would make use of concurrent get's to couch backend in a pool of threads to implement the get_docs() and get_all_docs() CouchDatabase backend methods. This commit replaces those by a simpler implementation use the `_all_docs` couchdb view api. It passes all needed IDs to the view and r etrieves all documents with content in the same request. A comparison between both implementations shows an improvement of at least 15 times for large number of documents. The table below shows the time for different implementations of get_all_docs() for different number of documents and threads versus _all_docs implementation: +-------+-----------------+------------------+-------------+ | | threads | _all_docs | improvement | +-------+-----------------+------------------+-------------+ | 10 | 0.0728030204773 | 0.00782012939453 | 9.3 | | 100 | 0.609349966049 | 0.0377721786499 | 16.1 | | 1000 | 5.86522197723 | 0.370730876923 | 15.8 | | 10000 | 66.1713931561 | 3.61764383316 | 18.3 | +-------+-----------------+------------------+-------------+
2016-08-01[refactor] simplify couch whats_changed calculationdrebs
2016-08-01[bug] use couch lock to atomize saving of documentdrebs
2016-08-01[feat] standardize metadata storage in couch backend.drebs
2016-08-01[feat] use a lock for updating couch gen datadrebs
2016-08-01[bug] fix order of multipart serialization when writing to couchdrebs
The couch backend makes use of attachments and multipart structure for writing the document to the couch database. For that to work, the order in which attachments are described must match the actual order in which attachments are written to the couch http stream. This was not being properly taken care of, and eventually the json serializer was arbitrarilly ordering the attachments description in a way that it didn't match the actual order of attachments writing. This commit fixes that by using json.dumps() sort_keys parameter and making sure conflicts are always written before content.
2016-08-01[feat] remove usage of design documents in couchdrebs
Design documents are slow and we already have alternatives to all uses we used to make of them, so this commit completelly removes all usage of design documents.
2016-07-25[feat] do not use couch views for sync metadatadrebs
When compared to plain couch document get, the use of the simplest view functions takes around double the time, while the use of the simplest list function can take more than 8 times: get 100 docs: total: 0.440337 secs mean: 0.004403 query 100 views: total: 0.911425 secs mean: 0.009114 query 100 lists: total: 3.711537 secs mean: 0.037115 Besides that, the current implementation of sync metadata storage over couch is dependent of timestamps of document puts, what can lead to metadata corruption if the clock of the system is changed for any reason. Because of these reasons, we seek to change the implementation of database metadata. This commit implements the storage of transaction log data on couch documents with special ids, in the form "gen-xxxxxxxxxx", where the x's are replaced by the generation index. Each generation document holds a dictionary containing the generation, doc_id and transaction_id for the changed document. For each modified document, a generation document is inserted holding the transaction metadata.
2016-07-25[feat] use _local couch docs for metadata storagedrebs
2016-07-13[style] pep8Kali Kaneko
2016-07-12[pkg] remove unneeded oauth codedrebs
2016-07-12[test] toxify testsdrebs
- move tests to root directory - split tests in different subdirectories - setup a small package with common test dependencies in /testing/test_soledad - add tox.ini that will: - install the test_soledad package and other test dependencies - install soledad common, client, server from the repository - run tests contianed in /testing/tests directory using pytest This commit also removes all oauth code from tests, as we have removed the u1db dependency (by importing it into the repo and naming it l2db) and don't neet oauth at all right now.
2016-07-12[refactor] make tests use l2db submoduleKali Kaneko
From this moment on, we embed a fork of u1db called l2db.
2016-07-12[refactor] fork u1dbKali Kaneko
2016-06-22pep8Kali Kaneko
2016-06-22[bug] fix test processing orderNavaL
This moves the reactor time to the loopingcall period. This is necessary as the decryption is now deferred to a thread. The test will exit before the task is executed otherwise.
2016-06-22[style] pep8 compatibility: indent and white spaceNavaL
It was breaking E126 and E202 before
2016-06-08[tests] avoid using get_all_docs on assertsVictor Shyba
EncryptedSyncTestCase.test_sync_very_large_files is still getting an excessive amount of memory on very slow machines (specially on old spinning magnetic disks). This commit checks each doc at a time instead of getting them all. More refinement is necessary for this test to pass on any machine.
2016-06-06[style] remove misused lambdasTulio Casagrande
Pep8 was warning about assignment of lambdas. These lambdas should be partials
2016-06-06[test] use inline deferreds in test for change passphrasedrebs
2016-06-06[test] turn test for _gen_secret into many unit testsdrebs
2016-06-06[bug] ensures docs_received table has the sync_id columnNavaL
For the case where the user already has data synced, this commit will migrate the docs_received table to have the column sync_id. That is required by the refactoring in the previous commits.
2016-06-06[refactor] encdecpool queries and testingVictor Shyba
This commit adds tests for doc ordering and encdecpool control (start/stop). Also optimizes by deleting in batch and checking for a sequence in memory before asking the local staging for documents.
2016-05-23[refactor] remove user_id argument from Soledad initCaio Carrara
The constructor method of Soledad was receiving two arguments for user id. One of them was optional with None as default. It could cause an inconsistent state with uuid set but userid unset. This change remove the optional user_id argument from initialization method and return the uuid if anyone call Soledad.userid method.
2016-05-16[style] pep8Kali Kaneko
2016-04-26[refactor] cleanup bootstrap processdrebs
2016-04-26[refactor] remove shared db locking from testsdrebs
2016-04-26[refactor] remove shared db locking from clientdrebs
Shared db locking was used to avoid the case in which two different devices try to store/modify remotelly stored secrets at the same time. We want to avoid remote locks because of the problems they create, and prefer to crash locally. For the record, we are currently using the user's password to encrypt the secrets stored in the server, and while we continue to do this we will have to re-encrypt the secrets and update the remote storage whenever the user changes her password.
2016-04-01[pkg] update to versioneer 0.16Kali Kaneko
2016-01-21[Fix] fix concurrency problem in test_sync_deferredFolker Bernitt
- Use dbsyncer (SQLCipherU1DBSync) instead of SQLCipherDatabase as only the first one supports multiple threads while syncing and is actually used by Soledad.sync
2015-12-22[docs] incomplete doc for security config parameterVictor Shyba
database_security parameter was either undocumented or incomplete. This commit adds a few more doc to make it consistent with latest changes. Closes #7689
2015-12-14[fix] remove trailing whitespace to please pep8Christoph Kluenter
2015-12-14[bug] fix failing tests after last events modificationKali Kaneko
2015-12-03[feat] set default to False on batching for nowVictor Shyba
All batching code has no effect by default with this commit. Since we know that this is a dangerous new feature we will enable them only on our test servers and check them manually before setting it as default or adding more configuration features. Use SyncTarget and server conf file to enable it for testing.
2015-12-03[feat] generation caching during a batchVictor Shyba
Generation cache was removed for simple processing and it should not got back, but during a batch the server wont change its generation. So a little trick to hold that temporary information until batch finishes is needed.
2015-12-03[feat] add configuration to disable batchingVictor Shyba
Batch support is optional. This commit adds a 'batching' configuration option to disable it.
2015-12-03[feat] checks staged docs inside batchVictor Shyba
This commit adds checking for consistency on batch. When a doc is needed during a batched sync and it doesnt exists on database, current code will make a partial batch to avoid processing like it doesnt exist.
2015-12-03[feat] put all docs at onceVictor Shyba
Using _bulk_docs api from CouchDB we can put all docs at a single request. Also, prefetching all ids removes the need to HEAD requests during the batch.
2015-12-03[feat] prepare server to handle batchesVictor Shyba
Created two methods on the backend to start and finish a batch. A dict of callbacks is available to defer actions for the last document, allowing temporary (changing often) metadata to be recorded only once. Using those methods we will also be able to put all docs in one go on the CouchDatabase implementation, but that is another step.
2015-12-01[test] shuffle and decrypts 5 docs 3 timesVictor Shyba
On real usage the docs will arrive shuffled and pool will be reused after many decrypts. This test asserts that everything ended up clear between execution and no inconsistency is left over for the next run.
2015-12-01[tests] encdec pool is being reused, adding a testVictor Shyba
This new test case will run the single insert test 5 times to ensure that using the same pool again is fine. This is needed due failures to shutdown the pool or inconsistency between syncs.
2015-11-24[bug] generation_info cant be cached per replicaVictor Shyba
This info can be changed by another syncing replica and would not reflect real database generation. That would be ok inside of the same sync, but can cause trouble on concurrent syncs. The other calls are ok, since they hold info that doesnt change during concurrent syncs or are only read/write by the replica syncing. A global cache could fit better this removed case, but for now let's stay on the safe side.
2015-11-23[bug] BackendNotReadyError breaks without statusBruno Wagner
The new BackendNotReadyError didn't have a status or a wire description, because of that, when you tried to use the leap.soledad.server package it would break trying to import this exception (because the annotation tries to use this variable). This was preventing soledad server from starting at all, after this change it works again
2015-11-23[bug] put a monkeypatch back for bigcouchVictor Shyba
Current code was tested on couch 1.6 and a monkeypatch got removed during refactor. This commit re-adds it, but in a separate module that is intended to hold temporary code for compatibility that can be removed on version upgrades.
2015-11-20[bug] do not register exception, breaksKali Kaneko
since the exception doesn't have the code and description, it breaks. we don't need those since the couch child exception describes them.
2015-11-19[bug] fix indirect dependency on couchdb for clientKali Kaneko
by subclassing the MissingDesignDocError, we don't have to import the soledad.common.couch submodule into the soledad.client.sync - Resolves: #7626
2015-11-03[refactor] remove exception based logicVictor Shyba
Creating a database was using a unnecessary complex try/except logic. Simplifying it should make the purpose more clear.
2015-11-03[style] pep8Victor Shyba
2015-11-03[feat] add caching to a place that was missingVictor Shyba
When retrieving other replica info the cache wasnt being used, but when saving it was. This commit applies caching on get as well.