summaryrefslogtreecommitdiff
path: root/common/src/leap
AgeCommit message (Collapse)Author
2015-09-24[feat] use one doc per remote replica in sync logdrebs
Before this change, we used a complicated update handler for storing the sync state on the couchdb backend. That update handler was implemented as an attempt to make couchdb take care of some validation for the update of the sync log during the sync exchange, mainly to allow concurrent received documents insertion during a sync. Right now we rely on the remote sending one document at a time and do not support concurrent insertions in the remote database backed by couch. Because of that, the code removed by this commit was unneeded. And more: it was a bottleneck of the sync process because we were writing to an unique file and using unnecessary couch design docs processing for that. So this commit both simplifies the storage of remote sync and removes a bottleneck of the sync process. Conflicts: common/src/leap/soledad/common/couch.py common/src/leap/soledad/common/tests/test_couch.py
2015-09-24[feat] full caching for each sync_id sessionVictor Shyba
The CouchDB backend implementation was accessing CouchDB too many times for the same values. Those values are known inside the same sync_id, which is the id of current sync session. This commit adds caching for all redundant calls to Couch inside the same sync_id for each replica. Refactoring is still needed, but for now couch.py works normally as if caching is not present, while sync.py injects the cache as a attribute to enable it. This needs a simpler implementation.
2015-09-24[feat] adds caching for other gen and trans idVictor Shyba
There are two functions in couch.py used to save and retrieve the last know gen and trans id for the syncing replica. The get function is called very often, but is only set on one point. Added a simple caching to avoid queying couch for a value that we already have. If cache is empty, it just query as usual and fills it.
2015-09-24[refactor] uses ThreadPool instead of own implementationVictor Shyba
Python has a native ThreadPool implementation that fits our needs. Changing it to use this instead and making some calls simpler.
2015-09-24[refactor] couch.py should use superclass methodsVictor Shyba
_put_doc_if_newer is implemented on CommonBackend already. This was copied over to CouchBackend just to add ensure conflicts. We can do this before calling the super method instead.
2015-09-17[tests] test for ensure ddocs independentlyVictor Shyba
This tests the previous fix on ensuring a db that is missing a doc other than 'docs'.
2015-09-17[bug] ensure needs to check into all design docsVictor Shyba
This code only checks for 'docs' presence, while we have 3 design documents. If one of them is missing, but 'docs' is not, then it will not ensure the others. This is needed to properly ensure ddocs on create command line script.
2015-09-16Merge remote-tracking branch 'leapcode/pr/271' into developKali Kaneko
2015-09-16[test] fix tests with the new emit_asyncRuben Pollan
2015-09-16[tests] remove remaining dbs checkVictor Shyba
This was used during db isolation to make sure that everything created was destroyed, but it fails with -j (multiprocess). Removing it allows parallelism.
2015-09-14[style] pep8 fixesVictor Shyba
2015-09-13[tests] test_couch does not need a serverVictor Shyba
Removing unused code as this test case does not need a server.
2015-09-13[tests] change big tearDown for simple addCleanupVictor Shyba
Big tearDown logic can be replaced by a simple addCleanup. Also remove unused imports and fix a small typo on a database cleanup check.
2015-09-12[tests] subclass instead of copy test codeVictor Shyba
This test only defines a set of different scenarios, all other methods are the same as this subclass.
2015-09-11[tests] db3 is expected to be an attributeVictor Shyba
self.db3 is closed on tearDown. This test was creating it as a local variable, making close possibly fail.
2015-09-11[tests] simplify make_app and getTarget codeVictor Shyba
It was hardcoded for 'test', but the database name is now random. What is useful for test code is the replica_uid, database name for the SyncTarget is now coming from database name.
2015-09-11[tests] isolate LockResource tests using a mockVictor Shyba
'shared' has to be used as a DB name just because of a constant, but it is used on only one point. This changes mock this point to have unique names for better tests isolation. 'tokens' was removed as unnecessary.
2015-09-11[tests] extract db creation with doc ensure methodVictor Shyba
This was a duplicate, but also was getting on the way to improve isolation. With this small refactor it should be cleaner and have unique names.
2015-09-11[tests] use addCleanup to ensure db deletionVictor Shyba
Test case has a addCleanup method, which provides a way to clean resources up and express this need as soon as you create. We are now using it to simplify some logic on database deletion during the test and to make sure that as soon as it gets created a cleanup is there to delete after.
2015-09-11[tests] isolate database names, use uuidVictor Shyba
We are using a single CouchDB install, which may cause tests to overlap since many of them uses the same database name, hurting isolation. This change tries to use uuid on most of it. Also changes for couch_url and couch_port introduced by removal of CouchDB process.
2015-09-11[tests] get rid of CouchDB python processVictor Shyba
This process per test gives a lot of headache, this is why we are removing it. With it we would need to try to start and stop properly on each test case. This fails badly when a test fail and, depending on how it fails, it freezes my pc. Also, it is very heavy for a CI to run a database process for each test case.
2015-08-28[style] pep8 fixesKali Kaneko
2015-08-27[tests] fix variable name from refactorVictor Shyba
From: [refactor] removing getters and setters from couch.py _couch_url was a private variable with getter and setter doing the same as a public variable. It is accessed all over the code, so being private with getters and setters didnt make sense. This commit fixes the tests to also follow this style from now on.
2015-08-26[refactor] Move constructor, use isinstanceVictor Shyba
isinstance is better, as kaliy pointed out, and the constructor is also in a safer place on __init__.py to be explicit. Also re-apply a change from last rebase;
2015-08-26[bug] fixes small issues pointed by drebsVictor Shyba
* file headers * variable names * missing docstrings * prune_conflicts ** extra: tests failed on a 1-based index bug
2015-08-26[bug] check type before processingVictor Shyba
Necessary methods are on CouchDocument, but we accept a Document as well, in this case self._factory is needed. Will be simpler soon.
2015-08-26[refactor] _process_incoming_doc is a function nowVictor Shyba
This was being calculated inside CouchDatabase, but it is not a persistence responsibility. It clearly doesn't belong to this persistence layer and seeing both sides separated allow us to work better on both parts.
2015-08-26[refactor] prune conflicts is Document's responsibilityVictor Shyba
2015-08-26[refactor] simplify conflicts managementVictor Shyba
Adding, removing and checking conflicts is an operation done by the model, the Database shouldn't be aware of that. Fetching and saving also is not model's responsability. Repetition remove as well.
2015-08-26[refactor] remove unused parameterVictor Shyba
2015-08-26[refactor] removing getters and setters from couch.pyVictor Shyba
This is not needed, the behavior under them is the same as an assignment.
2015-08-26[refactor] simplify the case of a brand new docVictor Shyba
If a doc is not present on database at all, it will simply get inserted. This commit makes this clear and skips unnecessary checks.
2015-08-26[refactor] split put_doc_if_newer in two operationsVictor Shyba
Those two operations were mixed on put_doc_if_newer, extracting should make it more clear.
2015-08-25[style] pep8 fixesKali Kaneko
2015-08-24[refactor] remove simplejson dependency on commondrebs
2015-08-24[tests] add sqlite tests to sqlcipherdrebs
Add tests from u1db.tests.test_sqlite_backend.TestSQLitePartialExpandDatabase to soledad sqlcipher tests.
2015-08-24[tests] remove u1db test remote sync target moduledrebs
All tests in that module have been included in soledad's own test modules, so it can be removed so we have less unneeded skipped tests.
2015-08-24[tests] add u1db sync target tests to soledaddrebs
Modify leap.soledad.common.tests.test_sync_target to add: * u1db.tests.test_remote_sync_target.TestRemoteSyncTargets * u1db.tests.test_sync.DatabaseSyncTargetTests
2015-08-24[tests] add missing sync tests to sqlcipherdrebs
Some tests from the original u1db.tests.test_sync.SQLCipherDatabaseSyncTests were missing, this commit adds them back to the soledad sqlcipher tests.
2015-08-20[style] pep8 fixesKali Kaneko
2015-08-20[tests] adapt test for end-of-sync eventdrebs
The soledad sync method was changed to use twisted deferreds, but the test that checks for sync signal was not changed accordingly. This commit fixes that.
2015-08-20[tests] add u1db sync tests to soledad couch testsdrebs
2015-08-20[style] pep8'dKali Kaneko
2015-08-20[tests] add u1db sync tests to sqlcipher's testsdrebs
2015-08-19[style] pep8 cleanupsKali Kaneko
2015-08-19[bug] wait for db init on sync decrypter pooldrebs
Previous to this modification, the initialization of the sync decrypter pool could happen concurrently with other database operations. That could cause the pool to hang because it could be waiting for something that was mistakenly deleted because of the wrong order of database operations. This commit implements a standard which we already use in leap.keymanager and leap.mail which makes some methods wait for the initialization operation before they are actually called. Closes: #7386
2015-08-17[style] pep8 cleanupKali Kaneko
2015-08-17[tests] add mutex sync testsdrebs
2015-08-17[tests] fix test_sqlcipher_syncdrebs
2015-08-17[tests] fix test_sync_deferred testdrebs
* Add possibility of passing sync_db and sync_enc_pool to target creation. * Ensure defer_encryption and defer_decryption are True when setting up the test environment.