summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2015-09-24[feat] adds cache expirationVictor Shyba
Now each backend object will be retrieved from cache for sync.py and values will live for 3600 by default. That is changed via parameter if needed.
2015-09-24[feat] use one doc per remote replica in sync logdrebs
Before this change, we used a complicated update handler for storing the sync state on the couchdb backend. That update handler was implemented as an attempt to make couchdb take care of some validation for the update of the sync log during the sync exchange, mainly to allow concurrent received documents insertion during a sync. Right now we rely on the remote sending one document at a time and do not support concurrent insertions in the remote database backed by couch. Because of that, the code removed by this commit was unneeded. And more: it was a bottleneck of the sync process because we were writing to an unique file and using unnecessary couch design docs processing for that. So this commit both simplifies the storage of remote sync and removes a bottleneck of the sync process. Conflicts: common/src/leap/soledad/common/couch.py common/src/leap/soledad/common/tests/test_couch.py
2015-09-24[feat] full caching for each sync_id sessionVictor Shyba
The CouchDB backend implementation was accessing CouchDB too many times for the same values. Those values are known inside the same sync_id, which is the id of current sync session. This commit adds caching for all redundant calls to Couch inside the same sync_id for each replica. Refactoring is still needed, but for now couch.py works normally as if caching is not present, while sync.py injects the cache as a attribute to enable it. This needs a simpler implementation.
2015-09-24[feat] adds caching for other gen and trans idVictor Shyba
There are two functions in couch.py used to save and retrieve the last know gen and trans id for the syncing replica. The get function is called very often, but is only set on one point. Added a simple caching to avoid queying couch for a value that we already have. If cache is empty, it just query as usual and fills it.
2015-09-24[refactor] uses ThreadPool instead of own implementationVictor Shyba
Python has a native ThreadPool implementation that fits our needs. Changing it to use this instead and making some calls simpler.
2015-09-24[refactor] couch.py should use superclass methodsVictor Shyba
_put_doc_if_newer is implemented on CommonBackend already. This was copied over to CouchBackend just to add ensure conflicts. We can do this before calling the super method instead.
2015-09-24[feat] first draft of sync_state in memoryVictor Shyba
This commit changes sync_state to be in memory, with all tests passing. The memory variable for now is a dict with each key composed by source_replica_uid and sync_id, replicating CouchDB implementation. Next steps includes migrating this to Beaker and refactor/clean up code. Changed the module's INFO dict to use Beaker's caching and adapted methods to get and save from it. Still needs refactoring, all tests passes. Beaker is now using memory as default; It is configurable, but we aren't opening the possibility of config now for security. We need to check what can be misconfigured first. We are not sure if beaker will be the definitive solution for server side caching. This change isolates it with more granularity. In order to replace it, just change get_cache_for to return the proper caching object using another implementation. This caching object is supposed to behave as a dict.
2015-09-24[pkg] add Beaker as a server dependencyVictor Shyba
Soledad server will use Beaker as cache provider, starting with sync_state being in memory.
2015-09-18Merge branch 'pixelated:bug/ensure_ddocs_independency' into developRuben Pollan
2015-09-17[tests] test for ensure ddocs independentlyVictor Shyba
This tests the previous fix on ensuring a db that is missing a doc other than 'docs'.
2015-09-17[bug] ensure needs to check into all design docsVictor Shyba
This code only checks for 'docs' presence, while we have 3 design documents. If one of them is missing, but 'docs' is not, then it will not ensure the others. This is needed to properly ensure ddocs on create command line script.
2015-09-17[refactor] decrease verbosity of sync logsKali Kaneko
2015-09-16[pkg] fix typo in changelogKali Kaneko
2015-09-16Merge remote-tracking branch 'leapcode/pr/271' into developKali Kaneko
2015-09-16[test] fix tests with the new emit_asyncRuben Pollan
2015-09-16[tests] remove remaining dbs checkVictor Shyba
This was used during db isolation to make sure that everything created was destroyed, but it fails with -j (multiprocess). Removing it allows parallelism.
2015-09-16[bug] review some of the close methodsVictor Shyba
We are getting "too many files open" while running tests with 1024 max files open. This is a leak from close methods. Some of them should be fixed on this commit, but further investigation may be necessary.
2015-09-16[pkg] bump leap versions needed for clientKali Kaneko
2015-09-16[feat] use async events clientKali Kaneko
in this way we use the reactor pattern to dispatch the events, instead of having the overhead of running a separate client thread. - Resolves: #7274
2015-09-14[style] pep8 fixesVictor Shyba
2015-09-13[tests] test_couch does not need a serverVictor Shyba
Removing unused code as this test case does not need a server.
2015-09-13[tests] change big tearDown for simple addCleanupVictor Shyba
Big tearDown logic can be replaced by a simple addCleanup. Also remove unused imports and fix a small typo on a database cleanup check.
2015-09-12[tests] subclass instead of copy test codeVictor Shyba
This test only defines a set of different scenarios, all other methods are the same as this subclass.
2015-09-11[tests] db3 is expected to be an attributeVictor Shyba
self.db3 is closed on tearDown. This test was creating it as a local variable, making close possibly fail.
2015-09-11[tests] simplify make_app and getTarget codeVictor Shyba
It was hardcoded for 'test', but the database name is now random. What is useful for test code is the replica_uid, database name for the SyncTarget is now coming from database name.
2015-09-11[tests] isolate LockResource tests using a mockVictor Shyba
'shared' has to be used as a DB name just because of a constant, but it is used on only one point. This changes mock this point to have unique names for better tests isolation. 'tokens' was removed as unnecessary.
2015-09-11[tests] extract db creation with doc ensure methodVictor Shyba
This was a duplicate, but also was getting on the way to improve isolation. With this small refactor it should be cleaner and have unique names.
2015-09-11[tests] use addCleanup to ensure db deletionVictor Shyba
Test case has a addCleanup method, which provides a way to clean resources up and express this need as soon as you create. We are now using it to simplify some logic on database deletion during the test and to make sure that as soon as it gets created a cleanup is there to delete after.
2015-09-11[tests] isolate database names, use uuidVictor Shyba
We are using a single CouchDB install, which may cause tests to overlap since many of them uses the same database name, hurting isolation. This change tries to use uuid on most of it. Also changes for couch_url and couch_port introduced by removal of CouchDB process.
2015-09-11[tests] get rid of CouchDB python processVictor Shyba
This process per test gives a lot of headache, this is why we are removing it. With it we would need to try to start and stop properly on each test case. This fails badly when a test fail and, depending on how it fails, it freezes my pc. Also, it is very heavy for a CI to run a database process for each test case.
2015-09-09[docs] add download badgesKali Kaneko
2015-09-09[docs] fix broken pypi badgesKali Kaneko
2015-09-09[bug] check threadpool state before closing itVictor Shyba
Code is trying to close a closed threadpool. This raises errors on Twisted 15.4.
2015-09-08[docs] add documentation about soledad sync processKali Kaneko
2015-08-31[bug] emit dict instead of strIvan Alejandro
- Resolves: #7412
2015-08-28[bug] cleanup http_target.py file from refactordrebs
The http_target.py refactor started in 8adf2dedb74941352520d8de42326b0c59818728 forgot to remove the old file.
2015-08-28[style] pep8 fixesKali Kaneko
2015-08-27[tests] fix variable name from refactorVictor Shyba
From: [refactor] removing getters and setters from couch.py _couch_url was a private variable with getter and setter doing the same as a public variable. It is accessed all over the code, so being private with getters and setters didnt make sense. This commit fixes the tests to also follow this style from now on.
2015-08-26[bug] reduce memory usage by consuming single docVictor Shyba
Preparing many docs is useful for batching only. As we are sendind one by one I will leave prepare_one_doc method to do the encrypt as the docs goes to upload. Also fixes method name as kaliy suggested.
2015-08-26[refactor] Move constructor, use isinstanceVictor Shyba
isinstance is better, as kaliy pointed out, and the constructor is also in a safer place on __init__.py to be explicit. Also re-apply a change from last rebase;
2015-08-26[docs] Fix docstringsVictor Shyba
There were some missing or on incorrect format (sphinx) as drebs and kaliy pointed out.
2015-08-26[bug] fixes small issues pointed by drebsVictor Shyba
* file headers * variable names * missing docstrings * prune_conflicts ** extra: tests failed on a 1-based index bug
2015-08-26[bug] check type before processingVictor Shyba
Necessary methods are on CouchDocument, but we accept a Document as well, in this case self._factory is needed. Will be simpler soon.
2015-08-26[refactor] _process_incoming_doc is a function nowVictor Shyba
This was being calculated inside CouchDatabase, but it is not a persistence responsibility. It clearly doesn't belong to this persistence layer and seeing both sides separated allow us to work better on both parts.
2015-08-26[refactor] prune conflicts is Document's responsibilityVictor Shyba
2015-08-26[refactor] simplify conflicts managementVictor Shyba
Adding, removing and checking conflicts is an operation done by the model, the Database shouldn't be aware of that. Fetching and saving also is not model's responsability. Repetition remove as well.
2015-08-26[refactor] remove unused parameterVictor Shyba
2015-08-26[refactor] removing getters and setters from couch.pyVictor Shyba
This is not needed, the behavior under them is the same as an assignment.
2015-08-26[refactor] splits http_target into 4 modulesVictor Shyba
SoledadHTTPSyncTarget is composed of 4 main groups of responsibility: * api.py - Public and main methods of a SyncTarget * fetch.py - Document fetching logic * send.py - Document sending logic * support.py - Support functions and patches Previous single file had ~600 lines with those 4 logic groups mixed, making it harder to read and understand.
2015-08-26[refactor] simplify send_docs operationsVictor Shyba
Just extracted some common logic to create u1db formatted requests into RequestBody class and created new methods to represent operations done during send_docs. This also removes send_one_doc, but does not add batching yet. The single send behavior still the same, represented by the parameter passed into RequestBody 'remove' method.