summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2015-09-28[feat] conf for enabling db creation via custom shVictor Shyba
We can now use a custom script to create databases by setting a parameter 'create_cmd' on soledad configuration. This will set CouchServerState to use it on ensure_database.
2015-09-28[tests] CouchServerState tests for ensure_databaseVictor Shyba
Tests that Unauthorized is raised in any failure scenario, leaving user blind for tips on what happened during execution. This should lower chances of information disclosure on execution failure.
2015-09-28[feat] ensure_database is now able to call a cmdVictor Shyba
If CouchServerState is created with a create_cmd parameter, it can now use this parameter to invoke a command to create databases. A validator for database name is also used to ensure that command injection is not possible if user manages to manipulate database name argument.
2015-09-28[tests] Tests for command validation and executionVictor Shyba
Checks if arguments validation occurs properly and command execution brings back status code and stdout or stderr on some scenarios.
2015-09-28[feat] Validate and execute commands by subprocessVictor Shyba
This commit adds a way to validate and execute commands using an argument validator. Commands are executed via subprocess.
2015-09-27Merge branch 'feature/sync_state_in_memory' into developRuben Pollan
- Releases: 0.8.0
2015-09-24[style] pep8Victor Shyba
2015-09-24[refactor] init_caching instead of setting attrVictor Shyba
As meskio found commented, setting this attribute directly is ugly, CouchDatabase now has a init_caching method for setting up cache instance.
2015-09-24[feat] enable delayed commitsVictor Shyba
We use CouchDB with single doc read/write. Following this documentation about performance, we should get more performance by enabling couch to delay and commit later. See: http://guide.couchdb.org/draft/performance.html#single
2015-09-24[feat] adds cache expirationVictor Shyba
Now each backend object will be retrieved from cache for sync.py and values will live for 3600 by default. That is changed via parameter if needed.
2015-09-24[feat] use one doc per remote replica in sync logdrebs
Before this change, we used a complicated update handler for storing the sync state on the couchdb backend. That update handler was implemented as an attempt to make couchdb take care of some validation for the update of the sync log during the sync exchange, mainly to allow concurrent received documents insertion during a sync. Right now we rely on the remote sending one document at a time and do not support concurrent insertions in the remote database backed by couch. Because of that, the code removed by this commit was unneeded. And more: it was a bottleneck of the sync process because we were writing to an unique file and using unnecessary couch design docs processing for that. So this commit both simplifies the storage of remote sync and removes a bottleneck of the sync process. Conflicts: common/src/leap/soledad/common/couch.py common/src/leap/soledad/common/tests/test_couch.py
2015-09-24[feat] full caching for each sync_id sessionVictor Shyba
The CouchDB backend implementation was accessing CouchDB too many times for the same values. Those values are known inside the same sync_id, which is the id of current sync session. This commit adds caching for all redundant calls to Couch inside the same sync_id for each replica. Refactoring is still needed, but for now couch.py works normally as if caching is not present, while sync.py injects the cache as a attribute to enable it. This needs a simpler implementation.
2015-09-24[feat] adds caching for other gen and trans idVictor Shyba
There are two functions in couch.py used to save and retrieve the last know gen and trans id for the syncing replica. The get function is called very often, but is only set on one point. Added a simple caching to avoid queying couch for a value that we already have. If cache is empty, it just query as usual and fills it.
2015-09-24[refactor] uses ThreadPool instead of own implementationVictor Shyba
Python has a native ThreadPool implementation that fits our needs. Changing it to use this instead and making some calls simpler.
2015-09-24[refactor] couch.py should use superclass methodsVictor Shyba
_put_doc_if_newer is implemented on CommonBackend already. This was copied over to CouchBackend just to add ensure conflicts. We can do this before calling the super method instead.
2015-09-24[feat] first draft of sync_state in memoryVictor Shyba
This commit changes sync_state to be in memory, with all tests passing. The memory variable for now is a dict with each key composed by source_replica_uid and sync_id, replicating CouchDB implementation. Next steps includes migrating this to Beaker and refactor/clean up code. Changed the module's INFO dict to use Beaker's caching and adapted methods to get and save from it. Still needs refactoring, all tests passes. Beaker is now using memory as default; It is configurable, but we aren't opening the possibility of config now for security. We need to check what can be misconfigured first. We are not sure if beaker will be the definitive solution for server side caching. This change isolates it with more granularity. In order to replace it, just change get_cache_for to return the proper caching object using another implementation. This caching object is supposed to behave as a dict.
2015-09-24[pkg] add Beaker as a server dependencyVictor Shyba
Soledad server will use Beaker as cache provider, starting with sync_state being in memory.
2015-09-21[pkg] fold in changes0.7.3Ivan Alejandro
2015-09-21[bug] set the received active secret before saving local fileKali Kaneko
- bug: we were dumping the received secrets locally to disk *before* setting the received property for the active secret, and therefore the 'active_secret' was always marked as null. - refactor common code into an utility method.
2015-09-18Merge branch 'pixelated:bug/ensure_ddocs_independency' into developRuben Pollan
2015-09-17[tests] test for ensure ddocs independentlyVictor Shyba
This tests the previous fix on ensuring a db that is missing a doc other than 'docs'.
2015-09-17[bug] ensure needs to check into all design docsVictor Shyba
This code only checks for 'docs' presence, while we have 3 design documents. If one of them is missing, but 'docs' is not, then it will not ensure the others. This is needed to properly ensure ddocs on create command line script.
2015-09-17[refactor] decrease verbosity of sync logsKali Kaneko
2015-09-16[pkg] fix typo in changelogKali Kaneko
2015-09-16Merge remote-tracking branch 'leapcode/pr/271' into developKali Kaneko
2015-09-16[test] fix tests with the new emit_asyncRuben Pollan
2015-09-16[tests] remove remaining dbs checkVictor Shyba
This was used during db isolation to make sure that everything created was destroyed, but it fails with -j (multiprocess). Removing it allows parallelism.
2015-09-16[bug] review some of the close methodsVictor Shyba
We are getting "too many files open" while running tests with 1024 max files open. This is a leak from close methods. Some of them should be fixed on this commit, but further investigation may be necessary.
2015-09-16[pkg] bump leap versions needed for clientKali Kaneko
2015-09-16[feat] use async events clientKali Kaneko
in this way we use the reactor pattern to dispatch the events, instead of having the overhead of running a separate client thread. - Resolves: #7274
2015-09-14[style] pep8 fixesVictor Shyba
2015-09-13[tests] test_couch does not need a serverVictor Shyba
Removing unused code as this test case does not need a server.
2015-09-13[tests] change big tearDown for simple addCleanupVictor Shyba
Big tearDown logic can be replaced by a simple addCleanup. Also remove unused imports and fix a small typo on a database cleanup check.
2015-09-12[tests] subclass instead of copy test codeVictor Shyba
This test only defines a set of different scenarios, all other methods are the same as this subclass.
2015-09-11[tests] db3 is expected to be an attributeVictor Shyba
self.db3 is closed on tearDown. This test was creating it as a local variable, making close possibly fail.
2015-09-11[tests] simplify make_app and getTarget codeVictor Shyba
It was hardcoded for 'test', but the database name is now random. What is useful for test code is the replica_uid, database name for the SyncTarget is now coming from database name.
2015-09-11[tests] isolate LockResource tests using a mockVictor Shyba
'shared' has to be used as a DB name just because of a constant, but it is used on only one point. This changes mock this point to have unique names for better tests isolation. 'tokens' was removed as unnecessary.
2015-09-11[tests] extract db creation with doc ensure methodVictor Shyba
This was a duplicate, but also was getting on the way to improve isolation. With this small refactor it should be cleaner and have unique names.
2015-09-11[tests] use addCleanup to ensure db deletionVictor Shyba
Test case has a addCleanup method, which provides a way to clean resources up and express this need as soon as you create. We are now using it to simplify some logic on database deletion during the test and to make sure that as soon as it gets created a cleanup is there to delete after.
2015-09-11[tests] isolate database names, use uuidVictor Shyba
We are using a single CouchDB install, which may cause tests to overlap since many of them uses the same database name, hurting isolation. This change tries to use uuid on most of it. Also changes for couch_url and couch_port introduced by removal of CouchDB process.
2015-09-11[tests] get rid of CouchDB python processVictor Shyba
This process per test gives a lot of headache, this is why we are removing it. With it we would need to try to start and stop properly on each test case. This fails badly when a test fail and, depending on how it fails, it freezes my pc. Also, it is very heavy for a CI to run a database process for each test case.
2015-09-09[docs] add download badgesKali Kaneko
2015-09-09[docs] fix broken pypi badgesKali Kaneko
2015-09-09[bug] check threadpool state before closing itVictor Shyba
Code is trying to close a closed threadpool. This raises errors on Twisted 15.4.
2015-09-08[docs] add documentation about soledad sync processKali Kaneko
2015-08-31[bug] emit dict instead of strIvan Alejandro
- Resolves: #7412
2015-08-28[bug] cleanup http_target.py file from refactordrebs
The http_target.py refactor started in 8adf2dedb74941352520d8de42326b0c59818728 forgot to remove the old file.
2015-08-28[style] pep8 fixesKali Kaneko
2015-08-27[tests] fix variable name from refactorVictor Shyba
From: [refactor] removing getters and setters from couch.py _couch_url was a private variable with getter and setter doing the same as a public variable. It is accessed all over the code, so being private with getters and setters didnt make sense. This commit fixes the tests to also follow this style from now on.
2015-08-26[bug] reduce memory usage by consuming single docVictor Shyba
Preparing many docs is useful for batching only. As we are sendind one by one I will leave prepare_one_doc method to do the encrypt as the docs goes to upload. Also fixes method name as kaliy suggested.