summaryrefslogtreecommitdiff
path: root/common/src/leap/soledad
AgeCommit message (Collapse)Author
2015-12-03[feat] set default to False on batching for nowVictor Shyba
All batching code has no effect by default with this commit. Since we know that this is a dangerous new feature we will enable them only on our test servers and check them manually before setting it as default or adding more configuration features. Use SyncTarget and server conf file to enable it for testing.
2015-12-03[feat] generation caching during a batchVictor Shyba
Generation cache was removed for simple processing and it should not got back, but during a batch the server wont change its generation. So a little trick to hold that temporary information until batch finishes is needed.
2015-12-03[feat] add configuration to disable batchingVictor Shyba
Batch support is optional. This commit adds a 'batching' configuration option to disable it.
2015-12-03[feat] checks staged docs inside batchVictor Shyba
This commit adds checking for consistency on batch. When a doc is needed during a batched sync and it doesnt exists on database, current code will make a partial batch to avoid processing like it doesnt exist.
2015-12-03[feat] put all docs at onceVictor Shyba
Using _bulk_docs api from CouchDB we can put all docs at a single request. Also, prefetching all ids removes the need to HEAD requests during the batch.
2015-12-03[feat] prepare server to handle batchesVictor Shyba
Created two methods on the backend to start and finish a batch. A dict of callbacks is available to defer actions for the last document, allowing temporary (changing often) metadata to be recorded only once. Using those methods we will also be able to put all docs in one go on the CouchDatabase implementation, but that is another step.
2015-12-01[test] shuffle and decrypts 5 docs 3 timesVictor Shyba
On real usage the docs will arrive shuffled and pool will be reused after many decrypts. This test asserts that everything ended up clear between execution and no inconsistency is left over for the next run.
2015-12-01[tests] encdec pool is being reused, adding a testVictor Shyba
This new test case will run the single insert test 5 times to ensure that using the same pool again is fine. This is needed due failures to shutdown the pool or inconsistency between syncs.
2015-11-24[bug] generation_info cant be cached per replicaVictor Shyba
This info can be changed by another syncing replica and would not reflect real database generation. That would be ok inside of the same sync, but can cause trouble on concurrent syncs. The other calls are ok, since they hold info that doesnt change during concurrent syncs or are only read/write by the replica syncing. A global cache could fit better this removed case, but for now let's stay on the safe side.
2015-11-23[bug] BackendNotReadyError breaks without statusBruno Wagner
The new BackendNotReadyError didn't have a status or a wire description, because of that, when you tried to use the leap.soledad.server package it would break trying to import this exception (because the annotation tries to use this variable). This was preventing soledad server from starting at all, after this change it works again
2015-11-23[bug] put a monkeypatch back for bigcouchVictor Shyba
Current code was tested on couch 1.6 and a monkeypatch got removed during refactor. This commit re-adds it, but in a separate module that is intended to hold temporary code for compatibility that can be removed on version upgrades.
2015-11-20[bug] do not register exception, breaksKali Kaneko
since the exception doesn't have the code and description, it breaks. we don't need those since the couch child exception describes them.
2015-11-19[bug] fix indirect dependency on couchdb for clientKali Kaneko
by subclassing the MissingDesignDocError, we don't have to import the soledad.common.couch submodule into the soledad.client.sync - Resolves: #7626
2015-11-03[refactor] remove exception based logicVictor Shyba
Creating a database was using a unnecessary complex try/except logic. Simplifying it should make the purpose more clear.
2015-11-03[style] pep8Victor Shyba
2015-11-03[feat] add caching to a place that was missingVictor Shyba
When retrieving other replica info the cache wasnt being used, but when saving it was. This commit applies caching on get as well.
2015-11-03[refactor] resource logic encapsulationVictor Shyba
Creating a resource from a path to use get_json causes a lot of dirty code and unexplained things like response[2]. This commit extracts that logic into a helper to let it more clear about what is happening.
2015-11-03[refactor] token verification moved to couch module + testsVictor Shyba
Added tests for this token verification as it wasn't covered. Then moved it to the new couch module that implements a couch storage. The ServerState was chosen to hold the verify_token method. CouchServerState holds the current implementation, which is called on authentication middleware as the new test shows.
2015-11-03[refactor] move couch errors and expose methodsVictor Shyba
errors.py was holding a few specific CouchDB errors, now moved into couch.errors module. Also, some of CouchDatabase methods were declared as private, but external classes needs them.
2015-11-03[refactor] CouchDocument is now ServerDocumentVictor Shyba
As SoledadBackend is intended to be database agnostic, a new generic document is now used instead of the old one made for CouchDB. The only attribute that really relates to couch was couch_rev, removed on this commit as it can be set on CouchDatabase implementation when needed.
2015-11-03[refactor] split out backend from couch databaseVictor Shyba
First step of splitting classes across files on common. backend.py holds SoledadBackend (generic backend logic) couch/ is now a directory with old code inside __init__.py and CouchServerState on state.py Also removed mock IndexedSoledadBackend, since Soledad does not support indexing due to encryption on server side. Also fixed DesignDocUnknownError to show up what is the message of the original exception. It was being lost.
2015-11-03[refactor] separate SoledadBackend from CouchDatabaseVictor Shyba
CouchDatabase was renamed to SoledadBackend and a new class CouchDatabase was created to hold all couchdb code. This should make SoledadBackend less tied to database implementation. A few more separations are needed to split into modules.
2015-11-03[refactor] remove MultipartWriter monkey patchVictor Shyba
This monkey patch was added to separate headers from content and use them to PUT on couchdb. The original implementation makes it possible by using two parameters in the constructor.
2015-11-03[refactor] separate errors and model from couchVictor Shyba
CouchDocument is a subclass from SoledadDocument which holds conflicts, making the server side aware of them. There is a module called document on soledad.common that holds SoledadDocument. This commit moves CouchDocument to its proper place. Error raising logic and exceptions declarations are also moved into error module inside common.
2015-10-29[feat] adds --migrate-all to create-user-db scriptVictor Shyba
This parameter applies the ensure_database to all existing databases, making all of them use the latest design documents. This can be used to migrate security documents, update handlers or any other kind of design document.
2015-10-28[style] fix pep8 warnigsFolker Bernitt
2015-10-28[feat] read security doc from configurationVictor Shyba
LEAP Platform needs to granularly allow access on user database for other services, like mx. This is now possible by editing soledad-server.conf file. A new section 'database-security' was added and it is parsed during 'create-user-db' to be set on security design document, present on every per-user database.
2015-10-05[bug] fix leap mx by default false on ensure_ddocsVictor Shyba
ensure_ddocs is a privileged operation. The code was defaulting to True, which caused unprivileged code to fail. This commit changes it to False, forcing you to check your privileges and declare a new argument when calling in order to ensure that this behavior is only supposed to happen on privileged parts.
2015-10-02[bug] Migrate back to python-couchdb 0.8Victor Shyba
Wheezy is still at 0.8 and it is yet supported. This commit changes all necessary calls from python-couchdb 1.0 back to python-couchdb 0.8. We can migrate this back to simpler implementation with python-couchdb 1.0 when support for wheezy is dropped.
2015-09-28[test] Making test_command pass on Mac OS X.Gislene Pereira
2015-09-28[test] remove old mocksVictor Shyba
Those hardcoded mocks are leaking into other tests and are unnecessary.
2015-09-28[refactor] kaliy's review and pep8 fixesVictor Shyba
README with information about latest change, missing docs and licenses, variable naming and pep8.
2015-09-28[tests] tests for ensure_securityVictor Shyba
As the other tests does. Make sure that a fresh database gets proper security doc after calling ensure_security method.
2015-09-28[feat] ensure security documentVictor Shyba
Beyond ensuring ddocs, it is also necessary to ensure _security doc presence while creating a database. This document will tell couchdb to grant access to 'soledad' user as a member role and no one as admin.
2015-09-28[bug] ensure_database returns db and replica_uidVictor Shyba
ensure database needs to return a db and its replica_uid. Updated tests, doc and code to reflect that.
2015-09-28[tests] CouchServerState tests for ensure_databaseVictor Shyba
Tests that Unauthorized is raised in any failure scenario, leaving user blind for tips on what happened during execution. This should lower chances of information disclosure on execution failure.
2015-09-28[feat] ensure_database is now able to call a cmdVictor Shyba
If CouchServerState is created with a create_cmd parameter, it can now use this parameter to invoke a command to create databases. A validator for database name is also used to ensure that command injection is not possible if user manages to manipulate database name argument.
2015-09-28[tests] Tests for command validation and executionVictor Shyba
Checks if arguments validation occurs properly and command execution brings back status code and stdout or stderr on some scenarios.
2015-09-28[feat] Validate and execute commands by subprocessVictor Shyba
This commit adds a way to validate and execute commands using an argument validator. Commands are executed via subprocess.
2015-09-24[style] pep8Victor Shyba
2015-09-24[refactor] init_caching instead of setting attrVictor Shyba
As meskio found commented, setting this attribute directly is ugly, CouchDatabase now has a init_caching method for setting up cache instance.
2015-09-24[feat] enable delayed commitsVictor Shyba
We use CouchDB with single doc read/write. Following this documentation about performance, we should get more performance by enabling couch to delay and commit later. See: http://guide.couchdb.org/draft/performance.html#single
2015-09-24[feat] adds cache expirationVictor Shyba
Now each backend object will be retrieved from cache for sync.py and values will live for 3600 by default. That is changed via parameter if needed.
2015-09-24[feat] use one doc per remote replica in sync logdrebs
Before this change, we used a complicated update handler for storing the sync state on the couchdb backend. That update handler was implemented as an attempt to make couchdb take care of some validation for the update of the sync log during the sync exchange, mainly to allow concurrent received documents insertion during a sync. Right now we rely on the remote sending one document at a time and do not support concurrent insertions in the remote database backed by couch. Because of that, the code removed by this commit was unneeded. And more: it was a bottleneck of the sync process because we were writing to an unique file and using unnecessary couch design docs processing for that. So this commit both simplifies the storage of remote sync and removes a bottleneck of the sync process. Conflicts: common/src/leap/soledad/common/couch.py common/src/leap/soledad/common/tests/test_couch.py
2015-09-24[feat] full caching for each sync_id sessionVictor Shyba
The CouchDB backend implementation was accessing CouchDB too many times for the same values. Those values are known inside the same sync_id, which is the id of current sync session. This commit adds caching for all redundant calls to Couch inside the same sync_id for each replica. Refactoring is still needed, but for now couch.py works normally as if caching is not present, while sync.py injects the cache as a attribute to enable it. This needs a simpler implementation.
2015-09-24[feat] adds caching for other gen and trans idVictor Shyba
There are two functions in couch.py used to save and retrieve the last know gen and trans id for the syncing replica. The get function is called very often, but is only set on one point. Added a simple caching to avoid queying couch for a value that we already have. If cache is empty, it just query as usual and fills it.
2015-09-24[refactor] uses ThreadPool instead of own implementationVictor Shyba
Python has a native ThreadPool implementation that fits our needs. Changing it to use this instead and making some calls simpler.
2015-09-24[refactor] couch.py should use superclass methodsVictor Shyba
_put_doc_if_newer is implemented on CommonBackend already. This was copied over to CouchBackend just to add ensure conflicts. We can do this before calling the super method instead.
2015-09-17[tests] test for ensure ddocs independentlyVictor Shyba
This tests the previous fix on ensuring a db that is missing a doc other than 'docs'.
2015-09-17[bug] ensure needs to check into all design docsVictor Shyba
This code only checks for 'docs' presence, while we have 3 design documents. If one of them is missing, but 'docs' is not, then it will not ensure the others. This is needed to properly ensure ddocs on create command line script.