summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2015-12-03[feat] add configuration to disable batchingVictor Shyba
Batch support is optional. This commit adds a 'batching' configuration option to disable it.
2015-12-03[feat] checks staged docs inside batchVictor Shyba
This commit adds checking for consistency on batch. When a doc is needed during a batched sync and it doesnt exists on database, current code will make a partial batch to avoid processing like it doesnt exist.
2015-12-03[feat] put all docs at onceVictor Shyba
Using _bulk_docs api from CouchDB we can put all docs at a single request. Also, prefetching all ids removes the need to HEAD requests during the batch.
2015-12-03[feat] prepare server to handle batchesVictor Shyba
Created two methods on the backend to start and finish a batch. A dict of callbacks is available to defer actions for the last document, allowing temporary (changing often) metadata to be recorded only once. Using those methods we will also be able to put all docs in one go on the CouchDatabase implementation, but that is another step.
2015-12-03[feat] Adds support to batching limited by sizeVictor Shyba
u1db provides batching by default. Current Soledad HTTPS Sync Target was stuck at 1 doc per request. This commit adds batching capability, limiting the size to a predefined value. Default limit size: 500kB
2015-12-02[docs] fix run-tests snippetKali Kaneko
2015-12-01[style] fix pep8Victor Shyba
2015-12-01[bug] concurrency bug while querying and insertingVictor Shyba
This line was missing an yield and without it we end up inserting a document that is being retrieved and bad things happen. This is the core fix from yesterday debugging session. During sequential syncs the pool was inserting and querying at the same time and sometimes repeating or failing to delete documents.
2015-12-01[test] shuffle and decrypts 5 docs 3 timesVictor Shyba
On real usage the docs will arrive shuffled and pool will be reused after many decrypts. This test asserts that everything ended up clear between execution and no inconsistency is left over for the next run.
2015-12-01[bug] fire callback after reseting instance varsVictor Shyba
If we reset the vars after firing the finish callback, other thread can pick up a dirty state on due concurrency.
2015-12-01[tests] encdec pool is being reused, adding a testVictor Shyba
This new test case will run the single insert test 5 times to ensure that using the same pool again is fine. This is needed due failures to shutdown the pool or inconsistency between syncs.
2015-11-24[bug] generation_info cant be cached per replicaVictor Shyba
This info can be changed by another syncing replica and would not reflect real database generation. That would be ok inside of the same sync, but can cause trouble on concurrent syncs. The other calls are ok, since they hold info that doesnt change during concurrent syncs or are only read/write by the replica syncing. A global cache could fit better this removed case, but for now let's stay on the safe side.
2015-11-24Merge branch 'pixelated/develop' into developRuben Pollan
2015-11-23[bug] BackendNotReadyError breaks without statusBruno Wagner
The new BackendNotReadyError didn't have a status or a wire description, because of that, when you tried to use the leap.soledad.server package it would break trying to import this exception (because the annotation tries to use this variable). This was preventing soledad server from starting at all, after this change it works again
2015-11-23[bug] put a monkeypatch back for bigcouchVictor Shyba
Current code was tested on couch 1.6 and a monkeypatch got removed during refactor. This commit re-adds it, but in a separate module that is intended to hold temporary code for compatibility that can be removed on version upgrades.
2015-11-20[bug] do not register exception, breaksKali Kaneko
since the exception doesn't have the code and description, it breaks. we don't need those since the couch child exception describes them.
2015-11-19[bug] fix indirect dependency on couchdb for clientKali Kaneko
by subclassing the MissingDesignDocError, we don't have to import the soledad.common.couch submodule into the soledad.client.sync - Resolves: #7626
2015-11-17[feat] add script to measure batch of uploadsKali Kaneko
- Releases: 0.8.0
2015-11-12Merge branch 'refactor/isolated_couchdb_code' into developRuben Pollan
- Releases: 0.8.0
2015-11-03[refactor] remove exception based logicVictor Shyba
Creating a database was using a unnecessary complex try/except logic. Simplifying it should make the purpose more clear.
2015-11-03[style] pep8Victor Shyba
2015-11-03[feat] add caching to a place that was missingVictor Shyba
When retrieving other replica info the cache wasnt being used, but when saving it was. This commit applies caching on get as well.
2015-11-03[refactor] resource logic encapsulationVictor Shyba
Creating a resource from a path to use get_json causes a lot of dirty code and unexplained things like response[2]. This commit extracts that logic into a helper to let it more clear about what is happening.
2015-11-03[refactor] token verification moved to couch module + testsVictor Shyba
Added tests for this token verification as it wasn't covered. Then moved it to the new couch module that implements a couch storage. The ServerState was chosen to hold the verify_token method. CouchServerState holds the current implementation, which is called on authentication middleware as the new test shows.
2015-11-03[refactor] move couch errors and expose methodsVictor Shyba
errors.py was holding a few specific CouchDB errors, now moved into couch.errors module. Also, some of CouchDatabase methods were declared as private, but external classes needs them.
2015-11-03[refactor] CouchDocument is now ServerDocumentVictor Shyba
As SoledadBackend is intended to be database agnostic, a new generic document is now used instead of the old one made for CouchDB. The only attribute that really relates to couch was couch_rev, removed on this commit as it can be set on CouchDatabase implementation when needed.
2015-11-03[refactor] split out backend from couch databaseVictor Shyba
First step of splitting classes across files on common. backend.py holds SoledadBackend (generic backend logic) couch/ is now a directory with old code inside __init__.py and CouchServerState on state.py Also removed mock IndexedSoledadBackend, since Soledad does not support indexing due to encryption on server side. Also fixed DesignDocUnknownError to show up what is the message of the original exception. It was being lost.
2015-11-03[refactor] separate SoledadBackend from CouchDatabaseVictor Shyba
CouchDatabase was renamed to SoledadBackend and a new class CouchDatabase was created to hold all couchdb code. This should make SoledadBackend less tied to database implementation. A few more separations are needed to split into modules.
2015-11-03[refactor] remove MultipartWriter monkey patchVictor Shyba
This monkey patch was added to separate headers from content and use them to PUT on couchdb. The original implementation makes it possible by using two parameters in the constructor.
2015-11-03[refactor] separate errors and model from couchVictor Shyba
CouchDocument is a subclass from SoledadDocument which holds conflicts, making the server side aware of them. There is a module called document on soledad.common that holds SoledadDocument. This commit moves CouchDocument to its proper place. Error raising logic and exceptions declarations are also moved into error module inside common.
2015-10-30[feat] bump twisted dependency to 12.3.0Ruben Pollan
Needed after the SSL monkeypatch removal at 3b869fb7f
2015-10-29[feat] remove SSL tsafe monkeypatchVictor Shyba
This was added for a Twisted 12 bug that should be gone by now.
2015-10-29[feat] adds --migrate-all to create-user-db scriptVictor Shyba
This parameter applies the ensure_database to all existing databases, making all of them use the latest design documents. This can be used to migrate security documents, update handlers or any other kind of design document.
2015-10-28[style] fix pep8 warnigsFolker Bernitt
2015-10-28[feat] read security doc from configurationVictor Shyba
LEAP Platform needs to granularly allow access on user database for other services, like mx. This is now possible by editing soledad-server.conf file. A new section 'database-security' was added and it is parsed during 'create-user-db' to be set on security design document, present on every per-user database.
2015-10-28Merge branch 'release/0.7.x' into developIvan Alejandro
2015-10-28[pkg] fold in changes0.7.4release/0.7.xIvan Alejandro
2015-10-23Merge branch 'pixelated/develop' into developRuben Pollan
- Release: 0.8.0
2015-10-19[bug] remove instance cachingVictor Shyba
couchdb library relies on garbage collector to close remaining connections. Somehow, caching the instance is avoiding gc to call __del__ on underlying couchdb sessions.
2015-10-14[bug] reduce overall sync session caching to 120sVictor Shyba
It was 3600s, but closing connections seems to yet depend on garbage collection and now causes server to leak file handlers. 120s should be enough to a sync session finish. Also, lowering this value will only make very long syncs use more of couch every 2 minutes, while raising this value will keep memory busy for useless time.
2015-10-13[bug] handle MissingDesignDocError after get_sync_infoVictor Shyba
MissingDesignDocError raised on get_sync_info due to a missing design document will be handled by the server during sync. Ensure is now False by default, and thus database creation can deliver an empty one that will be ensured during sync, following the ensure parameter.
2015-10-13[feat] handle DatabaseDoesNotExist during syncVictor Shyba
This error raises while getting info on target (or server) replica. On previous implementation there was nothing to do here, but now that we have db creation in place this error should be handled just like u1db original implementation. The reason is that db creation occurs during sync exchange, but before this we try to ask for info and the code that checks for info raises an error that will be used to signal the client that a database will be created and that it must save the replica uid returned by server, after sync exchange (where we send/fetch documents).
2015-10-06Merge branch 'leapcode/pr/287' into developRuben Pollan
- Release: 0.8.0
2015-10-05[bug] handle MissingDesignDocError after get_sync_infoVictor Shyba
MissingDesignDocError raised on get_sync_info due to a missing design document will be handled by the server during sync. Ensure is now False by default, and thus database creation can deliver an empty one that will be ensured during sync, following the ensure parameter.
2015-10-05Merge branch 'pixelated/develop' into developRuben Pollan
- Releas: 0.8.0
2015-10-05[bug] fix leap mx by default false on ensure_ddocsVictor Shyba
ensure_ddocs is a privileged operation. The code was defaulting to True, which caused unprivileged code to fail. This commit changes it to False, forcing you to check your privileges and declare a new argument when calling in order to ensure that this behavior is only supposed to happen on privileged parts.
2015-10-05Merge branch 'soledad_configdir' into developRuben Pollan
2015-10-05[feat] Move config dir to /etc/soledadvarac
- Resolves: #7509
2015-10-02[bug] Migrate back to python-couchdb 0.8Victor Shyba
Wheezy is still at 0.8 and it is yet supported. This commit changes all necessary calls from python-couchdb 1.0 back to python-couchdb 0.8. We can migrate this back to simpler implementation with python-couchdb 1.0 when support for wheezy is dropped.
2015-10-02[pkg] pin beaker and couchdb for wheezyVictor Shyba
Wheezy has python-couchdb 0.8 and python-beaker 1.6.3. Pinning them to avoid false positives on tests.