summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2015-10-28Merge branch 'release/0.7.x' into developIvan Alejandro
2015-10-28[pkg] fold in changes0.7.4release/0.7.xIvan Alejandro
2015-10-23Merge branch 'pixelated/develop' into developRuben Pollan
- Release: 0.8.0
2015-10-19[bug] remove instance cachingVictor Shyba
couchdb library relies on garbage collector to close remaining connections. Somehow, caching the instance is avoiding gc to call __del__ on underlying couchdb sessions.
2015-10-14[bug] reduce overall sync session caching to 120sVictor Shyba
It was 3600s, but closing connections seems to yet depend on garbage collection and now causes server to leak file handlers. 120s should be enough to a sync session finish. Also, lowering this value will only make very long syncs use more of couch every 2 minutes, while raising this value will keep memory busy for useless time.
2015-10-13[bug] handle MissingDesignDocError after get_sync_infoVictor Shyba
MissingDesignDocError raised on get_sync_info due to a missing design document will be handled by the server during sync. Ensure is now False by default, and thus database creation can deliver an empty one that will be ensured during sync, following the ensure parameter.
2015-10-13[feat] handle DatabaseDoesNotExist during syncVictor Shyba
This error raises while getting info on target (or server) replica. On previous implementation there was nothing to do here, but now that we have db creation in place this error should be handled just like u1db original implementation. The reason is that db creation occurs during sync exchange, but before this we try to ask for info and the code that checks for info raises an error that will be used to signal the client that a database will be created and that it must save the replica uid returned by server, after sync exchange (where we send/fetch documents).
2015-10-06Merge branch 'leapcode/pr/287' into developRuben Pollan
- Release: 0.8.0
2015-10-05[bug] handle MissingDesignDocError after get_sync_infoVictor Shyba
MissingDesignDocError raised on get_sync_info due to a missing design document will be handled by the server during sync. Ensure is now False by default, and thus database creation can deliver an empty one that will be ensured during sync, following the ensure parameter.
2015-10-05Merge branch 'pixelated/develop' into developRuben Pollan
- Releas: 0.8.0
2015-10-05[bug] fix leap mx by default false on ensure_ddocsVictor Shyba
ensure_ddocs is a privileged operation. The code was defaulting to True, which caused unprivileged code to fail. This commit changes it to False, forcing you to check your privileges and declare a new argument when calling in order to ensure that this behavior is only supposed to happen on privileged parts.
2015-10-05Merge branch 'soledad_configdir' into developRuben Pollan
2015-10-05[feat] Move config dir to /etc/soledadvarac
- Resolves: #7509
2015-10-02[bug] Migrate back to python-couchdb 0.8Victor Shyba
Wheezy is still at 0.8 and it is yet supported. This commit changes all necessary calls from python-couchdb 1.0 back to python-couchdb 0.8. We can migrate this back to simpler implementation with python-couchdb 1.0 when support for wheezy is dropped.
2015-10-02[pkg] pin beaker and couchdb for wheezyVictor Shyba
Wheezy has python-couchdb 0.8 and python-beaker 1.6.3. Pinning them to avoid false positives on tests.
2015-10-02[bug] increase http request timeout time to 90sKali Kaneko
this is a workaroud to reduce the chances of failed sync due to timeouts. this should be properly tackled by: 1. implementing proper cancellable for the sync operation. 2. implementing a retry count at the level of a single request, handled internally by soledad. in this way we can remove the retries logic from the soledadbootstrapper in the bitmask client. - Related: #7382
2015-10-02[bug] do not signal sync completion if failedKali Kaneko
- Related: #7503
2015-10-02[bug] increase http request timeout time to 90sKali Kaneko
this is a workaroud to reduce the chances of failed sync due to timeouts. this should be properly tackled by: 1. implementing proper cancellable for the sync operation. 2. implementing a retry count at the level of a single request, handled internally by soledad. in this way we can remove the retries logic from the soledadbootstrapper in the bitmask client. - Related: #7382
2015-10-02[bug] do not signal sync completion if failedKali Kaneko
- Related: #7503
2015-10-01Merge branch 'pixelated/develop' into developRuben Pollan
- Release: 0.8.0
2015-10-01[feat] read netrc path from configuration fileVictor Shyba
netrc file was hardcoded inside create-user-db. Now it reads the path from /etc/leap/soledad-server.conf as done on server process. The new configuration property is called 'admin_netrc'.
2015-09-30[docs] fix rst link markupKali Kaneko
2015-09-29[docs] added couchdb link to READMEvarac
2015-09-28Merge branch 'feat/create_db_using_cmd' into developRuben Pollan
- Releases: 0.8.0
2015-09-28[style] pep8Victor Shyba
2015-09-28[test] Making test_command pass on Mac OS X.Gislene Pereira
2015-09-28[test] remove old mocksVictor Shyba
Those hardcoded mocks are leaking into other tests and are unnecessary.
2015-09-28[refactor] kaliy's review and pep8 fixesVictor Shyba
README with information about latest change, missing docs and licenses, variable naming and pep8.
2015-09-28[feat] handle DatabaseDoesNotExist during syncVictor Shyba
This error raises while getting info on target (or server) replica. On previous implementation there was nothing to do here, but now that we have db creation in place this error should be handled just like u1db original implementation. The reason is that db creation occurs during sync exchange, but before this we try to ask for info and the code that checks for info raises an error that will be used to signal the client that a database will be created and that it must save the replica uid returned by server, after sync exchange (where we send/fetch documents).
2015-09-28[tests] tests for ensure_securityVictor Shyba
As the other tests does. Make sure that a fresh database gets proper security doc after calling ensure_security method.
2015-09-28[feat] ensure security documentVictor Shyba
Beyond ensuring ddocs, it is also necessary to ensure _security doc presence while creating a database. This document will tell couchdb to grant access to 'soledad' user as a member role and no one as admin.
2015-09-28[feat] script for user db creationVictor Shyba
Added a simple script for user db creation and design docs creation. It uses a netrc from /etc/couchdb/couchdb-admin.netrc and same validator used on couch.py for database names.
2015-09-28[bug] ensure_database returns db and replica_uidVictor Shyba
ensure database needs to return a db and its replica_uid. Updated tests, doc and code to reflect that.
2015-09-28[feat] conf for enabling db creation via custom shVictor Shyba
We can now use a custom script to create databases by setting a parameter 'create_cmd' on soledad configuration. This will set CouchServerState to use it on ensure_database.
2015-09-28[tests] CouchServerState tests for ensure_databaseVictor Shyba
Tests that Unauthorized is raised in any failure scenario, leaving user blind for tips on what happened during execution. This should lower chances of information disclosure on execution failure.
2015-09-28[feat] ensure_database is now able to call a cmdVictor Shyba
If CouchServerState is created with a create_cmd parameter, it can now use this parameter to invoke a command to create databases. A validator for database name is also used to ensure that command injection is not possible if user manages to manipulate database name argument.
2015-09-28[tests] Tests for command validation and executionVictor Shyba
Checks if arguments validation occurs properly and command execution brings back status code and stdout or stderr on some scenarios.
2015-09-28[feat] Validate and execute commands by subprocessVictor Shyba
This commit adds a way to validate and execute commands using an argument validator. Commands are executed via subprocess.
2015-09-27Merge branch 'feature/sync_state_in_memory' into developRuben Pollan
- Releases: 0.8.0
2015-09-24[style] pep8Victor Shyba
2015-09-24[refactor] init_caching instead of setting attrVictor Shyba
As meskio found commented, setting this attribute directly is ugly, CouchDatabase now has a init_caching method for setting up cache instance.
2015-09-24[feat] enable delayed commitsVictor Shyba
We use CouchDB with single doc read/write. Following this documentation about performance, we should get more performance by enabling couch to delay and commit later. See: http://guide.couchdb.org/draft/performance.html#single
2015-09-24[feat] adds cache expirationVictor Shyba
Now each backend object will be retrieved from cache for sync.py and values will live for 3600 by default. That is changed via parameter if needed.
2015-09-24[feat] use one doc per remote replica in sync logdrebs
Before this change, we used a complicated update handler for storing the sync state on the couchdb backend. That update handler was implemented as an attempt to make couchdb take care of some validation for the update of the sync log during the sync exchange, mainly to allow concurrent received documents insertion during a sync. Right now we rely on the remote sending one document at a time and do not support concurrent insertions in the remote database backed by couch. Because of that, the code removed by this commit was unneeded. And more: it was a bottleneck of the sync process because we were writing to an unique file and using unnecessary couch design docs processing for that. So this commit both simplifies the storage of remote sync and removes a bottleneck of the sync process. Conflicts: common/src/leap/soledad/common/couch.py common/src/leap/soledad/common/tests/test_couch.py
2015-09-24[feat] full caching for each sync_id sessionVictor Shyba
The CouchDB backend implementation was accessing CouchDB too many times for the same values. Those values are known inside the same sync_id, which is the id of current sync session. This commit adds caching for all redundant calls to Couch inside the same sync_id for each replica. Refactoring is still needed, but for now couch.py works normally as if caching is not present, while sync.py injects the cache as a attribute to enable it. This needs a simpler implementation.
2015-09-24[feat] adds caching for other gen and trans idVictor Shyba
There are two functions in couch.py used to save and retrieve the last know gen and trans id for the syncing replica. The get function is called very often, but is only set on one point. Added a simple caching to avoid queying couch for a value that we already have. If cache is empty, it just query as usual and fills it.
2015-09-24[refactor] uses ThreadPool instead of own implementationVictor Shyba
Python has a native ThreadPool implementation that fits our needs. Changing it to use this instead and making some calls simpler.
2015-09-24[refactor] couch.py should use superclass methodsVictor Shyba
_put_doc_if_newer is implemented on CommonBackend already. This was copied over to CouchBackend just to add ensure conflicts. We can do this before calling the super method instead.
2015-09-24[feat] first draft of sync_state in memoryVictor Shyba
This commit changes sync_state to be in memory, with all tests passing. The memory variable for now is a dict with each key composed by source_replica_uid and sync_id, replicating CouchDB implementation. Next steps includes migrating this to Beaker and refactor/clean up code. Changed the module's INFO dict to use Beaker's caching and adapted methods to get and save from it. Still needs refactoring, all tests passes. Beaker is now using memory as default; It is configurable, but we aren't opening the possibility of config now for security. We need to check what can be misconfigured first. We are not sure if beaker will be the definitive solution for server side caching. This change isolates it with more granularity. In order to replace it, just change get_cache_for to return the proper caching object using another implementation. This caching object is supposed to behave as a dict.
2015-09-24[pkg] add Beaker as a server dependencyVictor Shyba
Soledad server will use Beaker as cache provider, starting with sync_state being in memory.