| Age | Commit message (Collapse) | Author |
|---|
|
All batching code has no effect by default with this commit. Since we
know that this is a dangerous new feature we will enable them only on
our test servers and check them manually before setting it as default
or adding more configuration features.
Use SyncTarget and server conf file to enable it for testing.
|
|
Generation cache was removed for simple processing and it should not got
back, but during a batch the server wont change its generation. So a
little trick to hold that temporary information until batch finishes is
needed.
|
|
Batch support is optional. This commit adds a 'batching' configuration
option to disable it.
|
|
This commit adds checking for consistency on batch. When a doc is needed
during a batched sync and it doesnt exists on database, current code
will make a partial batch to avoid processing like it doesnt exist.
|
|
Using _bulk_docs api from CouchDB we can put all docs at a single
request. Also, prefetching all ids removes the need to HEAD
requests during the batch.
|
|
Created two methods on the backend to start and finish a batch. A dict of
callbacks is available to defer actions for the last document, allowing
temporary (changing often) metadata to be recorded only once.
Using those methods we will also be able to put all docs in one go on
the CouchDatabase implementation, but that is another step.
|
|
On real usage the docs will arrive shuffled and pool will be reused
after many decrypts. This test asserts that everything ended up clear
between execution and no inconsistency is left over for the next run.
|
|
This new test case will run the single insert test 5 times to ensure
that using the same pool again is fine. This is needed due failures
to shutdown the pool or inconsistency between syncs.
|
|
This info can be changed by another syncing replica and would not
reflect real database generation. That would be ok inside of the same
sync, but can cause trouble on concurrent syncs.
The other calls are ok, since they hold info that doesnt change during
concurrent syncs or are only read/write by the replica syncing. A global
cache could fit better this removed case, but for now let's stay on the
safe side.
|
|
The new BackendNotReadyError didn't have a status or a wire
description, because of that, when you tried to use the
leap.soledad.server package it would break trying to import
this exception (because the annotation tries to use this
variable). This was preventing soledad server from starting
at all, after this change it works again
|
|
Current code was tested on couch 1.6 and a monkeypatch got removed
during refactor. This commit re-adds it, but in a separate module that
is intended to hold temporary code for compatibility that can be removed
on version upgrades.
|
|
since the exception doesn't have the code and description, it breaks.
we don't need those since the couch child exception describes them.
|
|
by subclassing the MissingDesignDocError, we don't have to import the
soledad.common.couch submodule into the soledad.client.sync
- Resolves: #7626
|
|
Creating a database was using a unnecessary complex try/except logic.
Simplifying it should make the purpose more clear.
|
|
|
|
When retrieving other replica info the cache wasnt being used, but when
saving it was. This commit applies caching on get as well.
|
|
Creating a resource from a path to use get_json causes a lot of dirty
code and unexplained things like response[2]. This commit extracts that
logic into a helper to let it more clear about what is happening.
|
|
Added tests for this token verification as it wasn't covered. Then moved
it to the new couch module that implements a couch storage.
The ServerState was chosen to hold the verify_token method.
CouchServerState holds the current implementation, which is called on
authentication middleware as the new test shows.
|
|
errors.py was holding a few specific CouchDB errors, now moved into
couch.errors module. Also, some of CouchDatabase methods were declared
as private, but external classes needs them.
|
|
As SoledadBackend is intended to be database agnostic, a new generic
document is now used instead of the old one made for CouchDB. The only
attribute that really relates to couch was couch_rev, removed on this
commit as it can be set on CouchDatabase implementation when needed.
|
|
First step of splitting classes across files on common.
backend.py holds SoledadBackend (generic backend logic)
couch/ is now a directory with old code inside __init__.py and
CouchServerState on state.py
Also removed mock IndexedSoledadBackend, since Soledad does not support
indexing due to encryption on server side.
Also fixed DesignDocUnknownError to show up what is the message of the
original exception. It was being lost.
|
|
CouchDatabase was renamed to SoledadBackend and a new class
CouchDatabase was created to hold all couchdb code. This should make
SoledadBackend less tied to database implementation. A few more
separations are needed to split into modules.
|
|
This monkey patch was added to separate headers from content and use
them to PUT on couchdb. The original implementation makes it possible by
using two parameters in the constructor.
|
|
CouchDocument is a subclass from SoledadDocument which holds conflicts,
making the server side aware of them. There is a module called document
on soledad.common that holds SoledadDocument. This commit moves
CouchDocument to its proper place.
Error raising logic and exceptions declarations are also moved into error
module inside common.
|
|
This parameter applies the ensure_database to all existing databases,
making all of them use the latest design documents.
This can be used to migrate security documents, update handlers or any
other kind of design document.
|
|
|
|
LEAP Platform needs to granularly allow access on user database for
other services, like mx. This is now possible by editing
soledad-server.conf file. A new section 'database-security' was added
and it is parsed during 'create-user-db' to be set on security design
document, present on every per-user database.
|
|
ensure_ddocs is a privileged operation. The code was defaulting to True,
which caused unprivileged code to fail. This commit changes it to False,
forcing you to check your privileges and declare a new argument when
calling in order to ensure that this behavior is only supposed to happen
on privileged parts.
|
|
Wheezy is still at 0.8 and it is yet supported.
This commit changes all necessary calls from python-couchdb 1.0 back to
python-couchdb 0.8. We can migrate this back to simpler implementation
with python-couchdb 1.0 when support for wheezy is dropped.
|
|
|
|
Those hardcoded mocks are leaking into other tests and are unnecessary.
|
|
README with information about latest change, missing docs and licenses,
variable naming and pep8.
|
|
As the other tests does. Make sure that a fresh database gets proper
security doc after calling ensure_security method.
|
|
Beyond ensuring ddocs, it is also necessary to ensure _security doc
presence while creating a database.
This document will tell couchdb to grant access to 'soledad' user as a
member role and no one as admin.
|
|
ensure database needs to return a db and its replica_uid. Updated tests,
doc and code to reflect that.
|
|
Tests that Unauthorized is raised in any failure scenario, leaving user
blind for tips on what happened during execution. This should lower
chances of information disclosure on execution failure.
|
|
If CouchServerState is created with a create_cmd parameter, it can now
use this parameter to invoke a command to create databases. A validator
for database name is also used to ensure that command injection is not
possible if user manages to manipulate database name argument.
|
|
Checks if arguments validation occurs properly and command execution
brings back status code and stdout or stderr on some scenarios.
|
|
This commit adds a way to validate and execute commands using an
argument validator. Commands are executed via subprocess.
|
|
|
|
As meskio found commented, setting this attribute directly is ugly,
CouchDatabase now has a init_caching method for setting up cache
instance.
|
|
We use CouchDB with single doc read/write. Following this documentation
about performance, we should get more performance by enabling couch to
delay and commit later.
See: http://guide.couchdb.org/draft/performance.html#single
|
|
Now each backend object will be retrieved from cache for sync.py and
values will live for 3600 by default. That is changed via parameter if
needed.
|
|
Before this change, we used a complicated update handler for storing the sync
state on the couchdb backend. That update handler was implemented as an
attempt to make couchdb take care of some validation for the update of the
sync log during the sync exchange, mainly to allow concurrent received
documents insertion during a sync.
Right now we rely on the remote sending one document at a time and do not
support concurrent insertions in the remote database backed by couch. Because
of that, the code removed by this commit was unneeded. And more: it was a
bottleneck of the sync process because we were writing to an unique file and
using unnecessary couch design docs processing for that. So this commit both
simplifies the storage of remote sync and removes a bottleneck of the sync
process.
Conflicts:
common/src/leap/soledad/common/couch.py
common/src/leap/soledad/common/tests/test_couch.py
|
|
The CouchDB backend implementation was accessing CouchDB too many times
for the same values. Those values are known inside the same sync_id,
which is the id of current sync session.
This commit adds caching for all redundant calls to Couch inside the
same sync_id for each replica.
Refactoring is still needed, but for now couch.py works normally as if
caching is not present, while sync.py injects the cache as a attribute
to enable it. This needs a simpler implementation.
|
|
There are two functions in couch.py used to save and retrieve the last
know gen and trans id for the syncing replica. The get function is
called very often, but is only set on one point. Added a simple caching
to avoid queying couch for a value that we already have.
If cache is empty, it just query as usual and fills it.
|
|
Python has a native ThreadPool implementation that fits our needs.
Changing it to use this instead and making some calls simpler.
|
|
_put_doc_if_newer is implemented on CommonBackend already. This was
copied over to CouchBackend just to add ensure conflicts. We can do this
before calling the super method instead.
|
|
This tests the previous fix on ensuring a db that is missing a doc other
than 'docs'.
|
|
This code only checks for 'docs' presence, while we have 3 design
documents. If one of them is missing, but 'docs' is not, then it will
not ensure the others.
This is needed to properly ensure ddocs on create command line script.
|
